“The explosion of the word-of-mouth channels and the adoption of online verification have forever changed the marketing landscape, “ this is how one of the executives of Cone Inc. PR and marketing agency described the trends in modern e-commerce after their extensive study of 2011. We, at 3Beds have seen it in action, our model of rating the airbeds largely relies on using reviews to find the best air mattress.

So, we decided to quantify what we’ve seen and this case study is a result of that effort.

We looked into the existing research and came away with one conclusion:

Online reviews are the new word-of-mouth.

It’s only after we truly realized how the power of user feedback translated from the real world where a friend would recommend something to the world of e-commerce that we started adjusting our rating formulas to make them what they are today.

The results of the BrightLocal research concluded that 88% of people responded “YES” when asked if reading a positive/negative review of local businesses influenced their decision.

That gave us further proof that we need to dig deeper in pursuit of better understanding of how a user interacts with online feedback.

 

The results of the BrightLocal study:

Why understanding behavioral economics is important for what we do

Imagine this, you have one very satisfied customer who shared a great review of the product and another that, let’s say, wasn’t as pleased.

Knowing that a negative review has approximately twice as much impact, do you simply include them product ratings as equals or do you include adjusting factors to more precisely depict the quality of the product?

The nature of the work we do here at 3Beds.com requires deep a understanding of the decision-making process, the flow of thoughts that follows every purchase and especially, how they are influenced by the feedback of other users.

Our product ratings consist of three parameters:

• In-house ratings (these are objective tests that have a clear quantification)
• The ratings of our featured testers which are, again, pretty straightforward
• Product ratings from the user reviews we gather

The formulas that we use to rate a product have been tweaked over time from a mere average of the three to a well-balanced formula that adjusts to the importance of these factors. Especially if a person is getting ready to spend a substantial amount of money on a luxury air mattress.

So, here’s what we did:

1. We grew the sample of people who contacted us over the years and did a study on how they react to different ratios of positive vs. negative reviews until we reached a sample that was big enough (1000 people) to test some of our assumptions
2. We asked them how likely are they to get the products (strategically distributed across different ratios of positive vs. negative feedback)
3. We cross-referenced the results to tweak our rating formulas

 

Here’s what we’ve concluded:

The methodology and rules we followed

Once every two months we sent the users the link to the products we chose and asked them to answer one simple question:

“What would you say are the chances are you’d get the product if you needed it right now?”

To minimize any room for statistical error in the process we stuck to a few rules:

1. To eliminate personal preference – we always used the same people for the polls
2. To eliminate the influence of the product itself, we always used the same product and kept track of how the user affinity changed as the number and ratio of reviews changed
3. We eliminated any products that changed price over the course of the research
4. We eliminated products whose ratio of positive vs. negative feedback fluctuated more than 5 % over time (recent reviews are perceived as most relevant)
5. We took into account the ratio of new reviews vs. total existing reviews
6. We choose 10 products from the same price range ($100-120) since, naturally, items with a higher price tag were more thoroughly researched and the influence of reviews on tipping the purchase decision scales increased

We realized we had to adjust

The whole thing was long overdue and we knew it, but the idea was additionally fueled in 2011 when the Cone research (Cone inc. PR and marketing agency) proved that user reviews were gaining traction in e-commerce.

Their Online Influence Trends research showed that approximately two thirds of users changed their mind influenced by negative reviews.

How it influenced the way we rate products

Our goal is to reach a number (Overall Rating) that accurately describes the quality of a product. One of the three components (and the most intricate one) is interpreting user reviews.

In rating a product like an air mattress, there’s significant space for products that are flawed on arrival and if the data is used “raw” it can distort the image about a product.

Over the years, our in-hose tests became increasingly reliable, so what we did is tweak the formulas we use for user reviews that are included in the final rating by adding adjusting factors until we reached formulas that accurately depicted what we saw in our quality testing over the years.

Some of the criteria that we sort and rate the products in:

• Most durable air mattress

Best by airbed brand:

• Coleman
• Aerobed
• Intex
• SoundAsleep Dream Series

Top choices by size:

• Twin air mattresses
• Queen airbeds
• King size
• California King

Top choices by other criteria:

• Toddler air mattresses, folding beds and cots

What we do here at 3Beds

Having in mind the power of the user opinion, we have shaped a system of rating that is as close to science as analyzing products as air mattresses can be:

• We get the products and put them through the scrutiny of our tests (and rate them in 5 quality categories)
• We send out the beds to our featured testers (without any special compensation) for their impressions and ratings in the same 5 categories
• We look into reviews and contact people online to simply ask about the airbeds in question
• We combine the 3 methods and reach our final rating
• We sort the ratings into guides based on different criteria

The tweaks were small and we made sure that they don’t favor a product, our only goal was to minimize the influence of perception to a quality rating of a product, that’s was meant to be a clear, quantifiable value.

Is this where it stops?

The market is a living and breathing organism, both in supply and demand and the way people make their decisions. This implies that our process has to remain a living ever-changing “organism” that described the quality of the product at any given moment.

There is a number of moving parts and our responses and adjustments will and can never stop.