Overall Scores and Weighted Averages
Overall Scores and Weighted Averages
November 14, 2014 // In SR Help // By
Not all reviews are created equal. It’s hard to know what to trust, especially in an industry rampant with pseudo-science and marketing hype.
We understand this here at SR. Our Overall Score calculation takes into account that some reviews simply shouldn’t be counted. We also adjust each product’s score for quantity of reviews, so each product score is a number you can count on.
I wasn't prepared to do math today. Just tell me what this means.
The overall score we give to each product is something we are very confident about. If you are comparing a product that has an 8.5 overall score and a 9.5 overall score, the 9.5 is probably just going to be better. Plain and simple as that. You don't have to spend time seeing if the reviews are legit or not. We already did that.
How everyone else does it (the wrong way)
The traditional method is to just add up the score from each review, and divide by the total number of reviews.
Overall Score = sum( Rating ) / (Number of Reviews)
This creates a completely useless set of overall product scores.
“Product A” may only have one review (from the company rep) giving it a 10/10, and that will result in a product score of 10/10. Does that mean that “Product A” is the best? Even better than “Product B” with 1,000 reviews and a score of 9.8/10?
The entire “test team” from some unknown brand may have been summoned to give their product 20 perfect reviews. The product now has a score of 10/10 with 20 reviews. Does this mean it’s the best product?
Clearly other review sites haven’t thought about these problems, or they simply don’t care. We have taken measures to make sure only the best products make it to the top, and made it impossible for reps to boost their ratings with fake reviews.
How we do it (the right way)
The first step is to look at the Member Trust for every review. Here is the basic formula we use:
Overall Score = sum( Trust * Rating ) / sum( Trust )
So a member with 0% trust won't get their vote counted at all. A member with 100% trust will get their vote counted 100%. Two members with a trust score of 50% each will have their combined votes count the same as one member with 100% trust. You get the idea.
But wait. We're not done. We adjust for a few more things before spitting out the final "Product Score". The adjustments we do include:
- Review age - a review that is four days old is weighted higher than review that is 4 years old. Manufacturers will often reformulate and change prices without telling anyone, so the freshness of a review is important.
- Sum of Member Trust - If a product has 3 reviews, all from members with 0% trust, it is hard for us to confidently give that product a score, so we just give it ?/10 while we wait for more trustworthy reviews. Also, if a product has just one 10/10 review from a member with 100% trust, it's still hard for us to confidently give that product a perfect 10. So we adjust for total trust % of all reviews for that product combined.
Why is there no average score?
“Overall Score: ?/10 (Only 7 Reviews - Waiting on more Trustworthy Reviews)”
Basically, there are no trustworthy reviews on this product/brand/retailer yet. We only give Overall Scores that we feel confident about. If this item has not yet been reviewed by a reputable member on the site, we don't feel confident about giving it an overall score.
This makes it much more difficult for company owners and reps to maliciously manipulate the overall scores. We have seen many companies in the past either sign up themselves and try to boost their own scores or send their army of devout followers to flood the site with low-quality, one-hitter-quitter reviews.
At SR, we value building member trust and reputation. If you are just going to join, post one review and then never come back again, it is really hard for us to trust your opinion. We haven't had a chance to get to know you. So if a product has ONLY these types of reviews, its as good as having no reviews at all.
How can I get an Overall Score?
Great question! It's hard for visitors to trust a product that has a ?/10 rating. You can wait around with your fingers crossed, or you can get your products reviewed using the SR TROOPs.