If I was to ask someone that was a quality rater at Google what their interface looks like when they were asked to rate side by side search engine results they would probably say it looks something like this.
As you can see, the above image is fairly self-explanatory and you can see that a Google quality rater is asked to rate which search result is best.
If I was to ask the same quality rater if this was the only way they rate side by side search results they would say no and would then go on to describe an interface that would look something like this…
Advanced Side By Side Rating
Raters are asked to rate which side has the overall better results but are then asked to visit each page and rate them individually.
Raters then leave feedback to explain their reasoning for each individual result but are also asked to give an overview of all the results.
They can also flag websites as being spam, porn, foreign language, malicious or if the page didn’t load.
Usually raters are given 5 results each side to rate but occasionally more results can be displayed and sometimes only 1 result is displayed in one column and 5 results in the other – perhaps this is because Google are thinking about delivering only one result to users for certain searches, quality raters are also instructed that more results isn’t always better.
Quality raters are also told to make sure that diverse results are delivered in the search engine results page. So, for example, if you were to search for something like dogs, Google would want to serve results that has information on dog breeds, dog images, dogs for sale and perhaps a dog ecommerce site.
On the other hand, if a user is looking for something very specific such as buy dog accessories online then the entire page should be filled with ecommerce results as the searcher probably has a very specific intent.
It is also worth mentioning that quality raters are also asked to rate search results from different search verticals such as Google News, Image, Video and Shopping etc. So on occasion they may be comparing image results, video results and news results.
On other occasions they might be served with 2 results for the same website but they have different page titles and / or meta descriptions and the quality rater is asked to rate the best result.
As we have known for a long time, quality raters have been used by Google to give feedback on which search results are best.
It is very interesting to hear that quality raters are comparing results from different search verticals as surely the preference of what type of result a searcher wants to see will differ depending on the individual and could even be influenced by things such as the type of device that they are searching on.
I’ve noticed Google testing result in the past and when they only delivered one result for certain queries. Perhaps this is something that they are moving toward and it would be very interesting if this was to happen.
As Google has instructed its raters to comment on whether or not results were diverse enough, it may influence the way you target broad keywords. If you want to rank in a particular competitive niche for a certain keyword it may be worth looking at the top 10 results and building a website that has a totally different theme compared to the current results.