Google Begins Tackling Domain Crowding & SERP Diversity

It’s no secret that Google’s SERPs have been full of domain crowding for quite some time now. It’s a massive issue in some niches where a dominant domain can essentially take over the entirety of the first page of results.

The good news is that Google seems to be making a strong attempt to fix things lately.

What is domain crowding?

For those that haven’t come across this term before, I should probably quickly explain it. A domain crowd is anywhere you see sequential results in the SERPs from the same domain. Here’s an example in the [hotel reviews in glasgow] SERP where tripadvisor.co.uk has 3 sequential results on page 1:

Domain crowding became a bigger issue in the SERPs after the notorious Bigfoot Update that Dr. Pete wrote about on SEOmoz back in June.

The process

As I’ve mentioned before, we track around 50,000 unique keywords daily across G/Y!/B so we have a good set of data to play with. As with my EMD Update post a couple of months back, I’ve used a random selection of 5000 keywords to create an unbiased dataset.

All of this SERP analysis has been carried out on Google UK, but given the fact that flux seems to mimic US SERPs these days I think it’s safe to assume that algorithmic updates normally happen in unison now (although this is not necessarily true for ‘red button’ updates like Panda or Penguin refreshes).

I deliberately looked at quite a large timeframe for this study (blue box above) as there were mumblings of SERP diversity changes over the period of a few days despite SERPmetrics not reporting any single stand-out high volatility day during it. For these reasons, I’ve compared data from Nov 30th with Dec 6th.

Also, Mozcast showed a definite uptick in domain diversity over this timeframe so it certainly warranted further investigation (update: as well as a stormy weather day on the 5th).

For each of the 5000 keywords, I looked at the top 100 organic results and made a record any time I came across a ‘crowd’ of results (2 or more sequential results from the same hostname).

For each domain crowd I recorded the start position, the finish position and the number of results spanned. I didn’t look specifically at pages and, so, domain crowds could span them.

Anyway, enough of the rambling - let’s look at the results.

The results

Over the test period, occurrences of domain crowding (2 or more sequential results from the same hostname) have fallen from 51,384 to 43,606. That’s a difference of:

The actual number of results within a domain crowd fell from 156,038 down to 135,222 - a similar drop of -15.34%; but this alone doesn’t tell the whole story. Many people will say that 2 sequential results isn’t really crowding, so let’s look at the effect on different size crowds:

The number of occurrences for 2-result crowds far outweighs the bigger blocks so this isn’t easy to see, but you can fairly accurately determine at a glance that the numbers have dropped significantly. A better view would likely be the % change rather than actual numbers though:

Now it’s easy to see that there is a strong drop of around 15% for each crowd size, but this diminishes from the 7-block upwards which only saw a change of around -5%.

Personally, I would have expected to see increases in the lower sized crowds when the bigger crowds were being dispersed but these numbers tend to imply that multi-result crowds of all sizes have been reduced to singular results in a lot of instances.

Really? I didn’t notice much change

Well, there might be a good reason for that. I split the data into page buckets (1-10, 11-20 etc) and it flagged up an interesting anomaly. Domain crowding on Page 1 of the SERPs only saw a -5.5% change; so it seems most of the diversity clean-up happened below position 11: