Google’s Indexation Cap »

Posted by randfish

Over the past 2 years, SEOmoz has worked with quite a number of websites whose primary goal (or primary problem) in SEO has been indexation - getting more of their pages included in Google’s index so they have the opportunity to rank well. These are, obviously, long tail focused sites that earn the vast majority of their visits from queries that bring in 5 or fewer searches each day. In this post, I’m going to tackle the question of how Google determines the quantity of pages to index on a site and how sites can go about improving these metric.

First, a quick introduction to a truth that I’m not sure Google’s shared very publicly (though they may have discussed it on panels or formally on the web somewhere I haven’t seen) - that is - the concept that there’s an "indexation cap" on the number of URLs from a website that Google will maintain in their main index. I was skeptical about this until I heard it firsthand from a Googler being described to a webmaster. Even then, I didn’t feel like the principle was "confirmed," but after talking to a lot of SEOs working at very large companies, some of whom have more direct interactions with the search quality team, this is, apparently, a common point of discussion and something Google’s been more open about recently.

The "indexation cap" makes sense, particularly as the web is growing exponentially in size every few years, often due to the production of spam and more legitimate, but no less index-worthy content on sites of all sizes and shapes. I believe that many site owners started noticing that the more pages they produced, even with very little "unique" content, the more traffic Google would send and thus, abuse was born. As an example, try searching using Google’s "last 24 hours" function:

Seriously, go have a look; the quantity of "junk" you wouldn’t want in your search engine’s index is remarkable

Since Tom published the post on Xenu’s Link Sleuth last night, Google’s already discovered more than 250 pages around the web that include that content or mentions of it. If, according to Technorati, the blogosphere is still producing 1.5 million+ posts each week, that’s conservatively growing the web by ~20 billion pages each year. It should come as no surprise that Google, along with every other search engine, has absolutely no desire to keep more than, possibly, 10-20% of this type of content (and anyone who’s tried re-publishing in this fashion for SEO has likely felt that effect). Claiming to have the biggest index size may actually be a strike against relevancy in this world (according to Danny Sullivan, it’s been a dead metric for a long time).

So - long story short - Google (very likely) has a limit it places on the number of URLs it will keep in its main index and potentially return in the search results for domains.

The interesting part is that, in the past 3 months, the number of big websites (I’ll use that to refer to sites with an excess of 1 million unique pages) we’ve talked to, helped through Q+A or consulted with that have lost wide swaths of indexation has skyrocketed, and we’re not alone. The pattern is usually the same:

• One morning, you wake up, and 40% of your search traffic is gone with no signal as to what’s happened
• Queue panicking executives, investors and employees (oh, and usually the poor SEO team, too)
• Enter statistics data, showing that rankings for big terms aren’t down (or, maybe down a little), but that the long tail has gotten a lot shorter
• Re-consideration request goes to Google
• Somewhere between 10 to 40 days later, a message arrives saying:

We’ve processed your reconsideration request for http://xyz.com.

We received a request from a site owner to reconsider how we index the following site: http://xyz.com

We’ve now reviewed your site. When we review a site, we check to see if it’s in violation of our Webmaster Guidelines. If we don’t find any problems, we’ll reconsider our indexing of your site. If your site still doesn’t appear in our search results, check our Help Center for steps you can take.

• This email, soon to be recognized by the Academy of Nonsense for its pre-eminent place among the least helpful collection of words ever assembled, spurs bouts of cursing and sometimes, tragically, termination of SEO or marketing managers. Hence, we at SEOmoz take it pretty personally (as this group includes many close friends & colleagues).
• Calls go out to the Google AdWords reps, typically consisting of a conversation that goes something like:
  Exec: "We spent $10 million @#$%ing dollars with you last month and you can’t help?"
  AdWords Rep: "I’m sorry. We wish we could help. We just don’t have any influence on that side of the business. We don’t know anyone there or talk to anyone there."
  Exec: "Get me your boss on the phone. Now."
  Repeat ad nauseum until you reach level of management commensurate with spend of the exec’s company (or their connections)
  Exec: "Can you get me some answers?"
  AdWords Boss: "They won’t tell me much, but apparently they’re not keeping as many pages in the index from your site as they were before."
  Exec: "Yeah, we kind figured that part out. Are they going to put us back in."
  AdWords Boss: "My understanding is no."
  Exec: "So what am I supposed to do? We’re not going to have money to buy those $10 million in ads next month, you know."
  AdWords Boss: "You might try talking to someone who does SEO."
• At this point, consultants receive desperate email or phone messages

To help site owners facing these problems, let’s examine some of the potential metrics Google looks at to determine indexation (note that these are my opinions, and I don’t have statistical or quantitative data to back them up at this time):

1. Importance on the Web’s Link Graph
   We’ve talked previously about metrics like a domain-level calculation of PageRank (Domain mozRank is an example of this). It’s likely that Google would make this a backbone of the indexation cap estimate, as sites that tend to be more important and well-linked-to by other important sites tend to also have content worthy of being in the index.
2. Backlink Profile of the Domain
   The profile of a site’s links can look at metrics like where those links come from, the diversity of the different domains sending links (more is better) and why those links might exist (methods that violate guidelines are often getting caught and filtered so as not to provide value).
3. Trustworthiness of the Domain
   Calculations like TrustRank (or Domain mozTrust in Linkscape) may make their way into the determination. You may not have as many links, but if they come from sites and pages that Google trusts heavily, your chances for raising the indexation cap likely go up.
4. Rate of Growth in Pages vs. Backlinks
   If your site’s content is growing dramatically, but you’re not earning many new links, this can be a signal to the engine that your content isn’t "worthy" of ongoing attention and inclusion.
5. Depth & Frequency of Linking to Pages on the Domain
   If your home page and a few pieces of link-targeted content are earning external links while the rest of the site flounders in link poverty, that may be a signal to Google that although users like your site, they’re not particularly keen on the deep content - which is why the index may toss it out.
6. Content Uniqueness
   Uniqueness is a constantly moving target and hard to nail down, but basically, if you don’t have a solid chunk of words and images that are uniquely found on one URL (ignoring scrapers and spam publishers), you’re at risk. Google likely runs a number of sophisticated calculations to help determine uniqueness, and they’re also, in my experience, much tougher on pages and sites that don’t earn high quantities of external links to their deep content with this analysis.
7. Visitor, CTR and Usage Data Metrics
   If Google sees that clicks to your site frequently result in a click of a back button, a return to the SERPs and the selection of another result (or another query) in a very short time frame, that can be a negative signal. Likewise, metrics they gather from the Google toolbar, from ISP data and other web surfing analyses could enter into this mix. While CTR and usage metrics are noisy signals (one spammer with a Mechanical Turk account can swing the usage graph pretty significantly), they may be useful to decide which sites need higher levels of scrutiny.
8. Search Quality Rater Analysis + Manual Spam Reports
   If your content is consistently reported as being low value or spam by users and or quality raters, expect a visit from the low indexation cap fairy. This may even be done on a folder-by-folder basis if certain portions of your site are particularly egregious while other material is index-worthy (and that phenomenon probably holds true for all of the criteria above as well).

Now let’s talk about some leading indicators that can help to show if you’re at risk:

• Deep pages rarely receive external links - if you’re producing hundreds or thousands of pages of new content and fewer than "dozens" earn any external link at all, you’re in a sticky situation. Sites like Wikipedia, the NYTimes, About.com, Facebook, Twitter and Yahoo! have millions of pages, but they also have dozens to hundreds of millions of links, and relatively few pages that have no external links. Compare that against your 10 million page site with 400K pages in the index (which is more pages than what Google reports indexing on Adobe.com, one of the best linked-to domains on the web).
• Deep pages don’t appear in Google Alerts - if Google Alerts is consistently passing you by (not reporting, this can be (but isn’t universally) an indication that they’re not perceiving your pages as being unique or worthy enough of the main index in the long run.
• Rate of crawling is slow - if you’re updating content, links and launching new pages multiple times per day, and Google’s coming by every week, you’re likely in trouble. XML Sitemaps might help, but it’s likely you’re going to need to improve some of those factors described above to get in good graces for the long term.

There’s no doubt that indexation can be a vexing problem, and one that’s tremendously challenging to conquer. When the answer to the "how do we get those pages back?" is "make the content better, more unique, stickier and get a good number of diverse domains to link regularly to each of those millions of URLs," there’s going to be resistance and a search for easier answers. But, like most things in life, what’s worth having is hard to get.

As always, I’m looking forward to your thoughts (and your shared experiences) on this tough issue. I’m also hopeful that, at some point in the future, we’ll be able to run some correlations on sites that aren’t fully indexed to show how metrics like link counts or domain importance may relate to indexation numbers.

Do you like this post? Yes No

Xenu’s Link Sleuth - More Than Just A Broken Links Finder »

Posted by Tom_C

There are literally a bazillion SEO tools on the internet (literally!), this post discusses just one such tool; Xenu’s Link Sleuth. Many people in the SEO industry are already aware of this tool but many people I’ve spoken to only treat the tool as a broken link finder. It’s so much more than that.

This post is aimed at those who haven’t heard of it before and those who do use it regularly - there are lots of nifty features that solve all kinds of SEO-problems and hopefully beginners and advanced alike will learn something from this post.

What is Xenu?

Xenu’s Link Sleuth is a FREE download (everyone loves free) that runs on all versions of Windows (but not quite on Macs unfortunately). It’s a lightweight download and I’ve never had issues with it crashing or hanging. In a nutshell it’s a site crawler and once you point it at a URL it will crawl around the site and spit out a report at the end. It’s main focus and branding is all about finding broken links on your site (so where you link internally to a 404 error) but I’ve found that I use it to solve a whole host of different SEO-related issues which I will explain below.

Problem - How do I find broken links on my site?

This is the most basic use of Xenu in my opinion, but also the most common use. Simply point the program at the homepage of your site, check ’skip external’ to avoid it crawling the entire web, and set it going!

Click here to view a sample report provided by Xenu for the Distilled site (note that this is a sample report only, not run across the whole site).

You can see that there is a handy section which reports any broken links that it finds, though in this case I’ve chosen a rather poor example since there are no broken links on the homepage of Distilled :-)

Problem - How do I get a crawl of my site into microsoft Excel?

The answer to this one, as you may have guessed is also Xenu! Simply choose the following menu option once the report is run:

Click here for a google docs of a sample report from the Distilled site. As you can see you get some really useful data such as:

• The status code of all pages crawled
• The type of page crawled
• The title tag of each page crawled

Problem - How do I check the length of my title tags across my whole site?

Looking at the above data sheet - simply filter for html pages and then check the length of the column titled "Title" - this will give you the length of the title tag. Filter for any above 65 and bingo - there’s your to-do list!

Problem - How do I analyse my site’s information architecture?

Yep, you guessed it - Xenu will do this too. This one requires a little more explaining however. Firstly, you see that in the spreadsheet above there is a column for "level" - what this column tells you is the number of links away from the initial link that you entered the crawled page is. So in the example sheet all the pages have a level of 1 since I restricted the crawl to just those pages 1 link away from the homepage.

This is really useful information as it tells you how many clicks it takes to get to a given page on your site from the homepage. Useful information! Especially in a large site where you have multiple levels of information architecture and several different types of navigation. Below is a quick screenshot of a report run 3 levels deep on the site. I’ve pivot-tabled the data (zomg - excel ftw) and selected the following options:

Of course, the beauty of pivot tables is that I can double click each of those rows and see which pages are contained within each level. This is of course, a pretty basic application of the data. But you see that once you start getting more data you can do more powerful things.

The second application of the very same data is the useful links in/links out column which looks like this:

There are other ways of getting this data for your site, Linkscape does it for example, but the good thing about Xenu is that you get the data structured in Excel and you have all the other page metrics alongside it. There’s plenty more you can do with this but at a very crude level you can use it to identify pages with more than 100 links on the page across your site!

Taking this data to the next level - here’s a glimpse at what’s possible, an analysis of type of page vs number of internal links shows you that for this site (not the distilled site) the money pages are getting very few internal links compared to top level pages and something is broken in the information architecture:

Problem - How do I find any 302 redirects on my site?

Xenu to the rescue! In order to catch redirects on your site you need to modify one of the settings on the crawl preferences to "treat redirects as errors":

Then, when you run the report and export to excel redirects will no longer get the status code 200 but will get the true status code, be it 301 or 302! Perfect.

Problem - How do I check the indexing of a test version of my site?

Xenu of course! If your test version lies at a public URL such as testsite.distilled.co.uk then you can just point Xenu at that URL. However, if that’s not an option then you can even run Xenu off a local HTML file which is pretty nifty:

Problem - How do I generate an XML sitemap for my site?

Although there are many many ways of generating an XML sitemap for your site, Xenu does this in a quite nice (if not particularly customisable) way. This is perfect for small site owners with limited technical knowledge I think:

Problem - How do I find images missing alt text?

If only Xenu would do such a thing…. Wait, it does! Simply filter your excel download to image files, then the "Title" column is the alt text of the image:

Well that’s just a few of the many many applications of the Xenu tool - hopefully it’s inspired you to go out and give it a try - I know I use it a lot for all kinds of things. I mean, once you get your data into Excel the world is literally your oyster. Mmmmmm data oysters.

But wait! That’s not all - I reached out to Rich Baxter as I know he’s a very knowledgeable and smart SEO and he uses Xenu a lot. I asked him if he had any killer tips and here’s his killer tip. Thanks a lot Rich for getting me this at short notice:

Crawling web directories, looking for errors (By Rich Baxter)

Xenu’s not just a great tool to look inside your own site, it’s also pretty powerful for crawling external resources like directories, particularly if you’re looking for a domain to buy.

Try crawling dmoz.org, being sure to restrict Xenu’s access to “editors.dmoz.org”, but allow the crawler to “check external links”.

Quite quickly you’ll start finding “not found” URL errors from directory entries that might have been forgotten, on domains that may not yet have expired. Just sort by “status” in the crawl results table in Xenu. Here’s one I found earlier. I’m pretty sure that with the right offer via SEDO, the owner of fridgemagnet.org.uk (with its 634 sub domain links) might be interested in selling before the domain expires.

I’ve always found the “Copy URL”, Google cache and Wayback Machine links invaluable on a right mouse click on the results you’re interested in:

As a side note: If you are crawling external resources, try to be a good citizen and crawl slowly. Set your maximum threads to a very low level, so as not to get your IP banned by your target host.

Thanks Rich! Great tips. Let’s get link sleuthing! If anyone has any other creative/useful uses for Xenu please share them in the comments.

In other news, there’s still time to get your FREE SES Chicago Pass by purchasing a year of PRO! (If Chicago’s not your thing, SES will let you exchange the pass for any SES Event in 2010). SES just raised their prices to $1995 for a pass, so $799 for an entire year of PRO and a full-access SES Pass is an awesome deal

You’ve also still got 7 days left to get super-low release pricing on the brand-new SEOmoz Advanced SEO Training Series: Tips, Tricks & Tactics six-disk DVD set! Until December 6th, you can get this killer new series for 20% off and Free Shipping anywhere in the world. Hundreds of people have already ordered and supplies are limited, so take advantage while the price is low, and this limited-edition set is still available.

Do you like this post? Yes No

SEOmoz’s First Ever PRO Webinar Dec. 10th: We Need Your Feedback & Suggestions »

Posted by randfish

First off, I hope everyone among our US (and expat) readers had a great holiday weekend, filled with tryptophan and football (I know mine was). Second, I’m very excited to announce that SEOmoz PRO is launching our first ever webinar on Thursday Dec. 10th at 11am Pacific (2pm Eastern, 7pm London/UK). We’ve heard from a number of our members that they’d like to do some live reviewing of strategies and recommendations and get questions tackled in this format. I’ll be running the webinar personally, but I haven’t quite decided on a topic, and that’s one thing I need your help with.

Below is an embedded Google form (they’re pretty spiffy) with three short questions. We’ll use your feedback to help determine the content and format for the webinar, as well as gauge interest level.

We’ll have another blog post in the next few days announcing details (based on your requests + votes), as well as an email to all PRO members with a registration link.

Of course, if you have anything to add in the comments or any recommendations, we’d love to hear from you there, too. If this webinar goes well, we’re certainly planning to make it a monthly event for PRO members, and possibly offer some free webinars to the entire community. A future subject that folks have been asking about is training on the SEOmoz toolset - that’s something we want to do, but we have some changes + additions coming in January, so we’ll get those released first, then follow up.

Thanks for your feedback and happy holiday season! 

Do you like this post? Yes No

Whiteboard Friday - Analytics for SEO »

Posted by great scott!

Happy belated Thanksgiving my American moz-Comrades, and welcome to Black Whiteboard Friday (and just Friday to everyone else)! 

This week we’re looking at analytics. No, not the normal stuff you probably pay attention to, but a few sneaky little metrics that can give you some great SEO insight.  Since it’s now officially the "Holidays," I’m not going to reveal too much; rather, I’ll let the video serve as an early gift, ready to be opened! Suffice to say, prepare to start looking at your referral traffic, abandonment rates, browse rates, and more in a whole new light :)

SEOmoz Whiteboard Friday - Analytics for SEO from Scott Willoughby on Vimeo.

Since it is Black Friday, I want to remind you of some killer deals that are still available for you or your favorite SEO!

There’s still time to get your FREE SES Chicago Pass by purchasing a year of PRO! We’ve only got about a dozen passes left, so you should probably hurry. SES just raised their prices to $1995 for a pass, so $799 for an entire year of PRO and a full-access SES Pass is an awesome deal (and if Chicago’s not your thing, SES will let you exchange the pass for any SES Event in 2010).

You’ve also still got 10 days left to get super-low release pricing on the brand-new SEOmoz Advanced SEO Training Series: Tips, Tricks & Tactics six-disk DVD set! Until December 6th, you can get this killer new series for 20% off and Free Shipping anywhere in the world. Hundreds of people have already ordered and supplies are limited, so take advantage while the price is low, and this limited-edition set is still available.

Do you like this post? Yes No

This Week in Search for 11/26/09 »

Posted by Sam Niccolls

• Google Displaying Site Hierarchies in SERPs:
  Google provides yet another reason to have sound site architecture.

• Google Displaying Movie Times in SERPs:
  In addition to the many other things Google is testing, movie showtimes are being shown in the SERPs.

• ACC Approves Microsoft Yahoo Deal in Canada & Australia:
  After a 40 day review the ACC approved the deal which will allow Bing to power Yahoo search results in Canada & Australia.

• Michael Gray - How Bing & WSJ Could Team Up on Google:
  Whether you agree with Gray or not, his post aggregates some of the most relevant articles following Rupert Murdoch’s announcement several weeks ago.

• SEOmoz Advanced Training Series on DVD:
  The latest release of the training series has arrived. And if you order the 6-disc DVD set before December 6th you will get 20% off, plus free shipping.

• Avinash Kaushik - Measuring Social Media:
  How should you track success on sites like Twitter? In a forward-thinking post on social media analytics Avinash offers several suggestions. On a related note, Econsultancy also has a list of 35 Social Media KPIs worth looking over.

• How Web Analytics Helped the Obama Campaign Win:
  Dan Siroker left Google to become the Director of Analytics for the Obama Campaign. In his hour long presentation he shares some amazing statistics & interesting tactics they used to help Obama win the election.

• Cuil Theory Poster:
  Licenseplate’s tongue and cheek poster captures the essence of Cuil.   

• Link Builders Share Advanced Queries:
  You’ve probably used the Advanced Search Operator Guide (it’s a useful resource if you haven’t), but Garrett French’s compilation includes 21 advanced queries used by expert link builders to identify link opportunities.

• 69 Sexy Portfolio Designs:
  The designs are not all conversion focused, but Smashing Magazine’s list of 69 portfolio designs has some elegant layouts that’ll provide idea inspiration.

• Google News - Eric Enge Interviews Josh Cohen:
  Maile Ohye talked about Google News SEO and best practices a number of months back, but Enge’s interview with Google News’ Senior Business Product Manager offers some useful Google News insights.

• Get Elastic - Increase Average Order Value:
  GoDaddy might be the extreme when it comes to upsells during their checkout process, but if you have an e-commerce site, testing the location of your "continue shopping" button on your checkout pages might lead to increases in average order value.

• Online Dating Trends:
  How much do looks matter on dating sites? OK Cupid helps answer the question with interesting user behavior data.

• 21 Tools for Spying on Your Competitors:
  Bryan Eisenberg’s compilation of competitive analysis tools is a useful one. Although admittedly, if it were my list I might add a few other SEO tools, too. :-)

• Feedburner Integration with Google Analytics:
  Google Analytics users now have better campaign tracking and subscriber consumption statistics for their Feedburner feeds. 

• Page Load Times & AdWords Quality Score:
  Matt Cutts recently talked about how page load times will play an even bigger role in organic search, but ROI Revolution shares a few other reasons to identify and fix long page load times.

• e-Marketer - Cautious Optimism for Holiday Sales:
  Good news for e-commerce sites from e-marker. The holiday season could yield big online wins for retailers.

• Google Analytics API - Development Growth Continues:
  Juice Analytics’ flow visualization is yet another creative extension of the rapidly growing Google Analytics API.    

• Linkbait - Mint Aggregates Finance Related Twitter Streams:
  Mint has great design and their blog has it’s fair share of great linkbait, but their new Twitter aggregation is also intriguing.

• Luna Metrics - GA Plugin for Copying Goals & Filters Across Multiple Profiles:
  If you’re a GA user and you have to set up 20 profiles with the exact same goals, the Luna Metrics plugin is a great time saver.

Top YOUmoz entries:

1. Annual Local Demand Using Google Keyword Tool by identity
   

2. Best Practices for Setting Up a Twitter Account by danazarrella
   

3. SEO Daily Workout by Martini81Vette

4. Actionable SEO Insights from Google Analytics by kim_toomey 
   

5. Google Apps - Getting Efficient at SEO  by The Lost Agency
   

Do you like this post? Yes No

Favorite Tidbits from PubCon 2009 »

Posted by jennita

Photo courtesy of Michael Dorausch

It has been a couple weeks since PubCon and yes, it’s taken me this long to recuperate! Bare with me here as I reminisce about my favorite tidbits over the 3 days of sessions. PubCon does an amazing job of putting together a schedule that covers a multitude of tracks and topics. Which usually means that I often end up with a case of "session envy." You know, when you’re sitting in a session and you start to see tweets about how great another session is. It doesn’t even mean that the one you’re in is bad, it’s just that you want to be in two places at once.

Many people were live blogging the event and obviously there have been quite a few recaps of PubCon as well. I realize I’m a little late to the game. :)  But, let’s get to the good stuff! While there were many great speakers and presentations there were a couple that stood out to me. What I was looking for were answers to questions that we often get either through comments on the blog or through Q & A. With that said, these are my favorite take-aways from PubCon 2009.

Day 1 - How Do You Optimize For Universal and Personal Search?

This was a great session, full of valuable information. The biggest takeaway for me, came from  Bruce Clay. He took the road of sticking to a very specific topic, image and video content and explained how to get them indexed.

Since these are topics that come up quite a bit in Q & A I was excited to get some specific information on interesting ways to index images and videos. Thanks to Virginia Nussey over at Bruce Clay sending me the actual presentation (woot!).

Case Study - Indexing Images

The first thing Bruce spoke about was a case study they did involving getting images with text indexed. Here are the steps they took:

• Take a newspaper article
• Scan it to create a jpg
• Include that image into a PDF
• Submit that PDF to Google
• Search for info in the article as a PDF and the article is found.

So what happened? Well the PDF was indexed AND the text from the image was also indexed. Take a look at this SERP.

The result above seems to only come up when using some data centers but not all. :) Also, the search query contains text directly from the article which is an image in the PDF. It shows up in the title and meta description. Pretty sweet eh?

Obviously this isn’t going to work for you in all cases, but it’s an interesting way of getting images indexed and could be useful in a number of scenarios.

Case Study - Indexing Video

Next, Bruce went through another case study where they got a video indexed. Here are the steps they took.

• Take a Video
• Run it through our processor
• Edit the text transcript and save with the image
• When playing the video, search for words
• Jump into the Video

You can see this most clearly at Google audio indexing in the Labs section. Do a search for "economy" then notice how it shows you where in the video the word "economy" is spoken. I can see this being the wave of the future. How awesome would it be to have your videos come up in a search for words that were spoken within the video. Yea, pretty cool.

What I really liked about this presentation was that the information was displayed as a case study. If you read SEOmoz often you know how much we like to test theories out and put them to action. And although indexing images and video in this way may not be new to everyone, it’s new to some people and could be a great way to enhance your sites.

Day 2 - SEO/SEM Tools

This was seriously one of my favorite sessions. The speakers (including Rand) had a ton of great information but the one that really stood out to me was Jim Boykin of We Build Pages. He gave away some really great tips but the one I liked the most had to do with how to find out what keywords your competitors are targeting. Here are the steps to take:

• Go to the Google Adwords Keyword Tool
• Enter in the URL of your competitor’s sitemap
• check box for " Include other pages on my site linked from this URL"
• When you get results, Change "Match Type" to "Exact Match"to see actual searches and volume for specific phrases.

Obviously not only does this help with checking out which keywords your competitors are focused on, but it can also help you make sure you’re site is also focusing on the right keywords.

Day 3 - The Search Engine Smackdown

For me the third day was my favorite. I actually wanted to attend every session in the first slot. I ended up watching the first part of the Link Buying session and the second part of the site review with Matt Cutts. I have to admit also that I loved the fact that the PubCon team programmed Matt Cutts doing a site audit at the same time as the Link Buying session… pretty darn ingenious if you ask me! Rand posted information about the site review, so I wont go into detail but it was a great session.

Although I think the best part came after the session when Matt took to shaving Evan Fishkin’s head due to a bet they made while at the SEOmoz Werewolf party earlier in the week. And although that alone was pretty great, I loved the fact that while Matt was shaving Evan’s head, people were asking him questions and he was answering as if it were a regular Q & A. Here’s a short clip of the shaving!

Ok, ok back to the Search Engine Smackdown. I should admit that I debated whether I should head back to the hotel after a long 3 days or go to the final session, and I’m glad I went! These are the topics I found most interesting… and yes they’re all from Matt Cutts.

Google Social Search
Here’s the information straight from the site:

Sign in to Google and do a search. If there’s relevant web content written by people in your social circle, it will automatically show up at the bottom of your search results under a section called "Results from people in your social circle."

To see even more social content you can click the "Show options" link at the top of the results page and then click on the "Social" link.

What is your social circle? It’s a combination of your Gmail chat buddies, your Gmail contacts friends, family and co-worker groups, and people you’re publicly connected to on other social sites (such as Twitter and FriendFeed). Learn more about social search.

I can see myself using this quite a bit. Most of my job is about "social searches" in general,so this one is right up my alley! It was a fun find for me.

Testing for Speed

As Rand mentioned in his post, Google representatives have mentioned several times that page load time is important and Matt himself said at this session that although they haven’t used speed as a ranking factor in the past, that they’re thinking about adding it in the future. He gave some resources on how to check the speed and ways to make it faster.

• http://code.google.com/speed/ - Ways to clean up your site’s speed
• webpagetest.org - Let’s you input a URL, select test location and other configuration options, then test!
• http://code.google.com/closure/ - a JavaScript compiler

Wrap it up

I’m sure if you follow me on Twitter you know quite well how I feel about Vegas. I really wish there was a good way to clone myself so I could attend multiple sessions at once. With up to 7 tracks going on at the same time, it’s difficult to pick one. In the end PubCon was a hit, and it was great to meet many of our SEOmoz members and see some old friends. See you again next year!

Kate Morris, Dana Lookadoo, Amanda Stewart, Jen Lopez, Kristy Bolsinger, Lyndsay Walker, Joanna Lord, Manda Otto
Thanks to Dana Lookadoo for all the great photos!

Do you like this post? Yes No

Illustrating the Long Tail »

Posted by randfish

The long tail of search demand has been around since the dawn of web search and, since that time, search marketers have been attempting to tap into the powerful stream that high quantities of unique content can provide. I recently came across some great data from Hitwise (about 1 year old, but still highly relevant) showing off just how substantive the long tail can be. Bill Tancer’s post - Sizing Up the Long Tail - gives some stats:

…the head and body together only account for 3.25% of all search traffic! In fact, the top terms don’t account for much traffic:

• Top 100 terms: 5.7% of the all search traffic
• Top 500 terms: 8.9% of the all search traffic
• Top 1,000 terms: 10.6% of the all search traffic
• Top 10,000 terms: 18.5% of the all search traffic

This means if you had a monopoly over the top 1,000 search terms across all search engines (which is impossible), you’d still be missing out on 89.4% of all search traffic. There’s so much traffic in the tail it is hard to even comprehend. To illustrate, if search were represented by a tiny lizard with a one-inch head, the tail of that lizard would stretch for 221 miles.

Top 10,000 Search Terms by Percentage of All Search Traffic

The truth is my research is still greatly understating the true size of the tail because:
• The Hitwise sample contains 10 million U.S. Internet users and a complete data set would uncover much larger portions of the long tail.
• The data set I used filtered out adult searches.
• I only looked at 3-months worth of data (which were some of the slower months for search engines).

To help put this in perspective, I made a few spiffy charts that can help to illustrate these points:

In this first chart, you can see a representation of Hitwise’s data from the four chunks Bill broke down.

In this next representation, I’m showing the classic "long tail" style curve, but color-coded to help show the various areas of keyword demand. Note that you could conceptually say that the 9,000 of the top 10,000 terms should technically fit into the chunky middle. Bill classified them thusly in his post, but I tend to think that at those demand levels, we’re still talking about "head" of the curve figures.

For both of these graphics, there’s a large, high-res version available by clicking the chart. You can find lots, lots more on our Free Charts page :-)

Do you like this post? Yes No

Google Link: Command - Busting the Myths »

Posted by randfish

I’m a big Google fan - my wife often sleeps in their t-shirts, I speak on panels with Googlers all the time and I’ve even got a Google water bottle for working out (which happens all of once a month these days). However, I am NOT a fan of the Google link command, and I’m shocked by the number of folks who operate in and around the SEO, webdev and technology industries who haven’t realized this.

Here’s what Google themselves have to say on the matter:

You can perform a Google search using the link: operator to find a sampling of links to any site. For instance, [link:www.google.com] will list web pages that have links pointing to the Google home page. Note there can be no space between the "link:" and the web page URL.

To see a much larger sampling of links to any verified site in Webmaster Tools:

1. On the Webmaster Tools Home page, click the site you want.
2. Under Your site on the web, click Links to your site.

Note: Not all links to your site may be listed. This is normal.

Here’s what Matt Cutts (head of Google’s Webspam team) had to say in a video on the subject:

The short answer is that historically, we only had room for a very small percentage of backlinks because web search was the main part and we didn’t have a ton of servers for link colon queries and so, we have doubled or increased the amount of backlinks that we show over time for link colon, but it is still a sub-sample. It’s a relatively small percentage. And I think that that’s a pretty good balance, because if you just automatically show a ton of backlinks for any website then spammers or competitors can use that to try to reverse engineer someone’s rankings.

Google themselves is telling us not to pay too much attention to the link command, but that doesn’t seem to be stopping folks. Let the myth busting commence.

Myth #1 - The Google Link Command Returns Accurate Numbers

Nope. Not even close. Google themselves say the numbers aren’t accurate and that they’re showing a small sub-sample. The numbers show this as well. Check your link counts with the Google link command vs. the number inside Google’s Webmaster Tools (when you verify your account, you’ll see them shown). Here’s the stats for SEOmoz, for example:

Google’s link command claims 1,590 links. Let’s see what Webmaster Tools says:

Hmm… 381,403 seems slightly larger than 1,590. In fact, the link command is showing me 0.4% of what Webmaster Tools says exists. Running this analysis on another few domains that we have access to in Webmaster Tools, I saw numbers ranging from 0.1% to 4.4% (meaning there’s not even any consistency between in the percentage of links from the two counts). 

Myth #2 - The Google Link Command Returns Important Links

Tragically, a long time ago (pre-2004), Google did show only important links via the link: command, which created the myth that exists to this day. In fact, the links shown in the link: command have no particular importance or relevance. They are truly a random sample, including links that are nofollowed, links from pages that have had PageRank penalties applied to them as well as links that do pass link juice and value.

Myth #3 - The Google Link Command Returns Links in Some Kind of Order

No one in SEO has been able to show any ordering of any kind in the Google link: command’s results. Important, well-known websites may be listed on page 2 or page 20 of the results, and it is likewise with spam, scrapers and low quality sites that Google’s likely not counting. In Site Explorer and the web results, Yahoo! appears to do some type of ordering, tending to show more important links, pages and sites before less important ones (though not with great consistency). Unfortunately, many SEOs suspect that, should Microsoft’s deal to power Yahoo! with Bing results go through, Yahoo! is unlikely to maintain their own web index (and thus, link, linkdomain and site explorer will be gone).

As exemplified above, Google appears to be very random indeed when showing link: results.

Myth #4 - The Google Link Command Returns a Numerically Representative Count of Links

This is possibly the myth that’s most disturbing of all, primarily because so many operators in the SEO field belive it and track the link: command count as a reliable, useful metric. Nothing could be further from the truth - and here’s some data to help back it up:

I collected the data above spur of the moment, so I won’t try to claim great statistical integrity. However, looking at Google’s link: command results, the best I can say is that Google has some relationship to the others within 1-2 orders of magnitude, though they may be directionally inaccurate much of the time as well. Just look at the NYTimes.com for example - Google claims they have 2/3rds the links that Salon.com has, yet Yahoo! and Linkscape agree that, in fact, NYTimes.com has 6X+ Salon.com’s link total.

These are not numbers you want to hang your hat (or any crucial business decisions) on.

Myth #5 - The Google Link Command Tracks Accurately Over Time

Unfortunately, I don’t have data points I can show, but our observations over time indicate that Google’s link count in Webmaster Tools might rise, along with the Yahoo! and Linkscape link counts, yet the Google link: command will show lower numbers. The reverse is sometimes also the case. Without directional consistency, even when compared against their own counts, it’s very hard to take the Google link: count seriously.

Myth #6 - The Google Link Command is Up to Date

Most SEOs & webmasters have noticed that the Google link: counts update infrequently, inconsistently and most often in correlation with toolbar PageRank updates (another data point I’ll need to takcle in a future post). These updates from Google occur every 2-10 months with little warning about when they’re coming or have happened. If you watch sites like closely, they’ll report many of these as they occur.

The next time someone tells you their Google link: command numbers as a metric for SEO, competitive analysis or anything else, make sure they read this post. Google’s not nearly as up-front with the information as they should be (honestly, removing the link command would save so much time and effort for poor site owners who get needlessly confused), but hopefully as a community, we can help build more awareness around this issue.

Do you like this post? Yes No

SEO and Social Media Benchmarking »

Posted by RobOusbey

Introduction

We’ll often rattle off various metrics quite casually in conversation, but it’s easy to forget that others (such as your clients!) might not know whether these are big numbers or small ones. For example: “We’ve just published a guest post on a site with Domain mozTrust 5.67. Mr_Gadget mentioned it to his 64 thousand followers, and it managed to get over 1,000 Diggs.“

This post should be useful to anyone who needs to orientate themselves around such numbers. The Website Benchmarks section shows metrics for around half a dozen sites from ten different niches (based on the list originally prepared for comparing SEOMoz Trifecta data.) This is followed by Social Media Benchmarks, to give an idea of the impact and influence of votes and people on three popular social websites.

Links and Traffic: Website Benchmarks

This section uses three particular metrics:

• Domain mozRank: the strength of a website, based on the sites & pages that link to it,
• Domain mozTrust: the trustworthiness of a website, based on links from trusted sites & pages,
• Unique Vistors: Compete.com’s estimate of the number of monthly unique visitors to a site.

N.B.: The ‘moz’ metrics use data exposed by SEOMoz’s Linkscape, and are on an exponential scale from 0 - 10. Data collected in 2006 suggested that the no visitor metric - Compete included - could be used to precisely estimate unique visitors to blogs at the time.

Social Media Benchmarks

Digg

Reddit

Twitter

Do you like this post? Yes No

Whiteboard Friday - Content & Technology Licensing »

Posted by great scott!

Looking for a super-powerful tactic to build lots of high-quality links? Well we’ve got a winner for you! Licensing your content and/or data can be an immensely powerful, highly scalable strategy for building powerful links and brand awareness alike.  It’s incredibly effective for folks who have quality content or data and want to leverage that material into a great link building solution. 

Be warned though: there are important rules to consider in order to avoid potential duplicate content issues as well as cannibalization.  You want your content licensing working for you, not against you, so watch this week’s WBF to learn how you can manage licensing arrangements to best reap the benefits…

SEOmoz Whiteboard Friday - Content & Technology Licensing from Scott Willoughby on Vimeo.

Do you like this post? Yes No

The NEW Advanced SEO Training Series: Tips, Tricks & Tactics now Available on DVD »

Posted by great scott!

Ready to upgrade your SEO skills, learn actionable new strategies, and get ready to make 2010 the year you totally rock online? Well, we’ve got just the thing to help you finish out the year with the latest tactics to take your SEO to the next level.

That’s right, the SEOmoz Advanced SEO Training Series is back with an all new DVD Series: Tips, Tricks & Tactics!  Whether you’re a fan of the series, or this is your first time experiencing SEOmoz Training DVDs, this new six DVD set is one you won’t want to miss. Filmed before a live audience in Seattle, Tips, Trick & Tactics features over a dozen world-class instructors teaching high-level, actionable strategies on 17 advanced topics.

This brand new edition of the Advanced SEO Training Series includes over 10 hours of immediately actionable material designed to really push your SEO skills to the next level. These advanced strategies will help you launch your SEO plan or reinvigorate your existing efforts with fresh tactics that will change the way you think about online marketing.

Don’t miss out on your opportunity to get the latest and greatest SEO strategies from the world’s top experts. The information you get in Tips, Tricks & Tactics is the sort of in-depth, high-level stuff these pros don’t normally share at conferences, or even online. We actively encourage them to share their best advice, and most effective strategies that they use to be at the top of the SEO game…and they really deliver!

Until Sunday, December 6th you can get the Advanced SEO Training Series: Tips, Tricks & Tactics 6 DVD Set for just $399 (that’s $100 off the regular price of $499) and we’ll include FREE Shipping.  That’s less than 1/3 of what most industry conferences cost, and you can share this among your entire team (no travel expenses either).

Want to save even more?  SEOmoz PRO Members only pay $199 during launch ($249 regularly). Do the math and you’ll see that this is a great opportunity to get the brand new Advanced SEO Training Series, try SEOmoz PRO and save a ton of money at the same time.  Order soon, because the special release pricing will only last until December 6th and these DVD Series’ sell out every year (we only do a limited run of each DVD series). Get SEOmoz PRO and save a ton of money at the same time.  Order soon, because the special release pricing will only last until December 6th and these DVD Series’ sell out every year (we only do a limited run of each DVD series). Get Tips, Tricks & Tactics now, and rev up your SEO in the new year!

What’s covered on Tips, Tricks & Tactics?  Tons!  This series features 15 Instructors (including Rand Fishkin, Todd Malicoat, Vanessa Fox, Ben Jesson, David Mihm, and more) speaking on almost every topic important to the advanced search marketer…

Disk One

• SEO for Content
• Advanced Site Architecture
• Advanced Keyword Research

Disk Two

• Local Search Strategies
• Alternative SEO Tools
• News & Image Search Strategies
• Advanced Linkscape Tactics

Disk Three

• How to Design a Link Building Campaign
• How to Launch a Startup

Disk Four

• History of SEO
• Future of SEO
• High ROI Social Media

Disk Five

• Researching the Algorithms
• How to Win SEO Budget
• Conversion Rate Optimization

Disk Six

• SEO Business Strategies
• Avoiding Search Engine Penalties

Still want more? We have a limited quantity of free passes to SES Chicago ($1,995 value) if you purchase a new 1-year PRO Membership! That means you can get the Tips, Tricks & Tactics DVD Training Series, a full year of SEOmoz PRO, and an all-access pass to SES Chicago for almost $1,000 less than just the conference pass alone! Not going to Chicago? SES will let you exchange your free Chicago pass for any of their other conferences in 2010, including London, West, New York, or any other!

What Did Live Attendees have to Say About Tips, Tricks & Tactics?

"If last year was ‘tech school’ this year is ‘Grad School’! Really powerful presentations."

"I’m the SEO for a $200 million company and I plan to use the knowledge I gained from this Training to radically change the foundations of my company."

"I’ve seen the future…This is absolutely essential to businesses and did nothing short of blowing me away."

"I’d recommend it to anyone but my competitors."

"I felt like I had a serious wake-up call to remember why I’m optimizing websites in the first place."

"It’s nice to hear about theories and logic, but these presentations with examples definitely make the ‘putting it into practice’ part much easier."

"I’ve been doing SEO for four years now, and I found a lot of things in here that you just can’t find online."

"This is very high-level, actionable stuff!"

"There’s a lot of information! It really gives you the edge."

Get Tips, Tricks & Tactics now and get the edge for the new year!

Do you like this post? Yes No

Show What You Know: The SEO Keyword Research Quiz »

Posted by Danny Dover

  Psst! Word on the street is that you are a big deal. Some say that working with you is even more fun than changing a co-workers primary Google language to Swahili¹. Well now, it is time to put your reputation to the test!

All of us at SEOmoz are proud to announce the brand new Keyword Research Quiz. This lean, mean, question asking machine was designed to test your sugar infused SEO brain by covering all corners of the Keyword Research world. The questions are weighted by difficulty to make the harder questions worth more points than their easier counterparts. The result is that keyword research beginners walk away with their tails between their legs and the true Dark Lords of the SERPs can return to their rightful place at the front of the Star Trek IMAX line. (Well that was rude…)

If you are able to finish the quiz, you’ll have the chance to share your triumph by letting your competitors know how long they should cry in angst at night.

---

¹Changing the default language to Swahili is extra funny/mean because it changes the language options in Google’s preferences drop down to their Swahili translation. For example: ‘English’ is changed to ‘Kiingereza’. Obviously this makes it very difficult to switch back.

Do you like this post? Yes No

Whiteboard Friday - Special Wednesday Edition: Link Growth Patterns »

Posted by great scott!

Happy Wednesday, Whiteboard Friday Fans!  You may have noticed that there was no new Whiteboard Friday on, well, Friday. That’s because a sizable battalion of the mozSquad went to Vegas for PubCon and, alas, I didn’t bring the camera with me (it’s actually prohibited by law–what happens in Vegas, and all that). Nonetheless, Rand and I feel horribly guilty about leaving you sans-WBF last week and wanted to make it up to you. Thus, I present you with this Special Wednesday Edition of WBF.

What makes it "special" other than titular anachronism?  First off, you!  You’re special and we love you, so thanks for watching. Second, I’m trying out a new video embed service called Vzaar. They were generous enough to hook me up with an account and let me try them out, so any feedback about audio/video quality, smoothness of streaming, compatibility, etc. would be greatly appreciated.
UPDATE: I switched the embed back to Vimeo since a lot of people seemed to be having trouble with Vzaar…thanks for the feedback!
So, what is the topic of this week’s WBFSWE? Link Growth Patterns!  Rand looks at how some typical link growth patterns tend to be interpreted by the engines.  Did you launch your site with a massive PR blitz and then stop all efforts? The resultant spike and sudden drop-off in link acquisition could cause you some serious problems in the SERPs.  There’s got to be a better way, right?  Indeed!  Rand will also show you how to construct a staged link acquisition strategy that will provide a much more natural link-growth curve that the engines will trust and respect.

Have fun, and we’ll see you in a couple of days for a good ol’ proper Whiteboard Friday.

SEOmoz Whiteboard Friday - Link Growth Patterns from Scott Willoughby on Vimeo.

Do you like this post? Yes No

5 Simple Tips for Better SEO Value from Your Feeds »

Posted by randfish

I’ve been connecting with a lot of site owners who are re-entering or ramping up their efforts in the blogosphere. I suspect this has something to do with the focus on content creation + linkbait in the SEO world’s dialogue as well as the potential new traffic streams bloggers are feeling from the surge of linking via Twitter. Whatever the case, there’s a few critical pieces that can help make for greater SEO value from blogging and feeds in general (and most of these haven’t been covered in my previous posts on blog optimization).

#1 - Control Your Own Feed

It’s hard to write something better than Danny Sullivan’s terrific piece on Staying Master of Your Feed Domain. The concept is that you can utilize services like Feedburner, but you want those feed URLs to originate from your domain (so you keep the link juice you’re earning):

To make this work, you need your hosting provider to create a CNAME entry for a new subdomain you’ll create. If they can’t do that easily for you, find a new hosting provider. I highly recommend ours, Tiger Technologies. Cheap, easy for you to do this yourself, plus Digg-tested.

For me, I simply make a subdomain called feeds for any domain I’m dealing with. Since searchengineland.com is our main domain, our feed domain is feeds.searchengineland.com.

Once I’ve created this, the MyBrand magic lets FeedBurner take control of where the domain points to. That let’s me turn the FeedBurner feed address for us into http://feeds.searchengineland.com/searchengineland.

But wait — I thought it was about keeping control? Relax. I’m giving them control because I want to. If they went all evil, I’d just change the CNAME record and point that subdomain to wherever I want. I own the domain. I control where it ultimately points to.

Sadly, SEOmoz doesn’t do this, and it’s a pain to switch (though at some point, it may be worth that trouble). If you’re new to feed publishing or are early in the game, it makes a lot of sense to move now, before it becomes more painful.

#2 - Get Your Feed Listed Across the Web

There are some great directory lists like this one from TopRank Blog and this one from Ari Paparo. However, my advice here would be to go after not just the generic lists, but the more specific feed lists, aggregators, portals and yes, other blogs & news sites that can put your posts in front of an audience that’s passionate about your topic.

In the technology field, for example, places like Alltop, Techmeme, PopURLs, even the NYTimes technology page list feeds from a variety of sources. Those are amazing links and incredible sources of traffic, too (Alltop recently entered SEOmoz’s top 30 referring domains for traffic to the site). If you’re committed to getting the most out of your feed, you need to identify the portals in your niche that command share, traffic and page views, make a feed worthy of being posted and get their attention. Emails are surprisingly effective, but nothing beats an in-person conversation.

#3 - Use Absolute URLs in Your Feed

Scrapers, both good and bad, are going to scoop up your feed and re-publish it, including the links. If you use absolute URLs in your markup (e.g. http://www.seomoz.org/blog/rand-loves-the-nfl) rather than relative URLs (e.g. /rand-loves-the-nfl) your chances of getting link equity and PageRank back from those who re-publish goes up significantly. Note that this is a general disagreement with JohnMu (who posted on this topic last year, though not specifically as it relates to feeds).

#4 - Record Feed CTR & Links You Earn as "Conversions"

Through feed tracking, you can determine the posts that received the greatest/fewest clickthroughs. You can also use your web analytics or tools like Linkscape, Yahoo!, Technorati or Blogscape’s SMM Prototype to see how many links each post has earned (Backtweets is another good one if you want to record tweets). Treat those links andd clicks like a conversion - write more posts like the ones that have success and shy away from the posts that don’t earn much love/attention. Great bloggers don’t start out great (I certainly didn’t). They learn over time what’s successful and effective and get consistently on that track.

#5 - Full Text Feeds are Generally Better for SEO

The argument over partial text vs. full text tends to be about earning the clicks and interactions on your site (full text means people can read off-site and may never click through, while partial text really annoys some subscribers), but from a raw SEO perspective, full text has a few benefits.

• All things being equal, you tend to get more subscribers with full text than partial, which boosts your numbers, gives you wider distribution and increases the liklihood you’ll earn a link from those readers.
• Full text feeds get re-published in full, and that means links further down in the content potentially pass value back to you.
• Blog and feed lists are sometimes picky about partial feeds, and may opt not to include your site.
• Potential distribution partners like full text, because it gives them the opportunity to keep the visitor on their site (but if these deals get done, they almost always mean link juice back to you).

Obviously, business goals may overrule this recommendation, but it’s wise to be aware of the possible impact.

Since it’s a short list, I’d love if anyone in the comments can link to posts or recommendations (yes, even if it’s your own stuff!) that can also be helpful on this subject.

Do you like this post? Yes No

Strategies to Spend Money & Earn ROI from Link Campaigns (without "buying links") »

Posted by randfish

Last week at Pubcon Las Vegas, I presented on How to Buy Links with Maximum Juice and Minimum Effort with fellow panelists Roger Montti (Martinibuster), Aaron Wall (SEOBook) and Todd Malicoat (Stuntdubl). I was a bit of an odd choice for this discussion, as I’d only recently announced SEOmoz’s Stance on Paid Links & Link Ads, but Pubcon’s organizers decided it would be interesting to have a divergent point-of-view.

Below is my presentation, which covers the perspective I come from and why I’m so risk-averse as well as strategies I recommend to capture value from investing in link acquisition campaigns:

Pubcon 2009 Paid Links

Not surprisingly I had a lot of people talk to me (and email me) after the presentation and express some really valuable opinions and questions. The presentations started late due to a misfunctioning projector, meaning there was no time for formal Q+A. I thought I’d take the opportunity in this post to address some of those missed questions.

Do you ever recommend link buying for any site? What about hyper-competitive industries?

Because of my distaste for risk of any kind when it comes to Google’s webspam team, my answer is consistent - no. I don’t ever suggest that businesses buy links from brokers or in the form of link ads that carry the primary intent of boosting a site’s ranking. To be fair, many of my colleagues who practice SEO in competitive industries (dating, gaming, pharmaceutical, real estate, e-commerce, etc.) don’t agree and do engage in buying links to boost their rank. I even know folks at Fortune 500s who use link brokers successfully for specific pages and targeted keywords (this group is probably in the lowest risk category).

Despite these examples and my respect for my colleagues, whenever I’m asked, I’m going to give the same reply - it’s my belief that in the long run, your money will be better spent on link acquisition that runs no risk of being flagged as manipulative by Google. The penalties and problems of link buying simply outweigh the benefits in my mind, so while I have no problem with paid links from a moral, ethical or legal standpoint (nofollow is most definitely not a way to disclose advertising to consumers as per the FTC’s guidelines), the pragmatist in me says link buying isn’t the way to success at Google.

What about directories that require a payment?

The short answer is - it depends. I’d wager a lot of money that some directories which do require payments pass great link equity. These include sites like:

• The Yahoo! Directory
• The Better Business Bureau Directory
• SEMPO’s Member Directory
• Apple’s Web Apps Directory

Then there’s the opposite end of the spectrum of directories that exist primarily for the purpose of selling PageRank. Google took action against many of these a couple years back and I suspect they continue to identify and discount their links as new ones crop up. In 2007, I wrote a lengthy post on What Makes a Good Directory and I’d still stand by nearly all of that today.

The message here is that just because a site requires payment to get a link doesn’t make it a "paid link" that Google will penalize or discount. As with many things in life, SEO and the web, there are shades of gray and nuances that require paying attention. If stuff like this were simple, SEO would be, too, and we know that’s not the case.

If I see my competitors engaging in link buying, how can I compete if I don’t do it, too?

I think a big misnomer with link analysis comes up when people scroll through a list of their competition’s links via something like Yahoo! Site Explorer. There’s no metrics indicating whether the link is passing juice, no metric for trustworthiness or quality, just a notation that a link exists on the page. Even if you’re using something more advanced like Linkscape, there’s nothing to say which links Google counts and which they don’t. You can easily get pulled into the idea that paid links are what’s propping up the competition’s rankings, when in fact, it’s a few great natural links that are doing all the heavy lifting.

I remember a site clinic several years back featuring a Google’s webspam chief, Matt Cutts. He was reviewing a site’s link profile on stage using an internal tool and commented that while Google saw several hundred links to the site, only three (yes 3 out of hundreds!) were passing link equity. Cearly, the search giant does a tremendous amount of filtering on the web’s link graph, so don’t presume to be sure which links are passing value.

Even if you feel very confident that paid links are winning the battle for your archnemesis, I recommend taking the low-risk road. In the long run, they’re likely to get penalized/devalued and you’re likely to overtake them with a link profile that’s clean and continually increasing in value.

Where do you draw the line between money that’s spent to acquire a link indirectly (as with event sponsorship, ads that turn into links, etc.)

This gets at the crux of the issue, but I think I’ve got a reasonably good methodology for determining which links requiring funds fit with Google’s guidelines and which violate them. I like these three questions:

1. Does the organization offering the link tout SEO, PageRank, customizable anchor text or Google rankings as either a portion or the whole of the benefit you’ll receive by paying this money?
2. Does the money go towards little else besides the link itself?
3. Does the organization/website provide links via this acquisition methodology (whether that’s an event sponsorship, a charitable donation, an advertising relationship, etc.) to the more aggressive side of the SEO/web marketing field (niches like porn, pills, casino, legal, real estate, etc.) often with anchor text heavy links?

If the answer to any of these is a definite "yes," the source is likely to fit into Google’s "suspicious" pile and possibly will lose the ability to pass link equity in the future (or already has).

How can you be sure that linkbait and viral content won’t be treated the same as paid links by Google in the future?

Just a couple months back, I wrote about Why Linkbait is a Tactic the Search Engines Will Always Value, so it’s probably not worth re-hashing here. Certainly, there are ways to be manipulative about virtually anything in the link acquisition world, and Google may well take action against some forms of these, but I believe natural links acquired through great content are going to stand the test of time (and are likely to benefit from future ranking signals, whatever they may be).

This is just Google FUD - we shouldn’t let them dictate how to do our jobs!

But we already do! The only reason we try to build these links, research the right keywords, create and submit XML sitemaps, etc. is because Google is dictating the way their crawling, processing and ranking systems work. In their ecosystem - the one that drives 85%+ of all search traffic on the web - there are guidelines, best practices, rules and regulations. If you want to play on their court, you’ve got to abide by those rules or be ready to face the consequences. I’m not ready for those consequences and thus, have low risk tolerance and the attitude you’re reading about.

None of this is to say that a more risk-heavy appetite and more gray-black hat methodologies for link acquisition aren’t worth trying; just make sure you do it on sites you’re willing to get tossed out of the playground.

As always, I’m looking forward to the conversation in the comments.

Do you like this post? Yes No

Advanced Link Analysis Charts »

Posted by willcritchlow

Bored of sorting massive lists of links in all kinds of different directions to understand the link profile of a new site?

Struggle to understand how to gather actual insights about link profiles from lists of thousands of links and persuade management of the actions needed?

Don’t panic. Help is at hand.

I’m going to share some data visualisation tips today that I reckon I could use to beat up on Rand in a presentation-off (umm, again). We have recently been doing some deep dives into clients’ and prospects’ link profiles which gave me an excuse to mash up some again). We have recently been doing some deep dives into clients’ and prospects’ link profiles which gave me an excuse to mash up some Linkscape API data in Excel. I’ve used Linkscape data, but you could use any link analysis tool you like as long as you can get some metric to sort the linking domain by (I have used domain mozTrust in most of the examples below). Equally, I’ve used Excel, but you can use any data analysis package you like. If you want to use Excel, you will need the Data Analysis Toolpak (for the histogram function).

I’ll get into how to make the charts in a minute, but first I’m going to just show you some pretty pictures:

Impress the boss

This one is of questionable use (I think there are better ways of actually visualising the data) but it’s pretty, and bosses like pretty (allegedly). This is a surface chart of number of linking domains by domain mozTrust shown across 4 data points - all links, links to the homepage and links to the next two strongest pages:

The bit of insight this does give us at a glance is that the vast majority of the site’s very low DmT links go to the homepage and that the most trusted domains linking to the site (DmT >= 8) don’t link to the homepage or the next two strongest pages.

The same chart just showing links to the homepage compared to all links which shows the top end a litle more clearly:

Gathering insights

I think this data is actually easier to see as a line chart like this (locations A and B are the top two strongest pages on the site after the homepage):

What we just about see here is some bumps up at the top end of the DmT scale in the light blue line which is the same bit of insight I mentioned above.

Drilling down

Diving into this data to show only the top end of the DmT scale, we get:

And we see that although the homepage and these top two location pages are the most powerful pages on the site, they are not the ones with the links from the biggest / most trusted sites. This is an area for further examination that would be hard to discover by looking at endless lists of links.

This is just an example of the kind of insight you can gather. I’m showing off tools and techniques here rather than specific insights. I’ll leave you to do your own playing to discover interesting things about your clients and competitors. I didn’t know what I was going to find when I started diving into the data for this site. You likely won’t know either, but graphs are great discovery tools. Sometimes, of course you find nothing of interest:

Comparing just the top two pages doesn’t give us any very meaningful insights except that the big links out at 6.5-7 DmT to location A probably explain why it’s more powerful than B. It might be more insightful at a lower granularity.

Equally, I haven’t yet learnt to understand the meaning that I am sure is buried in charts like this one:

This is the number of links to a whole site by the mR of the linking page. Like the mythical guys who can understand network traffic by watching LEDs blink on routers, I’d love to be able to look at this kind of chart and really understand things. The closest I’ve got so far is that I think these charts should look roughly smooth in the absence of manipulation. If we assume that the difficulty of acquiring a link is roughly correlated to its strength and that we get links at a rate inversely proportional to their difficulty, then I think this chart should look roughly like a Poisson distribution:

Which this one does, so I’m happy.

Persuading management / bosses

The next thing that some of these charts helps with is making the case to management when you know something is true, but they need more persuading. This next example takes two different sites (neither of them is the site above) that are in different industries but have remarkably similar link characteristics at the macro level (don’t ask me how I found these sites - I am just that sad). The spider chart shows how similar they are:

However, if we dig in a little further, we find quite a difference behind the scenes:

The red site seems to have loads more decent links (mR 4, 5, 6) than the blue site. So how does the blue site end up with similar domain metrics?

It’s all about the relatively small number of very powerful links the blue site has. Zooming in on mR 6 & 7 links:

If you were just to look at this chart, you might imagine that the red site was getting more juice passed via these links than the blue site is. However, you’d be being fooled by the logarithmic scale. In terms of total juice passed by just these mR 6 and 7 links, the actual story is:

In other words, the blue site is competing almost purely on the basis of the big mR 7 links it has that the red site doesn’t. That’s kinda interesting in terms of strategy generation isn’t it?

How do you do this analysis?

Pretty much everything in this post was generated using the histogram function in Excel running over Linkscape API data. It’s pretty straightforward with the online help. The only gotchas I noticed that you might need to know about were:

1. Align the ‘bins’ (which are the x-axis values on most of the charts above) either with mR / mT intervals (e.g. 1, 2, 3, 4, …) or go much more granular (e.g. 0.1, 0.2, 0.3, ….). Anything in between tends to generate artifacts
2. The bin range has to be on the same sheet as the data - if you try to pull in a bin range from another sheet, it fails silently
3. If you want to do the surface chart, you need to do some interpolation between your points. In the examples above, I just did a linear interpolation (i.e. drawing a straight line between the different page levels) - so if the homepage has 100 mR 2 links and the next page has 50 mR 2 links, I just created 10 imaginary pages with 55, 60, 65, 70… mR 2 links to spread the surface out far enough to see it. This may not be the best way of doing things. I’d love to hear from anyone who has a better method

Thanks to foliovision for the photo from the ProSEO seminar.

Technorati Tags

linkbuilding, analysis, visualisation, visualization, data

Do you like this post? Yes No

New & Interesting Insights Into Google Rankings & Spam from Pubcon »

Posted by randfish

Tonight’s post comes via the Pubcon conference in Las Vegas and is likely of interest to many in the webmaster and search communities. Today, during the Interactive Site Review Session, Google’s head of Web Spam, Matt Cutts, along with Vanessa Fox of NinebyBlue and Derrick Wheeler of Microsoft took thorough dives into a number of sites. The session was well coverd on Twitter, and in live form by Barry Schwartz at SERoundtable.

Matt Cutts and Vanessa Fox on the Site Review Panel (photo credit: davecolorado.com)

A few points in particular stood out and are worthy of coverage:

• Blocking Internet Archive may be a Negative Signal
  Matt Cutts noted that spammers very frequently block archive.org from crawling/storing their pages and few reputable sites engage in this. Thus, it’s a potential spam signal to search engines. SEO Theory has a good writeup on when and why there may be legitimate reasons to do this, but webmasters seeking to avoid scrutiny may want to take heed.
  _
• Web Page Load Time can Positively Influence Rankings
  Maile Ohye actually mentioned this at SMX East in New York, but Matt Cutts repeated it again today. In a nutshell - while slow page load times won’t negatively impact your rankings, fast load times may have a positive effect. This comes on a day when the Google Chrome blog introduced their new SPDY research project. I’m particularly happy about this news, because it’s also true that load times have a positive second-order effect on SEO. Pingomatic recently published some excellent research on load times from Akamai noting the expectations of users for faster web browsing have doubled in the past 2 years. In addition, fast loading pages are, in my opinion, considerably more likely to earn links, retweets and other forms of sharing than their slow-loading peers. This tool from Pingdom is a great place to start testing your own site.
  _
• It May be Easier to Walk Away from Banned Domains
  Sites that Google’s webspam team has severely penalized or banned entirely from the index can be very difficult to re-include, and thus, Matt suggested that "walking away" and "starting over" may be a more prudent strategy. In my opinion, this is largely due to link profile issues - if your site has a "spammy" link profile, it’s tough to ask an engineer to sort out the wheat from the chaffe manually (or algorithmically) and stop counting only the bad links. Thus, re-consideration requests may not be as effective a use of time as registering a new site and trying to re-build a more trusted presence.
  _
• Repetition of Keywords in Internal Anchor Text (particularly in footers) is Troubling
  During a specific site’s review, Matt noted that keyword usage in the anchor text of many internal links, particularly in the footer of a website, is seen as potentially manipulative. Yahoo!’s search engineers have noted this in the past and we at SEOmoz have seen specific cases where removal of keyword-stuffed internal links from a footer had immediate impacts on Google rankings (removing what appeared to be large negative ranking penalties sitewide).
  _
• Having Multiple Sites Targeting Subsections of the Same Niche can be Indicative of Spam
  Matt Cutts today mentioned that "having multiple sites for different areas of the same industry can be a red flag to Google." Though Googlers have mentioned this before, today’s site review panel brought renewed attention to both Google’s ability and proclivity for carefully considering not only an individual site, but all the other sites owned by that registrant/entity/person. Given Google’s tremendous amount of data on web usage behavior, many SEOs suspect that they track beyond simply domain registration records.

I also presented at Pubcon today - on a panel called Linkfluence: How to Buy Links with Maximum Juice and Minimum Risk (live SERoundtable coverage here) - as the counterpoint speaker (on why not to buy links). I’ll try to have that presentation in written format early next week on the blog.

p.s. I was asked by a large number of attendees at the conference about our venture capital fundraising experience. I expect to be able to write about that very soon and certainly appreciate all the support. :-)

p.p.s. For those who are interested, my brother, Evan Fishkin (who works at Portent Interactive) had his head shaved by Google’s webspam chief. On a personal note, I must say I was particularly impressed with Matt’s ability to shave a head without nicks or cuts, and his foresight in bringing proper equipment. Unfortunately, I’m not fully briefed on why this occurred, but I do know that my little brother was in terrible need of a trim (photo of my shocked observance of the event here & more photos/video here).

Do you like this post? Yes No

This Week in Search for 11/12/09 »

Posted by Sam Niccolls

• Google Acquires AdMob for $750 Million:
  Just another day at the office on Monday for Omar Hamoui and the AdMob team who got acquired by Google for $750 million.  

• Possible Implications of Google’s New Music Search:
  Google will soon launch a new music search allowing searchers to preview songs directly from the SERPs.

• Bing Launches Video Portal with Help from MSN:
  YouTube may still be the 800 lb gorilla, but Bing now provides over 900 TV shows and access to high-quality videos from within a sleek new UI.

• Bing Unveils Wolfram|Alpha Powered Search:
  Whether you want to look up nutrition facts or find out your body mass index (perhaps before the holidays), Bing’s Wolfram|Alpha powered results give you the information you want… or don’t want, as the case may be.

• Amazon Design Analysis:
  Just copy Amazon for e-commerce, right? Well, there are certainly reasons Amazon brought in 19 billion in revenue in 2008, but before you borrow too heavily, Jared Spool’s SlideShare presentation analyzing Amazon design elements is worth a look.

• 50 Beautiful & Creative Blog Designs:
  Smashing Magazine shares a list of great blog designs that is great idea fodder for bloggers.

• Deep Links Are Disproportionately Valuable:
  Eric Enge serves up some food for thought… do deep links pass a disproportionately high amount of link juice?     

• Win Big by Doing Just a Few Things Well:
  As shown by Chipotle, Pinkberry, and others, simplicity can be an effective path to executing and marketing successful products. It’s difficult to do, but as 37 Signals highlights, often times less is more.

• Bill Slawski - How the Engines Might Be Using Query & Click Logs:
  SEO by the Sea looks at Google’s recent patent and shares several ways the engines might be using query and click logs to influence rankings.

• AdWords Announces Sitelinks:
  Similar to sitelinks in the organic listings, AdWords now offers richer paid ads with sitelinks.

• Heatmaps for Display Ads & Product Pages:
  Hot or not? … it’s a simple enough question, but few marketers use heatmaps to inform display ads or product pages. As Jonathan Mendez and Linda Bustos each point out, heatmaps are a great way to do like Snoop Dogg and "drop, drop it like it’s hot." 

• Smart E-mail Marketing Tips:
  Shay Howe’s blog post full of e-mail tips is a useful hit list for marketing managers in charge of designing and delivering effective e-mails.

• New Google Flu Shot Locator:
  Google moves the local search needle through continued improvements to their flu shot locator, which will soon be added to Flu.gov.

• AdSense - Rolls Out New Features:
  From an improved UI to enhanced dashboards, AdSense advertisers now have access to more robust channel and ad unit reporting. 

• Why You Should Use the NoArchive Meta Tag:
  Dave Naylor’s stance on Google’s cache can be argued, but he provides an interesting point of view, as well as several valid points.

• Marissa Mayer Named Glamour Magazine Woman of the Year:
  Google’s 34-year-old VP joins Michelle Obama, Maya Angelou, Serena Williams and Rihanna among Glamour Magazine’s women of the year. And yes, the Mozzers currently in Vegas are projecting that our very own Sarah Bird will top Glamour’s list in 2010.

• Monetizing Mobile Content:
  A timely subject on the heels of Google’s AdMob acquisition, eMarketer’s study sheds light on the growth of U.S. mobile users and ways content owners could monetize mobile users.  

• Using Position Distribution to Manage Branded Keywords in Paid Search:
  Siddharth Shah’s Search Engine Land post provides several reasons you might want to use position distribution as a key factor when managing branded paid search ads.   

• You Don’t Need Tons of Traffic to Optimize Your Site:
  Brendan Regan, of the now Bryan Eisenberg-less Futurenow Blog, dispels the myth of needing minimum traffic volume to make website optimizations.

Top YOUmoz entries:

1. * Web Analytics & Segmentation for Better Conversion Optimization by Web Analytics & Segmentation for Better Conversion Optimization by philou2803

2. * What Make a Link Worthy Post - Part 2 by What Make a Link Worthy Post - Part 2 by chenry

3. Dear Generation X: Social Media from the Eyes of Your Successors by trycm
   

4. Google Analytics Alternatives - Measuring Beyond Last Click Wins by John Santangelo

5. AdWords Tracking Proven to be Misleading by William E 
   

6. Case Study: How Building a Site for Users Improved Our Rankings by csaliba

* Indicates blog post was promoted to the SEOmoz Blog

Do you like this post? Yes No

Is Social Media ROI Unmeasurable? »

Posted by Dr. Pete

I’m reporting live from Pubcon Las Vegas this week, along with some of the SEOmoz team. To be honest, we’ve struggled a bit with how to cover the conference here on the blog. As someone who only hits a couple of conferences per year, I know how annoying it can be to have to hear how great an event is that you already regret not being able to go to. On top of that, sometimes information that seems brilliant in context just doesn’t translate into a quick blog blurb or Tweet. So, in the interest of providing value to those of you who aren’t here at Pubcon, we’re going to try to take some deeper dives into the content, hopefully providing some of that context you may be missing.

Is That An Elephant?

No, I’m not trying to distract you. These first two days of sessions, I couldn’t help but feel that there was an elephant in the room with us during the social media sessions. The enthusiasm for social media (and especially Twitter) has been stronger than ever, but we all seem reluctant to dampen that enthusiasm by talking about an uncomfortable fact - very few of us have really found a way to measure social media success. Sure, there are internal metrics for any given platform – Twitter followers, for example – but without something external to tie it to, those are little more than high scores in the social media video game.

The B-word

Of course, the default answer is always "branding". Unfortunately, much like "engagement", branding is too often just a distraction, an intangible excuse we use to avoid the fact that we have nothing to measure. Ironically, during a session that had nothing to do with social media, I heard something close to an answer during Q&A. No matter what you think branding is, find a way to measure it. Here are just a few possiblities:

• Direct brand mentions
• Links with brand-related anchor text
• Branded search volume

Where’s there a number, there’s a path to calculating ROI.

Target a Response

At this morning’s keynote, we had a chance to hear from the marketing departments of various Vegas hotels. Like the rest of us, these marketers are learning as they go, trying to figure out how to use Twitter and Facebook to drive real business value. Most of the hotel marketing departments see social media as a direct-response channel, and that’s certainly a start. Put out a special offer through social media channels, and you can measure the response. Where there’s a measurable response, there’s ROI.

MGM Grand’s marketing head hinted at another possibility – their employees monitor Twitter to spot dissatisfied hotel guests, dispatching staff to help solve the problem. What’s the natural next step? Measure this response. How many problems did they intercept? How many were they able to solve? What does solving one customer’s problem equal in real dollars? All of these questions can be answered, and from those answers comes tangible value.

Find a Comparison

Finally, during a session about how social media and search intersect, we heard a great example from Lee Odden about how to put a value on social media. Lee mentioned that his firm drives about 15-20 major media mentions per month from social media. He estimates that this equates to paying a PR firm $10,000/month. This may not sound like metrics in the traditional sense, but it’s an entirely valid approach. PR costs money to generate, and social media has replaced that value.

Just Measure It

When it comes to measuring social media ROI, what are we really afraid of? If I start measuring, will I have to admit that being a 307th-level Maniac on Facebook Mafia Wars isn’t providing solid business value? Stop making excuses, stop mumbling about branding, and find a way to quantify social media success in real dollars.
 

Do you like this post? Yes No

Headsmacking Tip #16: Meet the Linkerati in Person »

Posted by randfish

We all know about the Linkerati by now - how to identify them, how they’re segmented and why they’re the secret to SEO. Yet, time and again, I see link builders and companies pass up amazing opportunities to earn links and attention from those who have the best ability to help your content/brand spread virally.

I’ll lay out two scenarios below to help illustrate this point:

Scenario 1: Emailing a Prominent Blogger/Writer/Journalist/Site Owner/Social Media Personality/Etc. Hoping for a Link

You/your company:

• Identify a list of Linkerati that may be relevant/interested in your business/content
• Send a carefully crafted email to each individual, hoping to attract their attention and interest
• Follow up with those who reply (and maybe those who don’t) with emails or even a phone call
• Request a review of your product/tool/site/idea

Scenario 2: Meeting that Same Person First, then Following Up

You/your company:

• Identify prominent (or even relatively less known) Linkerati in your city or at an event you’re attending
• Schedule a meeting / invite them to coffee or to tour your offices (or even just go to an event you know they’ll be at)
• Introduce yourself politely and humbly and mention you’re a fan. Exchange business cards, have a chat and let them know what you do (also helps if you can find some non-work related topics to bond on as well)
• Follow up with an email thanking them for the meeting and asking if they’d take a look at your product/tool/site/idea

I’d argue that while Scenario 1 is more scalable, it’s also potentially damaging in the long run. When you first introduce your work to someone who can help it spread, you have that single chance to make a first impression. If the relationship matters and you’re seeking a high "conversion rate" for attracting attention from the Linkerati, use Scenario 2.

The beauty of these links is that they not only create value for SEO, but often attract second-order effects like increased brand awareness, links/tweets from the followers & fans of the Linkerati, and improved odds that you’ll be positively remembered and introduced when someone mentions they need "X" (whatever it is your product/tool/site/idea does).

In-person connections have always been powerful attractors of value for me in the SEO, social media and startup worlds and when I see early stage (and mature, later-stage companies) engage in this fashion, it’s almost always positive. Just make sure you’re professional, candid, friendly and never over-bearing in your interactions; chances are you’ll get much more than a link.

Do you like this post? Yes No