Diagrams for Solving Crawl Priority & Indexation Issues
Posted by randfish
Yesterday night I stayed up way too late authoring a post on Google’s Indexation Cap. Today, despite getting up way too early, I wanted to follow up and answer some of the questions from the comments, Twitter and my email. I think SEOs who read the post rightly asked for more direction in solving this problem - a fair request. Below, I’ve done my best to tackle these problems visually, as I believe we all think about site architecture and crawling issues in a visual structure.
First off, here’s a sample site hieararchy to set down the concept and give the colors I’m using in the following diagrams more context:
Next, I’ve illustrated in a more representative fashion, how those hieararchies might look on a website, and noted the external link potential of each:
In this next piece, I’m trying to explain a very important concept and something that’s frequently misunderstood by SEOs. Once upon a time, search spiders would crawl the web largely recursively - hitting a homepage that had been submitted to its index (remember way back when search engines had submission?!), then crawling in an outward fashion based on the links they discoverd there. That hasn’t been the case for a long time, and as we all see with crawl paths (if you’re looking at the requests Google/Yahoo!/Bing make to your domain), multiple entry points are nearly universal and crawling pushes "outward" from those priority URLs. It looks a bit like Minesweeper, right? :-)
Finally, I’ve got a graphic to help understand how to positively approach these problems and solve them.
There are certainly more recommendations that can be provided around these issues, and I look forward to a discussion of them in the comments.
p.s. I covered site architecture and navigation in a good bit of detail at the PRO Training this summer, but I like this image format so much, I think I might re-craft something new for next year. It feels like structuring sites properly is still a big pain point for SEOs (but possibly that’s less to do with lack of knowledge and more to do with lack of influence during the design phase?)
You may also enjoy reading:
Comments are closed.
Thank you for reading Online Marketing News (OMN). We're serious about providing a great user experience. That's why we employ human editors to review each and every syndication feed and press release to ensure only the most relevant content makes it on the Online Marketing News website. If you would like to have your blog featured on OMN, please let us know by introducing yourself in the comments below.