Search Engines & Marketing

Search Engine OptimizationGooglebot, crawling, & indexingWork with Google
Search Engine Optimization is a technique that will help you achieve higher ranking and more targeted traffic to your website.

Search engine optimization has often been promoted as a secret skill known only by a few technical gurus. However, the real art of search engine optimization has more to do with employing common sense strategies in the web design process and website maintenance.

search & marketing

Googlebot, crawling, & indexing

Whenever you create a new website or blog for your business, the first thing you probably want to happen is have people find it. And, of course, one of the ways you hope they will find it is through search. But typically, you have to wait around for the Googlebot to crawl your website and add it (or your newest content) to the Google index.

So the question is: how do you ensure this happens as quickly as possible? Here are the basics of how website content is crawled and indexed, plus some great ways to get the Googlebot to your website or blog to index your content sooner rather than later.

  • The Googlebot is simply the robotic search software that Google sends out to collect information about documents on the web to add to Google’s searchable index.
  • Crawling is the process where the Googlebot goes around from website to website, finding new and updated links to report back to Google.
  • Indexing is the processing of the information gathered by the Googlebot from its crawling activities. Once documents are processed, they are added to Google’s searchable index if they are determined to be quality content. During indexing, the Googlebot processes the words on a page and where those words are located. Information such as page titles, categories, and image attributes are also analyzed during indexing.

Work with Google

So how does the Googlebot find new content on the web? It starts with web pages captured during previous crawl processes and adds in sitemap data provided by webmasters. As it browses web pages previously crawled, it will also detect links upon those pages to add to the list of pages to be crawled.

Hence, new content on the web is discovered through sitemaps and links.

The best way to ensure that your new content is discovered quickly is simply by sharing it on social media networks through status updates, especially on Google+.

Update Social Profiles
As mentioned previously, bots get to your site via links. Add a link to your new website on your Twitter, Facebook, Google+, LinkedIn, Pinterest, YouTube, or company profiles.
Share Your New Website Link
Once you have added your new website link to a social profile, share it in a status update on those networks (or click the LIKE button on your own page). That will still alert search engines that are tracking social signals. For instance simply doing a short post like “Hey as you know I have been working on Well we launched it today …” For Pinterest, pin an image from the website. For YouTube, consider creating a video introducing your new website and include a link to it in the video’s description. Many professional associations have web pages that include a page of URLs and e-mail addresses for members. Submit yours to your own associations, clubs, departments, etc.
Create Offsite Content
Again, to help in the link building process, get some more links to your new website by creating content on other websites, such as submitting guest posts to blogs in your niche and articles to quality article directories, and press releases to services that offer SEO optimization and distribution.
Build a Sitemap
A sitemap is a simple document on your website that lists each page on your website. It tells search engines when new pages have been added and how often to check back for changes on specific pages. If your website is built on WordPress, you can install the Google XML Sitemaps plugin and have it automatically create and update your sitemap for you as well as submit it to search engines.
Submit your Sitemap to Google Webmaster Tools
The first place you should submit your sitemap for a new website is Google Webmaster Tools. If you don’t already have one, simply create a free Google Account, then sign up for Webmaster Tools. Add your new site to Webmaster Tools, then go to Optimization > Sitemaps and add the link to your website’s sitemap to notify Google about it and the pages you have already published.
Submit to Blog Directories
TopRank has a huge list of sites you can submit your RSS feed and blog to. This will help you build even more incoming links. If you aren’t ready to do them all, at least start with Technorati as it is one of the top blog directories. Once you have a good amount of content, also try Alltop.
Install Google Analytics
You want to do this for tracking purposes anyway, but it also might give Google the heads up that a new website is on the horizon.
Setup Your RSS with Feedburner
Feedburner is Google’s own RSS management tool. Sign up or in to your Google account and submit your feed with Feedburner by copying your blog’s URL or RSS feed URL into the “Burn a feed” field. In addition to your sitemap, this will also notify Google of your new blog and each time that your blog is updated with a new post.
Put it in Print
Be sure to put your web address on your letterhead, business cards, brochures, and as part of the signature line in your e-mails.

Once your website or blog is indexed, you’ll start to see more traffic from Google search.

Back to top