Among the specialized bots Google uses include: The web spider Googlebot, the Adsense spider MediaBot, the image spider ImageBot, the AdWords spider AdsBot, the RSS feed spider FeedFetcher-Google, and Googlebot-Mobile spider for mobile devices. MSN & Yahoo, the other two of the 'big three' have their own proprietary versions of spiders.
Why is it important for an Internet Marketer to know how spiders crawl your website?
A search engine crawler is your best visitor. Giving a crawler easy and uninhibited movement in your website is necessary for good search engine rankings.
An astute marketer should also desire to see how search engines see his or her site. This may be accomplished by a Lynx Viewer which is a text-mode web browser. Additionally, a Lynx Viewer can help you determine if your web pages are accessible to the vision impaired, an assemblage of visitors that should not be ignored ---yes, there are millions of visually impaired people surfing the Internet regularly.
Even though you must design your website with your visitors in mind first, it is crucial that you accord the search engines top level priority too, since the vast majority of these visitors will arrive via search engines. Practice good SEO (Search Engine Optimization) but not at the expense of your visitors' experience -- it is a balancing act that must be accomplished with prudence.
Also of significant importance is the fact that web browser standards are not yet fully harmonized. A web page that looks great in Internet Explorer might look atrocious in a Mozilla based browser like Firefox or Netscape. Additionally, with the proliferation of hand held devices for browsing the Internet, compliance with W3C standards is becoming more and more critical. A marketer must therefore be conversant with the intricacies of cross-browser design -- designing for one browser (IE) is no longer ideal, as the Google backed FireFox is eating up Microsoft's browser turf at an alarming rate.
Anybody can "whip up" a web page in FrontPage without sufficient knowledge of HTML, but may not be able detect and correct the messy code that FrontPage generates underneath the page, some of which is proprietary to Microsoft. Consequently a website that looks superb in Microsoft Internet Explorer may look and load dreadfully in Opera and/or some other browser, denying you visitor traffic.
Never use a Word Processor to design your website. Word Processing software generate tremendous amounts of code that is not search engine friendly. If you cannot hand-code using a text editor then it is necessary that you use authentic and industry standard web design software that incorporate the most up to date design principles. Macromedia's Dreamweaver and the latest version of Microsoft FrontPage are good candidates with Dreamweaver getting my partisan nod.
A first-rate design strategy should include the use of CSS (Cascading Style Sheets) and valid XHTML, the most current in the HTML generation of standards. Websites designed in strict W3C standards tend to be lighter, faster and cross-browser compatible. This is not to insinuate that table based design is going anywhere anytime soon, for it is my humble disputation that if strict W3C standards were to be enforced in browsers, 95% percent of websites would go out of business, furthermore the lack of inter-browser synchronization just worsens things.
"The double-digit increase in online search activity marks a significant milestone in the evolution of Internet consumer behavior," says Ken Cassar, senior director of analytics at Nielsen/NetRatings. "Online search is the primary tool most people rely on to do everyday research," he says.
So, from a marketing perspective unless your site appears in the top 30 listings of the major search engines & directories, you will hardly get any worthwhile traffic. Therefore, Search Engine Positioning is vital to your marketing success on the web!