Abandoned by Google! Googlebot, Wherefore Art Thou Googlebot?



As a search engine optimization specialist I often optimizeexisting web pages for small business clients, upload them tothe site and see pages re-indexed by Google within a week.This only happens with existing business sites that have beenonline for a few years. Google seems to be updating theirindex as often as every other week at this point and olderestablished sites that are already indexed seem to be re-crawled on that twice a month schedule on a fairly routinebasis.

Two clients that hired me for recent work saw their rankingsshoot to the top for a newly targeted search phrase in aweekend when I did optimization on a Thursday and they wereranked instantly by Saturday. Now keep in mind that thisdoesn't happen for everyone, only those that have been onlinefor some period and already have significant content thatsimply needs tweaking and proper title and metatag informationadded. They usually have relatively good existing PageRank anddo well for other RELEVANT search phrases already. I offer thatwarning only to avoid instilling false hopes in anyone hopingto achieve the same instant ranking boost overnight.

Those clients that do succeed in this way are often thrilledwith the results accomplished in such short order. I'd loveto be able to offer that type of ranking boosts to everyone,but some are more equal than others when it comes to easy,inexpensive SEO tune-ups that rev up your rankings overnight.Your mileage may vary.

WHY DO NEW SITES SUFFER?

What is going on with newer sites that don't get crawled formonths? I've got a client, a newer attorney directory thatoffers tons of great information in the form of articles onspecific areas of law, links to incredibly valuable andrelevant legal sites and over 600,000 attorneys listed bypractice area and state. Yet the site has not been re-crawledby Google for over 3 months! Now this would not be such a bigissue for many sites, but this site is relatively new and we'veoptimized all the titles, tags & page text, created a completesite map and placed links to all these resources on the frontpage.

I know that the site is not being crawled because Google'scached copy of the front page shows it before we did thework four months ago, without the new links and withouttitle tags. We've submitted the site by hand, (manually)once a month for three months via the Google Add URL page.http://www.google.com/addurl.html When the hand submissionfailed to get it re-indexed for four months, we submittedthe sitemap page, which has not been crawled at all. Googleshows only ONE page on this site, when in fact it hasthousands of pages, a sitemap and dozens static pages!

Part of the problem is that this site must be dynamic, sincea database of over 632,000 attorneys must be accessed,retrieved and served for any of those law firms searched forto be returned to the site visitor. Google warns owners ofdynamic sites that Googlebot may not crawl dynamicallygenerated pages with "?"" question marks in the URL. This isto avoid crashing the server with too many concurrent pagerequests from Google's spider.http://www.google.com/webmasters/2.html#A1

The solution to this dynamic URL problem has been discussedwidely in search engine forums and solutions have been bandiedabout including software provided by SEO's, URL re-writetechniques for dynamic pages on APACHE servershttp://www.alistapart.com/articles/urls/ and PHP pageshttp://www.stargeek.com/php-seo.php to generate search enginefriendly URL's. Others recommend simply adding static HTMLsitemap pages as alternatives for the search engine spiders.In this instance the client's developer simply said "Ican'tdo that (PHP solution) on this server". So we resorted toputting up the static HTML sitemap pages with hard-coded

URLS to the main 54 pages of the site athttp://lawfirm411.com/Law-Firm-411-sitemap.html This shouldget at least those fifty pages crawled by Googlebot, butGoogles' spider appears not to be crawling this site at all.How do we know this? See for yourself by using the followingquery in the search box at Google: allinurl:www.lawfirm411.comwhere the result page shows ONE page in the results. If youtry that query on your own site (replace your own domain namefor lawfirm411.com), you'll see the results lists ALL yourpages.

The site home page was crawled by Google four months ago, whenthey took their "Cached Snapshot" of the page. You can seethis by visiting the Google cached page here:http://66.102.7.104/search?sourceid=navclient&;ie=UTF-8&q=cache:www.lawfirm411.comwhere the date of this snapshot is "Apr 20, 2004 07:42:19 GMT"and they haven't been back since. The page in that snapshothas none of the newly added links, an outdated title tag, andold content.

This problem is not unique to this site. One client we workedwith two years ago had a dynamically generated, framed site!Those two site structures have always given search enginestrouble. Their site was not crawled at all and only the frontpage showed up. Our solution was to create a second domain(owned by the client), which had static HTML pages thatprecisely mirrored the content of the client's framed,dynamically generated site. Guess what happened afterGooglebot crawled the static site? Google indexed the framedsite in full and then banned the static site from the index!Not an approach we advocate, but the one that worked for thisclient.

We're still searching for ways to get Googlebot back toLawFirm411.com before creating that new static site, butdecided to share this odd experience with the SEO communitybefore going to any extremes. Google provides over 70% ofmost search engine referred traffic to ALL of our clientsand we realized we can't site idly by and see a major clientlanguish because Googlebot didn't like what it found at theclient site on the first visit four months ago.This issue dogs newer sites in other places as well. The OpenDirectory Project has also become notoriously slow in addingnew sites to the directory and in this case, has not pickedup this site even after 6 regular monthly submissions. Theweb playing field may have begun tilting toward older,established sites and away from new ones.

------------------------------------------------------------

Mike Banks Valentine is the SEO for http://www.lawfirm411.com

Contact him at http://www.seoptimism.com/SEO_Contact.htm

Improve Your Small Business Online at our Ecommerce Tutorial

http://website101.com/Free-Tutorials/index.html

------------------------------------------------------------

MORE RESOURCES:

Sponsored
RELATED ARTICLES
The Secret To Getting Indexed In Yahoo
Getting indexed in Yahoo has become very difficult in the last few months. The indexing robot, Yahoo!Slurp, has become erratic.
How Google Indexes Content From Your Web Directory
In a fluke, I was able to notice something about the way Google indexes content from web directories. Excluding your template, the most important line of code is the first title you add to your main body.
Blogging, Spamming, and Blog Spam
Email marketing once proved to be immensely effective, but the greedy and idiotic polluted the well by spamming the planet with everything from weight-loss products to sexual enhancement drugs and beyond. Because of the stench, filters and laws have been created to attempt to fix the problem, but still the Internet is polluted with more and more junk each day.
How Important is PageRank, Really?
Webmasters can spend most of their waking hours doing everything they can to raise their Google PageRank. It is common knowledge that PageRank, which is largely based upon the number and quality of backlinks a webpage has, is an important factor in how well a particular webpage ranks within the Google search results.
Link Building for Hilltop
Hilltop is one of the major concepts underpinning Google's search algorithm, yet its workings and implications are often misunderstood.After the infamous Florida Update, many webmasters were aghast as their rankings plummeted; and again, when the mysterious "sandbox" was implemented, some webmasters could not get a Web site to rank well, period.
How To Write Effective And Unique Articles That Are Optimized for the Search Engines
It is a well known fact that writing, distributing and posting articles to various location on the web is an extremely effective way to generate targeted traffic and establish yourself as an experienced marketer in your field. Unique articles that help solve the reader's problems, assists the reader in advancing their marketing acumen, or otherwise generally inform tend to stay posted on many web sites throughout the Internet and usually generate targeted traffic for the writer.
What is Your Sales Message?
So you built a website, you have a product, but it's not selling. What's your sales message? Does your site scream out to potential customers "BUY ME!!" or does it simply say "Hi, we're here, we have a product, if you want one.
The Smart Way To Get Listed On Google!
So you want to get listed on Google? And you want a GOOD listing on Google? "But my website is not popular and is too new to be noticed by Google," you say?Well, I have a solution to your problem.There are couple ways to get listed on Google.
Straight Talk on Search Engine Optimization
To get your website listed well in the major search engines, you have to perform search engine optimization (SEO). .
Not Ranking High Enough on Search Engines? Here's Some Help!
Search engines are frugal things. (Froogle, too, haha! Sorry, lame joke) They take many, many things into consideration when ranking your pages.
When to Hire a Professional SEO Firm?
In efforts to increase sales and profitability, more and more companies are turning to online marketing initiatives - specifically, search engine optimization. Search Engine Optimization or (SEO) is the art and science of blending technical and marketing skills together into a finely tuned website that is both search engine-friendly, ranks well for specific keywords and phrases, and is tailored to your audience from a buyer and seller perspective.
Seek Engines: What If Seek Had Bumped Out Search?
In the early days of the World Wide Web, when things were just getting started many geek speak terms were jockeying for position to be the standards for years to come. I used to drive by a California-based company every day that had named itself YKK and think, "What a pity.
Link Popularity
In today scenario when we talk about Search Engine Optimization, we also talk about one of the most important aspect of SEO, which is Link Building. But there are different types, aspects and limitations of Link Building, which would be discussed now under1.
Search Engine Traffic Domination - How to Turn the Search Engines into Your 24/7 Salesmen
Getting your website returned by the search engines as a high result is a significant goal by most webmasters because higher search results means higher traffic and revenues. However, there are some tips you can utilize to help your website gain better rankings by the search engines and ways to turn the search engines into your 24/7 salesmen.
SEO #1: Choosing THE Keywords to Optimize for
This is the first lesson out of 6 that teaches you the most important elements of search engine optimization. You should read one of these 6 courses EVERY day so that you can have enough time to "digest" all the information and put them to the test.
Ten Steps To A Well Optimized Website - Step 3: Site Structure
Welcome to part three in this search engine positioning series. Last week we discussed the importance and considerations that much be made while creating the content that will provide the highest ROI for your optimization efforts.
How Do I Submit My Site To Search Engines?
You've built up a good site, put in a lot of content, but none of the search engines lists your site? If you'd like to know how to go ahead with Search Engine Submission the right way, read on.How do I get my site to be included in search-engines? That's a question a lot of people ask me.
9 Ways to Keep Google Happy
A recent Google patent application has the SEO community buzzing. At a bare minimum this document reveals the direction Google is taking its future search criteria.
Release from Google Sandbox Only to Search the Playground
The Google Sandbox Effect has been discussed at length in ourcase study of a new website first crawled in May by Googlebot.We can now further the case study with indexing comparisonsand discuss interesting Googlebot crawler behavior afterrelease, at the 75 day mark, of the study website from thatvery confining Sandbox.
60 Day Sandbox for Google & AskJeeves; MSN Indexes Quickest, Yahoo Next
Search engine listing delays have come to be called the Google Sandboxeffect are actually true in practice at each of four top tiersearch engines in one form or another. MSN, it seems has theshortest indexing delay at 30 days.

Home | Site Map | Thai Hosting | Website Directory
  © 2007 ThaiIRC.in.th