Read Your Server Logs: The Only Source That Never Guesses

Every SEO tool you use is a model of your site. Your access log is not a model — it is a receipt for every request that ever hit the server, with a status code, a byte count, and a timestamp. If Googlebot fetched a URL at 3:14 a.m. and got a 500, that fact is sitting in a text file on your server right now. No sampling, no 28-day averaging, no estimate.

The format

Apache's combined log format is the default nearly everywhere, and every field has a fixed position:

66.249.66.1 - - [13/Aug/2026:03:14:22 +0000] "GET /blog/cache-headers HTTP/1.1" 200 18422 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
 ^                ^                            ^    ^                    ^   ^                                    ^
 $1 client IP     $4 timestamp                 $6   $7 path              $9  $10 bytes                            user agent
                                               method                    status

Which means awk is a perfectly good log analyzer. $7 is the path, $9 is the status code, and that covers most of what you want to know.

Verify the crawler first

Before you conclude anything, understand that the user agent string is a free-text field the client chooses. Anyone can claim to be Googlebot, and plenty of scrapers do precisely that to get past crude blocks. Filtering on the string alone gives you a mix of Google and liars.

Google's documented verification method is a reverse DNS lookup followed by a forward lookup:

# 1. What hostname does the IP claim?
host 66.249.66.1
# -> 1.66.249.66.in-addr.arpa domain name pointer
#    crawl-66-249-66-1.googlebot.com

# 2. Does that hostname resolve back to the same IP?
host crawl-66-249-66-1.googlebot.com
# -> crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Both steps are required. Step one alone can be forged by anyone controlling a reverse DNS record; the round trip cannot. The hostname must end in googlebot.com or google.com. Google also publishes its crawler IP ranges as JSON, which is the better approach if you are verifying in bulk rather than by hand.

Four one-liners

Status code distribution

awk '{print $9}' access.log | sort | uniq -c | sort -rn

Your baseline. A healthy site is mostly 200 and 304, with a modest tail of 404. Any meaningful number of 5xx is the most urgent thing in the file, because crawlers slow down against a server that errors.

What Googlebot is actually spending requests on

grep 'Googlebot' access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -30

This is the money query. You will often find the top entries are not your important pages — they are sort-order parameters, a paginated archive forty pages deep, or a calendar widget generating dates into the next century.

Errors served to crawlers

awk '$9 ~ /^[45]/ {print $9, $7}' access.log \
  | sort | uniq -c | sort -rn | head -30

Add grep Googlebot in front to narrow it to crawler-visible breakage. A 404 that Googlebot keeps requesting usually means something on your own site still links to it.

Crawl volume over time

grep 'Googlebot' access.log \
  | awk -F: '{print $2":00"}' | sort | uniq -c

Requests per hour. A sudden drop after a deploy is a signal worth chasing — it often traces back to a robots.txt change or a spike in server errors.

What the numbers usually tell you

Common log findings and what they mean

Pattern	Likely cause
Parameterized URLs dominate crawl	A faceted navigation trap. Filters are generating an effectively infinite URL space.
Many 301s in the crawl	Internal links point at pre-redirect URLs. Every one is a wasted round trip.
Important pages rarely crawled	Buried in the link graph, or crawl capacity is going elsewhere.
URLs crawled that are not in your sitemap	Either your sitemap is incomplete or something is generating URLs you did not intend.
5xx clustered in time	Resource limits under concurrent crawl. Shared hosting process caps do this.
Crawling of /assets/ dropped to zero	Something started blocking them — and rendering is now broken.

Redirect chains, found cheaply

Crawlers following two hops to reach your content is pure waste. Find the offenders:

awk '$9 == 301 || $9 == 308 {print $7}' access.log \
  | sort | uniq -c | sort -rn | head -20

Other crawlers in your logs

Googlebot is not the only visitor. You will also see Bingbot, various SEO tool crawlers, uptime monitors, and AI training crawlers such as GPTBot and ClaudeBot. Some respect robots.txt, some do not, and a few generate real load.

awk -F'"' '{print $6}' access.log \
  | sort | uniq -c | sort -rn | head -20

That prints user agent strings by volume. If a single non-search crawler is your heaviest traffic source, a robots.txt rule is the polite first step and a server-level block is the second. Deciding whether to allow AI crawlers is a business call, not a technical one — but you cannot make it if you do not know they are there.

Practicalities

• Find the file. On shared hosting it is usually under ~/logs/ or exposed in the control panel. Ask if it is not obvious — some hosts keep only a few days, which is worth knowing before you need history.
• Rotated logs are gzipped. Use zgrep and zcat rather than decompressing first.
• For anything recurring, use a real tool. goaccess gives you an interactive report from the same file in one command. The one-liners are for answering a specific question fast.
• Sample size matters. A single day is enough to spot a crawl trap and not enough to judge crawl frequency. Use a month for trends.

The half-hour version

1. Status code distribution. Anything 5xx gets fixed first.
2. Top 30 paths Googlebot requested. Are they your actual content?
3. All 4xx and 5xx served to verified crawlers.
4. Top redirected paths — then fix the internal links causing them.
5. User agents by volume, to see who else is on your server.

Nothing in that list requires a subscription, and it answers questions no dashboard can.