Log File Analysis for SEO: A Beginner’s Guide

If you want to know what Googlebot really does on your site, not what tools guess it does, you need your server logs. Log file analysis for SEO means reading the raw record of every request your server received. You filter it down to search engine and AI crawlers, then use it to spot wasted crawling, errors and pages that bots ignore.

You do not need expensive software to start. In my experience, a week of logs, a spreadsheet or a few terminal commands, and a clear list of questions get you most of the value.

What a Server Log File Actually Is

Every time a browser or bot asks your server for something (a page, an image, a CSS file, a robots.txt file), the server writes one line to its access log. Nothing is sampled and nothing depends on JavaScript running. So the log captures visits that analytics tools never see, including every crawler hit.

That is the key difference from GA4. Analytics tracks people whose browsers run your tracking code. Logs track every request. That is why they are the only first-hand record of how crawlers move through your site.

Most servers (Apache, Nginx, LiteSpeed) use a version of the “combined” log format. A single Googlebot request looks something like this:

66.249.66.1 - - [11/May/2026:08:14:22 +0000] "GET /services/seo-audit/ HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

It looks dense, but it is just six or seven fields separated by spaces.

The Log Fields That Matter for SEO

You can ignore most of a log line. These are the fields I’d learn first:

• IP address (66.249.66.1): who made the request. You need this to confirm that a “Googlebot” visit really came from Google.
• Timestamp ([11/May/2026:08:14:22 +0000]): when it happened, including the time zone offset. Useful for crawl frequency and for matching spikes to site releases or outages.
• Request method and URL (GET /services/seo-audit/): what was requested. This is the field you group and count most often.
• Status code (200): how the server answered. 200 is fine, 301/302 are redirects, 404 is not found, 410 is gone, and 5xx means the server failed.
• Bytes (18342): the size of the response. Very small responses on important pages can mean bots are getting a blank or broken page.
• User agent (the long string at the end): what the client says it is. Googlebot, Bingbot, GPTBot and others name themselves here.

The referrer field (the "-" before the user agent) is usually empty for crawlers, so it matters less for this job. Some hosts and CDNs also add response time. I keep that if it is there.

How to Get Your Log Files

Where your logs live depends on how your site is hosted. Ask for at least two to four weeks of data if you can. A single day is too noisy to judge crawl patterns.

Shared Hosting With cPanel

In cPanel, look for Raw Access under the Metrics section. You can download compressed archives (.gz) for each domain. Many cPanel hosts delete raw logs after a short time unless you tick the option to archive them. So switch that on now, even if you won’t analyse anything for a month.

Hostinger hPanel and Other Custom Panels

Hosts with their own control panels, such as hPanel, usually show access logs somewhere under the website’s analytics or advanced settings. Some only show a recent window in the browser and do not offer full raw downloads. If you cannot find a download option, ask support directly for “raw access logs” and how long they keep them.

VPS, Dedicated Servers and Managed WordPress Hosts

On a server you control, logs usually sit in /var/log/apache2/ or /var/log/nginx/. They rotate daily into numbered or compressed files. Managed WordPress hosts often provide logs through their dashboard or on request, and how long they keep them varies a lot.

CDN Logs

Your site may sit behind a CDN (a network that serves cached copies of your pages) such as Cloudflare, Fastly or Amazon CloudFront. Then many bot requests may be answered from the CDN’s cache and never reach your origin server. Your origin logs will undercount crawling. CDN providers offer their own request logs. But full raw log exports are often limited to higher-tier plans, so check what your plan includes before you build a process around it.

A practical tip: whichever source you use, check whether IPs are being replaced by the CDN’s or load balancer’s addresses. If every request comes from the same handful of IPs, your server is logging the proxy, not the visitor. You will need the forwarded client IP field instead. This is the first thing I check when log numbers look strange.

Verify That Googlebot Is Really Googlebot

Anyone can put “Googlebot” in a user agent string. Scrapers and SEO tools do it all the time. If you skip verification, fake bots hitting odd URLs can badly skew your analysis.

Google’s documented method for verifying requests from Google’s crawlers is a two-way DNS check:

1. Run a reverse DNS lookup on the IP from your log.
2. Confirm the hostname ends in googlebot.com, google.com or googleusercontent.com.
3. Run a forward DNS lookup on that hostname.
4. Confirm it points back to the original IP.

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The forward lookup matters, because anyone who controls an IP can set its reverse DNS to say anything. Only the round trip proves it.

For larger log files, Google also publishes its crawler IP ranges as JSON files (for example, common-crawlers.json for Googlebot). Matching IPs against those lists is faster than running DNS lookups on thousands of addresses.

Bing supports the same reverse-then-forward check, with hostnames ending in search.msn.com. Several AI companies publish IP ranges for their crawlers too. Check each provider’s own documentation before you trust a user agent.

What to Look for in Your Logs

Once you have filtered to verified search engine bots, work through these questions. Each one leads to a fix you can actually make.

Crawl Waste

Crawl waste is Googlebot spending requests on URLs that should never be indexed. Common causes:

• Faceted navigation and filter URLs (?colour=red&size=10&sort=price)
• Internal search result pages
• Session IDs, tracking parameters and calendar pages that go on forever
• Old staging or test folders that are still linked somewhere

Count how many Googlebot requests go to parameter URLs compared with clean URLs. If a large share of crawling lands on filters while new articles wait days to be crawled, make that your priority. Google’s own guidance says crawl budget is mainly a concern for very large or fast-changing sites. So a 200-page brochure site rarely has a true budget problem. Even there, though, logs show where bots are getting lost.

404 and 5xx Errors

A handful of 404s is normal, because bots recheck old URLs for a long time. What you are looking for is patterns:

• 404s on URLs that should exist, which points to broken internal links or a bad migration
• Many 404s from one section, which often means a template is creating bad links
• Any regular 5xx responses to Googlebot, which signal server strain

I treat server errors as more urgent than 404s. When Google sees repeated 5xx responses or timeouts, it slows its crawling to protect your server. Then important pages get revisited less often. My guide to HTTP status codes and SEO explains what each code tells Google.

Redirects and Redirect Chains

Filter for 301 and 302 responses served to Googlebot. If bots keep requesting URLs that redirect, your internal links, canonical tags or sitemap probably still point at old addresses.

Logs show each hop as a separate request, so you can spot chains. The old URL returns 301, the next one also returns 301, and only the third returns 200. Update links to point straight at the final URL.

Orphan Pages Being Crawled

Compare the URLs Googlebot requests with the URLs a site crawler (such as Screaming Frog or Sitebulb) finds by following your internal links. URLs that appear in logs but not in the crawl are often orphan pages. Google knows about them from old links, backlinks or an old sitemap, but nothing on your site links to them now.

Decide case by case. Link to the page properly from a relevant hub, redirect it, or let it return 404 or 410 if it no longer has a purpose. Our guide to building topic clusters with internal links covers how to bring useful orphans back into the structure.

The reverse check is just as useful. Some important pages appear in your crawl but get few or no Googlebot hits over several weeks. They are under-crawled. They usually need stronger internal links or a place in your XML sitemap.

Mobile Versus Desktop Googlebot

Google crawls most sites mainly with its smartphone crawler. In the user agent, look for “Android” and “Mobile” alongside “Googlebot”. If you see mostly desktop Googlebot, check which crawler Search Console reports for your site. But don’t read too much into small samples.

AI Crawlers

Logs are also the clearest way to see how AI companies use your content. The main user agents to filter for are:

• GPTBot: OpenAI’s crawler for collecting training data
• OAI-SearchBot: OpenAI’s crawler for showing sites in ChatGPT search results
• ChatGPT-User: requests made when a ChatGPT user’s action triggers a page fetch
• ClaudeBot: Anthropic’s crawler
• PerplexityBot: Perplexity’s crawler for its search index

It helps to separate training crawlers from search and user-triggered fetchers, because you may want to treat them differently in robots.txt. Our article on robots.txt and AI crawlers explains the options. Note that Google-Extended is a robots.txt control token, not a separate crawler, so you will not see it in your logs. Google’s crawling still shows up as Googlebot.

Check whether AI bots are hitting pages you blocked, whether they cause errors, and how much load they create. If they crawl heavily but you see no referral visits in analytics, I see that as a business decision worth discussing, not just a technical one.

Compare Your Logs With Search Console Crawl Stats

Search Console’s Crawl Stats report (under Settings) is the free shortcut. It shows total crawl requests, download size, average response time and host status. It also breaks these down by response code, file type, crawl purpose (discovery or refresh) and Googlebot type. It is only available for Domain properties or root-level URL-prefix properties. I explain each chart in my guide to reading the Crawl Stats report.

I use the two together:

• Crawl Stats gives a reliable overview and only covers Google, so there is no fake bot problem. But its example URLs are samples, not a full list.
• Logs give you every single URL, every bot, and exact timings. But they take more work and need verification.

Don’t expect the totals to match exactly. Google notes that its counts can differ slightly from server logs. And if your site sits behind a CDN, your origin logs will show fewer requests than Google reports. If the trend lines move in the same direction, your data is sound. If Crawl Stats shows thousands of daily requests and your logs show a few dozen Googlebot hits, something is wrong with how your logs are collected.

If you use Search Console’s reports more widely, our guide to the Search Console AI report builder covers building custom views quickly.

Simple Ways to Analyse Logs

Dedicated log analysers such as Screaming Frog Log File Analyser, JetOctopus, Botify or an ELK stack are worth it for large sites. For most small and mid-sized sites, a spreadsheet or the command line is enough to answer the important questions.

Option 1: A Spreadsheet

This works well for logs up to a few hundred thousand lines.

1. Unzip the log file and import it into Google Sheets or Excel, using a space as the delimiter (the character that splits columns). Quoted fields like the user agent stay in one column if you set the text qualifier to a double quote.
2. Name the columns: IP, timestamp, request, status, bytes, referrer, user agent.
3. Filter the user agent column to rows containing “Googlebot”.
4. Build a pivot table with URL as rows and count of requests as values. Add status code as a column to see errors by URL.
5. Add a second pivot by status code alone to see your overall error rate.

Spreadsheets struggle beyond about a million rows. That is where the command line takes over.

Option 2: The Command Line

On macOS or Linux, the Terminal already has everything you need. On Windows, the Windows Subsystem for Linux gives you the same tools. These examples assume the combined log format shown earlier, where the URL is field 7 and the status code is field 9.

Combine several days of compressed logs into one file:

zcat access.log.*.gz > all.log
cat access.log >> all.log

Pull out Googlebot requests:

grep "Googlebot" all.log > googlebot.log
wc -l googlebot.log

Status codes served to Googlebot, most common first:

awk '{print $9}' googlebot.log | sort | uniq -c | sort -rn

The 30 most-crawled URLs:

awk '{print $7}' googlebot.log | sort | uniq -c | sort -rn | head -30

Every URL that returned a 404 to Googlebot, with counts:

awk '$9 == 404 {print $7}' googlebot.log | sort | uniq -c | sort -rn

How many Googlebot requests went to parameter URLs:

awk '{print $7}' googlebot.log | grep -c "?"

Googlebot hits per day, to see crawl trends:

awk '{print substr($4, 2, 11)}' googlebot.log | uniq -c

Request counts for the main AI crawlers:

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot" all.log | sort | uniq -c | sort -rn

List the unique IPs claiming to be Googlebot, then check each one with a reverse lookup:

awk '{print $1}' googlebot.log | sort -u > gbot-ips.txt
while read ip; do echo "$ip $(host $ip | awk '{print $NF}')"; done < gbot-ips.txt

Any IP whose hostname does not end in googlebot.com, google.com or googleusercontent.com is not Google. Remove those lines before you draw conclusions. And do the forward lookup on anything that looks borderline.

If your server uses a custom log format, the field numbers will shift. Print one line with head -1 all.log and count the space-separated fields before you run the commands.

A Simple Monthly Log Review Routine

You don’t need to stare at logs every day. For most sites I think a monthly check is enough, plus an extra look after any migration, redesign or big content launch.

• Compare Googlebot requests per day against last month.
• Check the share of 3xx, 4xx and 5xx responses.
• List the top 20 crawled URLs and ask whether they are the pages you care about.
• Check that new pages from the past month have been crawled at least once.
• Review AI crawler volume and any requests to blocked paths.

One privacy point: access logs contain IP addresses, which can count as personal data in many regions. Share them only with people who need them, store them securely, and delete old files in line with your privacy policy.

Bringing Your Log File Analysis for SEO Together

Log file analysis for SEO is less about the tools and more about asking the right questions. Is Google crawling the pages that matter? Is it wasting time elsewhere? Is your server answering cleanly? Start with two to four weeks of logs and verify Googlebot properly. Then work through errors, redirects, crawl waste and orphan pages, in that order.

A good next step is to make sure the crawling you see lines up with how Google finds pages in the first place. Our explainer on how Google Search works fills in the crawling, indexing and ranking picture behind what your logs show you.