Glossary

robots.txt

A text file at the root of a website that tells search engine crawlers which pages they can and cannot access - the basic traffic directive for web crawlers.

See How It Works Book a Demo

Technical

What Is robots.txt?

The robots.txt file is a standard protocol for communicating with web crawlers. It uses "allow" and "disallow" directives to control which parts of a site different crawlers can access. For dealerships, proper robots.txt configuration is important for: blocking low-value pages from crawling (internal search results, admin pages, filtered SRP variations), allowing AI crawlers (GPTBot, ChatGPT-User, PerplexityBot) for GEO visibility, including a reference to the XML sitemap, and preventing crawl budget waste. A common mistake is blocking too much - some dealership website providers accidentally block important pages or entire sections from crawling.

Related Terms

Crawl Budget

The number of pages Google will crawl on your website within a given time period - critical for large dealership sites with thousands of inventory pages.

XML Sitemap

A file that lists all important pages on a website, helping search engines discover and crawl content efficiently - especially important for large inventory sites.

llms.txt

A plain text file placed at the root of a website that tells AI language models which pages are most important and what the site is about - like robots.txt for AI.

Deep-Dive Guides

/resources/dealership-website-optimization

/resources/dealership-seo-audit-2026

Back to Full Glossary

Diverse team of dealership professionals standing together

Don't Wait

Build Before You Need To

The teams gaining ground aren't reacting faster. They're building a content system that works for them even when they're not working on it.

That advantage grows every month.

Start Free

We Rise Together.