Glossary

robots.txt

A text file at the root of a website that tells search engine crawlers which pages they can and cannot access - the basic traffic directive for web crawlers.

See How It Works Book a Demo

Technical

What Is robots.txt?

The robots.txt file is a standard protocol for communicating with web crawlers. It uses "allow" and "disallow" directives to control which parts of a site different crawlers can access. For dealerships, proper robots.txt configuration is important for: blocking low-value pages from crawling (internal search results, admin pages, filtered SRP variations), allowing AI crawlers (GPTBot, ChatGPT-User, PerplexityBot) for GEO visibility, including a reference to the XML sitemap, and preventing crawl budget waste. A common mistake is blocking too much - some dealership website providers accidentally block important pages or entire sections from crawling.

Related Terms

Crawl Budget

The number of pages Google will crawl on your website within a given time period - critical for large dealership sites with thousands of inventory pages.

XML Sitemap

A file that lists all important pages on a website, helping search engines discover and crawl content efficiently - especially important for large inventory sites.

llms.txt

An optional plain-text file at a site root that lists important pages for some non-Google AI crawlers. Google Search ignores it for rankings and generative AI features.

Deep-Dive Guides

/resources/dealership-website-optimization

/resources/dealership-seo-audit-2026

Back to Full Glossary

Don't Wait

Master robots.txt and Beyond

Hrizn helps dealerships execute the strategies behind every term in this glossary - from AI-optimized content to automated schema markup.

That advantage grows every month.

See How It Works Book a Demo

We Rise Together.