The AI Black Box What Transparency to Demand
“Proprietary algorithm, trust us” is no longer an acceptable answer from an AI vendor in 2026. This page is four questions every AI-touching vendor should answer on record, the standard for what a good answer looks like, the shape of a bad one, and an explicit talk track that agencies can use with their own dealer clients.
"Proprietary algorithm, trust us" is not an acceptable answer from an AI-touching vendor in 2026. Four transparency questions every vendor should answer on record: (1) reproducibility, can we reproduce this metric using first-party data; (2) audit trails, what is the per-asset record for generated content; (3) source and data disclosure, what data flows where, how long is it retained, and with whom is it shared; (4) human oversight, where does AI decision-making end and human sign-off begin. Specificity in answers is credibility. Blanket "proprietary" is a signal. The questions are evaluation-phase; contract red flags are commit-phase; both are needed. Agencies can use this framework as a talk track with their dealer clients to show real evaluation work is being done.
Reproducibility is the minimum bar. If you cannot re-derive the metric, you cannot audit it.
Audit trails on AI-generated content protect you from hallucinated facts, compliance failures, and silent quality drift.
"Anonymized" is a word that means nothing without the technique (hashing, k-anonymity, differential privacy) and the aggregation unit.
A good partial answer is more credible than a blanket "proprietary" answer. Specificity is credibility.
The four questions are evaluation-phase. Contract red flags are commit-phase. Both are required.
Agencies that run the framework with dealer clients earn stronger relationships and raise ecosystem transparency standards.
“Proprietary Algorithm, Trust Us” Does Not Clear the Bar
Every AI-touching vendor on the market today makes two kinds of claims about their system: what it does (content generation, metric production, conversation handling, visibility optimization) and why it is better (scale, intelligence, proprietary training data, integration depth). Vendors generally defend the first kind of claim well. They almost always shield the second.
The shield is sometimes legitimate, as in an IP-protected core model. More often it is methodology opacity: “proprietary algorithm” used to deflect questions a competent vendor should have no trouble answering. The four questions on this page are the ones that separate the two cases. A vendor with a genuinely proprietary core model can still answer them. A vendor hiding behind “proprietary” will struggle.
This page is the buyer’s-side argument. Where the report decoder tears apart the metrics and contract red flags covers the commit-stage legal terms, this page is the evaluation-stage filter. Four questions, asked on record, before anyone signs anything.
What Every AI Vendor Should Answer on Record
Reproducibility
“Can we reproduce this metric ourselves, independent of your system, using first-party data? If not, exactly what is the reason it is non-reproducible?”
Reproducibility is the minimum bar for any measurement anyone pays for. If you cannot re-derive the number, you cannot audit it, compare it across time, or verify that it is moving because of your actions rather than vendor methodology drift. A vendor who will not disclose enough methodology to make reproduction possible is selling a narrative, not a measurement.
“Here is the data source, the calculation, the date range, and the parameters. You can run this yourself and should get a result within X percent of ours; here is why. And here is the raw underlying data we used.”
“The calculation is proprietary. It incorporates many signals in a way that would be hard to explain. You should trust the number because of our track record, industry reputation, or algorithm sophistication.”
Audit Trails for Generated Content
“For any content generated on my behalf, what is the audit trail? What source material was used, what prompts were issued, what model produced the output, and what human review passes were applied?”
Content generated by AI can inherit biases, fabricate claims, violate OEM compliance, or plagiarize upstream sources. Without an audit trail, you have no defense if the content becomes a problem later, and no insight into why quality is drifting. Audit trails are also the single best lever against hallucinated dealership facts in AI-generated content.
“Every piece of published content has a traceable record of source inputs, the model and version used, the prompt structure, any RAG retrieval sources, human editor IDs, and the approval timestamps. You can export the record for any asset, on request, in a standard format.”
“Content goes through our AI pipeline with human oversight. We cannot provide per-asset audit records because of the volume, but we guarantee quality at the system level.”
Source and Data Disclosure
“What data sources are consumed to produce deliverables or reports on my behalf, what data from me is consumed by your system, and where does my data go from there?”
Data flow is the fundamental mechanic of AI systems. If a vendor cannot clearly articulate what they ingest, where it goes, how long it is retained, and who has access to it, they are asking you to extend trust they have not explained. For dealerships, the stakes include customer lead data, inventory data, and conversational data from AI chat deployments.
“Here is the data inventory: what we pull from you, where it is stored, how long it is retained, who has access under what authorization, and the third parties with whom it is shared. Retention windows and deletion workflows are documented and honored.”
“We use a combination of public data, proprietary data, and dealer-supplied data. Specific sources and retention details are internal to how we operate the platform.”
Human Oversight Boundaries
“Where does AI decision-making end and human decision-making begin in your process? For publishing to my site, for reports that reach my executives, for customer-facing conversations, who signs off?”
AI systems can and do make consequential mistakes. A clear boundary between AI decisions and human decisions is the difference between a recoverable incident and a PR problem. It also tells you whether the vendor is genuinely operating with oversight or simply layering AI-generated outputs with AI-generated review.
“Here are the specific checkpoints where a named human role is the final decision-maker. Here is the escalation path if AI outputs trigger compliance, brand, or legal concerns. Here is the mean-time-to-human for issues the AI layer flags.”
“Our AI is trained with human feedback. Human oversight is built into the system. Exceptions are handled by our quality assurance process.”
What Changed Between 2023 and 2026
In 2023, “we use AI” was a differentiator. Vendors could reasonably wave a wand and call it a day. That period is over. Three shifts have raised the bar:
- AI is table stakes, not a differentiator. Every marketing-adjacent vendor uses AI now. The question is no longer whether, but how.
- AI failures are consequential and public. Hallucinated reviews, fabricated vehicle specs, OEM-non-compliant content, and AI chat agents saying the wrong thing to customers have all generated headlines. Dealers have seen enough to know the risk is real.
- Regulators and OEMs are writing rules. Emerging AI disclosure requirements, OEM AI governance policies, and industry-specific compliance frameworks all assume vendors can answer transparency questions. The ones who cannot will not clear those standards.
The dealer who asks transparency questions today is not being difficult. They are anticipating the standard that is arriving anyway, and getting ahead of vendor relationships that will not survive it.
How Agencies Can Use This With Their Dealer Clients
Agencies sit in a unique position: between the vendor ecosystem and the dealer. A dealer who receives vendor pitches from multiple directions, including through their agency, will often defer to the agency’s judgment on which ones to accept. That is a responsibility, and this page is the framework that makes the responsibility exercisable.
“We ran the four AI transparency questions with [vendor] last week. Here is what we heard on reproducibility, audit trails, data disclosure, and human oversight. Based on the answers, we recommend / do not recommend moving to contract, and here is why. Either way, we have the responses in writing. If the relationship changes in year two, we have the record.”
Three things this does. It shows the dealer that the agency is running a real evaluation process, not rubber-stamping vendor pitches. It creates a written record that survives agency staff changes and dealership ownership changes. And it raises the aggregate transparency standard of the vendor ecosystem, which benefits every client the agency serves.
Agencies that take this posture end up with stronger dealer relationships. Vendors know the agency is serious. Dealers know the agency is advocating for them, not executing vendor commissions in the background. The evaluation work is the value.
The Four Questions in Compact Form
Before we finalize our engagement, we would like written answers to four standard AI transparency questions we run with every vendor:
Reproducibility. Can we reproduce the metrics you report using first-party data? If not, what specifically prevents reproduction?
Audit trails. For content generated on our behalf, what is the per-asset audit record (source material, prompt, model, human review)? Can it be exported on request?
Source and data disclosure. What data flows from us to your system, where is it stored, how long is it retained, and with what third parties is it shared?
Human oversight. Where does AI decision-making end and human sign-off begin in your process, and who specifically holds that authority?
We are happy to answer any context questions that help you respond. Thank you for the transparency.
Common Questions About Running the Framework
How do I present these four questions to a vendor without making it adversarial?
Frame them as a standard vendor-review process, not an interrogation. "Before we finalize, we are running the four transparency questions we ask every AI-touching vendor. These are for our records as much as yours." Most vendors respond well. The ones who don’t are telling you something important.
What if a vendor partially answers but not completely?
Partial answers are common and usually acceptable, as long as the vendor is explicit about what they can and cannot disclose and why. “We can disclose data flow and retention; reproducibility is limited by the LLM API we use, which changes under us; audit trails are per-asset for content and per-batch for bulk operations.” That is a defensible answer. A vendor who gives that level of specificity is credible even when they cannot answer everything. A vendor who says “proprietary” across the board is not. See the AI visibility report decoder for the most common pattern of fully-proprietary responses.
Is this practical for a small dealership or independent dealer?
Yes. The four questions are written to be framework-neutral: any vendor, any size, any price point. An independent dealer is, if anything, more exposed to vendor-methodology failures than a 40-rooftop group, because the cost of a bad vendor decision represents a larger share of total marketing spend. The framing and the time investment are identical.
Where does this fit alongside the contract red flags page?
Transparency questions sit before the contract. They are part of the evaluation phase: what you learn from how a vendor answers determines whether you are comfortable moving to the contract phase at all. AI Visibility Contract Red Flags is the next step: turning the answers you got here into binding contractual protections. A vendor who answered the transparency questions well should have no trouble accepting the addendum on that page.
Do the questions change for OEM-driven AI programs?
The questions do not change, but the escalation does. OEM programs have a different commercial structure than direct-vendor relationships, so the answers often have to come from the OEM rep relaying from the program vendor. Get the answers in writing regardless. See OEM AI Programs Due Diligence for the expanded diligence set that accompanies these four questions in OEM contexts.
Build Before You Need To
The teams gaining ground aren't reacting faster. They're building a content system that works for them even when they're not working on it.
That advantage grows every month.
Start FreeWe Rise Together.