Thorsten Litzki · 26 August 2026
The Same 559 Domains, Scanned Twice. The Average Score Dropped.
In June 2026, I scanned 559 domains across 10 regulated German industry verticals with 183 infrastructure signals. Every single domain scored below the certification threshold. The highest score was 63 out of 100.
In August 2026, I ran the same scan again. Same methodology, same engine, same user agent. The average score dropped from 45.3 to 44.0. The ceiling rose to 70 for a single domain. The floor fell to 20. And 22 domains that had scored above 25 in June fell below that line. Zero domains passed. The web deteriorated.
This article presents the findings from a longitudinal infrastructure audit across 10 regulated German industry verticals. Two scans, 70 days apart, identical methodology. The goal: measure how far the open web has moved toward supporting autonomous systems.
The Dataset: 559 Domains, 10 Sectors, 183 Signals
The audit covered 559 domains across 10 sector lists: Automotive (50), B2B E-Commerce (30), Government Agencies (50), Digital Industry Association / BVDW (100), BaFin-regulated Financial Services (50), DACH Publishers (40), DAX 40 (40), German E-Commerce Top 100 (100), Healthcare (50), and Critical Infrastructure / KRITIS (50). 540 of these are unique domains; 19 appear in multiple lists.
The scoring engine evaluated each domain against 183 individual signals, grouped into 21 verification clusters across five categories: Content Quality, Machine Provisioning (the engine labels this cluster AI Readiness), Infrastructure, Authority, and Anchoring. The engine assigns a composite score from 0 to 100. A domain earns a CERTIFIED verdict only when every category meets a minimum floor of 70. Below that threshold: the verdict reads NOT READY.
Methodology: How the Scanner Works
The scanner operates as a lightweight HTTP client with a named user agent (SOVPBot/1.0) and a 15-second timeout per request. It fetches static HTML responses, parses structured data, evaluates HTTP headers, checks DNS records, and probes for machine-readable discovery files (llms.txt, robots.txt, agent endpoints). It runs entirely without a headless browser. Every signal comes from what the server delivers to a standard HTTP GET request.
155 domains blocked the scanner entirely. These domains earned a composite score based on available signals, but individual cluster checks that require page-level access returned null. The per-dimension counts below refer only to the evaluable subset.
What Changed in 70 Days
The status quo held. Scores dropped. The average composite score fell by 1.3 points. The distribution widened in both directions. One domain climbed to 70 (up from a June ceiling of 63). 22 domains dropped below 25 (zero had scored that low in June). The spread grew while the center sank.
Score Shift by Sector
| Sector | n | June | Aug | Δ | Direction |
| Automotive | 50 | 45.2 | 40.4 | −4.8 | Declined |
| B2B E-Commerce | 29 | 44.1 | 50.1 | +6.0 | Improved |
| Government | 50 | 43.2 | 39.7 | −3.4 | Declined |
| BVDW | 100 | 45.7 | 44.0 | −1.7 | Declined |
| BaFin Financial | 50 | 45.3 | 45.9 | +0.6 | Stable |
| DACH Publishers | 40 | 48.9 | 48.6 | −0.3 | Stable |
| DAX 40 | 40 | 45.1 | 42.4 | −2.8 | Declined |
| E-Commerce Top 100 | 100 | 46.7 | 44.7 | −2.0 | Declined |
| Healthcare | 50 | 42.3 | 41.6 | −0.7 | Stable |
| Critical Infra | 50 | 45.0 | 44.8 | −0.1 | Stable |
B2B E-Commerce is the only sector that improved meaningfully (+6.0). Automotive declined the hardest (−4.8), followed by Government Agencies (−3.4) and DAX 40 (−2.8). The sectors with the strongest compliance pressure (BaFin, KRITIS) stayed flat. The sectors with the least external pressure declined.
Distribution Shift
The quartile distribution shows the widening:
| Score Range | June | August | Shift |
| 0–25 | 0 | 22 | +22 |
| 25–50 | 422 | 379 | −43 |
| 50–75 | 137 | 158 | +21 |
| 75–100 | 0 | 0 | 0 |
Domains are moving out of the center in both directions. 22 fell into the lowest bracket. 21 climbed into the 50–75 range. The 75–100 bracket remains empty. The ceiling at 70 belongs to one domain. The structural barrier persists.
Where the Architecture Breaks: Category Averages
Four of the five scoring dimensions carry a clear interpretation for infrastructure verification. The fifth (Anchoring) primarily reflects domain age and self-identification. Its high values (79.4 in August) reveal the maturity of the sample, and little about autonomous readiness.
The four actionable categories:
Infrastructure leads at 66.1 (June: 62.5, +3.6). SSL certificates, HTTP headers, DNS configuration. The plumbing works and is improving. Most organizations have invested here because compliance frameworks demanded it for years.
Content Quality follows at 58.4 (June: 57.9, +0.5). Structured HTML, semantic markup, content-to-noise ratio. Acceptable for human consumption. Deterministic extraction requires more.
Authority sits at 48.8 (June: 50.0, −1.2). Domain age, web corpus indexing, backlink ecosystem. A neutral midpoint reflecting signals designed for search engines.
Machine Provisioning (labeled AI Readiness in the scoring engine) collapses to 10.6 (June: 11.5, −0.9). This is the structural gap. The category measures llms.txt adoption, agent discovery endpoints, and machine-readable policy declarations. The score declined between scans. Across both periods, this category trails every other by a factor of five.
Dimension by Dimension: What the Numbers Expose
1. Structured Data: 42% of Evaluable Domains Carry Zero JSON-LD
Of 400 domains where the structured data check completed successfully (WAFs blocked 159 and returned zero results), 168 (42%) contained zero JSON-LD markup. Zero Schema.org types. Zero semantic annotations. Zero parseable structure for a machine.
The remaining 232 domains (58%) had at least one JSON-LD block. Presence marks a starting point. Usability requires four additional criteria: unique Schema type definition, 100% field completeness, logical nesting hierarchy, and validated format.
For an agent processing a product page, the difference between a validated JSON-LD block and raw HTML is the difference between a database lookup and a RegEx gamble.
2. Compliance Signals: 18% Adopted llms.txt. The Debate Over Its Utility Continues.
100 of 559 domains (18%) serve an llms.txt file. 454 lack it. This is higher than expected for regulated German enterprises and suggests real awareness of the signal.
Adoption and utility are different questions. An Ahrefs study published in June 2026 analyzed server logs across 137.000 domains and found that 97% of llms.txt files received zero requests. Zero bots. Zero humans. Zero traffic. Of the 3% that saw any activity, AI retrieval bots accounted for roughly 1% of requests. The largest consumer category: SEO audit tools checking for the file’s existence.
Google Search has publicly stated that appearing in AI results works without llms.txt. John Mueller compared the file to the keywords meta tag. At the same time, Chrome Lighthouse now audits for it as part of its agentic browsing checks. Google occupies both sides of the argument simultaneously.
For this audit, the question is narrower than AI visibility. The llms.txt file signals architectural intent: a domain that publishes one has at least considered how machines should consume its content. The remaining 82% have published zero indication of that consideration. Actual AI readership of the file sits apart from its value as an infrastructure maturity signal. In this dataset, both the adopters and the non-adopters scored below the certification threshold.
3. Response Integrity
A server returning HTTP 200 for a missing page deceives your agent. Soft 404s, redirect chains, and inconsistent status codes poison automated workflows. The audit verifies that HTTP responses accurately reflect resource state. Honest infrastructure communicates failures as failures. The scan stores aggregate scores only. Per-domain soft-404 counts remain outside this dataset.
4. Crawlability
Can your agent reach the content without executing JavaScript? Many modern web applications hide their data behind client-side rendering. For a headless agent that processes the raw HTTP response, a page that renders in Chrome and delivers an empty div in the HTML source is a dead end. The current pipeline measures content quality as an aggregate. A standalone CSR measurement requires a dedicated scan with headless browser comparison, which falls outside this dataset’s scope.
5. Latency and Stability
An unstable server produces timeouts and incomplete data packets. For agent fleets that make parallel requests at high frequency, latency variance (jitter) is a cost multiplier. The audit measures both the mean and the standard deviation of response time. A domain that responds in 200ms on average and spikes to 4 seconds every fifth request breaks scheduling and inflates the cost per run.
155 Domains Block the Scanner Entirely
155 of 559 domains blocked the scanner (153 in June, 155 in August). The scanner identifies itself with a named user agent (SOVPBot/1.0). These blocks represent how regulated web infrastructure treats unknown automated systems: with silence.
A blocked request and a missing domain produce the same outcome for an agent. These 155 domains stay invisible to every autonomous system outside their allowlist.
From Probabilistic Ingestion to Deterministic Verification
Most agent pipelines today rely on probabilistic methods. RAG systems slice unstructured text into chunks, embed them, and hope the vector similarity captures the right meaning. This works well enough when the source material is clean. When the source material carries noise, ambiguous markup, or inconsistent metadata, RAG scales the noise proportionally.
Structured data anchors change this dynamic. A validated Schema.org entity provides a semantic truth that the vector embedding can be verified against. The combination of probabilistic retrieval and deterministic anchors reduces hallucinations to a technical minimum. One system finds the content. The other confirms it is correct.
The problem: this hybrid architecture only works when the source domain provides those anchors. 42% of evaluable domains in this dataset carry zero JSON-LD. At a Machine Provisioning average of 10.6, the machine-readable signals this architecture requires barely exist on the open web.
Building Your Own Infrastructure Filter
Waiting for the web to catch up means betting on others solving the problem first. Three steps, integrated directly into your data pipeline.
Step one: the automated pre-flight check. A lightweight service that inspects HTTP headers, validates llms.txt presence, and checks response integrity before every ingestion. This first gate eliminates unstable domains immediately.
Step two: the semantic deep dive. Your system validates existing Schema.org structures against your requirements. Only domains with 100% field completeness and validated JSON-LD pass this milestone.
Step three: the binary gate. Combine all results into a final decision module. The output is a PASS or FAIL. Zero percentages. Zero weighted averages that hide which 20% of the data is garbage. If the source fails, your agent stays away.
A minimal pre-flight check in Python covers the three gates in under 30 lines:
import requests, json
def preflight_check(domain: str) -> dict:
base = f"https://{domain}"
result = {'domain': domain, 'pass': True, 'checks': {}}
# Gate 1: Response integrity
try:
r = requests.get(base, timeout=10,
headers={"User-Agent": "InfraAuditBot/1.0"})
result['checks']['status'] = r.status_code
result['checks']['ttfb_ms'] = int(
r.elapsed.total_seconds() * 1000)
if r.status_code != 200:
result['pass'] = False
except requests.RequestException:
result['pass'] = False
return result
# Gate 2: llms.txt presence
llms = requests.get(f"{base}/llms.txt", timeout=5)
result['checks']['llms_txt'] = llms.status_code == 200
# Gate 3: JSON-LD validation
jsonld = [s for s in r.text.split('application/ld+json">')
if '"@type"' in s]
result['checks']['jsonld_found'] = len(jsonld) > 0
if jsonld:
try:
schema = json.loads(jsonld[0].split('</script>')[0])
result['checks']['schema_type'] = schema.get(
'@type', None)
except json.JSONDecodeError:
result['checks']['jsonld_valid'] = False
result['pass'] = False
# Binary verdict
if not result['checks'].get('llms_txt'):
result['pass'] = False
if not result['checks'].get('jsonld_found'):
result['pass'] = False
return resultThis filter is intentionally strict. In the 559-domain dataset, 454 domains (82%) fail on the llms.txt check alone. Of the 100 that serve llms.txt, many fail on JSON-LD validation. Zero domains reach CERTIFIED under the full scoring engine. The binary filter produces the same result as the comprehensive audit, at a fraction of the cost.
An 80% reliability score sounds reassuring in a slide deck. In production, it means one in five data points is inconsistent. Across a scaled pipeline processing thousands of sources, that 20% failure rate compounds into unpredictable downstream errors. Binary filtering eliminates this category of risk entirely.
The Structural Reality
Two scans. Seventy days apart. The same 559 domains. The average dropped. The floor dropped further. The ceiling barely moved. Zero domains passed.
Engineers built the web for humans looking at screens. Structured data is an afterthought. The signals that autonomous systems require (llms.txt, agent discovery, machine-readable policies) exist on a small minority of production domains. Even where they exist, industry data suggests they go unread. Machine Provisioning at 10.6 out of 100 is the clearest measure of this gap.
This will change. The economic pressure of agentic commerce guarantees it. Organizations that provision structured, verifiable, machine-readable infrastructure today will be the preferred data sources of every agent fleet tomorrow. Organizations that ignore this will become invisible to autonomous systems, regardless of their brand value or search ranking.
The ceiling of 70 is a priorities problem. SSL works. DNS works. HTTP works. The layer above the plumbing remains unbuilt: the structured invitation, the machine-readable policy, the verifiable proof that the data is what it claims to be.
How many of your agent’s data sources would survive a binary infrastructure audit?
About the Author
Thorsten Litzki is the founder of Litzki Systems LLC and the architect behind the Sovereign Validation Protocol (SOVP), a cryptographically signed infrastructure verification engine. His work focuses on the intersection of data engineering and deterministic web verification. He documents his research publicly on LinkedIn (linkedin.com/in/thorsten-litzki/) and at litzki-systems.com/sovp.