How publishers fight AI scraping: what Libération and The New York Times are doing

If you work with web data, you already know how easy it has become to collect it at scale. A short Python script, a pool of rotating proxies and a headless browser can pull thousands of pages before lunch. For news publishers, that same ease has turned into a business problem, because a growing share of the traffic landing on their articles is no longer human. Telling humans from bots has become a job in its own right.

What makes the news industry worth watching is that nobody has settled on a single playbook. Some newsrooms are betting on the courts and on licensing deals, others on technical control of their own traffic, and plenty are trying a bit of both. Two publishers show how far apart those strategies can sit.

Why news sites became such an attractive target

News content is fresh, structured and fact-checked, precisely what large language models need to train and to answer questions about current events. The catch is that a news site only earns money when people actually visit it. A scraper that lifts an article skips the ads, ignores the subscription prompt and never becomes a paying reader, while the publisher still covers the bandwidth and server load.

The old gatekeeper, robots.txt, only works when the visitor chooses to respect it, and it is a convention rather than something legally binding. The scrapers that ignore it rarely announce themselves: many run on headless browsers that render pages the way Chrome would, spread their requests across residential IP addresses and pace themselves to look like ordinary readers.

How DataDome spots AI scrapers on a news site

Datadome, a security platform that publishers use to manage bot traffic, starts from a simple observation: a scraper needs millions of requests to collect anything worthwhile, and every request leaves traces. Rather than relying on rate limits or IP blocklists, its detection reads three families of signals at once, since any single one can be evaded with enough effort.

The first is behavior. Server side, that means request speed, the order pages are visited in and how deep sessions go. In the browser, a JavaScript tag reads mouse paths, scrolling and typing rhythm, where a real reader is slightly chaotic and a bot stays mechanical even when randomized.

The second is reputation. Requests are scored against the history of their IP address, subnet and network operator. That matters because serious operations now rotate through residential proxies: DataDome’s researchers documented one attack spread across more than 45,000 residential IPs, each making fewer than ten requests a day, invisible to any rate limiter.

The third is fingerprinting. HTTP headers, the TLS handshake and browser characteristics betray headless browsers and automation frameworks such as Puppeteer, Playwright and Selenium, even with stealth plugins. For a publisher, the result is a decision in under two milliseconds on every request: verified crawlers and paying partners through, everything else challenged or blocked.

The New York Times: courts first, licensing second

The New York Times picked the most visible route. In December 2023 it sued OpenAI and Microsoft, arguing that millions of its articles had been used to train AI models without permission. The dispute is still open: in September 2026, all three parties set out their arguments before the judge, with a ruling still to come ahead of a possible trial.

The Times has also made clear that its objection is to unpaid use, not to AI as such. In May 2025 it signed its first licensing agreement tied to generative AI, a multiyear deal with Amazon covering its news articles, NYT Cooking and The Athletic, including their use to train Amazon’s own models.

It is a strategy for a publisher with deep pockets. If it succeeds, the precedent helps the whole industry, but it is slow and does nothing to stop a scraper hitting the site tonight.

Libération: taking control of the traffic itself

Libération, the French daily with more than a million readers a day and over 120,000 subscribers, reached the opposite conclusion about timing. Its CTO, Lucas Delcroix, has said plainly that legal answers from France or Europe could take years, and that the newsroom needed protection before its content was taken rather than after.

Part of the problem was visibility. Like many papers, Libération keeps some of its journalism behind a paywall, but its public pages were being copied at scale by AI-powered crawlers and copycat sites, and the technical team struggled to tell which tools sat behind which requests. Because the paper already ran on Arc XP, a CMS designed for newsrooms, it switched on that platform’s native DataDome integration, a set-up that took less than five minutes, according to the published Libération case study.

Once the traffic was classified, the scale became hard to ignore. Between January and May 2026, 565 million malicious requests were blocked, an average of roughly 113 million a month, while unauthorized scraping attempts rose by 86% over the same period. Just as useful, its engineers can now tell human readers apart from legitimate crawlers, contracted partners, unauthorized AI agents and scrapers, and decide which to let in. Delcroix describes the shift as going from victim to observer.

Two strategies that work better together

Placed side by side, the New York Times and Libération are less opposed than they first appear. A licensing deal only holds its value if a publisher can see who is accessing its content and enforce the terms for everyone else. An agreement with one AI company means little if dozens of unidentified bots can take the same articles for free through the back door. Detection, in that sense, is what turns a licensing policy into something enforceable.

For anyone who collects web data, the direction of travel is clear. Crawlers that identify themselves, follow the habits recommended in guides to responsible large-scale scraping and go through agreed channels are likely to keep their access, while those relying on disguise will find the open web noticeably less open.