The Situation
More and more users no longer search via traditional search engines. Instead, they get instant, generated answers from Google Overviews/AI Mode, ChatGPT, Perplexity, or Gemini — often without source attribution, without a click to a publisher’s site, and without any advertising contact.
For publishers, this means: carefully produced, paid-for, and editorially verified journalism is being used without control or fair exchange. That threatens both digital advertising models and the ability to fund quality journalism.
Why “Pay-per-Crawl” Isn’t Enough
When Cloudflare introduced its Pay-per-Crawl approach, it gave a glimmer of hope to many helpless publishers (same goes for other players like ProRata or TollBit). At last, someone was taking the issue seriously and the idea of controlling or pricing bot access is a step in the right direction.
But it’s not enough:
Micropayments don’t close the gap – Payments of just a few cents per article can’t cover the production costs required for high-quality reporting.
If AI platforms go ad-based (the more likely scenario, 1, 2), publishers lose twice – Advertising keeps users inside the AI platform’s interface, next to the platform’s own ads. Any revenue share will likely be small. Local newspapers have already learned that ads alone are far from sufficient to finance journalism, so there’s little reason to expect better results on someone else’s site.
If AI platforms go subscription-based, the pie is too small – Subscription income would have to be split across hundreds of publishers.
The illusion of reach – For local newspapers, it’s difficult to gain substantial new audiences through AI platforms. Many already have strong market penetration, leaving limited room for significant reach growth.
Loss of audience relationship and insight – When readers consume your content via an AI platform, the relationship shifts to the platform. Content is detached from its brand and context, surfaced in ways the publisher can’t control, and the ability to learn from audience data is lost.
Pay-per-Crawl addresses bots, not people. But every bot query ultimately comes from a human being (at least the ones that are send out for real-time scraping, like ChatGPT-User or Perplexity‑User).
A Third Way: A User-Based Bot Paywall
Between blocking and giving away content for free, there’s another option: When a bot requests content, it should carry the identity or entitlement of the human who triggered it.
So, let’s flip the logic: If someone has a subscription with a newspaper, they should be able to access content directly within AI platforms like Perplexity or ChatGPT. Non-subscribers don’t (or make a higher micro-payment).
Potential benefits:
Added value for subscribers – paying readers get modern, flexible access wherever they are, even in their favorite AI app.
Maintaining the customer relationship – newspaper stay in contact and keep insights into what our readers care about, inside and outside our own platforms.
Future product potential - Why not provide helpful GPTs, Gems, or similar? If monetized in the right way.
How It Could Work — Conceptually
When an AI platform generates an answer that includes protected content, the request should carry a verifiable signal representing the human who triggered it — not just the bot itself.
This signal could take the form of an anonymised access token or entitlement claim, issued when the user links their account with the AI platform once (via OAuth or a similar standard). When the AI requests an article, it forwards this signal to the content provider’s system. The publisher’s system then checks, in real time, whether that user is entitled to the full content. If yes, the article is delivered; if not, the system can respond with a teaser, a summary, or block the request entirely.
This can also be implemented in many forms. It does not have to be a "bot-paywall" at all. It could also be a direct integration with platform providers.
But conceptually it is similar to how programmatic advertising works: ad exchanges pass along consent and targeting data between trusted parties to decide, within milliseconds, what ad to serve. Here, instead of ad targeting data, the exchanged information is the user’s verified right to access the content.
Summary
This “user-based bot paywall” approach is not plug-and-play. It would require technical pilots, alignment on standards, and clear incentives for AI platforms to participate. We’d need to test real-world performance, ensure privacy compliance, and design commercial models that make cooperation worthwhile for all parties.
I believe it’s worth exploring and the sooner we start, the more we can shape the rules rather than adapt to them later. I’d be happy for any input, perspectives, or challenges to refine this idea further.