Machine Readable Content: The UAE Playbook for the Web AI Agents Actually Read

Machine readable content is the version of your website that AI agents actually read, and for most UAE brands it does not exist yet. On 31 July 2026, Time began selling advertising written for machines rather than people, placing sponsored FAQ blocks inside stripped-down markdown copies of its articles that AI crawlers fetch instead of the designed web page. Ally Bank and the Project Management Institute bought the first placements. The publisher’s chief operating officer says Time sees more bot traffic than human traffic on most days.

That is not a publishing curiosity. Cloudflare measured automated requests at 57.5% of all HTML web traffic in June 2026, against 42.5% from humans. The web now has two audiences, and only one of them has eyes.

Here is the short version for UAE businesses. Your site is being read far more often by language models than by customers, and those models are parsing a bloated HTML page built for browsers. Cleaning up what machines receive, structuring answers so they can be lifted verbatim, and deciding which crawlers you allow are now part of content strategy, not a developer’s side quest. This guide covers what Time built, what the crawl data shows, why zero of eleven industries passed a recent AI readability benchmark, what llms.txt honestly does, and five moves worth making this quarter.

Bots now outnumber humans on the open web

Cloudflare chief executive Matthew Prince published the figure on 3 June 2026: automated requests account for 57.5% of all HTML web traffic, leaving humans at 42.5%. Cloudflare Radar puts AI crawlers at 20.3% of verified bot traffic in May 2026, with AI search bots adding a further 6.5%, so roughly 26.7% of verified bot activity was AI related.

The uncomfortable part is what those bots are doing. Search purpose crawling, the kind that can end in a citation and a click, was under 10% of AI crawler requests in May 2026. The other 90% is extraction. Content taken to train or ground a model, with little or nothing sent back.

Cloudflare’s crawl-to-refer ratio makes the imbalance concrete. It counts how many pages a bot pulls from your site for every visitor its parent platform sends you.

Ratios from Cloudflare Radar data, compiled in the GEO Data Report 2026. Higher means the crawler takes more and gives back less.

Read that table twice before you decide to block everything. A 10,300 to 1 ratio looks like theft when you measure clicks. It looks different when you measure influence, because the model that read those 10,300 pages is the one answering your customer’s question about which supplier to shortlist.

What Time actually built, and who bought it

Time converted its web pages into markdown, a plain text format with the design, scripts and styling stripped out. It then blocked AI bots by default and built a whitelist. Approved crawlers get the markdown version, not the browser page. Digiday reported the change in July 2026, and on 31 July confirmed the next step: Time is selling advertising inside those markdown files.

The ads are not banners. Working with the ad tech firm Mobian, a brand brief becomes a block of FAQ style sponsored content, labelled as sponsored, and dropped into the markdown page. One per page. Time charges a premium on the argument that bot impressions against authoritative reporting are scarce. Mobian measures the result by querying AI engines with those same FAQ questions and tracking visibility, favourability and accuracy over time.

“Maybe it’s more important to influence the agent than even the human. When you influence ChatGPT, you’re influencing potentially all of ChatGPT.” Jonah Goodhart, co-founder and chief executive, Mobian.

Why markdown, and not just a faster site

TollBit, which runs this conversion for publishers, says a full HTML page can take over a minute to process while the markdown equivalent is fetched in about 0.25 seconds, with roughly 90% fewer tokens consumed. Fewer tokens spent on div soup means more of the model’s attention on your actual argument.

“AI can comprehend more of your article because they’re not spending money parsing out other HTML.” Toshit Panigrahi, chief executive, TollBit.

Time is not alone, but the rest are moving carefully. The Economist is exposing only marketing copy and B2B sales material to agents, because a subscription business cannot hand its journalism to a model for free. Le Monde is testing whether it can tell when an agent is acting for a paying subscriber. At least one large publisher is trialling WebMCP, the agent interface co-developed by Google and Microsoft. Mobian estimates only about 15% of brands currently publish markdown pages at all.

The obvious risk

Serving different content to machines than to people is, structurally, cloaking. Search engines have penalised that for two decades. The defence here is parity and disclosure: the markdown must say the same things as the page, and sponsored blocks must be labelled. If you go down this road, keep both versions honest. A model that later learns to discount undeclared sponsored FAQs will discount yours retroactively.

The readability gap: zero of eleven industries passed

Publishers are ahead because bots are already their biggest audience. Everyone else is behind, and there is now a number for it. Between 22 May and 8 July 2026, a benchmark study scored 399 websites across eleven industries on how readable they are to AI systems. Not one industry averaged above the 60 point “fair” threshold. The weighted mean was 41.3 out of 100.

Scores from a 399 site AI readability benchmark run 22 May to 8 July 2026. The eleventh category, a catch-all “other” group, scored 33.2. Bands: 85+ excellent, 70 to 84 good, 60 to 69 fair, below 60 needs work.

Look at the bottom of that list. Real estate at 34.3 and hospitality at 33.1 are two of the largest marketing categories in Dubai and Abu Dhabi, and they are the least legible to the systems now answering “best serviced apartments in Business Bay” or “which Dubai agency handles brand and performance together”. That is not a small gap. It is a queue nobody has joined yet.

What llms.txt does, and what it does not

Every AI visibility checklist doing the rounds tells you to publish an llms.txt file: a markdown index at the root of your domain pointing models at your important pages. Adoption has climbed. An SE Ranking study of 300,000 domains found 10.13% now have one. Shopify sits at 78.1%, though that is not developer enthusiasm, it is Shopify pushing the file to every store by default in spring 2026. WordPress, at 8.7%, is the more honest benchmark.

Now the part the checklists skip. In one analysis of more than 500 million AI bot visits over a 90 day window, just 408 requests fetched llms.txt at all. Anthropic and OpenAI reference the format in their developer documentation, so it has real value for agents working with your product docs. As a lever on AI search citations, the evidence is close to nil.

So ship it. It takes an afternoon and costs nothing. Then stop congratulating yourself, because the crawlers are still fetching your ordinary pages, and that is where the work is.

Machine readable content is now a UAE competitive advantage

Three things make machine readable content a sharper play in the UAE than in most markets. The first is demand. 58% of UAE working age professionals use AI tools heavily, second only to India globally, which means an unusually large share of local research, shortlisting and vendor comparison already happens inside a chat window rather than a results page.

The second is money. UAE digital ad spend is forecast to grow 15.2% in 2026 to reach $2.64 billion, and to hit roughly $4.30 billion by 2029. Spend is rising into channels where an AI layer increasingly sits between the ad and the answer. Being the source a model quotes is cheaper than outbidding everyone for the click that follows.

The third is that the competition has not started. Zero of eleven industries passed the readability benchmark. Almost no UAE B2B brand appears in AI answers for its own category. When a whole market is invisible, legibility is the differentiator, and it is a build, not a bid.

The payoff shows up in the numbers where brands have done the work. One Dubai brand lifted its AI referred traffic share from 9% to 24% within three months of a focused effort. Brands cited inside an AI Overview earned 120% more organic clicks per impression than uncited brands on the same results page, and 58% of marketers report that AI referred visitors convert better than standard organic traffic. Smaller volume, better intent.

Want to know what the bots see when they read your site? Talk to the Rothian Digital team and we will run the audit against your top pages.

Five moves for UAE brands this quarter

1. Find out what the crawlers are actually taking

Pull your server logs and segment by user agent: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot, Applebot-Extended. Count requests per bot, then count referrals from each platform in analytics. You now have your own crawl-to-refer ratio. Most UAE sites have never looked, and the answer usually surprises the marketing team.

2. Fix the pages before you build a second layer

A markdown mirror is the advanced move. The basic one is making the HTML you already serve legible: content in the initial response rather than injected by JavaScript, one claim per paragraph, headings that state the answer instead of teasing it, tables for anything comparative, and dates on everything. Roughly half of AI cited content is under thirteen weeks old, so an undated page is a page a model cannot trust.

3. Write in extractable units

Models lift passages, not pages. Give them clean ones. Lead every section with a direct answer, mark up FAQ blocks with schema, keep your key statistics in a sentence that stands alone without the paragraph around it, and name your sources inline. This is the same discipline behind our SEO and content strategy work, applied to a reader that never scrolls.

4. Set a deliberate bot policy

Blanket blocking in robots.txt is a decision with revenue attached, so make it consciously. A reasonable default for most UAE brands: allow the search and answer bots that cite, be selective about pure training crawlers, and revisit quarterly. If your content is your product, price access rather than giving it away. If your content is marketing for something else you sell, being read is the whole point.

5. Measure citations, not only clicks

Track how often your brand appears in ChatGPT, Gemini, Claude, Perplexity and Google AI Mode answers for your twenty commercial queries, and whether the mention is accurate and favourable. That is the metric Mobian sells against, and it is the one your board will ask about within a year. Clicks will keep falling. Influence is what you are buying.

The bottom line

Time did something small and revealing: it looked at who was reading, found it was mostly machines, built a version of the site for them, and then sold advertising in it. Every part of that sequence is available to a UAE brand except the last one, and the first three are where the value is.

You do not need a markdown mirror or an agent ad budget this quarter. You need to know what the bots take from your site, publish pages a model can quote without reformatting, and stop assuming the only reader worth designing for is the one with a browser. The brands that get this right will be the answer. Everyone else will be a link nobody clicks.

Frequently asked questions

---

Sources

1. Cloudflare, “The crawl before the fall of referrals: understanding AI’s impact on content providers”
2. Digiday, “Time has started serving ads to AI agents”, 31 July 2026
3. Digiday, “How Time and others are rebuilding parts of the web for AI agents”
4. GEO Data Report 2026, crawl-to-refer ratios for AI crawlers and LLM bots
5. AI Readability Benchmark 2026, 399 sites across eleven industries
6. Casey Burridge, “Does anyone actually have an llms.txt? I checked millions of websites”
7. United Arab Emirates Digital Ad Spend Business Report 2026
8. Mobian, Agent Ads