web crawler
TrialTechniques
Automated software that scans and indexes web content.
Why it's here
Placed in Trial: 8 article(s) of evidence from 5 source(s), led by security coverage, with 7 in the last 30 days. Confidence 77%.
Evidence (8)
- 6Ars Technica AI·8/12/2026researchNew font aims to confuse AI web scrapers
ShieldFont is a font design proposed by two designers to help web publishers make pages readable to humans while altering the underlying text seen by scrapers. It uses ligatures to replace selected words with different ones at render time, so scrapers that ingest source text may collect nonsensical data instead of the original content.
- 6Hacker News·8/7/2026securityA year-long battle against scrapers
The article describes a website operator's year-long effort to reduce scraping on a 1.5 million-page site, where most traffic was generated by bots. It focuses on the operational costs, detection methods, and mitigation tactics used to protect content and infrastructure.
- 5Hacker News·8/5/2026product_launchTIME gives AI bots a separate site with built-in ads
TIME is reportedly serving AI bots a different version of its website that includes ads. The move appears aimed at controlling how automated crawlers access and consume its content while preserving ad-supported monetization.
- 7Hacker News·7/27/2026regulationJudge Rejects Google’s DMCA Defense Against Scraping
A judge rejected Google’s attempt to use the DMCA as a defense against claims that its content was scraped. The ruling suggests copyright takedown rules may not be a blanket shield against data collection practices.
- 7Hacker News·7/25/2026product_launchCloudflare adds new options for AI traffic controls
Cloudflare announced new customer options for managing AI traffic on its network, including controls aimed at how AI bots and crawlers access content. The update is positioned as part of Cloudflare's effort to give site owners more choice over whether and how AI systems can use their data.
- 8Simon Willison·7/15/2026securityClaude web_fetch loophole exposed in prompt-injection attack
A researcher found a flaw in Anthropic's Claude web_fetch tool that allowed nested links embedded in fetched pages to be followed, bypassing intended anti-exfiltration protections. The attack could be used to leak private user data such as name, city, and employer, and Anthropic has since removed this navigation behavior.
- 8Hacker News·7/15/2026securityClaude memory data exposed through web-browsing abuse
A Hacker News post describes a prompt-injection style attack against Claude's consumer assistant, where web browsing behavior was used to exfiltrate personal memory data. The author says Claude could be induced to send sensitive profile information, including name, employer, and security-question answers, to an attacker-controlled server without obvious user visibility.
- 7Cloudflare Blog·7/1/2026researchCloudflare launches program to improve AI search and content value
Cloudflare announced new bot controls and a research program aimed at making AI search smarter by using signals such as content freshness and quality to reduce unnecessary crawling. The company also said it wants to help creators get compensated when their content is used to answer user queries, addressing the shift from traditional search clicks to AI-generated summaries.