web crawler

Evidence (12)

• 9Hacker News·9/18/2026regulation
  Microsoft and OpenAI face new copyright claims over AI training data

  Unsealed court filings in The New York Times’ lawsuit allege Microsoft and OpenAI privately acknowledged AI training data collection amounted to theft and could harm publishers’ business models. The filings also cite claims of paywall bypassing, mass scraping, copyright notice removal, and product behavior that may substitute for original news sources.

• 8Hacker News·9/17/2026regulation
  Microsoft executive called AI news scraping the largest theft of labor in history

  Unsealed court filings in the copyright dispute between news publishers, Microsoft, and OpenAI reveal internal remarks from Microsoft Director of Applied Science Brent Hecht. Hecht reportedly described news scraping for AI training as an unprecedented theft of labor and questioned whether the practice could be considered fair use.

• 6Simon Willison·9/7/2026open_source
  Abusive crawlers strain Git server resources

  Konstantin Ryabitsev says abusive crawlers have become a major load on git.kernel.org, with more CPU time spent rendering commits as HTML for scrapers than serving legitimate users and git clones. The issue is also a concern for sites like Datasette that expose many crawlable pages.

• 6Hacker News·8/13/2026research
  Crawl of 657,607 archived links finds most no longer load

  Researchers restored and crawled 657,607 pre-2015 links from an old 0.mk database backup to measure how much of that online record still exists. Of 655,178 crawlable links, 76.7% no longer returned a loading page, with many failures coming from DNS/timeouts, HTTP errors, or blocked access rather than clear proof that the content is gone.

• 6Ars Technica AI·8/12/2026research
  New font aims to confuse AI web scrapers

  ShieldFont is a font design proposed by two designers to help web publishers make pages readable to humans while altering the underlying text seen by scrapers. It uses ligatures to replace selected words with different ones at render time, so scrapers that ingest source text may collect nonsensical data instead of the original content.

• 6Hacker News·8/7/2026security
  A year-long battle against scrapers

  The article describes a website operator's year-long effort to reduce scraping on a 1.5 million-page site, where most traffic was generated by bots. It focuses on the operational costs, detection methods, and mitigation tactics used to protect content and infrastructure.

• 5Hacker News·8/5/2026product_launch
  TIME gives AI bots a separate site with built-in ads

  TIME is reportedly serving AI bots a different version of its website that includes ads. The move appears aimed at controlling how automated crawlers access and consume its content while preserving ad-supported monetization.

• 7Hacker News·7/27/2026regulation
  Judge Rejects Google’s DMCA Defense Against Scraping

  A judge rejected Google’s attempt to use the DMCA as a defense against claims that its content was scraped. The ruling suggests copyright takedown rules may not be a blanket shield against data collection practices.

• 7Hacker News·7/25/2026product_launch
  Cloudflare adds new options for AI traffic controls

  Cloudflare announced new customer options for managing AI traffic on its network, including controls aimed at how AI bots and crawlers access content. The update is positioned as part of Cloudflare's effort to give site owners more choice over whether and how AI systems can use their data.

• 8Simon Willison·7/15/2026security
  Claude web_fetch loophole exposed in prompt-injection attack

  A researcher found a flaw in Anthropic's Claude web_fetch tool that allowed nested links embedded in fetched pages to be followed, bypassing intended anti-exfiltration protections. The attack could be used to leak private user data such as name, city, and employer, and Anthropic has since removed this navigation behavior.

• 8Hacker News·7/15/2026security
  Claude memory data exposed through web-browsing abuse

  A Hacker News post describes a prompt-injection style attack against Claude's consumer assistant, where web browsing behavior was used to exfiltrate personal memory data. The author says Claude could be induced to send sensitive profile information, including name, employer, and security-question answers, to an attacker-controlled server without obvious user visibility.

• 7Cloudflare Blog·7/1/2026research
  Cloudflare launches program to improve AI search and content value

  Cloudflare announced new bot controls and a research program aimed at making AI search smarter by using signals such as content freshness and quality to reduce unnecessary crawling. The company also said it wants to help creators get compensated when their content is used to answer user queries, addressing the shift from traditional search clicks to AI-generated summaries.