AI companies have scraped the web. Their next target is data that shows how humans actually work.
What's happening
AI platforms and AI agents are shifting their attention from public web text to richer signals about real human behavior. Buyers now start searches in ChatGPT or Gemini instead of Google, but many sites were built for old crawlers and hide content in JavaScript, lack llms.txt, or block AI bots, so models skip them; visibilitycheck.ai offers a free visibility check. Major players including OpenAI, Meta, Google and others are pushing autonomous agents that can scour the internet to do tasks from booking reservations to managing inboxes, but websites are increasingly blocking or limiting those agents. Reporting highlights a broader move: after large-scale web scraping, AI companies are chasing data that reveals how humans actually work, which raises questions about what sites will allow agents to access and how the web must change to serve agents.
Why it's trending
Because autonomous AI agents and LLM-first search are becoming mainstream, and websites are changing how they expose content, creating friction and new incentives for AI to seek behavior-level data now.
Signal2.0× its usual volume, confirmed across 2 independent source types.
Story volume
Stories per dayAngles you could write
If you think AI just needs more words, you're wrong, AI firms are now hunting the data that shows how people actually do things, and that should worry product teams and privacy advocates.
+2 more angles for this topic with an account — all it takes is your email.
Original sources5
- Making the internet better for AI agents might make it worse for everyone else
This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here. Until recently, you could divide internet users into just two categories: bots and people. Websites have long allowed bots to perform automated tasks, like capturing the content of web pages, but rules…
MIT Tech ReviewOct 19 - Beware of AI Content for SEOPractical EcommerceOct 19
- Free AI Visibility Check
Buyers now start searches in ChatGPT or Gemini instead of Google. The problem is most sites were made for old crawlers, so they have no llms.txt, hide content in JavaScript, or block AI bots outright. That makes them expensive for AI to read, so the models skip them. What surprised you most about how sites get left out of AI answers? Get a free visibility check at
r/artificialOct 19
+2 more sources for this topic
Create an account to follow the full coverage in the live radar.
More rising in AI & Tech
- A hacker broke into John Deere and SharkNinja with an army of AI agentsSurging6.0Surging6.0 momentum
- Character.AI Just Filed For Bankruptcy, And Former Foe Disney May Buy It For PartsSurging6.0Surging6.0 momentum
- Report: FTC probes Anthropic, OpenAI after Trump dismissed AI fears as “hoax”Climbing3.9Climbing3.9 momentum
- Maxwell gives sub-optimal code a runtime glow-upSurging4.0Surging4.0 momentum
- ASML's strong orders ease fears AI spending boom is running out of steamSteady1.3Steady1.3 momentum
- Meta and Apple team up to make Muse AI models run better on MacsSteady0.9Steady0.9 momentum