AI companies have scraped the web. Their next target is data that shows how humans actually work.
What's happening
AI platforms and AI agents are shifting their attention from public web text to richer signals about real human behavior. Buyers now start searches in ChatGPT or Gemini instead of Google, but many sites were built for old crawlers and hide content in JavaScript, lack llms.txt, or block AI bots, so models skip them; visibilitycheck.ai offers a free visibility check. Major players including OpenAI, Meta, Google and others are pushing autonomous agents that can scour the internet to do tasks from booking reservations to managing inboxes, but websites are increasingly blocking or limiting those agents. Reporting highlights a broader move: after large-scale web scraping, AI companies are chasing data that reveals how humans actually work, which raises questions about what sites will allow agents to access and how the web must change to serve agents.
Why it's trending
Because autonomous AI agents and LLM-first search are becoming mainstream, and websites are changing how they expose content, creating friction and new incentives for AI to seek behavior-level data now.
Signal1.5× its usual volume, confirmed across 2 independent source types.
Momentum
Score per dayStory volume
Stories per dayAngles you could write
If you think AI just needs more words, you're wrong, AI firms are now hunting the data that shows how people actually do things, and that should worry product teams and privacy advocates.
+2 more angles for this topic with an account — all it takes is your email.
Original sources5
- Making the internet better for AI agents might make it worse for everyone else
This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here. Until recently, you could divide internet users into just two categories: bots and people. Websites have long allowed bots to perform automated tasks, like capturing the content of web pages, but rules…
MIT Tech ReviewOct 19 - Beware of AI Content for SEOPractical EcommerceOct 19
- Free AI Visibility Check
Buyers now start searches in ChatGPT or Gemini instead of Google. The problem is most sites were made for old crawlers, so they have no llms.txt, hide content in JavaScript, or block AI bots outright. That makes them expensive for AI to read, so the models skip them. What surprised you most about how sites get left out of AI answers? Get a free visibility check at
r/artificialOct 19
+2 more sources for this topic
Create an account to follow the full coverage in the live radar.
More rising in AI & Tech
- A closer look at Verano, AMD's first step toward a Zen-based superchip to rival Nvidia's Vera RubinClimbing3.9Climbing3.9 momentum
- Fireworks AI gets burned twice as customer data exposure widensClimbing3.1Climbing3.1 momentum
- Anthropic launches Claude Meetings to turn workplace discussions into plans and tasksClimbing3.1Climbing3.1 momentum
- Big Walk Used AI In 'The Final Stretch' Of Development, Kicking Off Intense Awards DebateClimbing2.8Climbing2.8 momentum
- Ring cameras go mobile as Amazon debuts AI-powered security drone and robotClimbing2.8Climbing2.8 momentum
- OpenAI rolls out new business profiles, making ChatGPT just a little bit more like GoogleSteady1.0Steady1.0 momentum