Why AI Agents Struggle to Use the Web
Anti-bot systems block the model before the research starts.
Hey, it’s Matt. I’m a former Bloomberg News reporter, and you’re reading AI Street, where I report on how Wall Street uses AI.
AI agents are supposed to search the web for us, but the internet wasn’t built for bots.
A couple of months ago, I bought a Mac Mini, downloaded NanoClaw (an operating system for agents), and set one up to gather SEC/CFTC/news/web data for a daily digest. I thought, naively, that connecting my agent to the internet would be straightforward. Instead, I wasted a few hours and spent 8 bucks in API calls trying to pull pages from a government website because it kept blocking my bot.
A lot of the focus with agentic AI is on the model. I think that’s in part because you don’t really think about the underlying infrastructure of the internet other than when you have to confirm your humanity by selecting which photos have a pedestrian crosswalk in them.
The word “bot” has a negative connotation: spam, fake traffic, credential attacks etc. Bots/agents are now performing tasks on behalf of actual humans. The current web was built to block most of that behavior. It’s not really designed to sort out legitimate agents from hostile bots. This, I think, has to change because more bots are coming. CEO Matthew Prince says bot traffic could exceed human traffic by 2027, up from about 20% before the generative AI era.
This may feel a little far afield for AI in finance, but I expect agents to be the new junior analysts, doing the rote work of running fundamental research, diligencing potential acquisitions, monitoring portfolio companies, etc. But firms need to know where their data is coming from and prove that provenance to regulators. That means having: source, timestamp, collection rights, login or proxy use, repeatability and a compliance trail.
We’re still in AI’s early days because we don’t have basic definitions yet: what exactly is an agent? It’ll take a while to build consensus. In February, NIST launched an AI Agent Standards Initiative and asked for input on agent security, identity and authorization, including how agents can operate securely on behalf of users. NIST’s typical timeline is years, not months.
The gap has opened a market for companies building the access, identity and compliance tools agents need before formal standards are settled.
I spent the last few weeks talking to folks building in this space to better understand why agents still struggle with a web built for humans, and what is being built to fix it.
In the rest of the piece for paid subscribers, I go through the emerging stack for an agentic web: access, retrieval and documentation.
For paid subscribers, I map the companies building this agentic web stack. If you’re burning through tokens or trying to figure out why agents still break on basic web tasks, consider becoming a paid subscriber.
