Vercel's AI SDK Ships Web Fetch Tool That Works Across any AI Model

retained text

Vercel's AI SDK Ships Web Fetch Tool That Works Across any AI Model
Vercel's AI SDK adds a first-party web fetch tool that returns pages as markdown, works with any function-calling model, and ships with domain allowlists.
- Vercel AI SDK adds a Web Fetch Tool that works with any function-calling model.
- Pages are returned as markdown using defuddle, cutting token usage versus raw HTML.
- Install via pnpm add @ai-sdk/tool-web-fetch, requires Node.js 22.13 or newer.
- Security controls: allowedDomains, blockedDomains, requireUrlInUserMessage, and DNS rebinding protection on Node.
- Supports HTML, text, JSON, XML, CSV; no PDFs, binaries, or JavaScript-rendered pages.
- Built-in pagination via offset with short-lived per-instance cache; default 50k char limit, 10 MiB cap.
Vercel’s AI SDK team has released a first-party tool that lets agents fetch publicly reachable web pages with any AI SDK model that supports tool calling. The Web Fetch docs describe an integration that accepts a URL, retrieves the page, removes common boilerplate, and returns content the model can process.
For server-rendered, text-based pages, the package can replace a custom scraping service, headless browser, or HTML-to-text pipeline. It runs as a standard AI SDK tool, so the same integration continues to work when an application changes model providers.
A small integration surface
Install the separate package, register the tool with generateText, streamText, or ToolLoopAgent, and set a step limit to prevent an unbounded tool loop:
npm install @ai-sdk/tool-web-fetchimport {
  experimental_createWebFetchTool as createWebFetchTool,
} from '@ai-sdk/tool-web-fetch';
import { generateText, isStepCount } from 'ai';
const result = await generateText({
  model,
  tools: {
    webFetch: createWebFetchTool(),
  },
  stopWhen: isStepCount(5),
  prompt: 'Summarize https://ai-sdk.dev/docs/introduction.',
});
console.log(result.text);
The exported factory currently carries an experimental_ prefix, so teams should treat its API as subject to change. The isStepCount(5) condition caps the number of model and tool steps during this run.
Cleaner input, fewer tokens
The tool requests Markdown by sending Accept: text/markdown. When a server returns HTML, the package uses Defuddle to extract the main content and remove navigation, footers, scripts, and other page furniture.
Removing HTML tags, inline styles, scripts, and tracking markup reduces the tokens consumed by each page. The savings grow when an agent fetches several URLs within one tool loop.
The model calls the tool with { url, offset? }. A successful response includes the requested URL, final URL after redirects, content type, title, extracted content, and pagination metadata. Failures return structured error information that the model can use to revise the URL or choose another source.
Long pages stay bounded
Each call returns up to the configured maxLength, which defaults to 50,000 characters, and applies a 10 MiB response ceiling. When a document exceeds the character limit, the response provides an offset that the model can pass into a subsequent call.
Processed documents are cached for a few minutes within each tool instance, allowing offset requests to continue reading without downloading and converting the page again. The cache belongs to that instance rather than serving as a shared application cache.
Guardrails for hostile content
Fetched text must be treated as untrusted input because a page can contain instructions intended to override the agent’s task. A compromised model could also place sensitive data in a query string and request an attacker-controlled URL.
- allowedDomains limits requests to specified domains and their subdomains.
- blockedDomains denies requests to listed domains.
- requireUrlInUserMessage permits only URLs found in user messages. URLs introduced by system prompts, assistant messages, fetched pages, or tool results remain ineligible.
- AI SDK tool approval can require a person to authorize each fetch.
Domain restrictions are checked again across redirects. The transport also sends GET requests without cookies or credentials, rejects URLs containing embedded credentials, and blocks private or reserved addresses such as localhost and cloud metadata endpoints.
On Node.js, the tool validates resolved IP addresses when the connection opens. This check reduces DNS rebinding risk, where a hostname initially resolves to a public address and later resolves to an internal service after validation.
Choose controls by workflow
- User-submitted URL summaries: Enable requireUrlInUserMessage, add an allowlist when the expected domains are known, and consider approval for sensitive environments.
- Citation-following research agents: Disable requireUrlInUserMessage only when the agent must follow links discovered in search results or fetched pages. Constrain egress with domain rules, a proxy, or network policy.
- Internal applications: Keep private address blocking in place and avoid passing session cookies or service credentials through a custom transport.
Where fetching stops
- No web discovery: The tool retrieves a known URL. Agents that need to find sources require a separate search tool.
- No JavaScript execution: Client-rendered applications may return an empty shell or incomplete content because the package does not run a browser.
- No authenticated browsing: Requests omit cookies and credentials, so pages behind a sign-in flow are generally unavailable.
- No binary extraction: PDF-heavy workflows still need a document parser or dedicated ingestion pipeline.
- Imperfect content detection: Automatic extraction can select the wrong section on unusual layouts. Known page structures can use an option such as defuddleOptions.contentSelector: 'article'.
Portability without provider lock-in
Provider-native browsing features vary in their search, retrieval, citation, and execution behavior. The AI SDK implementation focuses on fetching known URLs inside the application’s process, giving developers direct control over domain policy, logging, caching, and network routing.
A custom fetch implementation can route traffic through a proxy or existing egress layer. Normal model, bandwidth, and compute costs still apply, while the application avoids depending on a provider-specific browsing interface.
The package also complements the Bash Tool docs, which cover command execution in environments such as Vercel Sandbox or the in-memory just-bash runtime. One tool retrieves web content; the other runs code against files and command-line programs.
A production checklist
- Pin package versions while the factory remains experimental.
- Set stopWhen and choose a suitable maxLength.
- Enable domain restrictions and requireUrlInUserMessage where the workflow allows them.
- Log requested URLs, final redirect URLs, content types, tool errors, and approval decisions.
- Test redirects, oversized documents, client-rendered pages, malformed content, and prompt injection attempts.
- Provide separate tools for search, PDFs, authenticated pages, or browser-rendered applications.
For existing AI SDK applications, integration requires one package and one tool registration. Production deployment also requires explicit egress rules, bounded tool loops, and testing against the page types the agent will encounter.

document fetched 2026-10-24 15:22