The most useful scraper is one your agent can call as a tool. Instead of a script you run by hand, you deploy a small MCP server with a scrape tool: the agent asks for a page, the backend fetches it, extracts what matters, checkpoints the result to the durable store, and returns only what is new. The agent decides when to scrape. The backend remembers what it has already seen.
What does a scrape-on-demand tool look like?
About thirty lines. This one fetches a listings page, pulls out listing links, and keeps a dedupe set on the project's durable /persist disk, the same checkpointing pattern from our scraper guides:
import { defineMcp } from "@denkopsai/sdk/mcp";
import { z } from "zod";
const STATE = "/persist/seen.json";
export default defineMcp({
name: "listings-scraper",
version: "1.0.0",
tools: {
scrape_listings: {
description: "Fetch the listings page and return links not seen in earlier runs",
input: z.object({ url: z.string().url() }),
handler: async ({ url }) => {
const seen: string[] = (await Bun.file(STATE).exists()) ? await Bun.file(STATE).json() : [];
const html = await (await fetch(url)).text();
const links = [...html.matchAll(/href="(\/listing\/[^"]+)"/g)].map((m) => m[1]);
const fresh = links.filter((l) => !seen.includes(l));
await Bun.write(STATE, JSON.stringify([...new Set([...seen, ...links])]));
return { found: links.length, new: fresh };
},
},
},
});Deploy it and the tool is live at your project's /mcp endpoint. Any MCP client that supports remote servers can call it: "check the listings page and tell me what's new" becomes one tool call, and the answer is only the fresh links, because the backend carries the memory.
Why checkpoint to the durable store?
Because a scraper without memory re-reports everything, every time. The dedupe set is the difference between "here are 4 new listings" and "here are all 212 listings again", and it is only useful if it survives the process.
On DenkOps every project gets a durable disk at /persist that lives through restarts and redeploys. Write the seen-set there and the tool picks up exactly where it left off, even right after you ship a new version of the scraper itself. For richer state than one JSON file, the SDK's built-in key-value store sits on the same disk with get, set and TTL semantics.
What about outbound traffic?
This is the part most people skip, and the part that matters most when an agent holds the steering wheel. On DenkOps, outbound calls from a project are limited to an allowlist you control. Before the tool can fetch example.com, you add that host in the project's Egress panel, and anything else the process tries to reach is blocked and logged. The dashboard shows recently blocked hosts, so a legitimate new target is one click to allow.
For an agent-driven scraper this is exactly the guardrail you want. The agent chooses when to scrape and which allowlisted page to hit, but it cannot be talked into fetching arbitrary hosts, and a compromised dependency cannot quietly send your data somewhere new. We covered the reasoning in depth in zero-trust egress for agents; a scraper is the pattern where it pays off first, because scrapers are outbound traffic by definition.
Keep the tool honest about scope
Two habits make an agent-driven scraper pleasant to operate:
- Return deltas, not dumps. The agent's context is expensive. Returning only new links keeps tool results small and keeps the agent's follow-up questions sharp.
- Validate the input. The Zod schema above rejects non-URLs before your handler runs, so a confused agent gets a clear validation error instead of a stack trace.
Combined with the allowlist, the result is a tool with a well-defined blast radius: one site, one extraction, one growing checkpoint file.