An async scraper built on httpx can fetch dozens of pages concurrently, but that speed is wasted if every deploy wipes its progress file. DenkOps mounts a durable /persist disk on every slot, so a FastAPI-triggered crawl can write its visited-URL index there and pick up mid-crawl after a restart instead of starting over.
import httpx, json, os
from fastapi import FastAPI
app = FastAPI()
STATE = "/persist/crawl_state.json"
@app.post("/scrape")
async def scrape():
seen = json.load(open(STATE)) if os.path.exists(STATE) else []
async with httpx.AsyncClient(timeout=10) as client:
r = await client.get("https://example.com/listings")
seen.append(r.url.path)
json.dump(seen, open(STATE, "w"))
return {"total_seen": len(seen)}Deploy it: install the plugin, say "deploy on DenkOps", and get a live SSL URL.
Start on DenkOps →Yes, write the state file to /persist. It's a durable disk that survives restarts and redeploys, so an httpx crawl resumes instead of restarting cold.
Yes, through the zero-trust egress gateway, which permits and logs outbound HTTPS calls to any target site your scraper needs to reach.