Cutting a $250-a-day scraping bill to $50

I work on a data product that tracks what 18 websites publish. Once a day it visits each site, logs in where it has to, and turns what it finds into structured records. Customers open those records to see what's live today and what changed since yesterday.

In June that daily run cost about $250. Now it costs about $50. Roughly $37 of that is AI calls and $10 is hosted browsers. The browser provider we used to pay every day gets $0, and paid proxy traffic is down to about 1 GB.

Flip between June and now:

It took four months of small changes to get there. I held all of them to one rule: customers can't lose data because of a cost cut. So every change ran on staging first, got compared against the old path, and shipped behind a flag I could switch off.

How a daily run works

For each site, the pipeline opens a hosted browser and loads every page we watch. AI steps pull structured items out of each listing page. Then the pipeline opens each item's detail page, extracts its fields, and works out what's new, what changed and what disappeared.

We paid for proxy traffic by the gigabyte, for browser time by the session minute on two providers, and for AI calls by the token on every listing page and every detail page. I ended up working on all of them.

Part 1: fewer bytes through the proxy

On these sites, images and video make up most of the bytes, and we pay for every byte that goes through the proxy. Most pages only needed their text, though. The scraper never looked at the images.

So on text-only pages I block image and media requests at the browser protocol level, with CDP Fetch interception. It took more care than I expected. Interception is set per CDP connection, and some pages are read in several passes that reconnect in between. If the block isn't re-applied on each reconnect, it silently stops after the first pass. I also made it fail open: if setup breaks, the page loads normally, because I'd rather pay a bit more than miss a page. Pages where the pipeline needs the images don't get blocked at all.

Those image pages wasted money in a different way. The scraper held the browser session open while a vision step classified each image, and that step doesn't use the browser. In August, 266 of 1,287 of those sessions hit the provider's 300-second cap. When a session died partway, the rest of its items were saved with no classification and no error.

Now it's two passes. The first downloads every image through the session, then the session closes and the second pass classifies from the saved bytes. Sessions are shorter, and the blank rows went away.

Part 2: fewer model calls

Detail pages were where most of the AI money went. Every day the pipeline opened every live item's detail page and ran a full structured extraction on it, even though most items are the same as yesterday. Most of that spend produced records we already had.

Distilling the page first

A detail page carries a lot of noise: navigation, footers, cookie text, legal boilerplate. A distiller step goes first and cuts each page down to a short brief with only the content that matters.

The distiller's instructions took a few rounds. My first version had no lean either way on what to keep, and it threw away real content. A mistake in one direction costs a few extra tokens. A mistake in the other loses a record for good, and nothing reports it. So I tilted the instructions toward keeping things.

A two-stage change gate

Once every page had a brief, I could put a gate in front of the expensive extraction:

The first check is a hash. If the page's raw content hashes the same as the stored baseline, nothing changed and no model gets called. If the bytes differ, say from a rotating widget or a new timestamp, a small cheap comparator reads today's brief next to the stored one. It returns a structured verdict saying whether the meaning changed, with a reason.

I made two choices so the gate can't drift into bad data. Any failure goes straight to full extraction: a database error, a missing file, a distiller failure, a verdict that doesn't parse. The gate only skips when it's sure.

The second choice is about the baseline. When the gate says "not changed", the stored hash moves forward, but the stored brief stays pinned to the one the saved details came from. If each day compared against yesterday's brief, one wrong verdict would quietly become the new reference, and small changes could pile up without anyone seeing them. With the pin, every comparison is against what's actually in the database.

On the last staging run before production, the gate skipped 84 of 150 detail extractions, 73% of the pages it looked at. Each skipped page kept its existing details exactly as they were.

One call instead of many

Listing pages used to be extracted in groups of 10 items. That pattern came from another part of the pipeline, where long lists really did break single calls. Listing pages don't have that problem, and every group re-sent the same system prompt of roughly 1,000 tokens. I swapped the fan-out for one call that takes numbered blocks.

I tested it on three sites, two runs each, with identical captures and the same model:

The single call came out 46% cheaper and didn't drop anything.

The diff engine re-read everything

After extraction, a diff engine works out what's new, what changed and what's gone. It ran once per page, and each run re-read the site's whole history. A site with 8 watched pages read its full history 8 times per run.

Running it once per site looked like an easy fix, until I found out the re-read was holding something up. The table had no unique key on item identity. If an item showed up on two pages, the only thing keeping it from being inserted twice was the second run reading what the first had just written. So I shipped duplicate merging in memory as its own change first, and only then moved the engine to one run per site per phase.

Part 3: one browser provider on our own proxy

The network was the biggest piece. Sites ran on two hosted browser providers, and each one sent traffic through its own paid residential proxy. One handled the logged-out scrapes and the other the logged-in ones, so any site that needed both had us paying for two browser setups a day.

We already had our own proxy in the country we scrape from. The plan was to put every site on one browser provider and send its traffic through that proxy, without losing a page along the way.

Routing per domain

Proxy choice used to be a single switch for the whole deployment. Flipping it would have moved every page at once. I wrote a short RFC and replaced the switch with a routing table in the database, where each domain has a primary route and an optional fallback. The scheduler reads that table to pick which browser runs each leg of a site's scrape. Every browser session also records which route it took and which rule chose it, so I can trace any run afterwards.

Falling back without losing the login

Sometimes our proxy refuses a connection. When that happens, the page reopens on the provider's proxy with the same saved login, so a logged-in leg keeps going. I had to wire this into each kind of page separately. The fiddliest was the raw network capture, which reads data from a page's own API calls: its response listener has to be attached again on every attempt. Without that, the fallback page loads fine and records nothing.

A failure with no error

The case that worried me most never threw an error. On some sites, our proxy got a "verifying you are human" page before the real one. That page loads successfully, so nothing fails. A scraper that reads it too early sees an empty page, and then the diff marks everything stored for that page as gone.

Each domain can now have a wait, so the session holds until the check clears before it reads anything. And a pass that runs after a failed pass can add items but never retire them, so one bad read can't wipe out yesterday's data.

Moving the sites over

For every site, a sweep script opened each watched page on both routes and compared the text side by side. The differences were cookie banners, countdown clocks and live data, about what I expected. When a page needed an extra host, like a CDN, an image server or an anti-bot script, I added it to the proxy's allow-list and ran the sweep again.

Sites moved in three batches over about a week, and the last 10 went on 13 October after a final sweep on the live proxy. 16 of the 18 now run on our proxy. Two sites block it, and so does one subdomain of a third, so those stay on the provider's proxy. That's the 1 GB a day we still pay for.

One result surprised me. On one site, the provider's proxy had been getting content meant for a neighbouring country, and ours got the correct local version. So for that site the cheaper route turned out to be the more accurate one too.

Where it landed

AI calls are about $37 of the $50 now, and they mostly run only where something changed. The team message on the first day under $50 ended with "tears in my eyes!", which is still my favourite review of anything I've shipped.

What I'd tell myself at the start

Before cutting anything, find out what each dollar buys. A lot of this bill paid for repeat work: extracting pages that hadn't changed, re-reading history we'd just read, downloading images nobody looked at.

Any shortcut should fall back to the expensive path when it's unsure. The change gate, the resource blocking and the proxy fallback all do that, and it's the reason I was comfortable shipping them to production.

And look hardest for failures that don't throw. A bot-check page loads without any error, so nothing in the logs points at it.