Why Image Scrapers Silently Fail: CORS, Tainted Canvases, and Blob URLs
You write a small script that walks a page, grabs every img, fetches the src, and writes the bytes to disk. It works on nine sites out of ten. On the tenth you get a folder full of 0-byte files, or thirty SecurityError: The operation is insecure lines in the console, or images that appear in the DOM but 404 the moment you try to download them.
Almost nobody debugs this correctly, because all three failures look the same from the outside — "the download didn't work" — while they come from three completely different layers. Here is how I learned to tell them apart, and why each one breaks.
Failure one: the bytes never leave the server
The first layer is plain HTTP. A request for https://cdn.example.com/hero.jpg returns 403 Forbidden when it arrives without a Referer header, because the host runs hotlink protection and only serves images to pages it recognises. Browser-tab requests always carry a referrer, so the image looks perfectly fine in the page — and a script that tries to re-request the same URL from a bare fetch context gets rejected.
The diagnostic is boring and reliable: compare the same URL with and without a referrer.
curl -sI "https://cdn.example.com/hero.jpg" | head -3
curl -sI -H "Referer: https://example.com/gallery" "https://cdn.example.com/hero.jpg" | head -3
If the second returns 200 and the first 403, you have a referrer problem, not a parsing problem. Some CDNs also key on User-Agent, Accept, or a signed query parameter with an expiry — which is why an image that downloaded yesterday returns 403 today.
Failure two: CORS gives you an opaque response you cannot read
This is the one that wastes the most time, because the request succeeds. fetch() resolves, response.ok is false, response.status is 0, response.headers is empty, and response.body is null. That is an opaque response: the browser performed the request, but the Same-Origin Policy forbids your JavaScript from reading a single byte of it, because the target server did not send Access-Control-Allow-Origin.
The trap is mode: "no-cors". People add it to silence the error, then wonder why the file they wrote is empty. no-cors does not grant access — it explicitly asks for the unreadable version:
- fetch(url) with no CORS headers: TypeError, not a single readable byte.
- fetch(url, {mode: "no-cors"}): an opaque response, status 0, body null, still nothing you can write.
- fetch(url) where the server sends Access-Control-Allow-Origin: a readable ArrayBuffer and a real file.
- background fetch with host permission: a readable ArrayBuffer, no server opt-in required.
That last row is the whole reason browser extensions exist as a category. An extension that declares host_permissions for the target origin can fetch cross-origin bytes from its background context without the server ever opting in, because the user already granted that access at install time. In Manifest V3 this only holds in the service worker or extension pages — a fetch() issued from a content script is still subject to the page's CORS rules. So the working architecture is: content script enumerates candidates and posts URLs to the background, background fetches and returns ArrayBuffers, and the content script stays a pure observer.
Failure three: the canvas is tainted and refuses to encode
Suppose you want to normalise formats — decode the image, re-encode as PNG, resize. You draw it into an OffscreenCanvas and call convertToBlob(), and get SecurityError: Tainted canvas may not be exported. Drawing a cross-origin image without CORS approval marks the canvas's origin-clean flag as false, and from that moment toBlob(), toDataURL() and getImageData() are all forbidden. The image can be displayed; it cannot be exported.
Two ways out. Either set crossOrigin = "anonymous" on the Image before assigning src (and accept that it fails if the server sends no CORS headers), or fetch the bytes yourself and decode them locally from a Blob — an image constructed from a same-origin object URL or from an ArrayBuffer you already own does not taint anything. The second path is what actually works at scale: fetch → Blob → createImageBitmap → draw → encode, with no network-dependent CORS negotiation inside the canvas at all.
And then there is failure zero, hiding behind all of it: blob: URLs. A blob: URL is not a location, it is a handle into the object URL store of the document that created it. It dies when that document is destroyed or when someone calls revokeObjectURL, it cannot be resolved from another tab or another agent cluster, and the resource it points at may be megabytes of in-memory ArrayBuffer with no HTTP representation at all. A scraper that treats img.src as ground truth collects a pile of dead handles. The fix is to resolve blob: and data: sources into real bytes first, and to keep those bytes rather than the string.
What a correct pipeline actually looks like
Once you know the failure taxonomy, the design writes itself:
1. For each candidate, record why it might fail — cross-origin, referrer-gated, blob-backed, already a data URL.
2. Try the cheapest path first: background fetch with host permission.
3. On 403, retry with a referrer rewritten to the page that hosted the image; on 401, drop it and record the reason instead of silently skipping.
4. If the response is opaque (status === 0), do not write a file — mark it "unreadable, CORS" so the failure list is honest.
5. Resolve blob:/data: to bytes, never to URLs.
6. Only then decode, hash and deduplicate.
The last step matters more than it sounds. The same asset usually appears four times on one page — a srcset candidate, a blob: preview, a <link rel="preload"> and the actual CDN original. Deduplicating by perceptual hash collapses those into one entry; deduplicating by URL does not.
Where OmniPic AI Studio Pro fits
I built all of the above into OmniPic AI Studio Pro, a browser extension that inspects a whole site instead of one page at a time. It runs the diagnosis layer for you: host-permission fetches in the background, referrer fallback for hotlink-protected CDNs, blob and data URL resolution, local decode and re-encode so the canvas is never tainted, pHash deduplication across the collection, and an explicit failure report that tells you which of the three layers refused — instead of a folder of 0-byte files.
Everything happens on your machine: no image is uploaded anywhere, no cloud account, no telemetry. Also included: CDN original restoration for OSS/COS/Qiniu thumbnails, on-device MobileNet embeddings for similar-image search, one-click export to Eagle or Billfish, and streamed ZIP packing for 2000+ files without stalling.
Available for Chrome, Edge and Firefox:
https://chromewebstore.google.com/detail/omnipic-studio-pro/cokoeaaaglgcljahaffjeijomhmkljlc
If you have ever debugged an opaque response at 2 a.m., you know exactly how much of this you do not want to rebuild.
OmniPic AI Studio Pro — the pro-grade engine behind a site-wide image inspection.
评论
发表评论