Every Image on the Page: URL Canonicalization, Lazy Loading, and srcset in a Site-Wide Crawler
If you've ever tried to "save all images from this page" with a naive loop over document.querySelectorAll('img'), you know how much it misses. Lazy-loaded images aren't in the DOM yet. Responsive images hide their real URLs inside srcset. Background images live in CSS, not in any <img> tag. And even when you do find a URL, the same image can appear under ten different URLs — with different query strings, sizes, and tracking parameters — so your "2,300 images" are really 800 unique ones.
A serious site-wide image crawler isn't a querySelector loop. It's a pipeline with three hard problems: finding every image, canonicalizing its URL, and surviving pages that load images on demand. This post walks through each one.
Problem 1: Finding images that aren't <img> tags
The first pass is the obvious one: img[src], img[srcset], <source> inside <picture>, and video[poster]. But three classes of images need special handling:
Lazy loading. Modern pages ship loading="lazy" plus a data-src / data-srcset convention: the src holds a placeholder (a 1x1 GIF or a blur-up thumbnail) and the real URL appears only in the data attribute, swapped in by JS when the element scrolls near the viewport. A crawler that reads only src will happily save 2,000 placeholder pixels. The fix is to union all candidate attributes: src, data-src, data-srcset, data-original, data-lazy-src, and so on — every site picks a different convention, so the matcher has to be generous.
IntersectionObserver for the rest. Some sites don't even put the real URL in the DOM until the image is about to enter the viewport. A thorough crawler scrolls the page (or uses IntersectionObserver with a large rootMargin) to trigger those load handlers, then re-scans the DOM for newly populated attributes.
CSS backgrounds. background-image: url(...) on elements, and ::before / ::after pseudo-elements, hold plenty of images. The crawler walks computed styles with getComputedStyle(el).backgroundImage and parses the url(...) tokens — including the -webkit-image-set / image-set() variants with their own resolution candidates.
Problem 2: URL canonicalization — one image, one key
The same physical file frequently appears as:
https://cdn.example.com/img/hero.jpg
https://cdn.example.com/img/hero.jpg?w=800&q=75
https://cdn.example.com/img/hero.jpg?q=75&w=800
https://cdn.example.com/img/hero.jpg#gallery-3
To deduplicate, every URL is normalized before it becomes an object-store key:
1. Scheme and host lowercased — HTTP:// → http://.
2. Default ports stripped — :443 on https, :80 on http.
3. Fragment removed — #... never identifies a resource on the server.
4. Query keys sorted and values preserved — ?q=75&w=800 and ?w=800&q=75 become the same string. (Sort keys, don't drop them: w and q genuinely change the bytes served.)
5. Known tracking parameters dropped — utm_source, utm_medium, fbclid, gclid, spm (Alibaba/Chinese CDNs append spm everywhere). These are noise added by marketing tooling, not by the image pipeline.
6. Path dot-segments resolved — /a/./b → /a/b, /a/x/../b → /a/b.
Only after this pipeline do two URLs that "look different" collapse into the same record. The canonical URL becomes the IndexedDB key, and the dedup becomes a single get() instead of a linear scan.
Problem 3: srcset — one image, many candidates
srcset is a list of URL + widthOrDensityDescriptor pairs:
<img src="hero.jpg" srcset="hero-480.jpg 480w, hero-800.jpg 800w, hero-1600.jpg 1600w" sizes="(max-width: 600px) 480px, 100vw">
A crawler has to decide: save every candidate (you might want the 1600w master for archival) or just the largest (the practical "original"). OmniPic-style tools save the largest candidate as the primary and record the rest as alternates. The parser must also handle the subtle cases: comma-separated candidates, x descriptors (2x, 3x), and whitespace-splitting that breaks on URLs containing commas — which is why you parse srcset with a real tokenizer, not split(',').
How the pieces fit together
A well-architected extension keeps the crawler in a content script (it has DOM access) and the heavy lifting in the service worker:
1. Content script enumerates candidates (tags + attributes + computed styles), normalizes each URL, and sends a compact manifest over runtime.sendMessage.
2. Service worker persists the manifest to IndexedDB, then processes images in chunks — fetch → createImageBitmap → pHash → classify — updating progress after each chunk so the job survives service-worker suspension.
3. Deduplication runs against the canonicalized URL set plus a perceptual-hash pass, so exact duplicates (same bytes, different URLs) and near-duplicates (resized copies) both collapse.
The manifest must be serializable JSON: no DOM nodes cross the message boundary, only { canonicalUrl, sourceUrl, srcsetCandidates[], width, height } records.
Why this is harder than it looks
The trap isn't any single step — it's the long tail. One site uses data-actualsrc; another serves images only after two scroll events; a third appends ?spm=xxx to every asset; a fourth uses AVIF in srcset but JPEG in src. A crawler that handles 95% of sites crashes to 60% on the long tail, and every miss is a designer who can't find the image they needed.
Getting the discovery, canonicalization, and dedup pipeline right — plus the scroll-triggered rescan, the srcset tokenizer, and the storage layer underneath — is exactly the kind of engineering that takes weeks to get solid. That's why OmniPic AI Studio Pro ships it as a finished pipeline: one-click site-wide inspection, automatic URL normalization, lazy-load triggering, srcset handling, CSS-background capture, pHash deduplication, CDN original-URL recovery (Aliyun OSS / Tencent COS / Qiniu), on-device AI classification and reverse search, and export to Eagle/Billfish or a streaming ZIP of 2,000+ images — all processed 100% locally, nothing uploaded.
Available on Chrome, Edge, and Firefox:
Get OmniPic AI Studio Pro: https://chromewebstore.google.com/detail/omnipic-studio-pro/cokoeaaaglgcljahaffjeijomhmkljlc
评论
发表评论