Every Image on the Page: URL Canonicalization, Lazy Loading, and srcset in a Site-Wide Crawler
If you've ever tried to "save all images from this page" with a naive loop over document.querySelectorAll('img'), you know how much it misses. Lazy-loaded images aren't in the DOM yet. Responsive images hide their real URLs inside srcset. Background images live in CSS, not in any <img> tag. And even when you do find a URL, the same image can appear under ten different URLs — with different query strings, sizes, and tracking parameters — so your "2,300 images" are really 800 unique ones. A serious site-wide image crawler isn't a querySelector loop. It's a pipeline with three hard problems: finding every image, canonicalizing its URL, and surviving pages that load images on demand. This post walks through each one. Problem 1: Finding images that aren't <img> tags The first pass is the obvious one: img[src], img[srcset], <source> inside <picture>, and video[poster]. But three classes of images need special handling: ...