博文

Why Your Image Crawler Collects Blur Placeholders (and Misses the Images Hiding in CSS)

图片
You run a crawl across a product gallery, get 2,000 files, and open the folder. A third of them are 40-pixel smears. Another chunk are solid rectangles. And yet the page you crawled clearly had a full-bleed hero image that never showed up in the results at all. Both halves of that outcome come from the same mistake: treating "an image element with a URL" as "the image." Modern pages separate the *presentation* of an image from the *asset* behind it, and they keep a second layer of images entirely outside <img> tags. If your collector only walks the DOM for src attributes, it samples the placeholder layer and misses the CSS layer. Three things that look like images but are not 1. Inline LQIP data URIs. A low-quality image placeholder is a real image, just a deliberately terrible one — often 20×20 pixels encoded as a base64 JPEG of 40 to 300 characters. It is inlined so it paints before any network request resolves. The src attribute is a complete, valid image. ...

Why Image Scrapers Silently Fail: CORS, Tainted Canvases, and Blob URLs

图片
 You write a small script that walks a page, grabs every img, fetches the src, and writes the bytes to disk. It works on nine sites out of ten. On the tenth you get a folder full of 0-byte files, or thirty SecurityError: The operation is insecure lines in the console, or images that appear in the DOM but 404 the moment you try to download them. Almost nobody debugs this correctly, because all three failures look the same from the outside — "the download didn't work" — while they come from three completely different layers. Here is how I learned to tell them apart, and why each one breaks. Failure one: the bytes never leave the server The first layer is plain HTTP. A request for https://cdn.example.com/hero.jpg returns 403 Forbidden when it arrives without a Referer header, because the host runs hotlink protection and only serves images to pages it recognises. Browser-tab requests always carry a referrer, so the image looks perfectly fine in the page — and a script that tr...

Alt Text Is a Label: Turning HTML Metadata into a Searchable Image Library

图片
 Alt Text Is a Label: Turning HTML Metadata into a Searchable Image Library You have 2,000 images and no labels. Tagging them by hand is a week of work; an auto-tagger that guesses "product photo" for everything is worse than nothing. Meanwhile the page you downloaded them from already carried the answer — in alt, in title, in JSON-LD, in the filename — and your collector threw all of it away. This post is about harvesting those free labels, ranking them by how much you can trust them, and turning them into a real retrieval layer. WHERE LABELS ACTUALLY LIVE Every image on the web sits inside markup that describes it, often several times over: - img alt — the accessible description. Not always present, not always useful, but frequently precise: "Nike Air Zoom Pegasus 40, white/orange, side view". - title and aria-label — parallel descriptions, sometimes the only one. - figure > figcaption — editorial captions, often the richest prose. - JSON-LD blocks — schema.org...

Every Image on the Page: URL Canonicalization, Lazy Loading, and srcset in a Site-Wide Crawler

 If you've ever tried to "save all images from this page" with a naive loop over document.querySelectorAll('img'), you know how much it misses. Lazy-loaded images aren't in the DOM yet. Responsive images hide their real URLs inside srcset. Background images live in CSS, not in any <img> tag. And even when you do find a URL, the same image can appear under ten different URLs — with different query strings, sizes, and tracking parameters — so your "2,300 images" are really 800 unique ones. A serious site-wide image crawler isn't a querySelector loop. It's a pipeline with three hard problems: finding every image, canonicalizing its URL, and surviving pages that load images on demand. This post walks through each one. Problem 1: Finding images that aren't <img> tags The first pass is the obvious one: img[src], img[srcset], <source> inside <picture>, and video[poster]. But three classes of images need special handling: ...
 When you crawl a site and collect 2,000 images, what does the browser actually have to remember about each one? The URL, dimensions, format, source page, download status, a pHash fingerprint for deduplication, and — if you're doing on-device AI — a 1024-dimensional feature vector. That's roughly 5-10 KB of structured metadata per image just for the vectors, before counting the image bytes themselves. localStorage gives you 5-10 MB of synchronous string storage. A single Float32Array of 1024 values, JSON-serialized, is about 8 KB per image — 2,000 images means 16 MB, already past the limit, and it would block the main thread on every read. Plain in-memory objects vanish on refresh. The Cache API is designed for network responses, not queryable records. For a local-first image tool, none of these fit. The right tool is IndexedDB — a real transactional database built into every browser. This post walks through its data model, why it scales to thousands of records without freezing...

On-Device AI, Explained: Depthwise Convolutions, MobileNet, and Why Running Inference in the Browser Is a Privacy Win

 Every time you tap "upload to cloud" for a photo analysis task — reverse image search, object classification, similar-image grouping — you're sending a real image of a real document, a product shot, or a personal screenshot to somebody else's server. The provider logs it, trains on it, or keeps it for "quality purposes." You have no idea. There is a different way: run the model in the browser, on the user's own machine, with the pixels never leaving the tab. This used to be a fantasy — a neural network needs gigabytes of compute, right? Not with the right architecture. This post explains the two engineering ideas that make on-device inference practical (depthwise separable convolutions and WebGPU), and why "local" is not just a performance story but a privacy architecture. The problem with a "normal" convolution A convolutional layer takes a W×H×C input and produces a W×H×N output feature map. A standard 3×3 convolution does this: pa...

Perceptual Hashing, Explained: Turning an Image into a 64-Bit Fingerprint with DCT

 If you've ever tried to deduplicate a folder of downloaded images with a regular hash, you already know the problem: MD5 and SHA-256 are exact hashes. Re-encode a JPEG at 85% quality, resize it by 2%, or strip its EXIF metadata, and the bytes change — the hash changes completely. Two images that look identical to a human produce two unrelated hashes. That's why image tools need a perceptual hash instead: a fingerprint that stays stable under compression, resizing, and color tweaks. The most widely used one is the pHash (perceptual hash) algorithm, which squeezes an image down to 64 bits and can tell you, with a single integer comparison, whether two images are visually "the same." Here's the complete algorithm, step by step, including the math and a practical JavaScript sketch. Step 1: Shrink to 32×32 The image is resized to 32×32 pixels. This does two things at once: it normalizes away resolution differences (a 4000×3000 photo and a 800×600 crop of it both becom...