Smart Image Classification in the Browser: A Layered Architecture from Rule-Based Heuristics to On-Device CNNs
Designers, e-commerce sellers, and content teams share the same pain: you collect hundreds of images from a website, and they land in one disorganized folder. Product shots mixed with UI mockups, icons mixed with photos, 4K originals mixed with 32px thumbnails. Before you can do anything useful — dedupe, search, export — you need to classify. But what does "classify images" actually mean in a browser, where you have limited memory, no Python ecosystem, and a hard performance budget? Let's break down a layered classification architecture that works entirely on-device.
## The Problem: Why Naive Classification Fails
The naive approach is to treat all images the same:
```javascript
// Naive: everything is just an <img>
document.querySelectorAll('img').forEach(img => {
saveToFolder('all-images', img.src);
});
```
This fails for three reasons:
1. **No dimensional awareness** — a 4K hero banner and a 32px favicon end up together
2. **No format awareness** — a 2MB PNG screenshot and a 20KB WebP photo are indistinguishable
3. **No semantic awareness** — you can't tell a product photo from a UI mockup without looking at the pixels
## Layer 1: Rule-Based Heuristics — The Fast Path
Before touching any neural network, a set of deterministic rules can classify 80% of images correctly with near-zero cost. These rules run in microseconds and require no model loading.
### Resolution-Based Classification
The first signal is dimensions. Reading an image's natural size is cheap:
```javascript
function classifyByResolution(img) {
const naturalWidth = img.naturalWidth || 0;
const naturalHeight = img.naturalHeight || 0;
const area = naturalWidth * naturalHeight;
if (naturalWidth >= 3840 || naturalHeight >= 2160) {
return { category: '4K-UltraHD', quality: 100 };
}
if (naturalWidth >= 1920 || naturalHeight >= 1080) {
return { category: '1080P-HD', quality: 80 };
}
if (area >= 200 * 200) {
return { category: 'medium', quality: 50 };
}
return { category: 'thumbnail', quality: 20 };
}
```
But there's a subtlety: a 4000px-wide decorative banner is not more "valuable" than a 1200px product photo. Resolution alone is insufficient — which is why we combine signals.
### Format-Based Classification
Different formats serve different purposes: PNG for screenshots/UI/transparency, JPEG for photos, WebP/AVIF for modern web-optimized images, GIF for animation, SVG for icons and logos.
```javascript
function classifyByFormat(url, contentType) {
const ext = url.split('.').pop().toLowerCase();
switch (ext) {
case 'svg': return { category: 'vector', subcategory: 'icon-or-illustration' };
case 'png': return { category: 'raster', transparency: true };
case 'jpg': case 'jpeg': return { category: 'raster', transparency: false };
case 'webp': case 'avif': return { category: 'modern-raster' };
case 'gif': return { category: 'animation' };
default: return { category: 'unknown', needsInspection: true };
}
}
```
### Color and Composition Heuristics
A few cheap pixel-level statistics add powerful signals. Using an offscreen canvas, we can sample the image without disrupting the page:
```javascript
async function analyzeColorProfile(imageUrl) {
const img = await loadImage(imageUrl);
const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d', { willReadFrequently: true });
// Downsample to 32x32 for fast analysis
canvas.width = 32;
canvas.height = 32;
ctx.drawImage(img, 0, 0, 32, 32);
const data = ctx.getImageData(0, 0, 32, 32).data;
let saturationSum = 0;
let brightnessSum = 0;
let transparentPixels = 0;
for (let i = 0; i < data.length; i += 4) {
const r = data[i], g = data[i+1], b = data[i+2], a = data[i+3];
if (a < 128) { transparentPixels++; continue; }
const max = Math.max(r, g, b);
const min = Math.min(r, g, b);
const brightness = max / 255;
const saturation = max === 0 ? 0 : (max - min) / max;
saturationSum += saturation;
brightnessSum += brightness;
}
const totalPixels = data.length / 4;
const avgSaturation = saturationSum / totalPixels;
const avgBrightness = brightnessSum / totalPixels;
const transparencyRatio = transparentPixels / totalPixels;
// Heuristic: low saturation + mid brightness + high transparency = UI/screenshot territory
if (transparencyRatio > 0.1) return { kind: 'ui-asset', confidence: 0.7 };
if (avgSaturation < 0.25 && avgBrightness > 0.4) return { kind: 'screenshot-or-ui', confidence: 0.6 };
if (avgSaturation > 0.5) return { kind: 'photo', confidence: 0.7 };
return { kind: 'unknown', confidence: 0.3 };
}
```
### Rule Engine Design: Weighted Score, Not Hard Rules
Hard if/else rules are brittle. A better design is a weighted scoring system that degrades gracefully on partial matches. Each rule adds points to candidate categories, and you pick the argmax — a perfect icon scores 4, but a partial match still lands in the right neighborhood.
## Layer 2: Lightweight CNN — The Semantic Path
Rule-based heuristics handle the obvious cases, but they fail on semantic distinctions: "is this a product photo of a chair or a photo of a landscape?" That's where a small convolutional neural network comes in.
### Why MobileNetV3-Small?
For on-device classification, MobileNetV3-Small is the sweet spot:
- **2.5M parameters** (~2.5MB in float32, ~1MB quantized to int8)
- **224×224 input** — same preprocessing pipeline as feature extraction for reverse image search
- **~15ms inference** with WebGPU acceleration
- **1000 ImageNet classes** that can be mapped to practical categories
### Quantization: The Critical Optimization
A full float32 MobileNet runs, but at 2.5MB of weights, model loading eats memory and startup time. Quantization to int8 reduces model size by 4x and speeds up inference 2-3x on most hardware, with only 0.5-2% accuracy loss for classification tasks.
### Combining Heuristics + CNN: The Cascade
The key insight: don't run the CNN on every image. The cascade architecture runs cheap rules first, and only escalates ambiguous cases to the neural network:
```javascript
async function classifyImageCascade(metadata, colorProfile) {
// Layer 1: Fast rules — handle 80% of cases
const ruleResult = classifyImage(metadata, colorProfile);
if (ruleResult.confidence >= 0.8) {
return ruleResult.category; // No neural network needed
}
// Layer 2: CNN — only for ambiguous cases (~20%)
if (ruleResult.category === 'unknown' || ruleResult.confidence < 0.8) {
const cnnResult = await runMobileNet(metadata.element);
return mapImageNetToCategory(cnnResult);
}
return ruleResult.category;
}
```
This cascade gives you speed (80% of images classified in microseconds), accuracy (ambiguous cases get the full CNN treatment), and battery/memory savings (CNN runs on 1 in 5 images).
## Layer 3: The Taxonomy — Designing Categories That Actually Work
Classification is only as good as your category system. A good taxonomy for web images is multi-dimensional — an image belongs to a format dimension AND a semantic dimension AND a resolution dimension. That's why classification should produce tags, not just a single folder:
```javascript
const imageTags = {
format: 'webp',
semantic: 'photo',
subSemantic: 'product',
resolution: '4K-UltraHD',
colorProfile: 'high-saturation',
dimensions: [4096, 3072],
fileSize: 1240000,
source: 'example.com',
capturedAt: Date.now()
};
```
Tag-based organization lets users filter by any dimension without duplicating files — the same image can appear in a "4K" view and a "product photos" view simultaneously.
## Implementation Challenges
Building this in a browser extension is harder than it sounds:
1. **Canvas tainting** — cross-origin images taint the canvas, making getImageData throw a SecurityError. Workaround: fetch with crossOrigin='anonymous' (requires CORS) or use extension host permissions to bypass CORS.
2. **Memory pressure** — decoding 50 large images simultaneously can blow past the browser's memory limits. Solution: sequential processing queue with a sliding window, releasing URL.revokeObjectURL immediately after processing.
3. **WebGPU availability** — fall back to WebGL2 or CPU (WASM) kernels when WebGPU is unavailable.
4. **Model caching** — the quantized MobileNet (~1MB) should be cached in IndexedDB to avoid re-downloading on every browser start.
5. **Background classification** — classification of 1000+ images must be chunked and run in idle time via requestIdleCallback to avoid janking the page.
## OmniPic: The Classification Engine, Built In
As you can see, production-grade image classification in the browser requires a layered architecture — fast rule-based heuristics for the common cases, an on-device quantized CNN for semantic distinctions, a multi-dimensional taxonomy, and careful performance engineering for memory and battery. That's a substantial amount of engineering.
OmniPic AI Studio Pro includes all of this as a core feature. When you crawl any website:
- **Smart auto-classification** — every collected image is automatically sorted into 4K Ultra HD, 1080P HD, medium, and thumbnail categories, with format detection and color-profile analysis
- **On-device AI understanding** — MobileNet-based semantic classification runs entirely in your browser via WebGPU, identifying product photos, UI mockups, icons, and more — no cloud, no data upload
- **Full-site crawling** — every image detected, from hero banners to lazy-loaded carousel images
- **pHash visual deduplication** — automatically filter duplicates even when images are cropped, recompressed, or watermarked
- **CDN original image restoration** — automatically recover full-resolution originals from Aliyun OSS, Tencent COS, and Qiniu thumbnail URLs
- **On-device AI reverse image search** — find similar images in your collection in under 50ms
- **Streaming ZIP export** — download 2000+ images in one ZIP without freezing your browser
- **Eagle/Billfish integration** — push classified images straight into your local design asset library
- **100% local processing** — your images and browsing activity never leave your device
## Get Started
OmniPic AI Studio Pro is available on all three major browser extension stores:
- **Chrome**: Chrome Web Store (https://chromewebstore.google.com/detail/omnipic-studio-pro/cokoeaaaglgcljahaffjeijomhmkljlc)
- **Edge**: Microsoft Edge Add-ons
- **Firefox**: Firefox Add-ons
Image classification doesn't need a cloud API or a Python backend. With a layered architecture — rule-based heuristics as the fast path, a quantized CNN as the semantic escalator, and a multi-dimensional taxonomy as the organizing principle — the entire pipeline can run in your browser in real time. OmniPic packages this engineering into a one-click extension that organizes every image you collect, entirely on your device.
评论
发表评论