Smart Image Classification in the Browser: A Layered Architecture from Rule-Based Heuristics to On-Device CNNs

 Designers, e-commerce sellers, and content teams share the same pain: you collect hundreds of images from a website, and they land in one disorganized folder. Product shots mixed with UI mockups, icons mixed with photos, 4K originals mixed with 32px thumbnails. Before you can do anything useful — dedupe, search, export — you need to classify. But what does "classify images" actually mean in a browser, where you have limited memory, no Python ecosystem, and a hard performance budget? Let's break down a layered classification architecture that works entirely on-device.


## The Problem: Why Naive Classification Fails


The naive approach is to treat all images the same:


```javascript

// Naive: everything is just an <img>

document.querySelectorAll('img').forEach(img => {

  saveToFolder('all-images', img.src);

});

```


This fails for three reasons:


1. **No dimensional awareness** — a 4K hero banner and a 32px favicon end up together

2. **No format awareness** — a 2MB PNG screenshot and a 20KB WebP photo are indistinguishable

3. **No semantic awareness** — you can't tell a product photo from a UI mockup without looking at the pixels


## Layer 1: Rule-Based Heuristics — The Fast Path


Before touching any neural network, a set of deterministic rules can classify 80% of images correctly with near-zero cost. These rules run in microseconds and require no model loading.


### Resolution-Based Classification


The first signal is dimensions. Reading an image's natural size is cheap:


```javascript

function classifyByResolution(img) {

  const naturalWidth = img.naturalWidth || 0;

  const naturalHeight = img.naturalHeight || 0;

  const area = naturalWidth * naturalHeight;

  

  if (naturalWidth >= 3840 || naturalHeight >= 2160) {

    return { category: '4K-UltraHD', quality: 100 };

  }

  if (naturalWidth >= 1920 || naturalHeight >= 1080) {

    return { category: '1080P-HD', quality: 80 };

  }

  if (area >= 200 * 200) {

    return { category: 'medium', quality: 50 };

  }

  return { category: 'thumbnail', quality: 20 };

}

```


But there's a subtlety: a 4000px-wide decorative banner is not more "valuable" than a 1200px product photo. Resolution alone is insufficient — which is why we combine signals.


### Format-Based Classification


Different formats serve different purposes: PNG for screenshots/UI/transparency, JPEG for photos, WebP/AVIF for modern web-optimized images, GIF for animation, SVG for icons and logos.


```javascript

function classifyByFormat(url, contentType) {

  const ext = url.split('.').pop().toLowerCase();

  switch (ext) {

    case 'svg': return { category: 'vector', subcategory: 'icon-or-illustration' };

    case 'png': return { category: 'raster', transparency: true };

    case 'jpg': case 'jpeg': return { category: 'raster', transparency: false };

    case 'webp': case 'avif': return { category: 'modern-raster' };

    case 'gif': return { category: 'animation' };

    default: return { category: 'unknown', needsInspection: true };

  }

}

```


### Color and Composition Heuristics


A few cheap pixel-level statistics add powerful signals. Using an offscreen canvas, we can sample the image without disrupting the page:


```javascript

async function analyzeColorProfile(imageUrl) {

  const img = await loadImage(imageUrl);

  const canvas = document.createElement('canvas');

  const ctx = canvas.getContext('2d', { willReadFrequently: true });

  

  // Downsample to 32x32 for fast analysis

  canvas.width = 32;

  canvas.height = 32;

  ctx.drawImage(img, 0, 0, 32, 32);

  const data = ctx.getImageData(0, 0, 32, 32).data;

  

  let saturationSum = 0;

  let brightnessSum = 0;

  let transparentPixels = 0;

  

  for (let i = 0; i < data.length; i += 4) {

    const r = data[i], g = data[i+1], b = data[i+2], a = data[i+3];

    if (a < 128) { transparentPixels++; continue; }

    

    const max = Math.max(r, g, b);

    const min = Math.min(r, g, b);

    const brightness = max / 255;

    const saturation = max === 0 ? 0 : (max - min) / max;

    

    saturationSum += saturation;

    brightnessSum += brightness;

  }

  

  const totalPixels = data.length / 4;

  const avgSaturation = saturationSum / totalPixels;

  const avgBrightness = brightnessSum / totalPixels;

  const transparencyRatio = transparentPixels / totalPixels;

  

  // Heuristic: low saturation + mid brightness + high transparency = UI/screenshot territory

  if (transparencyRatio > 0.1) return { kind: 'ui-asset', confidence: 0.7 };

  if (avgSaturation < 0.25 && avgBrightness > 0.4) return { kind: 'screenshot-or-ui', confidence: 0.6 };

  if (avgSaturation > 0.5) return { kind: 'photo', confidence: 0.7 };

  

  return { kind: 'unknown', confidence: 0.3 };

}

```


### Rule Engine Design: Weighted Score, Not Hard Rules


Hard if/else rules are brittle. A better design is a weighted scoring system that degrades gracefully on partial matches. Each rule adds points to candidate categories, and you pick the argmax — a perfect icon scores 4, but a partial match still lands in the right neighborhood.


## Layer 2: Lightweight CNN — The Semantic Path


Rule-based heuristics handle the obvious cases, but they fail on semantic distinctions: "is this a product photo of a chair or a photo of a landscape?" That's where a small convolutional neural network comes in.


### Why MobileNetV3-Small?


For on-device classification, MobileNetV3-Small is the sweet spot:

- **2.5M parameters** (~2.5MB in float32, ~1MB quantized to int8)

- **224×224 input** — same preprocessing pipeline as feature extraction for reverse image search

- **~15ms inference** with WebGPU acceleration

- **1000 ImageNet classes** that can be mapped to practical categories


### Quantization: The Critical Optimization


A full float32 MobileNet runs, but at 2.5MB of weights, model loading eats memory and startup time. Quantization to int8 reduces model size by 4x and speeds up inference 2-3x on most hardware, with only 0.5-2% accuracy loss for classification tasks.


### Combining Heuristics + CNN: The Cascade


The key insight: don't run the CNN on every image. The cascade architecture runs cheap rules first, and only escalates ambiguous cases to the neural network:


```javascript

async function classifyImageCascade(metadata, colorProfile) {

  // Layer 1: Fast rules — handle 80% of cases

  const ruleResult = classifyImage(metadata, colorProfile);

  if (ruleResult.confidence >= 0.8) {

    return ruleResult.category; // No neural network needed

  }

  

  // Layer 2: CNN — only for ambiguous cases (~20%)

  if (ruleResult.category === 'unknown' || ruleResult.confidence < 0.8) {

    const cnnResult = await runMobileNet(metadata.element);

    return mapImageNetToCategory(cnnResult);

  }

  

  return ruleResult.category;

}

```


This cascade gives you speed (80% of images classified in microseconds), accuracy (ambiguous cases get the full CNN treatment), and battery/memory savings (CNN runs on 1 in 5 images).


## Layer 3: The Taxonomy — Designing Categories That Actually Work


Classification is only as good as your category system. A good taxonomy for web images is multi-dimensional — an image belongs to a format dimension AND a semantic dimension AND a resolution dimension. That's why classification should produce tags, not just a single folder:


```javascript

const imageTags = {

  format: 'webp',

  semantic: 'photo',

  subSemantic: 'product',

  resolution: '4K-UltraHD',

  colorProfile: 'high-saturation',

  dimensions: [4096, 3072],

  fileSize: 1240000,

  source: 'example.com',

  capturedAt: Date.now()

};

```


Tag-based organization lets users filter by any dimension without duplicating files — the same image can appear in a "4K" view and a "product photos" view simultaneously.


## Implementation Challenges


Building this in a browser extension is harder than it sounds:


1. **Canvas tainting** — cross-origin images taint the canvas, making getImageData throw a SecurityError. Workaround: fetch with crossOrigin='anonymous' (requires CORS) or use extension host permissions to bypass CORS.

2. **Memory pressure** — decoding 50 large images simultaneously can blow past the browser's memory limits. Solution: sequential processing queue with a sliding window, releasing URL.revokeObjectURL immediately after processing.

3. **WebGPU availability** — fall back to WebGL2 or CPU (WASM) kernels when WebGPU is unavailable.

4. **Model caching** — the quantized MobileNet (~1MB) should be cached in IndexedDB to avoid re-downloading on every browser start.

5. **Background classification** — classification of 1000+ images must be chunked and run in idle time via requestIdleCallback to avoid janking the page.


## OmniPic: The Classification Engine, Built In


As you can see, production-grade image classification in the browser requires a layered architecture — fast rule-based heuristics for the common cases, an on-device quantized CNN for semantic distinctions, a multi-dimensional taxonomy, and careful performance engineering for memory and battery. That's a substantial amount of engineering.


OmniPic AI Studio Pro includes all of this as a core feature. When you crawl any website:


- **Smart auto-classification** — every collected image is automatically sorted into 4K Ultra HD, 1080P HD, medium, and thumbnail categories, with format detection and color-profile analysis

- **On-device AI understanding** — MobileNet-based semantic classification runs entirely in your browser via WebGPU, identifying product photos, UI mockups, icons, and more — no cloud, no data upload

- **Full-site crawling** — every image detected, from hero banners to lazy-loaded carousel images

- **pHash visual deduplication** — automatically filter duplicates even when images are cropped, recompressed, or watermarked

- **CDN original image restoration** — automatically recover full-resolution originals from Aliyun OSS, Tencent COS, and Qiniu thumbnail URLs

- **On-device AI reverse image search** — find similar images in your collection in under 50ms

- **Streaming ZIP export** — download 2000+ images in one ZIP without freezing your browser

- **Eagle/Billfish integration** — push classified images straight into your local design asset library

- **100% local processing** — your images and browsing activity never leave your device


## Get Started


OmniPic AI Studio Pro is available on all three major browser extension stores:


- **Chrome**: Chrome Web Store (https://chromewebstore.google.com/detail/omnipic-studio-pro/cokoeaaaglgcljahaffjeijomhmkljlc)

- **Edge**: Microsoft Edge Add-ons

- **Firefox**: Firefox Add-ons


Image classification doesn't need a cloud API or a Python backend. With a layered architecture — rule-based heuristics as the fast path, a quantized CNN as the semantic escalator, and a multi-dimensional taxonomy as the organizing principle — the entire pipeline can run in your browser in real time. OmniPic packages this engineering into a one-click extension that organizes every image you collect, entirely on your device.

评论

此博客中的热门博文

Running AI in the Browser: WebGPU-Accelerated MobileNet Feature Extraction for Real-Time Image Search

Why Image Scrapers Silently Fail: CORS, Tainted Canvases, and Blob URLs

Why Your Image Crawler Collects Blur Placeholders (and Misses the Images Hiding in CSS)