This page documents the internal processing pipelines and algorithms used by Eise.app. Useful for contributors or anyone curious about how it works.
┌─────────────────────────────────────────────────────────────────────────┐
│ FILE INPUT │
│ SER / AVI / Video / RAW photos (CR2, NEF, ARW, RAF, DNG) / Images │
└───────────────────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ FORMAT DETECTION │
│ (FileUploader.vue) │
│ │
│ Routes to appropriate reader based on file type and codec │
└───────────────────────────────────┬─────────────────────────────────────┘
│
┌───────────────────────┼───────────────────────┐
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Raw Bayer Path │ │ Video Path │ │ RGB/MJPEG Path │
│ │ │ │ │ │
│ useSerParser │ │ useFFmpegReader │ │ useAviReader │
│ useAviParser │ │ │ │ useImageReader │
│ ↓ │ │ FFmpeg.js │ │ │
│ useDebayerReader│ │ ↓ │ │ Already RGB │
│ ↓ │ │ PNG frames │ │ (no demosaic) │
│ GPU Demosaic │ │ ↓ │ │ │
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │
└───────────────────────┼───────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ GPU ANALYSIS │
│ (webgpu_analyze_worker.js) │
│ │
│ 1. Crop Detection - Sample frames, find planet bounds │
│ 2. Per-frame Analyze - Sharpness scoring, per-frame centering │
│ 3. Frame Selection - Keep best N% by sharpness │
└───────────────────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ GPU STACKING │
│ (webgpu_template_match.js + webgpu_stacking.js) │
│ │
│ 1. Template Matching - Find local shifts at alignment points │
│ 2. De-warping - Interpolate displacement map, warp frame │
│ 3. Accumulation - Weighted sum with brightness normalization │
└───────────────────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ POST-PROCESSING │
│ (PostProcessor.vue) │
│ │
│ Wavelet sharpening, RGB alignment, color correction, crop │
└─────────────────────────────────────────────────────────────────────────┘
| Format | Reader | Notes |
|---|---|---|
| SER | useSerParser → useDebayerReader | Recommended. Raw Bayer with GPU demosaic (VNG or bilinear) |
| AVI (Y800, DIB 8-bit) | useAviParser → useDebayerReader | Raw Bayer AVI from capture software |
| AVI (MJPEG) | useAviReader | Already RGB, GPU analysis only |
| AVI (BGR 24-bit) | useAviReader | Already RGB, GPU analysis only |
| MP4, MOV, WebM | useFFmpegReader | FFmpeg decode → PNG → GPU analysis |
| PNG, JPG, TIFF | useImageReader | Image sequences |
| Camera RAW (CR2, CR3, NEF, ARW, ORF, RW2, DNG) | useLibRawParser → useDebayerReader | LibRaw (WASM) reads the undebayered sensor mosaic; demosaic runs on the GPU like SER, so the full sensor bit depth is kept |
| Camera RAW without a Bayer grid (Fuji X-Trans RAF, linear DNG, Foveon) | useLibRawRgb → useImageReader | No 2x2 mosaic to read, so LibRaw demosaics these itself and they join the image pipeline |
The analysis phase processes frames to determine which ones to keep for stacking.
For each batch of frames:
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Read from │ │ Demosaic │ │ Detect │ │ Compute │
│ disk │ ──▶ │ (if Bayer) │ ──▶ │ bounds │ ──▶ │ sharpness │
│ │ │ │ │ │ │ (Tenengrad) │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
│
┌──────┴──────┐
│ │
┌─────▼─────┐ ┌─────▼─────┐
│ grayOnly │ │ VNG │
│ (fast) │ │ (quality)│
│ 4 fetch │ │ 33 fetch │
└───────────┘ └───────────┘
│
▼
Analysis uses grayOnly
Stacking uses VNG
| Method | Texture Fetches | Used For |
|---|---|---|
| grayOnly | 4 per pixel | Analysis phase (sharpness scoring) |
| Bilinear | ~8 per pixel | Fast preview, low quality stacking |
| VNG | ~33 per pixel | Stacking phase (full quality) |
The GPU computes two complementary sharpness metrics and combines them:
Tenengrad (Sobel gradient magnitude): ┌─────────────────┐ ┌─────────────────┐ │ Gx = [-1 0 1] │ │ Gy = [-1 -2 -1] │ │ [-2 0 2] │ │ [ 0 0 0] │ │ [-1 0 1] │ │ [ 1 2 1] │ └─────────────────┘ └─────────────────┘ Tenengrad = mean(Gx² + Gy²) Laplacian (second derivative): ┌─────────────────┐ │ [ 0 1 0] │ │ [ 1 -4 1] │ │ [ 0 1 0] │ └─────────────────┘ Laplacian = mean(lap²) Combined sharpness = √(Tenengrad × Laplacian)
The geometric mean combines edge detection (Tenengrad) with fine detail detection (Laplacian). Higher values indicate sharper frames.
Note: The CPU fallback path uses Tenengrad only (no Laplacian) for simplicity.
Critical for planetary stacking: the planet drifts across frames due to atmospheric refraction and mount drift. Each frame must be cropped with its own detected center.
Frame 1: Frame 50: Frame 100: ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ │ │ │ │ ● │ │ ● │ │ ● │ Planet drifts! │ │ │ │ │ │ └─────────┘ └─────────┘ └─────────┘ After per-frame centering: ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ │ │ │ │ ● │ │ ● │ │ ● │ Centered! │ │ │ │ │ │ └─────────┘ └─────────┘ └─────────┘
Selected frames are aligned and accumulated using local alignment points.
┌───────────────────────────────────┐ │ · · · · · · │ │ │ │ · · · · · · │ · = Alignment Point (AP) │ ████████████ │ │ · · ██ Planet ██ · · │ Each AP tracks local motion │ ██ ██ │ using template matching │ · · ████████████ · · │ │ │ Patch size: 20-50 pixels │ · · · · · · │ Search radius: 8-34 pixels │ │ │ · · · · · · │ └───────────────────────────────────┘
Normalized Cross-Correlation finds the best match position for each AP:
Σ[(ref - μref)(frame - μframe)]
NCC = ─────────────────────────────────────────
sqrt(Σ(ref - μref)²) × sqrt(Σ(frame - μframe)²)
NCC ranges from -1 to +1 (1 = perfect match)
Sub-pixel precision achieved via parabolic interpolation of the 3x3 peak neighborhood.
Displacement vectors from APs are interpolated across the frame:
Measured displacements: Interpolated displacement map:
←· ·→ · ←←←↖↖↑↑↗↗→→
←←←↖↖↑↑↗↗→→
←· · ·→ ←←←↖↖↑↑↗↗→→
←←↖↖↖↑↗↗↗→→
· ·→ · ←↖↖↖↖↑↗↗↗↗→
↖↖↖↖↖↑↗↗↗↗↗
Gaussian-weighted interpolation:
w(d) = exp(-d² / (2σ²)) where d = distance to AP
For each frame f with sharpness S:
weight = S / max_sharpness
brightness_scale = reference_brightness / frame_brightness
accumulator += warped_frame × brightness_scale × weight
weight_sum += weight
Final = accumulator / weight_sum
| File | Purpose |
|---|---|
composables/useDebayerReader.js | Unified raw Bayer processing (SER, raw AVI) |
composables/useFFmpegReader.js | Video decode via FFmpeg.js |
composables/useStacker.js | Stacking orchestration, GPU/CPU path selection |
public/webgpu_analyze_worker.js | GPU demosaic, sharpness, bounds detection |
public/webgpu_template_match.js | GPU template matching for alignment |
public/webgpu_stacking.js | GPU frame warping and accumulation |
public/gpu/shaders.js | All WGSL compute shaders |
The mode affects template matching search radius and cut-off frame rejection:
| Mode | AP Search Radius | Cut-off Rejection | Use Case |
|---|---|---|---|
| Planet | 8 pixels | Yes | Jupiter, Saturn, Mars - small motion, reject frames where planet touches edge |
| Surface | 34 pixels | No | Moon, Sun closeups - larger drift between frames, no defined edge |
AP Search Radius = how far (in pixels) to search around each alignment point when looking for the best template match. Larger radius handles more frame-to-frame motion but is slower.
┌─────────────────────────────────────────────────────────────────────────┐
│ WebGPU Available? │
└───────────────────────────────────┬─────────────────────────────────────┘
│
┌───────────────┴───────────────┐
│ │
▼ ▼
┌───────────────┐ ┌───────────────┐
│ GPU Path │ │ CPU Path │
│ (primary) │ │ (fallback) │
│ │ │ │
│ webgpu_* │ │ unified_ │
│ workers │ │ analyze_ │
│ │ │ worker.js │
│ Fast! │ │ OpenCV-WASM │
└───────────────┘ └───────────────┘
OpenCV uses inverted naming from industry standard:
| Industry (SER) | OpenCV | Layout |
|---|---|---|
| RGGB | BayerBG | R G |
| BGGR | BayerRG | B G |
| GRBG | BayerGB | G R |
| GBRG | BayerGR | G B |
Check out the GitHub repository. Key documentation files:
CLAUDE.md - Architecture overview for AI assistantsPROCESSING_PIPELINE.md - Detailed pipeline documentationPIPELINE_OPTIMIZATION_ANALYSIS.md - Performance optimization opportunities