*Markdown version of <https://eise.app/about/architecture/>*

> Technical documentation of Eise.app's processing pipeline, algorithms, and architecture. For contributors and curious users.

## Technical Architecture

This page documents the internal processing pipelines and algorithms used by Eise.app. Useful for contributors or anyone curious about how it works.

### High-Level Overview

┌─────────────────────────────────────────────────────────────────────────┐
│ FILE INPUT │
│ SER / AVI / Video / RAW photos (CR2, NEF, ARW, RAF, DNG) / Images │
└───────────────────────────────────┬─────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ FORMAT DETECTION │
│ (FileUploader.vue) │
│ │
│ Routes to appropriate reader based on file type and codec │
└───────────────────────────────────┬─────────────────────────────────────┘
                                    │
            ┌───────────────────────┼───────────────────────┐
            │ │ │
            ▼ ▼ ▼
   ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
   │ Raw Bayer Path │ │ Video Path │ │ RGB/MJPEG Path │
   │ │ │ │ │ │
   │ useSerParser │ │ useFFmpegReader │ │ useAviReader │
   │ useAviParser │ │ │ │ useImageReader │
   │ ↓ │ │ FFmpeg.js │ │ │
   │ useDebayerReader│ │ ↓ │ │ Already RGB │
   │ ↓ │ │ PNG frames │ │ (no demosaic) │
   │ GPU Demosaic │ │ ↓ │ │ │
   └────────┬────────┘ └────────┬────────┘ └────────┬────────┘
            │ │ │
            └───────────────────────┼───────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ GPU ANALYSIS │
│ (webgpu\_analyze\_worker.js) │
│ │
│ 1. Crop Detection - Sample frames, find planet bounds │
│ 2. Per-frame Analyze - Sharpness scoring, per-frame centering │
│ 3. Frame Selection - Keep best N% by sharpness │
└───────────────────────────────────┬─────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ GPU STACKING │
│ (webgpu\_template\_match.js + webgpu\_stacking.js) │
│ │
│ 1. Template Matching - Find local shifts at alignment points │
│ 2. De-warping - Interpolate displacement map, warp frame │
│ 3. Accumulation - Weighted sum with brightness normalization │
└───────────────────────────────────┬─────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ POST-PROCESSING │
│ (PostProcessor.vue) │
│ │
│ Wavelet sharpening, RGB alignment, color correction, crop │
└─────────────────────────────────────────────────────────────────────────┘

### File Format Support

| Format | Reader | Notes |
| --- | --- | --- |
| SER | useSerParser → useDebayerReader | Recommended. Raw Bayer with GPU demosaic (VNG or bilinear) |
| AVI (Y800, DIB 8-bit) | useAviParser → useDebayerReader | Raw Bayer AVI from capture software |
| AVI (MJPEG) | useAviReader | Already RGB, GPU analysis only |
| AVI (BGR 24-bit) | useAviReader | Already RGB, GPU analysis only |
| MP4, MOV, WebM | useFFmpegReader | FFmpeg decode → PNG → GPU analysis |
| PNG, JPG, TIFF | useImageReader | Image sequences |
| Camera RAW (CR2, CR3, NEF, ARW, ORF, RW2, DNG) | useLibRawParser → useDebayerReader | LibRaw (WASM) reads the undebayered sensor mosaic; demosaic runs on the GPU like SER, so the full sensor bit depth is kept |
| Camera RAW without a Bayer grid (Fuji X-Trans RAF, linear DNG, Foveon) | useLibRawRgb → useImageReader | No 2x2 mosaic to read, so LibRaw demosaics these itself and they join the image pipeline |

### Analysis Phase

The analysis phase processes frames to determine which ones to keep for stacking.

For each batch of frames:

┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Read from │ │ Demosaic │ │ Detect │ │ Compute │
│ disk │ ──▶ │ (if Bayer) │ ──▶ │ bounds │ ──▶ │ sharpness │
│ │ │ │ │ │ │ (Tenengrad) │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
                            │
                     ┌──────┴──────┐
                     │ │
               ┌─────▼─────┐ ┌─────▼─────┐
               │ grayOnly │ │ VNG │
               │ (fast) │ │ (quality)│
               │ 4 fetch │ │ 33 fetch │
               └───────────┘ └───────────┘
                     │
                     ▼
            Analysis uses grayOnly
            Stacking uses VNG

#### Demosaicing Methods

| Method | Texture Fetches | Used For |
| --- | --- | --- |
| grayOnly | 4 per pixel | Analysis phase (sharpness scoring) |
| Bilinear | ~8 per pixel | Fast preview, low quality stacking |
| VNG | ~33 per pixel | Stacking phase (full quality) |

#### Sharpness Calculation

Every frame is scored with Tenengrad, the mean squared Sobel gradient:

Tenengrad (Sobel gradient magnitude):
┌─────────────────┐ ┌─────────────────┐
│ Gx = \[-1 0 1\] │ │ Gy = \[-1 -2 -1\] │
│ \[-2 0 2\] │ │ \[ 0 0 0\] │
│ \[-1 0 1\] │ │ \[ 1 2 1\] │
└─────────────────┘ └─────────────────┘
Sharpness = mean(Gx² + Gy²)

Higher values indicate sharper frames. The GPU and CPU paths use the same metric and produce the same numbers.

**Note:** Earlier versions folded a Laplacian-variance term into the score as a geometric mean. Measuring how each term averages down with frame count showed the Laplacian was almost entirely reading noise rather than detail, so it was dropped.

#### Per-Frame Centering

Critical for planetary stacking: the planet drifts across frames due to atmospheric refraction and mount drift. Each frame must be cropped with its own detected center.

Frame 1: Frame 50: Frame 100:
┌─────────┐ ┌─────────┐ ┌─────────┐
│ │ │ │ │ │
│ ● │ │ ● │ │ ● │ Planet drifts!
│ │ │ │ │ │
└─────────┘ └─────────┘ └─────────┘

After per-frame centering:
┌─────────┐ ┌─────────┐ ┌─────────┐
│ │ │ │ │ │
│ ● │ │ ● │ │ ● │ Centered!
│ │ │ │ │ │
└─────────┘ └─────────┘ └─────────┘

### Stacking Phase

Selected frames are aligned and accumulated using local alignment points.

#### Alignment Point Grid

┌───────────────────────────────────┐
│ · · · · · · │
│ │
│ · · · · · · │ · = Alignment Point (AP)
│ ████████████ │
│ · · ██ Planet ██ · · │ Each AP tracks local motion
│ ██ ██ │ using template matching
│ · · ████████████ · · │
│ │ Patch size: 20-50 pixels
│ · · · · · · │ Search radius: 8-34 pixels
│ │
│ · · · · · · │
└───────────────────────────────────┘

#### Template Matching (NCC)

Normalized Cross-Correlation finds the best match position for each AP:

           Σ\[(ref - μref)(frame - μframe)\]
NCC = ─────────────────────────────────────────
       sqrt(Σ(ref - μref)²) × sqrt(Σ(frame - μframe)²)

NCC ranges from -1 to +1 (1 = perfect match)

Sub-pixel precision achieved via parabolic interpolation of the 3x3 peak neighborhood.

#### De-warping

Displacement vectors from APs are interpolated across the frame:

Measured displacements: Interpolated displacement map:

   ←· ·→ · ←←←↖↖↑↑↗↗→→
                                 ←←←↖↖↑↑↗↗→→
   ←· · ·→ ←←←↖↖↑↑↗↗→→
                                 ←←↖↖↖↑↗↗↗→→
   · ·→ · ←↖↖↖↖↑↗↗↗↗→
                                 ↖↖↖↖↖↑↗↗↗↗↗

Gaussian-weighted interpolation:
w(d) = exp(-d² / (2σ²)) where d = distance to AP

#### Accumulation

For each frame f with sharpness S:
    weight = S / max\_sharpness
    brightness\_scale = reference\_brightness / frame\_brightness

    accumulator += warped\_frame × brightness\_scale × weight
    weight\_sum += weight

Final = accumulator / weight\_sum

### Key Files

| File | Purpose |
| --- | --- |
| `composables/useDebayerReader.js` | Unified raw Bayer processing (SER, raw AVI) |
| `composables/useFFmpegReader.js` | Video decode via FFmpeg.js |
| `composables/useStacker.js` | Stacking orchestration, GPU/CPU path selection |
| `public/webgpu_analyze_worker.js` | GPU demosaic, sharpness, bounds detection |
| `public/webgpu_template_match.js` | GPU template matching for alignment |
| `public/webgpu_stacking.js` | GPU frame warping and accumulation |
| `public/gpu/shaders.js` | All WGSL compute shaders |

### Processing Modes

The mode affects template matching search radius and cut-off frame rejection:

| Mode | AP Search Radius | Cut-off Rejection | Use Case |
| --- | --- | --- | --- |
| Planet | 8 pixels | Yes | Jupiter, Saturn, Mars - small motion, reject frames where planet touches edge |
| Surface | 34 pixels | No | Moon, Sun closeups - larger drift between frames, no defined edge |

AP Search Radius = how far (in pixels) to search around each alignment point when looking for the best template match. Larger radius handles more frame-to-frame motion but is slower.

### GPU vs CPU Paths

┌─────────────────────────────────────────────────────────────────────────┐
│ WebGPU Available? │
└───────────────────────────────────┬─────────────────────────────────────┘
                                    │
                    ┌───────────────┴───────────────┐
                    │ │
                    ▼ ▼
           ┌───────────────┐ ┌───────────────┐
           │ GPU Path │ │ CPU Path │
           │ (primary) │ │ (fallback) │
           │ │ │ │
           │ webgpu\_\* │ │ unified\_ │
           │ workers │ │ analyze\_ │
           │ │ │ worker.js │
           │ Fast! │ │ OpenCV-WASM │
           └───────────────┘ └───────────────┘

### Bayer Pattern Reference

OpenCV uses inverted naming from industry standard:

| Industry (SER) | OpenCV | Layout |
| --- | --- | --- |
| RGGB | BayerBG | `R G G B` |
| BGGR | BayerRG | `B G G R` |
| GRBG | BayerGB | `G R B G` |
| GBRG | BayerGR | `G B R G` |

### Want to Contribute?

Check out the [GitHub repository](https://github.com/timing/eise.app). Key documentation files:

- `CLAUDE.md` - Architecture overview for AI assistants
- `PROCESSING_PIPELINE.md` - Detailed pipeline documentation
- `PIPELINE_OPTIMIZATION_ANALYSIS.md` - Performance optimization opportunities

