Edge Vision

#At a glance

#Lines of code by language

#Lines of code by area

Not counted: a cloned third-party SAM3 batch-inference repository, virtual environments and model weights.

#The problem

At the end of a progressive die line, an operator checks parts by eye. For the pilot part, a spring assembly, the defects are a missing spring or a protruding spring. Manual checks are slow and inconsistent, and an escape reaches the customer. The goal: a camera at the final station that calls good or defect on every part, with models a non-specialist can train for a new part from a phone.

 iPhone app ──► FastAPI server (RTX 5090) ──► trained model ──► ONNX ──► Hailo HEF ──► Pi 5 at the press
   capture         train · validate · scan                          quantize + compile      live GOOD/BAD overlay
                         │
                         └── Plex ERP: workcenters and job schedule

#What I built

#Server and review dashboard (FastAPI)

• 28 endpoints for part profiles, training image and video upload, training, validation, scanning (single image, burst or video), scan history and review, per-camera inspection and GPU status.
• A web review dashboard shows the raw scan, the normalized crop, an explanation heatmap and the nearest known-good reference. One button adds a scan to the good set.
• An optional second opinion from a local vision-language model (Qwen through LM Studio).
• A per-part model registry with manifests and a validation gate (95 % good pass and 95 % defect catch) before a model can be published.

#Mobile capture app (Expo / React Native)

• Turns an iPhone into a capture and scan client: create a part, upload training photos or video, start training, run validation, scan parts, tag defect reasons and pick a workcenter.
• Reaches the server from the shop floor through secure tunnels.

#Inference backends (pluggable)

• Golden master: pixel difference against a reference image.
• Multi-view reference: nearest-good-view matching on blurred low-resolution features, optionally with ResNet18 or ConvNeXt-Large classifiers for good/defect and defect reason. Explanation maps come from test-time augmentation and occlusion/gradient methods.
• YOLO: Ultralytics detection or segmentation models used as condition classifiers.
• Vision-language model: a fine-tuned Gemma model that answers in structured JSON.

#Model training

• YOLO: YOLO26 segmentation and detection (x and n sizes) at up to 1536 px on the workstation, and YOLOv8n for the edge.
• Gemma VLM fine-tune: LoRA (PEFT) on gemma-4-E2B-it, with JSON-output targets, a dataset exporter and an evaluator.
• Labeling tools: a ~1,870-line manual labeling tool, plus automatic video-frame labeling with a local Qwen VLM.
• Nightly training pipeline: reads a job queue on a network share and groups jobs by building, workcenter and part. It trains, exports to ONNX, validates, optionally compiles for Hailo, and publishes atomically to versioned archive/<version> and latest folders, with push notifications. Unit tests cover it.

#Edge deployment (Raspberry Pi 5 + Hailo-10H)

• Live inspection: Picamera2 feeding the Hailo accelerator, with a live GOOD PART / BAD PART / SEARCHING overlay, a custom decoder for raw YOLO outputs, and a HailoRT-NMS path. A CPU fallback and a static-image validator are included.
• Model compile: PyTorch → ONNX (opset 11) → Hailo Dataflow Compiler / Model Zoo → HEF, built in WSL with calibration sets.
• Compiler workaround: the compiler can't handle the TopK, GatherElements and Mod ops at the end of YOLO, so the graph is cut at the raw head convolutions and later at the final transpose, and decoding moves into custom code.
• Parity gate: compares PyTorch and ONNX results on the validation set before anything ships to the edge.

#ERP integration

• Calls the Plex Connect REST API (production and scheduling) to list workcenters and their last, current and next jobs. The server exposes this as /workcenters and /workcenters/{id}/schedule, with a 5-minute cache.
• The app uses it to tie each scan to a workcenter and job.
• Posting inspection results back to the ERP is the next planned step.

#Results

Main test part: the spring assembly, judged pass/fail on a validation set of ~709–758 images.

• Optimization: with the same model, per-image latency dropped from ~4.1 s to ~139 ms, about 30× faster, for a 0.5-point accuracy cost.
• Lesson on metrics: detection mAP50 (~0.70–0.74 for every YOLO variant) said little about pass/fail accuracy, which ranged from 71 % to 92 %. The app's own pass/fail gate turned out to be the metric that matters.
• Caveat: the Gemma checkpoint was chosen on the same validation split, so its figures are optimistic until tested on a held-out set.

#Status

• Working: the server, mobile capture → train → scan loop, review dashboard, YOLO and Gemma training on the workstation, Plex workcenter lookup and the per-part model registry.
• Experimental:
  • Hailo on the Pi: the quantized edge model did not yet hold its accuracy after compilation; the HailoRT-NMS build is the current candidate.
  • Nightly pipeline: built and unit-tested, not yet run in production.
  • Gemma: not yet deployed to the edge.
  • Validation gate: no model has passed the 95 % / 95 % gate yet.
• Planned: a hardware trigger (GPIO / PLC) and posting inspection results to the ERP.
• Overall: a working pilot with a clear path to the press.