floppyrj45/cosma-qc
2
0
Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
81 Commits 8 Branches 11 Tags
15b4ddfd707b96d47d90446ff587528cea427c6d
Commit Graph

81 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ubuntu
15b4ddfd70 stage01: ajoute collectors MAG (cosmag csv ar/av/side) + SSS (Kogger klf/bin) 
Manifest 01 enrichi avec deux nouveaux champs:
- sss_files: {klf,bin} relatifs sous raw_data/sss/**
- mag_files: {ar,av,side,other} csv sous raw_data/mag/**

Tests:
- 20260505-Lepradet: sss=1 mag=0 (mag vide attendu)
- 20260508-sttropez: sss=53 mag=73 (match scout)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-15 09:53:01 +00:00
Ubuntu
65bda7ff71 stage02: filtre strict v5 (pct=80 dur=60 depth=-3m) + stage02b diag plots 
Defaults plus stricts pour éliminer surface/yoyo:
- min_near_bottom_pct: 50 -> 80 %
- min_sustained_duration: 30 -> 60 s
- min_mission_depth: -2 -> -3 m
- min_displacement_m: 5.0 documenté (stage06+ futur)

Nouveau script 02b_runs_diag.py: 4-panel PNG par run OK+rejected
(rel_alt+threshold, MAVROS state, depth histo, verdict criteria)
+ index.html pour inspection visuelle Flag.

Test Lepradet: 5 -> 1 run OK (AUV210_run_00 79s -13m 81pct)
Page publiée: laboratoire.freeboxos.fr/02-runs-diag-lepradet/
 2026-05-15 09:53:01 +00:00
Ubuntu
2858217897 stage02: filtre pre-water runs avant première submersion réelle 
- find_first_submersion_epoch(): premier rel_alt < -2m pendant >= 5s continu
- detect_runs_state_based(): rejette runs avant first_sub_epoch, tronque chevauchants
- CLI: --first-submersion-depth (default -2.0) + --first-submersion-duration (default 5.0)
- JSON output: first_submersion_epoch + pre_water_rejected_count par AUV
- Lepradet: AUV212 run_00 tronqué 25s, AUV213 run_00 tronqué 11s
 2026-05-15 09:53:01 +00:00
Poulpe
568ff9469b auto-iter 2026-05-14: iteration-log iter10 — RoPE fix + PR#13 merge + .83 blocker 
Co-Authored-By: Poulpe <claude@nowyouknow.fr>
 2026-05-14 04:56:09 +00:00
Poulpe
2611a72aa2 auto-iter 2026-05-14: max_frame_num 1024→2048 fix RoPE overflow GX019817 
Root cause: 3D RoPE precomputed for max_frame_num+100=1124 positions.
GX019817 has 1357 frames after trim → index overflow → tensor mismatch.
2048 → supports up to 2148 frames (covers all current segments).

Co-Authored-By: Poulpe <claude@nowyouknow.fr>
 2026-05-14 04:52:20 +00:00
Poulpe (Gouvernance Gitea)
50ca77490d Merge pull request 'fix: 05_inference viser-kill + background-poll + offload_to_cpu from yaml' (#13) from fix/05-inference-viser-kill-offload into feature/auto-pipeline 2026-05-14 04:47:10 +00:00
Poulpe
503d6d64c2 iter-9: veille + stage06 path analysis 2026-05-13 23:03:19 +00:00
Poulpe
38dbcfd46f auto-iter 20260513-2231: GX019817 RoPE skip, 4 PLY done ready for stage06 auto-iter-20260513-2231 2026-05-13 23:02:31 +00:00
Poulpe
091ffeb2f6 chore: iter-8 log + veille (2026-05-13) 2026-05-13 16:44:18 +00:00
Poulpe
13323f2edf fix: 05_inference — kill stale demo.py + background poll exit viser + offload_to_cpu from yaml 
- kill_stale_demo_py() before each segment to prevent GPU contention from orphan processes
- Remote script runs demo.py in background via nohup, polls for PLY file every 30s, kills viser server once PLY written — prevents indefinite SSH block on viser listener
- offload_to_cpu now read from thresholds.yaml[inference] (default false for 24GB VRAM)
- timeout reads inference_timeout_s from yaml (already 10800s)
- min_frames guard included (from fix/05-inference-min-frames-timeout)

Root cause: demo.py starts viser server after writing PLY; SSH timed out → orphan; two orphans competed for GPU with offload_to_cpu → pure CPU inference = 6h+ for 493 frames
 2026-05-13 16:41:18 +00:00
Poulpe
c55700677e auto-iter 2026-05-13: offload_to_cpu=false (.84 24GB VRAM, no CPU offload needed) auto-iter-20260513-1639 2026-05-13 16:39:51 +00:00
Poulpe
ba92d68492 chore: iter-7 veille + log (2026-05-13) 2026-05-13 10:42:37 +00:00
Poulpe
c7c4431e72 auto-iter 2026-05-13: inference min_frames=32 + timeout 3h (was 2h) 
- min_frames_for_inference: 32 (RoPE/attention needs ≥32 frames)
- inference_timeout_s: 10800 (GX029818 timed out at 7200s with 493 frames)

Authored-by: Poulpe <claude@nowyouknow.fr>
auto-iter-20260513-1036 		2026-05-13 10:36:28 +00:00
Poulpe
1f1502e67c auto-iter 2026-05-12: log iter-5 + veille + merge PR#10 fix streaming params 2026-05-12 22:49:59 +00:00
Ubuntu
81752163d2 Merge branch 'fix/05-inference-yaml-params' into feature/auto-pipeline auto-iter-20260512-2246 2026-05-12 22:46:30 +00:00
Poulpe
c06dd774ac auto-iter 2026-05-12: log iter-4 + veille 2026-05-12 16:43:05 +00:00
Poulpe
3a6b058f0d fix: 05_inference.py lit thresholds.yaml[inference] au lieu de windowed hardcodé 
- Ajoute _load_inference_cfg() qui lit config/thresholds.yaml
- Mode/conf/keyframe_interval/max_frame_num depuis config (streaming par défaut)
- Valide par GX049839_v2: streaming+conf=1.5+kf=1 → 146M pts vs 0 pts en windowed sans conf_threshold
- Ajoute --offload_to_cpu (stable sur RTX 3090 .84)
 2026-05-12 16:38:33 +00:00
Poulpe
8880c28af9 auto-iter 2026-05-12: keyframe_interval 6→1 (streaming, validé GX049839_v2 146M pts) auto-iter-20260512-1637 2026-05-12 16:37:06 +00:00
Poulpe
df45fd155d auto-iter 2026-05-12: bottom_visible_pct_min 30→25 (GX019817 29% récupérable) auto-iter-20260512-1034 2026-05-12 10:34:19 +00:00
Poulpe
f0154d7ea5 auto-iter 2026-05-12: log iter-2 + veille auto-iter-20260512-0440-log 2026-05-12 04:40:56 +00:00
Poulpe
8b826b0827 auto-iter 2026-05-12: fix duplicate frame_extract key in thresholds.yaml auto-iter-20260512-0439 2026-05-12 04:39:03 +00:00
Poulpe
4f54d58cd3 auto-iter 2026-05-11: iteration-log + veille iter-1 2026-05-11 22:34:42 +00:00
Poulpe
06d4aa5d4d auto-iter 2026-05-11: bottom_visible_pct seuil 50→30 (avg=37.5%) auto-iter-20260511-2233 2026-05-11 22:33:12 +00:00
Poulpe
e09ef7886b feat(pipeline): stage 04b port trim_above_water from dispatcher 2026-05-11 14:08:30 +00:00
Ubuntu
82f71fcc96 feat: frame QC scoring + viser per-AUV button 
Stage 04 frame extract:
- New lib_frame_qc.py: per-frame Laplacian/contrast/blue-dominance scoring
- Classes: bottom_visible / water_no_bottom / turbid_water / out_of_water
- Sample 1/5 frames after extraction, write qc.json per segment
- Record metrics (frames_total, frames_bottom_visible, bottom_visible_pct)
- Mark job degraded when bottom_visible_pct < 50%

Per-AUV viser view:
- scripts/viser_auv.py loads all PLYs of an AUV, color per file
- POST /pipeline/missions/{id}/auvs/{auv}/view rsyncs ply -> worker
- launches viser on hashed port 9300+, returns URL
- _pipeline.html exposes AUV list, JS handler opens viser tab
 2026-05-11 11:05:37 +00:00
Ubuntu
1a4fffd2c1 feat: pipeline monitor + orchestrator stats dashboard 2026-05-11 10:55:44 +00:00
Ubuntu
e597407ee5 feat(pipeline): jalon 1-3 — ingest, USBL parse, filter 
Stages 01-03 opérationnels sur 20260505-Lepradet:
- 01_ingest: manifest auto, 3 AUVs vidéo, 3 AUVs bags, mapping AUV2xx↔AUV0xx
- 02_usbl_parse: MCAP (format incompatible firmware) → fallback serial CSV, 213 pts bruts
- 03_usbl_filter: MAD-3σ + moving-avg + Kalman optionnel, dégradé gracieux si null lat/lon
- orchestrator/db.py: SQLite schema missions/jobs/metrics idempotent
- config/: thresholds.yaml + default_params.yaml versionnés
- qa/checks.py: vérifications pass/fail/degraded par étape

Note: MCAP bags corrompus ou format non-standard firmware — lat/lon absent.
Statut degraded (pas crash). Nécessite investigation format MCAP spécifique.
 2026-05-11 10:25:27 +00:00
Ubuntu
18b3e0e8e9 fix: dispatcher passe frames_dir (job dir) pas parent a pre_decimate 2026-04-25 16:32:26 +00:00
Ubuntu
4db7b13bb0 feat: hook post-job cosma-nav + style dashboard + docker-compose update 2026-04-25 16:25:19 +00:00
Ubuntu
ffcb254fa8 fix: closes #10 — bouton QC pointe vers /nav?job=<id> 2026-04-25 16:25:03 +00:00
floppyrj45
0deddf0009 docs: create CLAUDE.md with Infrastructure section 2026-04-24 15:56:23 +00:00
Ubuntu
19a48abf9d feat: section évolutions récentes + pipeline workflow sur le dashboard 2026-04-24 09:28:13 +00:00
Ubuntu
8810d61540 fix: supprimer base href /cosma-qc/ — chemins absolus pour accès direct et via Caddy 2026-04-24 08:53:52 +00:00
Floppyrj45
76bba217dc feat: stitch.py --poses trajectory_world.h5 — T_init depuis poses monde, remplace RANSAC 2026-04-24 10:27:55 +02:00
Flag
df83454de6 fix: SCP utilise glob frame_*.jpg au lieu de -r (évite création sous-dossiers) 2026-04-24 08:22:02 +00:00
Flag
67643d05ff feat: dispatcher passe --save_poses à demo.py — génère lingbot_poses.npz (Plan 2) 2026-04-24 08:07:55 +00:00
Flag
4eb9f22813 feat: migration vers cosma-vm (.83) — dispatcher+dashboard, OpenVPN, docs infra mis à jour 2026-04-24 00:16:21 +00:00
Flag
c765e8cc40 fix: COPY docs/_build/html dans image Docker 2026-04-23 23:59:43 +00:00
Flag
7ed24b66b7 fix: race condition exit_file z620 — rm sync avant bg ffmpeg 2026-04-23 23:23:28 +00:00
Flag
352af149fd fix: viser_ply filtrage outliers statistiques — supprime gros pâtés bruités 2026-04-23 23:10:11 +00:00
Flag
fb38ff2192 feat: lien PLY téléchargement direct depuis dashboard (port 8300 worker) 2026-04-23 22:14:39 +00:00
Flag
325e5feb5f fix: SQLite busy_timeout + z620 SSH via worker relay 
- db(): timeout=30 + PRAGMA busy_timeout=5000 pour éviter locked sous charge parallèle
- _ssh_via(): helper routant SSH z620 via worker (.82→worker→z620)
- _extract_on_remote_host(): toutes les commandes z620 passent via worker
- resume probe: ffprobe z620 également via worker
- reset jobs 11/17/18/20 en queued pour relancer
 2026-04-23 22:03:37 +00:00
Flag
91b25f0aae feat: docs Sphinx pipeline + lien dashboard header 2026-04-23 21:17:49 +00:00
Flag
1bbb6c8e6d fix: stitch tourne sur le même worker que ses PLY (evite 404 cross-worker) 2026-04-23 21:09:59 +00:00
Flag
a505ec1bcd feat: resume extraction — markers par vidéo, pas de rm au départ, reprise au crash 2026-04-23 21:07:09 +00:00
Flag
42ba218f09 fix: extraction frames sur host distant (z620) sans SCP du MP4 entier 2026-04-23 21:02:05 +00:00
Flag
9e9eff6cc1 fix: bouton PLY — viewer-btn JS handler au lieu HTMX (evite JSON dans la page) 2026-04-23 20:52:25 +00:00
Flag
7f8c1d0d07 fix: segment_label end-time recalculé depuis video_duration_s (exiftool=0 à lingest) 2026-04-23 20:42:56 +00:00
Flag
71d82018b1 feat: dashboard — plongée label, serial visible, GLB link, is_file thumbnail fix 2026-04-23 20:33:46 +00:00
Flag
1b467c5f03 fix: cleanup stale demo.py + resolve_worker_video_source frames_dir + SSH keys repaired 2026-04-23 19:53:03 +00:00