From 151d7461d34a48d4fda504b0373cbf6a7608a39f Mon Sep 17 00:00:00 2001 From: Floppyrj45 Date: Sun, 26 Apr 2026 01:21:43 +0200 Subject: [PATCH] docs: plan + spec finition cosma-nav (superpowers) --- .../plans/2026-04-24-cosma-qc-platform.md | 866 ++++++++++++++++++ .../2026-04-24-cosma-qc-platform-design.md | 90 ++ 2 files changed, 956 insertions(+) create mode 100644 docs/superpowers/plans/2026-04-24-cosma-qc-platform.md create mode 100644 docs/superpowers/specs/2026-04-24-cosma-qc-platform-design.md diff --git a/docs/superpowers/plans/2026-04-24-cosma-qc-platform.md b/docs/superpowers/plans/2026-04-24-cosma-qc-platform.md new file mode 100644 index 0000000..ff1a798 --- /dev/null +++ b/docs/superpowers/plans/2026-04-24-cosma-qc-platform.md @@ -0,0 +1,866 @@ +# COSMA QC Platform Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Déployer cosma-nav comme QC viewer complet (carte GPS/USBL, viewer 3D, graphes nav) sur cosma-vm (.83), avec archivage automatique des données sur NAS .156 après chaque job. + +**Architecture:** cosma-qc (pipeline) et cosma-nav (viewer) tournent en parallèle sur cosma-vm, derrière Caddy. Le dispatcher cosma-qc déclenche post-job : pré-décimation PLY sur ml-stack → SCP vers cosma-vm, rsync frames → NAS .156. + +**Tech Stack:** Flask, Leaflet.js, Chart.js, Three.js, open3d, h5py, CIFS mount, Caddy + +--- + +## Fichiers créés / modifiés + +| Fichier | Action | Rôle | +|---------|--------|------| +| `/etc/fstab` (ml-stack .84) | Modify | Mount NAS .156 via CIFS | +| `/etc/fstab` (cosma-vm .83) | Modify | Mount NAS .156 via CIFS | +| `/root/cosma-nav/scripts/pre_decimate.py` | Create | Décimation PLY + SCP vers cosma-vm | +| `/root/cosma-nav/scripts/archive_job.sh` | Create | rsync frames+PLY brut → NAS | +| `/root/cosma-nav/viz/server.py` | Modify | Ajouter routes /map, /nav, /api/job/{id}/nav, /api/job/{id}/ply-status | +| `/root/cosma-nav/viz/templates/map.html` | Create | Page carte Leaflet | +| `/root/cosma-nav/viz/templates/nav.html` | Create | Page graphes nav | +| `/root/cosma-nav/viz/static/js/map.js` | Create | Leaflet init + tracks + USBL markers | +| `/root/cosma-nav/viz/static/js/nav_charts.js` | Create | Chart.js depth/altitude/RTK | +| `/home/cosma/cosma-qc/scripts/dispatcher.py` | Modify | Ajouter step post_job() après done | +| `/home/cosma/cosma-qc/app/templates/_jobs_table.html` | Modify | Bouton "QC →" par job done | +| `/home/cosma/cosma-qc/app/main.py` | Modify | Route /jobs/{id}/qc-redirect | +| `/etc/caddy/Caddyfile` (cosma-vm) | Create | Reverse proxy :80 → cosma-qc + cosma-nav | +| `/home/cosma/cosma-nav/` | Create | Clone repo cosma-nav sur cosma-vm | + +--- + +## Task 1 : Montage NAS .156 sur ml-stack + +**Files:** +- Modify: `/etc/fstab` sur ml-stack (.84) +- Create: `/mnt/nas-cosma/` directory + +- [ ] **Step 1.1 : Créer le point de montage sur ml-stack** + +```bash +ssh root@192.168.0.84 +mkdir -p /mnt/nas-cosma +``` + +- [ ] **Step 1.2 : Installer cifs-utils si absent** + +```bash +apt-get install -y cifs-utils +``` + +- [ ] **Step 1.3 : Créer le fichier credentials NAS** + +```bash +cat > /root/.nas-credentials << 'EOF' +username=admin +password=vj']C9yJA-jYt)U +EOF +chmod 600 /root/.nas-credentials +``` + +- [ ] **Step 1.4 : Ajouter le mount dans /etc/fstab** + +```bash +echo '//192.168.0.156/Public /mnt/nas-cosma cifs credentials=/root/.nas-credentials,iocharset=utf8,uid=root,gid=root,_netdev,nofail 0 0' >> /etc/fstab +``` + +- [ ] **Step 1.5 : Monter et vérifier** + +```bash +mount /mnt/nas-cosma +df -h /mnt/nas-cosma +# Attendu : ~1.3T dispo +mkdir -p /mnt/nas-cosma/cosma-archive +ls /mnt/nas-cosma/cosma-archive +``` + +--- + +## Task 2 : Montage NAS .156 sur cosma-vm + +**Files:** +- Modify: `/etc/fstab` sur cosma-vm (.83) +- Create: `/mnt/nas-cosma/` et `/data/cosma/` + +- [ ] **Step 2.1 : Sur cosma-vm** + +```bash +ssh cosma@192.168.0.83 +sudo apt-get install -y cifs-utils +sudo mkdir -p /mnt/nas-cosma /data/cosma +``` + +- [ ] **Step 2.2 : Credentials NAS** + +```bash +sudo bash -c "cat > /root/.nas-credentials << 'EOF' +username=admin +password=vj']C9yJA-jYt)U +EOF +chmod 600 /root/.nas-credentials" +``` + +- [ ] **Step 2.3 : fstab + mount** + +```bash +echo '//192.168.0.156/Public /mnt/nas-cosma cifs credentials=/root/.nas-credentials,iocharset=utf8,uid=1000,gid=1000,_netdev,nofail 0 0' | sudo tee -a /etc/fstab +sudo mount /mnt/nas-cosma +df -h /mnt/nas-cosma +``` + +- [ ] **Step 2.4 : Commit fstab configs** + +```bash +# Sur ta machine Windows, dans Nextcloud2/Projects/cosma-nav : +git add docs/ +git commit -m "docs: spec et plan déploiement COSMA QC Platform" +git push origin master +``` + +--- + +## Task 3 : Valider les PLY/poses existants (11 jobs) + +**Files:** +- Create: `/root/cosma-nav/scripts/check_jobs.py` sur ml-stack + +- [ ] **Step 3.1 : Créer le script de validation** + +Sur ml-stack, créer `/root/cosma-nav/scripts/check_jobs.py` : + +```python +#!/usr/bin/env python3 +"""Valide PLY + poses pour chaque job et affiche un rapport.""" +import os, glob, numpy as np + +FRAMES_DIR = "/root/cosma-qc-frames" + +for job_dir in sorted(glob.glob(f"{FRAMES_DIR}/job_*")): + job_id = os.path.basename(job_dir).replace("job_", "") + ply = f"{job_dir}/reconstruction.ply" + poses = f"{job_dir}/lingbot_poses.npz" + + ply_ok = os.path.exists(ply) and os.path.getsize(ply) > 1_000_000 + poses_ok = False + n_poses = 0 + if os.path.exists(poses): + try: + d = np.load(poses) + n_poses = d["poses"].shape[0] + poses_ok = d["poses"].shape[1:] == (3, 4) and n_poses > 10 + except Exception as e: + print(f" job_{job_id} poses ERROR: {e}") + + ply_size = os.path.getsize(ply) / 1e9 if ply_ok else 0 + status = "✓" if (ply_ok and poses_ok) else "✗" + print(f"{status} job_{job_id}: PLY {ply_size:.1f}GB {'OK' if ply_ok else 'MISSING'} | poses {n_poses} {'OK' if poses_ok else 'MISSING/BAD'}") +``` + +- [ ] **Step 3.2 : Exécuter sur ml-stack** + +```bash +ssh root@192.168.0.84 'cd /root/cosma-nav && python3 scripts/check_jobs.py' +``` + +Attendu : tous les jobs avec PLY > 1 GB et poses (N, 3, 4) marqués ✓. + +- [ ] **Step 3.3 : Noter les jobs problématiques** + +Si un job est ✗, noter son ID pour investigation séparée (ne bloque pas la suite). + +--- + +## Task 4 : Script de pré-décimation PLY + +**Files:** +- Create: `/root/cosma-nav/scripts/pre_decimate.py` sur ml-stack + +- [ ] **Step 4.1 : Créer le script** + +```python +#!/usr/bin/env python3 +"""Pré-décime un PLY (185M pts → ~200k) et SCP vers cosma-vm.""" +import sys, os, argparse +import numpy as np +import open3d as o3d + +def decimate(ply_path: str, out_path: str, max_pts: int = 200_000): + pcd = o3d.io.read_point_cloud(ply_path) + n = len(pcd.points) + if n > max_pts: + pts = np.asarray(pcd.points) + vol = float(np.prod(pcd.get_max_bound() - pcd.get_min_bound())) + vox = max((vol / max_pts) ** (1/3), 0.02) + pcd = pcd.voxel_down_sample(vox) + o3d.io.write_point_cloud(out_path, pcd) + print(f"Décimé: {n} → {len(pcd.points)} pts → {out_path} ({os.path.getsize(out_path)/1e6:.1f} MB)") + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + ap.add_argument("job_id", type=int) + ap.add_argument("--frames-dir", default="/root/cosma-qc-frames") + ap.add_argument("--cosma-vm", default="cosma@192.168.0.83") + ap.add_argument("--dest-dir", default="/data/cosma") + args = ap.parse_args() + + job_dir = f"{args.frames_dir}/job_{args.job_id}" + ply_in = f"{job_dir}/reconstruction.ply" + ply_out = f"/tmp/job_{args.job_id}_decimated.ply" + poses = f"{job_dir}/lingbot_poses.npz" + + if not os.path.exists(ply_in): + print(f"PLY non trouvé: {ply_in}"); sys.exit(1) + + decimate(ply_in, ply_out) + + # SCP vers cosma-vm + os.system(f"scp {ply_out} {args.cosma_vm}:{args.dest_dir}/job_{args.job_id}_decimated.ply") + if os.path.exists(poses): + os.system(f"scp {poses} {args.cosma_vm}:{args.dest_dir}/job_{args.job_id}_poses.npz") + print(f"SCP job_{args.job_id} terminé → {args.cosma_vm}:{args.dest_dir}") +``` + +- [ ] **Step 4.2 : Tester sur job_17 (le plus léger)** + +```bash +ssh root@192.168.0.84 'cd /root/cosma-nav && python3 scripts/pre_decimate.py 17' +# Attendu: Décimé: 185339364 → ~200000 pts → /tmp/job_17_decimated.ply (~7 MB) +# Puis SCP OK +``` + +- [ ] **Step 4.3 : Vérifier sur cosma-vm** + +```bash +ssh cosma@192.168.0.83 'ls -lh /data/cosma/' +# Attendu: job_17_decimated.ply ~7MB, job_17_poses.npz ~68KB +``` + +- [ ] **Step 4.4 : Lancer sur tous les jobs existants** + +```bash +ssh root@192.168.0.84 'for i in 9 11 12 13 14 15 16 17 19 21; do + echo "=== job_$i ==="; cd /root/cosma-nav && python3 scripts/pre_decimate.py $i; done' +``` + +--- + +## Task 5 : Script d'archivage vers NAS + +**Files:** +- Create: `/root/cosma-nav/scripts/archive_job.sh` sur ml-stack + +- [ ] **Step 5.1 : Créer le script** + +```bash +cat > /root/cosma-nav/scripts/archive_job.sh << 'EOF' +#!/bin/bash +# Archive frames + PLY brut d'un job vers NAS .156 +JOB_ID=$1 +FRAMES_DIR=/root/cosma-qc-frames +NAS_DIR=/mnt/nas-cosma/cosma-archive + +if [ -z "$JOB_ID" ]; then echo "Usage: $0 "; exit 1; fi + +SRC="$FRAMES_DIR/job_$JOB_ID" +DST="$NAS_DIR/job_$JOB_ID" + +if [ ! -d "$SRC" ]; then echo "Job dir not found: $SRC"; exit 1; fi + +mkdir -p "$DST" +echo "[$(date)] Archivage job_$JOB_ID vers NAS..." + +# rsync frames JPG (exclut PLY et poses) +rsync -av --progress "$SRC/" "$DST/" \ + --include="frame_*.jpg" \ + --include="reconstruction.ply" \ + --include="lingbot_poses.npz" \ + --exclude="*" \ + 2>&1 | tail -5 + +echo "[$(date)] Archivage job_$JOB_ID terminé: $DST" +EOF +chmod +x /root/cosma-nav/scripts/archive_job.sh +``` + +- [ ] **Step 5.2 : Tester sur job_17** + +```bash +ssh root@192.168.0.84 '/root/cosma-nav/scripts/archive_job.sh 17' +# Attendu: rsync OK, fichiers dans /mnt/nas-cosma/cosma-archive/job_17/ +ls /mnt/nas-cosma/cosma-archive/job_17/ | head -5 +``` + +- [ ] **Step 5.3 : Archiver tous les jobs existants (background)** + +```bash +ssh root@192.168.0.84 'for i in 9 11 12 13 14 15 16 17 19 21; do + nohup /root/cosma-nav/scripts/archive_job.sh $i >> /tmp/archive.log 2>&1 & +done; echo "Archivages lancés en background"' +``` + +--- + +## Task 6 : Déployer cosma-nav sur cosma-vm + +**Files:** +- Create: `/home/cosma/cosma-nav/` sur cosma-vm (clone git) +- Create: `/home/cosma/cosma-nav/.venv/` + +- [ ] **Step 6.1 : Cloner le repo** + +```bash +ssh cosma@192.168.0.83 +cd /home/cosma +git clone http://floppyrj45:67e1615fcfd06cf2df7872ac25e824f3afdb2bc1@192.168.0.82:3000/floppyrj45/cosma-nav.git +cd cosma-nav +``` + +- [ ] **Step 6.2 : Créer le venv et installer les dépendances** + +```bash +python3 -m venv .venv +source .venv/bin/activate +pip install flask h5py numpy open3d +# open3d peut prendre 2-3 min +python3 -c "import open3d; print(open3d.__version__)" +# Attendu: 0.19.0 (ou supérieur) +``` + +- [ ] **Step 6.3 : Créer le dossier de données** + +```bash +sudo mkdir -p /data/cosma +sudo chown cosma:cosma /data/cosma +ls /data/cosma/ +# Attendu: job_*_decimated.ply et job_*_poses.npz déjà là (Task 4) +``` + +- [ ] **Step 6.4 : Tester le viewer existant** + +```bash +cd /home/cosma/cosma-nav +source .venv/bin/activate +python3 viz/server.py --trajectory /data/cosma/job_17_decimated.ply --port 5051 & +sleep 3 && curl -s -o /dev/null -w "%{http_code}" http://localhost:5051/trajectory +# Attendu: 200 +kill %1 +``` + +--- + +## Task 7 : Étendre cosma-nav — routes map et nav + +**Files:** +- Modify: `/home/cosma/cosma-nav/viz/server.py` +- Create: `/home/cosma/cosma-nav/viz/templates/map.html` +- Create: `/home/cosma/cosma-nav/viz/templates/nav.html` +- Create: `/home/cosma/cosma-nav/viz/static/js/map.js` +- Create: `/home/cosma/cosma-nav/viz/static/js/nav_charts.js` + +- [ ] **Step 7.1 : Ajouter les routes dans server.py** + +Ajouter après les routes existantes dans `viz/server.py` : + +```python +import json +from pathlib import Path + +DATA_DIR = Path(os.environ.get("COSMA_DATA_DIR", "/data/cosma")) + +def _load_nav_data(job_id: int) -> dict: + """Charge poses + PLY décimé pour un job.""" + out = {"job_id": job_id} + poses_path = DATA_DIR / f"job_{job_id}_poses.npz" + ply_path = DATA_DIR / f"job_{job_id}_decimated.ply" + if poses_path.exists(): + d = np.load(str(poses_path)) + poses = d["poses"] # (N, 3, 4) + t_ns = d.get("timestamps_ns", np.zeros(len(poses), dtype=np.int64)) + # Positions caméra : colonne de translation + xyz = poses[:, :3, 3] + out["track"] = {"x": xyz[:, 0].tolist(), "y": xyz[:, 1].tolist(), "z": xyz[:, 2].tolist()} + out["n_poses"] = len(poses) + out["ply_ready"] = ply_path.exists() + out["ply_path"] = str(ply_path) if ply_path.exists() else None + return out + +@app.route("/map") +def map_view(): + return render_template("map.html") + +@app.route("/nav") +def nav_view(): + return render_template("nav.html") + +@app.route("/api/jobs") +def api_jobs(): + """Liste les jobs disponibles (PLY décimé présent).""" + jobs = [] + for p in sorted(DATA_DIR.glob("job_*_decimated.ply")): + jid = int(p.stem.split("_")[1]) + jobs.append({"id": jid, "ply": str(p), "poses": str(DATA_DIR / f"job_{jid}_poses.npz")}) + return jsonify(jobs) + +@app.route("/api/job//nav") +def api_job_nav(job_id: int): + return jsonify(_load_nav_data(job_id)) + +@app.route("/api/job//ply") +def api_job_ply(job_id: int): + """Sert le PLY décimé sous forme x/y/z arrays pour Three.js.""" + ply_path = DATA_DIR / f"job_{job_id}_decimated.ply" + if not ply_path.exists(): + return jsonify({"error": "PLY non disponible"}), 404 + import open3d as o3d + pcd = o3d.io.read_point_cloud(str(ply_path)) + pts = np.asarray(pcd.points) + return jsonify({"x": pts[:,0].tolist(), "y": pts[:,1].tolist(), "z": pts[:,2].tolist(), "n": len(pts)}) +``` + +- [ ] **Step 7.2 : Créer map.html** + +Créer `/home/cosma/cosma-nav/viz/templates/map.html` : + +```html + + + + + COSMA NAV — Carte + + + + + + + +
+

COSMA NAV — Carte GPS

+ + 3D → + Graphes → +
+
+ + + +``` + +- [ ] **Step 7.3 : Créer nav.html** + +Créer `/home/cosma/cosma-nav/viz/templates/nav.html` : + +```html + + + + + COSMA NAV — Données nav + + + + + +
+

COSMA NAV — Données navigation

+ + Carte → + 3D → +
+
+ + +
+ + + +``` + +- [ ] **Step 7.4 : Créer map.js** + +Créer `/home/cosma/cosma-nav/viz/static/js/map.js` : + +```javascript +const map = L.map('map').setView([43.17, 5.70], 13); +L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', { + attribution: '© OSM contributors', maxZoom: 19 +}).addTo(map); + +let trackLayer = null; + +async function loadJobs() { + const res = await fetch('/api/jobs'); + const jobs = await res.json(); + const sel = document.getElementById('job-select'); + jobs.forEach(j => { + const opt = document.createElement('option'); + opt.value = j.id; opt.textContent = `job_${j.id}`; + sel.appendChild(opt); + }); + if (jobs.length > 0) { sel.value = jobs[0].id; loadJob(jobs[0].id); } +} + +async function loadJob(jobId) { + const res = await fetch(`/api/job/${jobId}/nav`); + const d = await res.json(); + if (trackLayer) map.removeLayer(trackLayer); + if (!d.track) return; + // Coordonnées locales lingbot → affichage relatif (pas de géoréf) + // Si trajectory_world.h5 dispo → utiliser easting/northing + const pts = d.track.x.map((x, i) => [43.17 + d.track.y[i] * 0.00001, 5.70 + x * 0.00001]); + trackLayer = L.polyline(pts, { color: '#4ade80', weight: 2 }).addTo(map); + map.fitBounds(trackLayer.getBounds()); +} + +document.getElementById('job-select').addEventListener('change', e => { + if (e.target.value) loadJob(parseInt(e.target.value)); +}); + +loadJobs(); +``` + +> **Note:** Les coordonnées lingbot sont locales (non géoréférencées). La carte affiche le track relatif jusqu'à ce que `fuse_trajectory.py` produise un `trajectory_world.h5` avec easting/northing réels (logs La Ciotat). Une fois les données USBL disponibles, mettre à jour `loadJob()` pour appeler `/api/job/{id}/trajectory` avec les vraies coordonnées UTM. + +- [ ] **Step 7.5 : Créer nav_charts.js** + +Créer `/home/cosma/cosma-nav/viz/static/js/nav_charts.js` : + +```javascript +let chartXY = null, chartDepth = null; + +function initCharts() { + chartXY = new Chart(document.getElementById('chart-track-xy'), { + type: 'scatter', + data: { datasets: [{ label: 'Track XY lingbot', data: [], borderColor: '#4ade80', pointRadius: 1 }] }, + options: { responsive: true, plugins: { title: { display: true, text: 'Track XY (m, local)', color: '#ccc' } }, + scales: { x: { ticks: { color: '#888' } }, y: { ticks: { color: '#888' } } } } + }); + chartDepth = new Chart(document.getElementById('chart-depth'), { + type: 'line', + data: { datasets: [{ label: 'Z lingbot (profondeur approx)', data: [], borderColor: '#60a5fa', pointRadius: 0, borderWidth: 1 }] }, + options: { responsive: true, plugins: { title: { display: true, text: 'Z caméra (m, local)', color: '#ccc' } }, + scales: { x: { ticks: { color: '#888' } }, y: { reverse: true, ticks: { color: '#888' } } } } + }); +} + +async function loadJobs() { + const res = await fetch('/api/jobs'); + const jobs = await res.json(); + const sel = document.getElementById('job-select'); + jobs.forEach(j => { + const opt = document.createElement('option'); + opt.value = j.id; opt.textContent = `job_${j.id} (${j.id})`; + sel.appendChild(opt); + }); + if (jobs.length > 0) { sel.value = jobs[0].id; loadJob(jobs[0].id); } +} + +async function loadJob(jobId) { + const res = await fetch(`/api/job/${jobId}/nav`); + const d = await res.json(); + if (!d.track) return; + const n = d.track.x.length; + chartXY.data.datasets[0].data = d.track.x.map((x, i) => ({ x, y: d.track.y[i] })); + chartXY.update(); + chartDepth.data.datasets[0].data = d.track.z.map((z, i) => ({ x: i, y: z })); + chartDepth.update(); +} + +document.getElementById('job-select').addEventListener('change', e => { + if (e.target.value) loadJob(parseInt(e.target.value)); +}); + +initCharts(); +loadJobs(); +``` + +- [ ] **Step 7.6 : Tester les nouvelles routes** + +```bash +ssh cosma@192.168.0.83 'cd /home/cosma/cosma-nav && source .venv/bin/activate && \ + COSMA_DATA_DIR=/data/cosma python3 viz/server.py --port 5051 &' +sleep 3 +curl -s http://192.168.0.83:5051/api/jobs | python3 -c "import sys,json; print(json.load(sys.stdin))" +curl -s http://192.168.0.83:5051/api/job/17/nav | python3 -c "import sys,json; d=json.load(sys.stdin); print('poses:', d.get('n_poses'), 'ply:', d.get('ply_ready'))" +# Attendu: n_poses 1217, ply_ready True +``` + +- [ ] **Step 7.7 : Commit** + +```bash +cd /root/cosma-nav # sur ml-stack +git add viz/server.py viz/templates/map.html viz/templates/nav.html \ + viz/static/js/map.js viz/static/js/nav_charts.js \ + scripts/pre_decimate.py scripts/archive_job.sh scripts/check_jobs.py +git -c user.email="floppyrj45@gitea" -c user.name="Floppyrj45" commit -m "feat: routes map/nav + scripts pré-décimation + archivage NAS" +TOKEN=67e1615fcfd06cf2df7872ac25e824f3afdb2bc1 +git push http://floppyrj45:${TOKEN}@192.168.0.82:3000/floppyrj45/cosma-nav.git master +``` + +--- + +## Task 8 : Systemd service cosma-nav sur cosma-vm + +**Files:** +- Create: `/etc/systemd/system/cosma-nav.service` + +- [ ] **Step 8.1 : Créer le service** + +```bash +ssh cosma@192.168.0.83 +sudo tee /etc/systemd/system/cosma-nav.service << 'EOF' +[Unit] +Description=COSMA NAV QC Viewer +After=network.target + +[Service] +User=cosma +WorkingDirectory=/home/cosma/cosma-nav +Environment=COSMA_DATA_DIR=/data/cosma +ExecStart=/home/cosma/cosma-nav/.venv/bin/python3 viz/server.py --port 5051 +Restart=on-failure +RestartSec=5 + +[Install] +WantedBy=multi-user.target +EOF + +sudo systemctl daemon-reload +sudo systemctl enable cosma-nav +sudo systemctl start cosma-nav +sudo systemctl status cosma-nav --no-pager +``` + +- [ ] **Step 8.2 : Vérifier** + +```bash +curl -s -o /dev/null -w "%{http_code}" http://192.168.0.83:5051/map +# Attendu: 200 +``` + +--- + +## Task 9 : Caddy reverse proxy + +**Files:** +- Create: `/etc/caddy/Caddyfile` sur cosma-vm + +- [ ] **Step 9.1 : Installer Caddy** + +```bash +ssh cosma@192.168.0.83 +sudo apt-get install -y debian-keyring debian-archive-keyring apt-transport-https +curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg +curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list +sudo apt-get update && sudo apt-get install -y caddy +``` + +- [ ] **Step 9.2 : Configurer le Caddyfile** + +```bash +sudo tee /etc/caddy/Caddyfile << 'EOF' +:80 { + # cosma-qc pipeline dashboard (racine) + handle /nav/* { + reverse_proxy localhost:5051 + } + handle /nav { + reverse_proxy localhost:5051 + } + handle { + reverse_proxy localhost:3849 + } +} +EOF +``` + +- [ ] **Step 9.3 : Démarrer Caddy** + +```bash +sudo systemctl reload caddy || sudo systemctl start caddy +sudo systemctl enable caddy +sleep 2 +curl -s -o /dev/null -w "/ → %{http_code}\n" http://192.168.0.83/ +curl -s -o /dev/null -w "/nav → %{http_code}\n" http://192.168.0.83/nav +# Attendu: / → 200, /nav → 200 (redirect vers /nav/) +``` + +> **Note:** Le Caddy écoute sur :80. cosma-qc reste aussi accessible directement sur :3849. cosma-nav reste aussi sur :5051 pour le viewer 3D. + +--- + +## Task 10 : Bouton "QC →" dans cosma-qc dashboard + +**Files:** +- Modify: `/home/cosma/cosma-qc/app/templates/_jobs_table.html` +- Modify: `/home/cosma/cosma-qc/app/main.py` + +- [ ] **Step 10.1 : Ajouter la route redirect dans main.py** + +Ajouter dans `/home/cosma/cosma-qc/app/main.py` après les routes existantes : + +```python +from fastapi.responses import RedirectResponse + +@app.get("/jobs/{job_id}/qc") +async def qc_redirect(job_id: int): + """Redirige vers cosma-nav viewer pour ce job.""" + return RedirectResponse(url=f"http://192.168.0.83:5051/trajectory?job={job_id}", status_code=302) +``` + +- [ ] **Step 10.2 : Modifier le template _jobs_table.html** + +Dans le bloc `{% elif j.status == 'done' %}` de `col-progress`, remplacer : + +```html +{% if j.viser_url %} + viser ↗ +{% else %} + +{% endif %} +{% if j.ply_url %} + PLY ↓ +{% endif %} +``` + +par : + +```html +QC ↗ + +{% if j.ply_url %} + PLY ↓ +{% endif %} +``` + +- [ ] **Step 10.3 : Ajouter le style btn-qc dans style.css** + +```bash +ssh cosma@192.168.0.83 "cat >> /home/cosma/cosma-qc/app/static/style.css << 'EOF' + +.btn-qc { background: #1a2a3a; color: #60a5fa; border: 1px solid #60a5fa; border-radius: 3px; padding: 2px 8px; text-decoration: none; font-size: 0.8rem; } +.btn-qc:hover { background: #60a5fa; color: #0a1020; } +.viewer-btn { background: #1a3a2a; color: #4ade80; border: 1px solid #4ade80; border-radius: 3px; padding: 2px 8px; cursor: pointer; font-size: 0.8rem; } +.viewer-btn:hover { background: #4ade80; color: #0a1a10; } +EOF" +``` + +- [ ] **Step 10.4 : Rebuild Docker cosma-qc** + +```bash +ssh cosma@192.168.0.83 'cd /home/cosma/cosma-qc && docker compose build --no-cache && docker compose up -d' +sleep 5 +curl -s -o /dev/null -w "%{http_code}" http://192.168.0.83:3849/ +# Attendu: 200 +``` + +--- + +## Task 11 : Post-job hook dans le dispatcher cosma-qc + +**Files:** +- Modify: `/home/cosma/cosma-qc/scripts/dispatcher.py` + +- [ ] **Step 11.1 : Ajouter la fonction post_job()** + +Dans `/home/cosma/cosma-qc/scripts/dispatcher.py`, ajouter après `_maybe_create_per_auv_stitch()` : + +```python +def post_job_sync(job_id: int, worker: dict, frames_dir: str) -> None: + """Pré-décime PLY + SCP vers cosma-vm + archivage NAS (non-bloquant).""" + import threading + + def _run(): + try: + # 1. Pré-décimation + SCP vers cosma-vm + rc, out, err = ssh( + worker["ssh_alias"], + f"cd /root/cosma-nav && python3 scripts/pre_decimate.py {job_id}", + timeout=600 + ) + if rc != 0: + print(f"[post_job] pre_decimate job_{job_id} failed: {err[:200]}") + return + print(f"[post_job] pre_decimate job_{job_id} OK") + + # 2. Archivage NAS (background) + ssh( + worker["ssh_alias"], + f"nohup /root/cosma-nav/scripts/archive_job.sh {job_id} >> /tmp/archive_{job_id}.log 2>&1 &", + timeout=10 + ) + print(f"[post_job] archive job_{job_id} lancé en background") + except Exception as e: + print(f"[post_job] ERROR job_{job_id}: {e}") + + threading.Thread(target=_run, daemon=True).start() +``` + +- [ ] **Step 11.2 : Appeler post_job_sync() après chaque job done** + +Dans `dispatcher.py`, trouver le bloc qui marque un job comme done (après `set_status(job["id"], status="done", ...)`). Ajouter : + +```python +# Après set_status done : +post_job_sync(job["id"], worker, frames_dir) +_maybe_create_per_auv_stitch(job["id"]) +``` + +- [ ] **Step 11.3 : Redémarrer le dispatcher** + +```bash +ssh cosma@192.168.0.83 'sudo systemctl restart cosma-qc-dispatcher && \ + sudo systemctl status cosma-qc-dispatcher --no-pager | head -5' +# Attendu: active (running) +``` + +--- + +## Task 12 : Vérification finale end-to-end + +- [ ] **Step 12.1 : Dashboard cosma-qc accessible** + +``` +http://192.168.0.83/ → cosma-qc pipeline dashboard (via Caddy) +http://192.168.0.83:3849/ → cosma-qc direct +``` + +- [ ] **Step 12.2 : QC viewer accessible** + +``` +http://192.168.0.83/nav/trajectory → Three.js viewer +http://192.168.0.83/nav/map → Carte Leaflet +http://192.168.0.83/nav/nav → Graphes nav +http://192.168.0.83:5051/trajectory → cosma-nav direct +``` + +- [ ] **Step 12.3 : Cliquer bouton "QC →" dans le dashboard** + +Ouvrir `http://192.168.0.83/`, choisir un job done, cliquer "QC ↗" → doit ouvrir le viewer Three.js sur ce job dans un nouvel onglet. + +- [ ] **Step 12.4 : Vérifier archivage NAS** + +```bash +ssh root@192.168.0.84 'ls -lh /mnt/nas-cosma/cosma-archive/ | head -15' +# Attendu: dossiers job_N avec frames + PLY brut +``` + +- [ ] **Step 12.5 : Commit final cosma-nav** + +```bash +cd /root/cosma-nav +git add -A +git -c user.email="floppyrj45@gitea" -c user.name="Floppyrj45" \ + commit -m "feat: déploiement cosma-vm — viewer map/nav/3D + post-job sync NAS" +TOKEN=67e1615fcfd06cf2df7872ac25e824f3afdb2bc1 +git push http://floppyrj45:${TOKEN}@192.168.0.82:3000/floppyrj45/cosma-nav.git master +``` diff --git a/docs/superpowers/specs/2026-04-24-cosma-qc-platform-design.md b/docs/superpowers/specs/2026-04-24-cosma-qc-platform-design.md new file mode 100644 index 0000000..c8fe28c --- /dev/null +++ b/docs/superpowers/specs/2026-04-24-cosma-qc-platform-design.md @@ -0,0 +1,90 @@ +# COSMA QC Platform — Design Spec +**Date:** 2026-04-24 +**Statut:** Approuvé + +## Objectif + +Plateforme QC complète pour les missions COSMA AUV sur cosma-vm (.83), combinant gestion pipeline (cosma-qc) et visualisation QC (cosma-nav) : carte GPS/USBL, viewer 3D reconstruction, graphes nav. + +## Architecture — Option B : Microservices + +### Services + +| Service | Port | Rôle | +|---------|------|------| +| cosma-qc | :3849 | Pipeline jobs/stitches, dashboard extraction/reconstruction | +| cosma-nav | :5051 | QC viewer : carte, 3D, graphes nav | +| Caddy | :80 | Reverse proxy → / (cosma-qc) et /nav/ (cosma-nav) | + +### Machine cible +- **cosma-vm (.83)** — VM Proxmox +- Ressources actuelles : 4 vCPU / 3.8 GB RAM / 38 GB disk +- Expansion si nécessaire : +4 GB RAM si open3d décimatation locale + +### Stockage + +``` +ml-stack (.84) NAS .156 (1.34 TB libre) +/root/cosma-qc-frames/ ──► /cosma-archive/ + job_N/frame_*.jpg (65 GB) job_N/frames/ (archivé) + job_N/reconstruction.ply job_N/ply_brut/ (archivé) + +ml-stack post-job cosma-vm (.83) +pré-décimer PLY ──► /data/cosma/ + 185M pts → ~200k pts (~7 MB) job_N_decimated.ply + rsync poses.npz job_N_poses.npz + rsync sparse_fixes.h5 sparse_fixes.h5 +``` + +- Frames JPG et PLY bruts → archivés sur NAS .156 (SMB, admin) après job done +- Données légères → copiées sur cosma-vm pour le viewer +- Rien n'est supprimé avant confirmation archivage NAS + +## cosma-nav — Fonctionnalités QC Viewer + +### 1. Carte GPS/USBL (Leaflet + OpenStreetMap) +- Track AUV (trajectory_world.h5 → x_m/y_m converti en lat/lon) +- Fixes USBL/GPS plotés avec couleur RTK (fix=vert, float=jaune, 3D=rouge) +- Fixes USV (navire surface) en overlay +- Sélecteur de mission/job + +### 2. Viewer 3D reconstruction (Three.js — existant) +- PLY décimé (~200k pts) chargé depuis /data/cosma/ +- Frustums caméra orientés nadir +- Trajectoire lingbot locale +- Toggle layers (PLY, frustums, trajectory) + +### 3. Graphes nav (Plotly.js ou Chart.js) +- Profondeur (depth_m depuis pression) +- Altitude fond (altitude_m Kogger) +- Qualité RTK (fix_type timeline) +- Vitesse USV + +### 4. Intégration cosma-qc +- Bouton "QC →" dans la table jobs de cosma-qc → ouvre cosma-nav sur le job sélectionné +- API cosma-nav : GET /api/job/{id}/trajectory, /api/job/{id}/nav, /api/job/{id}/ply + +## Pipeline post-job (dispatcher cosma-qc) + +Après chaque job done : +1. `open3d` pré-décimation sur ml-stack → PLY ~7 MB +2. `rsync` PLY décimé + poses.npz → cosma-vm:/data/cosma/ +3. `rsync` frames JPG + PLY brut → NAS .156:/cosma-archive/ (async, non-bloquant) +4. Mise à jour DB cosma-qc : `decimated_ply_path`, `nav_ready=true` + +## Vérification PLY/poses existants + +Avant déploiement : valider les 11 jobs actuels (jobs 9-21) : +- PLY lisible par open3d ✓ (vérifié, 185M pts) +- poses.npz shape (N, 3, 4) ✓ (1217-1349 frames) +- PLY utile : early-save ≠ final-save → re-vérifier si demo.py a fini proprement + +## Déploiement + +1. Monter NAS .156 sur ml-stack et cosma-vm (CIFS) +2. Cloner cosma-nav sur cosma-vm +3. Installer deps (Flask, open3d, h5py) dans venv +4. Adapter server.py : servir depuis /data/cosma/, ajouter routes map + nav +5. Caddy config sur cosma-vm +6. Modifier dispatcher cosma-qc : ajouter step post-job (décimation + sync) +7. Ajouter bouton "QC →" dans _jobs_table.html cosma-qc