docs: plan + spec finition cosma-nav (superpowers)

This commit is contained in:
Floppyrj45
2026-04-26 01:21:43 +02:00
parent 0b3f5722c2
commit 151d7461d3
2 changed files with 956 additions and 0 deletions

View File

@@ -0,0 +1,866 @@
# COSMA QC Platform Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Déployer cosma-nav comme QC viewer complet (carte GPS/USBL, viewer 3D, graphes nav) sur cosma-vm (.83), avec archivage automatique des données sur NAS .156 après chaque job.
**Architecture:** cosma-qc (pipeline) et cosma-nav (viewer) tournent en parallèle sur cosma-vm, derrière Caddy. Le dispatcher cosma-qc déclenche post-job : pré-décimation PLY sur ml-stack → SCP vers cosma-vm, rsync frames → NAS .156.
**Tech Stack:** Flask, Leaflet.js, Chart.js, Three.js, open3d, h5py, CIFS mount, Caddy
---
## Fichiers créés / modifiés
| Fichier | Action | Rôle |
|---------|--------|------|
| `/etc/fstab` (ml-stack .84) | Modify | Mount NAS .156 via CIFS |
| `/etc/fstab` (cosma-vm .83) | Modify | Mount NAS .156 via CIFS |
| `/root/cosma-nav/scripts/pre_decimate.py` | Create | Décimation PLY + SCP vers cosma-vm |
| `/root/cosma-nav/scripts/archive_job.sh` | Create | rsync frames+PLY brut → NAS |
| `/root/cosma-nav/viz/server.py` | Modify | Ajouter routes /map, /nav, /api/job/{id}/nav, /api/job/{id}/ply-status |
| `/root/cosma-nav/viz/templates/map.html` | Create | Page carte Leaflet |
| `/root/cosma-nav/viz/templates/nav.html` | Create | Page graphes nav |
| `/root/cosma-nav/viz/static/js/map.js` | Create | Leaflet init + tracks + USBL markers |
| `/root/cosma-nav/viz/static/js/nav_charts.js` | Create | Chart.js depth/altitude/RTK |
| `/home/cosma/cosma-qc/scripts/dispatcher.py` | Modify | Ajouter step post_job() après done |
| `/home/cosma/cosma-qc/app/templates/_jobs_table.html` | Modify | Bouton "QC →" par job done |
| `/home/cosma/cosma-qc/app/main.py` | Modify | Route /jobs/{id}/qc-redirect |
| `/etc/caddy/Caddyfile` (cosma-vm) | Create | Reverse proxy :80 → cosma-qc + cosma-nav |
| `/home/cosma/cosma-nav/` | Create | Clone repo cosma-nav sur cosma-vm |
---
## Task 1 : Montage NAS .156 sur ml-stack
**Files:**
- Modify: `/etc/fstab` sur ml-stack (.84)
- Create: `/mnt/nas-cosma/` directory
- [ ] **Step 1.1 : Créer le point de montage sur ml-stack**
```bash
ssh root@192.168.0.84
mkdir -p /mnt/nas-cosma
```
- [ ] **Step 1.2 : Installer cifs-utils si absent**
```bash
apt-get install -y cifs-utils
```
- [ ] **Step 1.3 : Créer le fichier credentials NAS**
```bash
cat > /root/.nas-credentials << 'EOF'
username=admin
password=vj']C9yJA-jYt)U
EOF
chmod 600 /root/.nas-credentials
```
- [ ] **Step 1.4 : Ajouter le mount dans /etc/fstab**
```bash
echo '//192.168.0.156/Public /mnt/nas-cosma cifs credentials=/root/.nas-credentials,iocharset=utf8,uid=root,gid=root,_netdev,nofail 0 0' >> /etc/fstab
```
- [ ] **Step 1.5 : Monter et vérifier**
```bash
mount /mnt/nas-cosma
df -h /mnt/nas-cosma
# Attendu : ~1.3T dispo
mkdir -p /mnt/nas-cosma/cosma-archive
ls /mnt/nas-cosma/cosma-archive
```
---
## Task 2 : Montage NAS .156 sur cosma-vm
**Files:**
- Modify: `/etc/fstab` sur cosma-vm (.83)
- Create: `/mnt/nas-cosma/` et `/data/cosma/`
- [ ] **Step 2.1 : Sur cosma-vm**
```bash
ssh cosma@192.168.0.83
sudo apt-get install -y cifs-utils
sudo mkdir -p /mnt/nas-cosma /data/cosma
```
- [ ] **Step 2.2 : Credentials NAS**
```bash
sudo bash -c "cat > /root/.nas-credentials << 'EOF'
username=admin
password=vj']C9yJA-jYt)U
EOF
chmod 600 /root/.nas-credentials"
```
- [ ] **Step 2.3 : fstab + mount**
```bash
echo '//192.168.0.156/Public /mnt/nas-cosma cifs credentials=/root/.nas-credentials,iocharset=utf8,uid=1000,gid=1000,_netdev,nofail 0 0' | sudo tee -a /etc/fstab
sudo mount /mnt/nas-cosma
df -h /mnt/nas-cosma
```
- [ ] **Step 2.4 : Commit fstab configs**
```bash
# Sur ta machine Windows, dans Nextcloud2/Projects/cosma-nav :
git add docs/
git commit -m "docs: spec et plan déploiement COSMA QC Platform"
git push origin master
```
---
## Task 3 : Valider les PLY/poses existants (11 jobs)
**Files:**
- Create: `/root/cosma-nav/scripts/check_jobs.py` sur ml-stack
- [ ] **Step 3.1 : Créer le script de validation**
Sur ml-stack, créer `/root/cosma-nav/scripts/check_jobs.py` :
```python
#!/usr/bin/env python3
"""Valide PLY + poses pour chaque job et affiche un rapport."""
import os, glob, numpy as np
FRAMES_DIR = "/root/cosma-qc-frames"
for job_dir in sorted(glob.glob(f"{FRAMES_DIR}/job_*")):
job_id = os.path.basename(job_dir).replace("job_", "")
ply = f"{job_dir}/reconstruction.ply"
poses = f"{job_dir}/lingbot_poses.npz"
ply_ok = os.path.exists(ply) and os.path.getsize(ply) > 1_000_000
poses_ok = False
n_poses = 0
if os.path.exists(poses):
try:
d = np.load(poses)
n_poses = d["poses"].shape[0]
poses_ok = d["poses"].shape[1:] == (3, 4) and n_poses > 10
except Exception as e:
print(f" job_{job_id} poses ERROR: {e}")
ply_size = os.path.getsize(ply) / 1e9 if ply_ok else 0
status = "" if (ply_ok and poses_ok) else ""
print(f"{status} job_{job_id}: PLY {ply_size:.1f}GB {'OK' if ply_ok else 'MISSING'} | poses {n_poses} {'OK' if poses_ok else 'MISSING/BAD'}")
```
- [ ] **Step 3.2 : Exécuter sur ml-stack**
```bash
ssh root@192.168.0.84 'cd /root/cosma-nav && python3 scripts/check_jobs.py'
```
Attendu : tous les jobs avec PLY > 1 GB et poses (N, 3, 4) marqués ✓.
- [ ] **Step 3.3 : Noter les jobs problématiques**
Si un job est ✗, noter son ID pour investigation séparée (ne bloque pas la suite).
---
## Task 4 : Script de pré-décimation PLY
**Files:**
- Create: `/root/cosma-nav/scripts/pre_decimate.py` sur ml-stack
- [ ] **Step 4.1 : Créer le script**
```python
#!/usr/bin/env python3
"""Pré-décime un PLY (185M pts → ~200k) et SCP vers cosma-vm."""
import sys, os, argparse
import numpy as np
import open3d as o3d
def decimate(ply_path: str, out_path: str, max_pts: int = 200_000):
pcd = o3d.io.read_point_cloud(ply_path)
n = len(pcd.points)
if n > max_pts:
pts = np.asarray(pcd.points)
vol = float(np.prod(pcd.get_max_bound() - pcd.get_min_bound()))
vox = max((vol / max_pts) ** (1/3), 0.02)
pcd = pcd.voxel_down_sample(vox)
o3d.io.write_point_cloud(out_path, pcd)
print(f"Décimé: {n}{len(pcd.points)} pts → {out_path} ({os.path.getsize(out_path)/1e6:.1f} MB)")
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("job_id", type=int)
ap.add_argument("--frames-dir", default="/root/cosma-qc-frames")
ap.add_argument("--cosma-vm", default="cosma@192.168.0.83")
ap.add_argument("--dest-dir", default="/data/cosma")
args = ap.parse_args()
job_dir = f"{args.frames_dir}/job_{args.job_id}"
ply_in = f"{job_dir}/reconstruction.ply"
ply_out = f"/tmp/job_{args.job_id}_decimated.ply"
poses = f"{job_dir}/lingbot_poses.npz"
if not os.path.exists(ply_in):
print(f"PLY non trouvé: {ply_in}"); sys.exit(1)
decimate(ply_in, ply_out)
# SCP vers cosma-vm
os.system(f"scp {ply_out} {args.cosma_vm}:{args.dest_dir}/job_{args.job_id}_decimated.ply")
if os.path.exists(poses):
os.system(f"scp {poses} {args.cosma_vm}:{args.dest_dir}/job_{args.job_id}_poses.npz")
print(f"SCP job_{args.job_id} terminé → {args.cosma_vm}:{args.dest_dir}")
```
- [ ] **Step 4.2 : Tester sur job_17 (le plus léger)**
```bash
ssh root@192.168.0.84 'cd /root/cosma-nav && python3 scripts/pre_decimate.py 17'
# Attendu: Décimé: 185339364 → ~200000 pts → /tmp/job_17_decimated.ply (~7 MB)
# Puis SCP OK
```
- [ ] **Step 4.3 : Vérifier sur cosma-vm**
```bash
ssh cosma@192.168.0.83 'ls -lh /data/cosma/'
# Attendu: job_17_decimated.ply ~7MB, job_17_poses.npz ~68KB
```
- [ ] **Step 4.4 : Lancer sur tous les jobs existants**
```bash
ssh root@192.168.0.84 'for i in 9 11 12 13 14 15 16 17 19 21; do
echo "=== job_$i ==="; cd /root/cosma-nav && python3 scripts/pre_decimate.py $i; done'
```
---
## Task 5 : Script d'archivage vers NAS
**Files:**
- Create: `/root/cosma-nav/scripts/archive_job.sh` sur ml-stack
- [ ] **Step 5.1 : Créer le script**
```bash
cat > /root/cosma-nav/scripts/archive_job.sh << 'EOF'
#!/bin/bash
# Archive frames + PLY brut d'un job vers NAS .156
JOB_ID=$1
FRAMES_DIR=/root/cosma-qc-frames
NAS_DIR=/mnt/nas-cosma/cosma-archive
if [ -z "$JOB_ID" ]; then echo "Usage: $0 <job_id>"; exit 1; fi
SRC="$FRAMES_DIR/job_$JOB_ID"
DST="$NAS_DIR/job_$JOB_ID"
if [ ! -d "$SRC" ]; then echo "Job dir not found: $SRC"; exit 1; fi
mkdir -p "$DST"
echo "[$(date)] Archivage job_$JOB_ID vers NAS..."
# rsync frames JPG (exclut PLY et poses)
rsync -av --progress "$SRC/" "$DST/" \
--include="frame_*.jpg" \
--include="reconstruction.ply" \
--include="lingbot_poses.npz" \
--exclude="*" \
2>&1 | tail -5
echo "[$(date)] Archivage job_$JOB_ID terminé: $DST"
EOF
chmod +x /root/cosma-nav/scripts/archive_job.sh
```
- [ ] **Step 5.2 : Tester sur job_17**
```bash
ssh root@192.168.0.84 '/root/cosma-nav/scripts/archive_job.sh 17'
# Attendu: rsync OK, fichiers dans /mnt/nas-cosma/cosma-archive/job_17/
ls /mnt/nas-cosma/cosma-archive/job_17/ | head -5
```
- [ ] **Step 5.3 : Archiver tous les jobs existants (background)**
```bash
ssh root@192.168.0.84 'for i in 9 11 12 13 14 15 16 17 19 21; do
nohup /root/cosma-nav/scripts/archive_job.sh $i >> /tmp/archive.log 2>&1 &
done; echo "Archivages lancés en background"'
```
---
## Task 6 : Déployer cosma-nav sur cosma-vm
**Files:**
- Create: `/home/cosma/cosma-nav/` sur cosma-vm (clone git)
- Create: `/home/cosma/cosma-nav/.venv/`
- [ ] **Step 6.1 : Cloner le repo**
```bash
ssh cosma@192.168.0.83
cd /home/cosma
git clone http://floppyrj45:67e1615fcfd06cf2df7872ac25e824f3afdb2bc1@192.168.0.82:3000/floppyrj45/cosma-nav.git
cd cosma-nav
```
- [ ] **Step 6.2 : Créer le venv et installer les dépendances**
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install flask h5py numpy open3d
# open3d peut prendre 2-3 min
python3 -c "import open3d; print(open3d.__version__)"
# Attendu: 0.19.0 (ou supérieur)
```
- [ ] **Step 6.3 : Créer le dossier de données**
```bash
sudo mkdir -p /data/cosma
sudo chown cosma:cosma /data/cosma
ls /data/cosma/
# Attendu: job_*_decimated.ply et job_*_poses.npz déjà là (Task 4)
```
- [ ] **Step 6.4 : Tester le viewer existant**
```bash
cd /home/cosma/cosma-nav
source .venv/bin/activate
python3 viz/server.py --trajectory /data/cosma/job_17_decimated.ply --port 5051 &
sleep 3 && curl -s -o /dev/null -w "%{http_code}" http://localhost:5051/trajectory
# Attendu: 200
kill %1
```
---
## Task 7 : Étendre cosma-nav — routes map et nav
**Files:**
- Modify: `/home/cosma/cosma-nav/viz/server.py`
- Create: `/home/cosma/cosma-nav/viz/templates/map.html`
- Create: `/home/cosma/cosma-nav/viz/templates/nav.html`
- Create: `/home/cosma/cosma-nav/viz/static/js/map.js`
- Create: `/home/cosma/cosma-nav/viz/static/js/nav_charts.js`
- [ ] **Step 7.1 : Ajouter les routes dans server.py**
Ajouter après les routes existantes dans `viz/server.py` :
```python
import json
from pathlib import Path
DATA_DIR = Path(os.environ.get("COSMA_DATA_DIR", "/data/cosma"))
def _load_nav_data(job_id: int) -> dict:
"""Charge poses + PLY décimé pour un job."""
out = {"job_id": job_id}
poses_path = DATA_DIR / f"job_{job_id}_poses.npz"
ply_path = DATA_DIR / f"job_{job_id}_decimated.ply"
if poses_path.exists():
d = np.load(str(poses_path))
poses = d["poses"] # (N, 3, 4)
t_ns = d.get("timestamps_ns", np.zeros(len(poses), dtype=np.int64))
# Positions caméra : colonne de translation
xyz = poses[:, :3, 3]
out["track"] = {"x": xyz[:, 0].tolist(), "y": xyz[:, 1].tolist(), "z": xyz[:, 2].tolist()}
out["n_poses"] = len(poses)
out["ply_ready"] = ply_path.exists()
out["ply_path"] = str(ply_path) if ply_path.exists() else None
return out
@app.route("/map")
def map_view():
return render_template("map.html")
@app.route("/nav")
def nav_view():
return render_template("nav.html")
@app.route("/api/jobs")
def api_jobs():
"""Liste les jobs disponibles (PLY décimé présent)."""
jobs = []
for p in sorted(DATA_DIR.glob("job_*_decimated.ply")):
jid = int(p.stem.split("_")[1])
jobs.append({"id": jid, "ply": str(p), "poses": str(DATA_DIR / f"job_{jid}_poses.npz")})
return jsonify(jobs)
@app.route("/api/job/<int:job_id>/nav")
def api_job_nav(job_id: int):
return jsonify(_load_nav_data(job_id))
@app.route("/api/job/<int:job_id>/ply")
def api_job_ply(job_id: int):
"""Sert le PLY décimé sous forme x/y/z arrays pour Three.js."""
ply_path = DATA_DIR / f"job_{job_id}_decimated.ply"
if not ply_path.exists():
return jsonify({"error": "PLY non disponible"}), 404
import open3d as o3d
pcd = o3d.io.read_point_cloud(str(ply_path))
pts = np.asarray(pcd.points)
return jsonify({"x": pts[:,0].tolist(), "y": pts[:,1].tolist(), "z": pts[:,2].tolist(), "n": len(pts)})
```
- [ ] **Step 7.2 : Créer map.html**
Créer `/home/cosma/cosma-nav/viz/templates/map.html` :
```html
<!doctype html>
<html lang="fr">
<head>
<meta charset="utf-8">
<title>COSMA NAV — Carte</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css">
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
<link rel="stylesheet" href="/static/style.css">
<style>
#map { height: calc(100vh - 60px); }
#job-select { margin: 8px; padding: 4px 8px; background: #1a2a1a; color: #ccc; border: 1px solid #4ade80; border-radius: 3px; }
</style>
</head>
<body>
<header>
<h1>COSMA NAV — Carte GPS</h1>
<select id="job-select"><option value="">-- sélectionner job --</option></select>
<a href="/trajectory">3D →</a>
<a href="/nav">Graphes →</a>
</header>
<div id="map"></div>
<script src="/static/js/map.js"></script>
</body>
</html>
```
- [ ] **Step 7.3 : Créer nav.html**
Créer `/home/cosma/cosma-nav/viz/templates/nav.html` :
```html
<!doctype html>
<html lang="fr">
<head>
<meta charset="utf-8">
<title>COSMA NAV — Données nav</title>
<link rel="stylesheet" href="/static/style.css">
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script>
<style>
.charts { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; padding: 16px; }
canvas { background: #0d1f0d; border-radius: 6px; }
#job-select { margin: 8px; padding: 4px 8px; background: #1a2a1a; color: #ccc; border: 1px solid #4ade80; border-radius: 3px; }
</style>
</head>
<body>
<header>
<h1>COSMA NAV — Données navigation</h1>
<select id="job-select"><option value="">-- sélectionner job --</option></select>
<a href="/map">Carte →</a>
<a href="/trajectory">3D →</a>
</header>
<div class="charts">
<canvas id="chart-track-xy" height="300"></canvas>
<canvas id="chart-depth" height="300"></canvas>
</div>
<script src="/static/js/nav_charts.js"></script>
</body>
</html>
```
- [ ] **Step 7.4 : Créer map.js**
Créer `/home/cosma/cosma-nav/viz/static/js/map.js` :
```javascript
const map = L.map('map').setView([43.17, 5.70], 13);
L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', {
attribution: '© OSM contributors', maxZoom: 19
}).addTo(map);
let trackLayer = null;
async function loadJobs() {
const res = await fetch('/api/jobs');
const jobs = await res.json();
const sel = document.getElementById('job-select');
jobs.forEach(j => {
const opt = document.createElement('option');
opt.value = j.id; opt.textContent = `job_${j.id}`;
sel.appendChild(opt);
});
if (jobs.length > 0) { sel.value = jobs[0].id; loadJob(jobs[0].id); }
}
async function loadJob(jobId) {
const res = await fetch(`/api/job/${jobId}/nav`);
const d = await res.json();
if (trackLayer) map.removeLayer(trackLayer);
if (!d.track) return;
// Coordonnées locales lingbot → affichage relatif (pas de géoréf)
// Si trajectory_world.h5 dispo → utiliser easting/northing
const pts = d.track.x.map((x, i) => [43.17 + d.track.y[i] * 0.00001, 5.70 + x * 0.00001]);
trackLayer = L.polyline(pts, { color: '#4ade80', weight: 2 }).addTo(map);
map.fitBounds(trackLayer.getBounds());
}
document.getElementById('job-select').addEventListener('change', e => {
if (e.target.value) loadJob(parseInt(e.target.value));
});
loadJobs();
```
> **Note:** Les coordonnées lingbot sont locales (non géoréférencées). La carte affiche le track relatif jusqu'à ce que `fuse_trajectory.py` produise un `trajectory_world.h5` avec easting/northing réels (logs La Ciotat). Une fois les données USBL disponibles, mettre à jour `loadJob()` pour appeler `/api/job/{id}/trajectory` avec les vraies coordonnées UTM.
- [ ] **Step 7.5 : Créer nav_charts.js**
Créer `/home/cosma/cosma-nav/viz/static/js/nav_charts.js` :
```javascript
let chartXY = null, chartDepth = null;
function initCharts() {
chartXY = new Chart(document.getElementById('chart-track-xy'), {
type: 'scatter',
data: { datasets: [{ label: 'Track XY lingbot', data: [], borderColor: '#4ade80', pointRadius: 1 }] },
options: { responsive: true, plugins: { title: { display: true, text: 'Track XY (m, local)', color: '#ccc' } },
scales: { x: { ticks: { color: '#888' } }, y: { ticks: { color: '#888' } } } }
});
chartDepth = new Chart(document.getElementById('chart-depth'), {
type: 'line',
data: { datasets: [{ label: 'Z lingbot (profondeur approx)', data: [], borderColor: '#60a5fa', pointRadius: 0, borderWidth: 1 }] },
options: { responsive: true, plugins: { title: { display: true, text: 'Z caméra (m, local)', color: '#ccc' } },
scales: { x: { ticks: { color: '#888' } }, y: { reverse: true, ticks: { color: '#888' } } } }
});
}
async function loadJobs() {
const res = await fetch('/api/jobs');
const jobs = await res.json();
const sel = document.getElementById('job-select');
jobs.forEach(j => {
const opt = document.createElement('option');
opt.value = j.id; opt.textContent = `job_${j.id} (${j.id})`;
sel.appendChild(opt);
});
if (jobs.length > 0) { sel.value = jobs[0].id; loadJob(jobs[0].id); }
}
async function loadJob(jobId) {
const res = await fetch(`/api/job/${jobId}/nav`);
const d = await res.json();
if (!d.track) return;
const n = d.track.x.length;
chartXY.data.datasets[0].data = d.track.x.map((x, i) => ({ x, y: d.track.y[i] }));
chartXY.update();
chartDepth.data.datasets[0].data = d.track.z.map((z, i) => ({ x: i, y: z }));
chartDepth.update();
}
document.getElementById('job-select').addEventListener('change', e => {
if (e.target.value) loadJob(parseInt(e.target.value));
});
initCharts();
loadJobs();
```
- [ ] **Step 7.6 : Tester les nouvelles routes**
```bash
ssh cosma@192.168.0.83 'cd /home/cosma/cosma-nav && source .venv/bin/activate && \
COSMA_DATA_DIR=/data/cosma python3 viz/server.py --port 5051 &'
sleep 3
curl -s http://192.168.0.83:5051/api/jobs | python3 -c "import sys,json; print(json.load(sys.stdin))"
curl -s http://192.168.0.83:5051/api/job/17/nav | python3 -c "import sys,json; d=json.load(sys.stdin); print('poses:', d.get('n_poses'), 'ply:', d.get('ply_ready'))"
# Attendu: n_poses 1217, ply_ready True
```
- [ ] **Step 7.7 : Commit**
```bash
cd /root/cosma-nav # sur ml-stack
git add viz/server.py viz/templates/map.html viz/templates/nav.html \
viz/static/js/map.js viz/static/js/nav_charts.js \
scripts/pre_decimate.py scripts/archive_job.sh scripts/check_jobs.py
git -c user.email="floppyrj45@gitea" -c user.name="Floppyrj45" commit -m "feat: routes map/nav + scripts pré-décimation + archivage NAS"
TOKEN=67e1615fcfd06cf2df7872ac25e824f3afdb2bc1
git push http://floppyrj45:${TOKEN}@192.168.0.82:3000/floppyrj45/cosma-nav.git master
```
---
## Task 8 : Systemd service cosma-nav sur cosma-vm
**Files:**
- Create: `/etc/systemd/system/cosma-nav.service`
- [ ] **Step 8.1 : Créer le service**
```bash
ssh cosma@192.168.0.83
sudo tee /etc/systemd/system/cosma-nav.service << 'EOF'
[Unit]
Description=COSMA NAV QC Viewer
After=network.target
[Service]
User=cosma
WorkingDirectory=/home/cosma/cosma-nav
Environment=COSMA_DATA_DIR=/data/cosma
ExecStart=/home/cosma/cosma-nav/.venv/bin/python3 viz/server.py --port 5051
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable cosma-nav
sudo systemctl start cosma-nav
sudo systemctl status cosma-nav --no-pager
```
- [ ] **Step 8.2 : Vérifier**
```bash
curl -s -o /dev/null -w "%{http_code}" http://192.168.0.83:5051/map
# Attendu: 200
```
---
## Task 9 : Caddy reverse proxy
**Files:**
- Create: `/etc/caddy/Caddyfile` sur cosma-vm
- [ ] **Step 9.1 : Installer Caddy**
```bash
ssh cosma@192.168.0.83
sudo apt-get install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt-get update && sudo apt-get install -y caddy
```
- [ ] **Step 9.2 : Configurer le Caddyfile**
```bash
sudo tee /etc/caddy/Caddyfile << 'EOF'
:80 {
# cosma-qc pipeline dashboard (racine)
handle /nav/* {
reverse_proxy localhost:5051
}
handle /nav {
reverse_proxy localhost:5051
}
handle {
reverse_proxy localhost:3849
}
}
EOF
```
- [ ] **Step 9.3 : Démarrer Caddy**
```bash
sudo systemctl reload caddy || sudo systemctl start caddy
sudo systemctl enable caddy
sleep 2
curl -s -o /dev/null -w "/ → %{http_code}\n" http://192.168.0.83/
curl -s -o /dev/null -w "/nav → %{http_code}\n" http://192.168.0.83/nav
# Attendu: / → 200, /nav → 200 (redirect vers /nav/)
```
> **Note:** Le Caddy écoute sur :80. cosma-qc reste aussi accessible directement sur :3849. cosma-nav reste aussi sur :5051 pour le viewer 3D.
---
## Task 10 : Bouton "QC →" dans cosma-qc dashboard
**Files:**
- Modify: `/home/cosma/cosma-qc/app/templates/_jobs_table.html`
- Modify: `/home/cosma/cosma-qc/app/main.py`
- [ ] **Step 10.1 : Ajouter la route redirect dans main.py**
Ajouter dans `/home/cosma/cosma-qc/app/main.py` après les routes existantes :
```python
from fastapi.responses import RedirectResponse
@app.get("/jobs/{job_id}/qc")
async def qc_redirect(job_id: int):
"""Redirige vers cosma-nav viewer pour ce job."""
return RedirectResponse(url=f"http://192.168.0.83:5051/trajectory?job={job_id}", status_code=302)
```
- [ ] **Step 10.2 : Modifier le template _jobs_table.html**
Dans le bloc `{% elif j.status == 'done' %}` de `col-progress`, remplacer :
```html
{% if j.viser_url %}
<a class="btn-viser" href="{{ j.viser_url }}" target="_blank" rel="noopener">viser ↗</a>
{% else %}
<button class="viewer-btn" data-view-url="jobs/{{ j.id }}/view" title="Charger PLY dans viser">PLY ↗</button>
{% endif %}
{% if j.ply_url %}
<a class="btn-ply-dl" href="{{ j.ply_url }}" download>PLY ↓</a>
{% endif %}
```
par :
```html
<a class="btn-qc" href="jobs/{{ j.id }}/qc" target="_blank" title="Ouvrir QC viewer">QC ↗</a>
<button class="viewer-btn" data-view-url="jobs/{{ j.id }}/view" title="PLY viser standalone">PLY ↗</button>
{% if j.ply_url %}
<a class="btn-ply-dl" href="{{ j.ply_url }}" download>PLY ↓</a>
{% endif %}
```
- [ ] **Step 10.3 : Ajouter le style btn-qc dans style.css**
```bash
ssh cosma@192.168.0.83 "cat >> /home/cosma/cosma-qc/app/static/style.css << 'EOF'
.btn-qc { background: #1a2a3a; color: #60a5fa; border: 1px solid #60a5fa; border-radius: 3px; padding: 2px 8px; text-decoration: none; font-size: 0.8rem; }
.btn-qc:hover { background: #60a5fa; color: #0a1020; }
.viewer-btn { background: #1a3a2a; color: #4ade80; border: 1px solid #4ade80; border-radius: 3px; padding: 2px 8px; cursor: pointer; font-size: 0.8rem; }
.viewer-btn:hover { background: #4ade80; color: #0a1a10; }
EOF"
```
- [ ] **Step 10.4 : Rebuild Docker cosma-qc**
```bash
ssh cosma@192.168.0.83 'cd /home/cosma/cosma-qc && docker compose build --no-cache && docker compose up -d'
sleep 5
curl -s -o /dev/null -w "%{http_code}" http://192.168.0.83:3849/
# Attendu: 200
```
---
## Task 11 : Post-job hook dans le dispatcher cosma-qc
**Files:**
- Modify: `/home/cosma/cosma-qc/scripts/dispatcher.py`
- [ ] **Step 11.1 : Ajouter la fonction post_job()**
Dans `/home/cosma/cosma-qc/scripts/dispatcher.py`, ajouter après `_maybe_create_per_auv_stitch()` :
```python
def post_job_sync(job_id: int, worker: dict, frames_dir: str) -> None:
"""Pré-décime PLY + SCP vers cosma-vm + archivage NAS (non-bloquant)."""
import threading
def _run():
try:
# 1. Pré-décimation + SCP vers cosma-vm
rc, out, err = ssh(
worker["ssh_alias"],
f"cd /root/cosma-nav && python3 scripts/pre_decimate.py {job_id}",
timeout=600
)
if rc != 0:
print(f"[post_job] pre_decimate job_{job_id} failed: {err[:200]}")
return
print(f"[post_job] pre_decimate job_{job_id} OK")
# 2. Archivage NAS (background)
ssh(
worker["ssh_alias"],
f"nohup /root/cosma-nav/scripts/archive_job.sh {job_id} >> /tmp/archive_{job_id}.log 2>&1 &",
timeout=10
)
print(f"[post_job] archive job_{job_id} lancé en background")
except Exception as e:
print(f"[post_job] ERROR job_{job_id}: {e}")
threading.Thread(target=_run, daemon=True).start()
```
- [ ] **Step 11.2 : Appeler post_job_sync() après chaque job done**
Dans `dispatcher.py`, trouver le bloc qui marque un job comme done (après `set_status(job["id"], status="done", ...)`). Ajouter :
```python
# Après set_status done :
post_job_sync(job["id"], worker, frames_dir)
_maybe_create_per_auv_stitch(job["id"])
```
- [ ] **Step 11.3 : Redémarrer le dispatcher**
```bash
ssh cosma@192.168.0.83 'sudo systemctl restart cosma-qc-dispatcher && \
sudo systemctl status cosma-qc-dispatcher --no-pager | head -5'
# Attendu: active (running)
```
---
## Task 12 : Vérification finale end-to-end
- [ ] **Step 12.1 : Dashboard cosma-qc accessible**
```
http://192.168.0.83/ → cosma-qc pipeline dashboard (via Caddy)
http://192.168.0.83:3849/ → cosma-qc direct
```
- [ ] **Step 12.2 : QC viewer accessible**
```
http://192.168.0.83/nav/trajectory → Three.js viewer
http://192.168.0.83/nav/map → Carte Leaflet
http://192.168.0.83/nav/nav → Graphes nav
http://192.168.0.83:5051/trajectory → cosma-nav direct
```
- [ ] **Step 12.3 : Cliquer bouton "QC →" dans le dashboard**
Ouvrir `http://192.168.0.83/`, choisir un job done, cliquer "QC ↗" → doit ouvrir le viewer Three.js sur ce job dans un nouvel onglet.
- [ ] **Step 12.4 : Vérifier archivage NAS**
```bash
ssh root@192.168.0.84 'ls -lh /mnt/nas-cosma/cosma-archive/ | head -15'
# Attendu: dossiers job_N avec frames + PLY brut
```
- [ ] **Step 12.5 : Commit final cosma-nav**
```bash
cd /root/cosma-nav
git add -A
git -c user.email="floppyrj45@gitea" -c user.name="Floppyrj45" \
commit -m "feat: déploiement cosma-vm — viewer map/nav/3D + post-job sync NAS"
TOKEN=67e1615fcfd06cf2df7872ac25e824f3afdb2bc1
git push http://floppyrj45:${TOKEN}@192.168.0.82:3000/floppyrj45/cosma-nav.git master
```

View File

@@ -0,0 +1,90 @@
# COSMA QC Platform — Design Spec
**Date:** 2026-04-24
**Statut:** Approuvé
## Objectif
Plateforme QC complète pour les missions COSMA AUV sur cosma-vm (.83), combinant gestion pipeline (cosma-qc) et visualisation QC (cosma-nav) : carte GPS/USBL, viewer 3D reconstruction, graphes nav.
## Architecture — Option B : Microservices
### Services
| Service | Port | Rôle |
|---------|------|------|
| cosma-qc | :3849 | Pipeline jobs/stitches, dashboard extraction/reconstruction |
| cosma-nav | :5051 | QC viewer : carte, 3D, graphes nav |
| Caddy | :80 | Reverse proxy → / (cosma-qc) et /nav/ (cosma-nav) |
### Machine cible
- **cosma-vm (.83)** — VM Proxmox
- Ressources actuelles : 4 vCPU / 3.8 GB RAM / 38 GB disk
- Expansion si nécessaire : +4 GB RAM si open3d décimatation locale
### Stockage
```
ml-stack (.84) NAS .156 (1.34 TB libre)
/root/cosma-qc-frames/ ──► /cosma-archive/
job_N/frame_*.jpg (65 GB) job_N/frames/ (archivé)
job_N/reconstruction.ply job_N/ply_brut/ (archivé)
ml-stack post-job cosma-vm (.83)
pré-décimer PLY ──► /data/cosma/
185M pts → ~200k pts (~7 MB) job_N_decimated.ply
rsync poses.npz job_N_poses.npz
rsync sparse_fixes.h5 sparse_fixes.h5
```
- Frames JPG et PLY bruts → archivés sur NAS .156 (SMB, admin) après job done
- Données légères → copiées sur cosma-vm pour le viewer
- Rien n'est supprimé avant confirmation archivage NAS
## cosma-nav — Fonctionnalités QC Viewer
### 1. Carte GPS/USBL (Leaflet + OpenStreetMap)
- Track AUV (trajectory_world.h5 → x_m/y_m converti en lat/lon)
- Fixes USBL/GPS plotés avec couleur RTK (fix=vert, float=jaune, 3D=rouge)
- Fixes USV (navire surface) en overlay
- Sélecteur de mission/job
### 2. Viewer 3D reconstruction (Three.js — existant)
- PLY décimé (~200k pts) chargé depuis /data/cosma/
- Frustums caméra orientés nadir
- Trajectoire lingbot locale
- Toggle layers (PLY, frustums, trajectory)
### 3. Graphes nav (Plotly.js ou Chart.js)
- Profondeur (depth_m depuis pression)
- Altitude fond (altitude_m Kogger)
- Qualité RTK (fix_type timeline)
- Vitesse USV
### 4. Intégration cosma-qc
- Bouton "QC →" dans la table jobs de cosma-qc → ouvre cosma-nav sur le job sélectionné
- API cosma-nav : GET /api/job/{id}/trajectory, /api/job/{id}/nav, /api/job/{id}/ply
## Pipeline post-job (dispatcher cosma-qc)
Après chaque job done :
1. `open3d` pré-décimation sur ml-stack → PLY ~7 MB
2. `rsync` PLY décimé + poses.npz → cosma-vm:/data/cosma/
3. `rsync` frames JPG + PLY brut → NAS .156:/cosma-archive/ (async, non-bloquant)
4. Mise à jour DB cosma-qc : `decimated_ply_path`, `nav_ready=true`
## Vérification PLY/poses existants
Avant déploiement : valider les 11 jobs actuels (jobs 9-21) :
- PLY lisible par open3d ✓ (vérifié, 185M pts)
- poses.npz shape (N, 3, 4) ✓ (1217-1349 frames)
- PLY utile : early-save ≠ final-save → re-vérifier si demo.py a fini proprement
## Déploiement
1. Monter NAS .156 sur ml-stack et cosma-vm (CIFS)
2. Cloner cosma-nav sur cosma-vm
3. Installer deps (Flask, open3d, h5py) dans venv
4. Adapter server.py : servir depuis /data/cosma/, ajouter routes map + nav
5. Caddy config sur cosma-vm
6. Modifier dispatcher cosma-qc : ajouter step post-job (décimation + sync)
7. Ajouter bouton "QC →" dans _jobs_table.html cosma-qc