docs: spec pipeline GDrive → replay USV/AUV dans viewer
This commit is contained in:
Flagabat
@@ -0,0 +1,184 @@
|
||||
# Design : GDrive Pipeline Replay
|
||||
|
||||
**Date :** 2026-04-27
|
||||
**Repo :** cosma-nav-tools
|
||||
**Branche :** feat/flag-local
|
||||
|
||||
---
|
||||
|
||||
## Objectif
|
||||
|
||||
Bouton dans le viewer 8765 qui déclenche un sync GDrive → pipeline Python → affichage replay des signaux USV et AUV sur la même page, synchronisés avec le slider 24h existant.
|
||||
|
||||
---
|
||||
|
||||
## Architecture globale
|
||||
|
||||
```
|
||||
GDrive (G:)
|
||||
└─ 06-Operations/06-Sorties/#XX-lieu/YYYYMMDD-lieu/raw_data/
|
||||
├─ SHIP/*.csv (USV nav + USBL + full)
|
||||
└─ SUB/*.mcap (AUV ROS2) + bin/
|
||||
|
||||
↓ rclone sync (déclenché par bouton viewer)
|
||||
|
||||
.83 /data/sorties/#XX/raw/
|
||||
├─ SHIP/
|
||||
└─ SUB/
|
||||
|
||||
↓ pipeline Python (scripts existants cosma-nav-tools/tools/)
|
||||
|
||||
.83 /data/sorties/#XX/processed/
|
||||
├─ usv.json.gz ← signaux USV downsamplés LTTB
|
||||
├─ auv_AUV010.json.gz ← signaux AUV010 downsamplés LTTB
|
||||
├─ auv_AUVxxx.json.gz ← un fichier par AUV détecté
|
||||
└─ tracks.geojson ← USV + AUV tracks (Leaflet)
|
||||
|
||||
↓ servi par
|
||||
|
||||
port 8767 pipeline-runner FastAPI (cosma-nav-tools/pipeline_runner/)
|
||||
port 8765 viewer/index.html ← page unique, fetch 8766 + 8767
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Section 1 : Pipeline-runner (port 8767)
|
||||
|
||||
### Structure
|
||||
|
||||
```
|
||||
cosma-nav-tools/
|
||||
pipeline_runner/
|
||||
main.py ← FastAPI app + endpoints
|
||||
runner.py ← logique rclone + orchestration scripts
|
||||
config.py ← chemins GDrive, output dir, rclone remote
|
||||
docker-compose.yml ← service pipeline-runner port 8767
|
||||
data/sorties/ ← gitignored
|
||||
```
|
||||
|
||||
### Endpoints
|
||||
|
||||
| Méthode | Route | Description |
|
||||
|---------|-------|-------------|
|
||||
| `GET` | `/sorties` | Liste sorties détectées sur GDrive (scan dossier) |
|
||||
| `POST` | `/run/{sortie_id}` | Lance rclone pull + pipeline complet |
|
||||
| `GET` | `/events/{sortie_id}` | SSE stream progress (étape + %) |
|
||||
| `GET` | `/sorties/{id}/usv` | Données USV processed (JSON gzip) |
|
||||
| `GET` | `/sorties/{id}/auvs` | Liste AUVs disponibles pour cette sortie |
|
||||
| `GET` | `/sorties/{id}/auv/{auv_id}` | Données AUV processed (JSON gzip) |
|
||||
| `GET` | `/sorties/{id}/tracks` | GeoJSON tracks USV + AUV |
|
||||
|
||||
### Flow interne POST /run/{sortie_id}
|
||||
|
||||
```
|
||||
1. rclone sync GDrive/#id → /data/sorties/#id/raw/ SSE: "sync" 0→50%
|
||||
2. parse_usv_nav.py + extract_usv_pwm.py SSE: "usv_parse" 50→65%
|
||||
3. parse_kogger_usbl.py + merge_nav_usbl.py SSE: "usbl_merge" 65→80%
|
||||
4. extract_mcap_signals.py (par AUV détecté) SSE: "auv_parse" 80→90%
|
||||
5. usbl_to_json.py SSE: "usbl_json" 90→95%
|
||||
6. downsample LTTB + écriture .json.gz SSE: "write" 95→100%
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Section 2 : Format données
|
||||
|
||||
### Signaux (usv.json.gz, auv_*.json.gz)
|
||||
|
||||
```json
|
||||
{
|
||||
"meta": {
|
||||
"sortie": "#71-golrest",
|
||||
"date": "2026-04-16",
|
||||
"vehicle": "USV001",
|
||||
"t_start": 1713268477,
|
||||
"t_end": 1713282877
|
||||
},
|
||||
"signals": {
|
||||
"yaw": [{ "t": 1713268477, "v": 142.3 }, ...],
|
||||
"heading": [...],
|
||||
"gps_status": [{ "t": ..., "v": "3D_FIX" }, ...],
|
||||
"battery_v": [...],
|
||||
"usbl_dist": [...],
|
||||
"usbl_angle": [...],
|
||||
"motor_1": [...],
|
||||
"motor_2": [...],
|
||||
"auv_status": [{ "t": ..., "v": "MISSION" }, ...]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `t` = epoch UNIX secondes
|
||||
- Max **4000 points par signal** via LTTB (Plotly optimal)
|
||||
- Valeurs discrètes (status, mode) = strings → step-plot Plotly
|
||||
- Endpoint optionnel `/range?from=&to=` pour zoom fin sur brutes
|
||||
|
||||
### Tracks
|
||||
|
||||
- **GeoJSON** FeatureCollection : une Feature LineString par véhicule
|
||||
- Poids estimé : ~300KB gzip pour 4h à 1Hz
|
||||
|
||||
---
|
||||
|
||||
## Section 3 : Viewer 8765 — extensions
|
||||
|
||||
### Layout (page unique)
|
||||
|
||||
```
|
||||
datebar ← inchangé
|
||||
header ← + dropdown sortie + [Sync & Process] + progress bar SSE
|
||||
map Leaflet ← inchangé
|
||||
slider 24h ← shared X axis, curseur vertical sur tous les graphs
|
||||
─────────────────────────────────────────────
|
||||
USV PANEL
|
||||
[Yaw] [Heading]
|
||||
[GPS status] [Battery voltage]
|
||||
[USBL dist] [USBL angle]
|
||||
[Motor 1] [Motor 2]
|
||||
[AUV status: disarm/mission/goto — step-plot]
|
||||
─────────────────────────────────────────────
|
||||
AUV PANEL ← tabs [AUV010] [AUV011] … si multi-AUV
|
||||
[Pitch/Roll/Yaw — 3 traces] [Depth]
|
||||
[Altitude] [Obstacle dist]
|
||||
[USBL dist] [USBL angle]
|
||||
[Battery voltage] [Arm/Disarm/Mode — step-plot]
|
||||
[Motors ×6 PWM — 1 graph multi-trace M1…M6]
|
||||
```
|
||||
|
||||
### Synchronisation slider
|
||||
|
||||
- Slider 24h existant → événement → `updateCursor(t)` sur tous les graphs Plotly via `Plotly.relayout` shapes
|
||||
- Données chargées une fois en mémoire au click de sortie, pas de re-fetch au scrub
|
||||
|
||||
### Bouton Sync & Process
|
||||
|
||||
- `POST /run/{sortie_id}` → ouvre SSE `/events/{sortie_id}`
|
||||
- Progress bar inline dans le header (ex: `rclone 34% → usv_parse…`)
|
||||
- À 100% → fetch automatique USV + AUV + tracks → render graphs
|
||||
|
||||
---
|
||||
|
||||
## Section 4 : Déploiement .83
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml (ajout)
|
||||
pipeline-runner:
|
||||
build: ./pipeline_runner
|
||||
ports:
|
||||
- "8767:8767"
|
||||
volumes:
|
||||
- /data/sorties:/data/sorties
|
||||
- /mnt/gdrive:/mnt/gdrive # rclone mount ou rclone exec
|
||||
environment:
|
||||
- GDRIVE_REMOTE=gdrive:Cosma - Internal/06-Operations/06 - Sorties
|
||||
- OUTPUT_DIR=/data/sorties
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Hors scope (v1)
|
||||
|
||||
- Authentification / accès multi-user
|
||||
- HDF5 archivage (peut venir en v2)
|
||||
- Replay animé temps-réel (curseur qui avance automatiquement)
|
||||
- Tests unitaires pipeline
|
||||
Reference in New Issue
Block a user