185 lines
5.6 KiB
Markdown
185 lines
5.6 KiB
Markdown
# Design : GDrive Pipeline Replay
|
||
|
||
**Date :** 2026-04-27
|
||
**Repo :** cosma-nav-tools
|
||
**Branche :** feat/flag-local
|
||
|
||
---
|
||
|
||
## Objectif
|
||
|
||
Bouton dans le viewer 8765 qui déclenche un sync GDrive → pipeline Python → affichage replay des signaux USV et AUV sur la même page, synchronisés avec le slider 24h existant.
|
||
|
||
---
|
||
|
||
## Architecture globale
|
||
|
||
```
|
||
GDrive (G:)
|
||
└─ 06-Operations/06-Sorties/#XX-lieu/YYYYMMDD-lieu/raw_data/
|
||
├─ SHIP/*.csv (USV nav + USBL + full)
|
||
└─ SUB/*.mcap (AUV ROS2) + bin/
|
||
|
||
↓ rclone sync (déclenché par bouton viewer)
|
||
|
||
.83 /data/sorties/#XX/raw/
|
||
├─ SHIP/
|
||
└─ SUB/
|
||
|
||
↓ pipeline Python (scripts existants cosma-nav-tools/tools/)
|
||
|
||
.83 /data/sorties/#XX/processed/
|
||
├─ usv.json.gz ← signaux USV downsamplés LTTB
|
||
├─ auv_AUV010.json.gz ← signaux AUV010 downsamplés LTTB
|
||
├─ auv_AUVxxx.json.gz ← un fichier par AUV détecté
|
||
└─ tracks.geojson ← USV + AUV tracks (Leaflet)
|
||
|
||
↓ servi par
|
||
|
||
port 8767 pipeline-runner FastAPI (cosma-nav-tools/pipeline_runner/)
|
||
port 8765 viewer/index.html ← page unique, fetch 8766 + 8767
|
||
```
|
||
|
||
---
|
||
|
||
## Section 1 : Pipeline-runner (port 8767)
|
||
|
||
### Structure
|
||
|
||
```
|
||
cosma-nav-tools/
|
||
pipeline_runner/
|
||
main.py ← FastAPI app + endpoints
|
||
runner.py ← logique rclone + orchestration scripts
|
||
config.py ← chemins GDrive, output dir, rclone remote
|
||
docker-compose.yml ← service pipeline-runner port 8767
|
||
data/sorties/ ← gitignored
|
||
```
|
||
|
||
### Endpoints
|
||
|
||
| Méthode | Route | Description |
|
||
|---------|-------|-------------|
|
||
| `GET` | `/sorties` | Liste sorties détectées sur GDrive (scan dossier) |
|
||
| `POST` | `/run/{sortie_id}` | Lance rclone pull + pipeline complet |
|
||
| `GET` | `/events/{sortie_id}` | SSE stream progress (étape + %) |
|
||
| `GET` | `/sorties/{id}/usv` | Données USV processed (JSON gzip) |
|
||
| `GET` | `/sorties/{id}/auvs` | Liste AUVs disponibles pour cette sortie |
|
||
| `GET` | `/sorties/{id}/auv/{auv_id}` | Données AUV processed (JSON gzip) |
|
||
| `GET` | `/sorties/{id}/tracks` | GeoJSON tracks USV + AUV |
|
||
|
||
### Flow interne POST /run/{sortie_id}
|
||
|
||
```
|
||
1. rclone sync GDrive/#id → /data/sorties/#id/raw/ SSE: "sync" 0→50%
|
||
2. parse_usv_nav.py + extract_usv_pwm.py SSE: "usv_parse" 50→65%
|
||
3. parse_kogger_usbl.py + merge_nav_usbl.py SSE: "usbl_merge" 65→80%
|
||
4. extract_mcap_signals.py (par AUV détecté) SSE: "auv_parse" 80→90%
|
||
5. usbl_to_json.py SSE: "usbl_json" 90→95%
|
||
6. downsample LTTB + écriture .json.gz SSE: "write" 95→100%
|
||
```
|
||
|
||
---
|
||
|
||
## Section 2 : Format données
|
||
|
||
### Signaux (usv.json.gz, auv_*.json.gz)
|
||
|
||
```json
|
||
{
|
||
"meta": {
|
||
"sortie": "#71-golrest",
|
||
"date": "2026-04-16",
|
||
"vehicle": "USV001",
|
||
"t_start": 1713268477,
|
||
"t_end": 1713282877
|
||
},
|
||
"signals": {
|
||
"yaw": [{ "t": 1713268477, "v": 142.3 }, ...],
|
||
"heading": [...],
|
||
"gps_status": [{ "t": ..., "v": "3D_FIX" }, ...],
|
||
"battery_v": [...],
|
||
"usbl_dist": [...],
|
||
"usbl_angle": [...],
|
||
"motor_1": [...],
|
||
"motor_2": [...],
|
||
"auv_status": [{ "t": ..., "v": "MISSION" }, ...]
|
||
}
|
||
}
|
||
```
|
||
|
||
- `t` = epoch UNIX secondes
|
||
- Max **4000 points par signal** via LTTB (Plotly optimal)
|
||
- Valeurs discrètes (status, mode) = strings → step-plot Plotly
|
||
- Endpoint optionnel `/range?from=&to=` pour zoom fin sur brutes
|
||
|
||
### Tracks
|
||
|
||
- **GeoJSON** FeatureCollection : une Feature LineString par véhicule
|
||
- Poids estimé : ~300KB gzip pour 4h à 1Hz
|
||
|
||
---
|
||
|
||
## Section 3 : Viewer 8765 — extensions
|
||
|
||
### Layout (page unique)
|
||
|
||
```
|
||
datebar ← inchangé
|
||
header ← + dropdown sortie + [Sync & Process] + progress bar SSE
|
||
map Leaflet ← inchangé
|
||
slider 24h ← shared X axis, curseur vertical sur tous les graphs
|
||
─────────────────────────────────────────────
|
||
USV PANEL
|
||
[Yaw] [Heading]
|
||
[GPS status] [Battery voltage]
|
||
[USBL dist] [USBL angle]
|
||
[Motor 1] [Motor 2]
|
||
[AUV status: disarm/mission/goto — step-plot]
|
||
─────────────────────────────────────────────
|
||
AUV PANEL ← tabs [AUV010] [AUV011] … si multi-AUV
|
||
[Pitch/Roll/Yaw — 3 traces] [Depth]
|
||
[Altitude] [Obstacle dist]
|
||
[USBL dist] [USBL angle]
|
||
[Battery voltage] [Arm/Disarm/Mode — step-plot]
|
||
[Motors ×6 PWM — 1 graph multi-trace M1…M6]
|
||
```
|
||
|
||
### Synchronisation slider
|
||
|
||
- Slider 24h existant → événement → `updateCursor(t)` sur tous les graphs Plotly via `Plotly.relayout` shapes
|
||
- Données chargées une fois en mémoire au click de sortie, pas de re-fetch au scrub
|
||
|
||
### Bouton Sync & Process
|
||
|
||
- `POST /run/{sortie_id}` → ouvre SSE `/events/{sortie_id}`
|
||
- Progress bar inline dans le header (ex: `rclone 34% → usv_parse…`)
|
||
- À 100% → fetch automatique USV + AUV + tracks → render graphs
|
||
|
||
---
|
||
|
||
## Section 4 : Déploiement .83
|
||
|
||
```yaml
|
||
# docker-compose.yml (ajout)
|
||
pipeline-runner:
|
||
build: ./pipeline_runner
|
||
ports:
|
||
- "8767:8767"
|
||
volumes:
|
||
- /data/sorties:/data/sorties
|
||
- /mnt/gdrive:/mnt/gdrive # rclone mount ou rclone exec
|
||
environment:
|
||
- GDRIVE_REMOTE=gdrive:Cosma - Internal/06-Operations/06 - Sorties
|
||
- OUTPUT_DIR=/data/sorties
|
||
```
|
||
|
||
---
|
||
|
||
## Hors scope (v1)
|
||
|
||
- Authentification / accès multi-user
|
||
- HDF5 archivage (peut venir en v2)
|
||
- Replay animé temps-réel (curseur qui avance automatiquement)
|
||
- Tests unitaires pipeline
|