feat: migration vers cosma-vm (.83) — dispatcher+dashboard, OpenVPN, docs infra mis à jour
This commit is contained in:
Flag
BIN
docs/_build/html/.doctrees/environment.pickle
vendored
BIN
docs/_build/html/.doctrees/environment.pickle
vendored
Binary file not shown.
BIN
docs/_build/html/.doctrees/infrastructure.doctree
vendored
BIN
docs/_build/html/.doctrees/infrastructure.doctree
vendored
Binary file not shown.
164
docs/_build/html/_sources/infrastructure.rst.txt
vendored
164
docs/_build/html/_sources/infrastructure.rst.txt
vendored
@@ -6,49 +6,68 @@ Réseau LAN — 192.168.0.0/24
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ LAN 192.168.0.0/24 │
|
||||
│ │
|
||||
│ .82 CORE Dispatcher (systemd) + FastAPI :3849 │
|
||||
│ Gitea + Grafana + InfluxDB + Caddy │
|
||||
│ │
|
||||
│ .84 ml-stack GPU worker RTX 3090 24GB │
|
||||
│ .87 gpu GPU worker RTX 3060 12GB │
|
||||
│ │
|
||||
│ .168 z620 Proxmox host HP Z620 │
|
||||
│ SSD → /mnt/portablessd (MP4 bruts) │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
┌───────────────────────────────────────────────────────────────┐
|
||||
│ LAN 192.168.0.0/24 │
|
||||
│ │
|
||||
│ .82 CORE Gitea + Grafana + InfluxDB + Caddy │
|
||||
│ Reverse-proxy → cosma-vm pour /cosma-qc │
|
||||
│ │
|
||||
│ .83 cosma-vm Dispatcher cosma-qc (systemd) │
|
||||
│ Dashboard FastAPI :3849 │
|
||||
│ OpenVPN server :1194 UDP │
|
||||
│ │
|
||||
│ .84 ml-stack GPU worker RTX 3090 24GB │
|
||||
│ .87 gpu GPU worker RTX 3060 12GB │
|
||||
│ │
|
||||
│ .168 z620 Proxmox host HP Z620 — SSD MP4 bruts │
|
||||
│ .101 poxML Proxmox hyperviseur — héberge cosma-vm (VM 202)│
|
||||
└───────────────────────────────────────────────────────────────┘
|
||||
|
||||
|
||||
Nœud core (.82)
|
||||
----------------
|
||||
Nœud cosma-vm (.83) — serveur principal cosma-qc
|
||||
--------------------------------------------------
|
||||
|
||||
**Rôle :** orchestrateur central du pipeline.
|
||||
VM #202 sur Proxmox poxML (.101). 4 vCPU, 4 GB RAM, 40 GB SSD.
|
||||
User : ``cosma`` (avec docker group et sudo).
|
||||
|
||||
Services actifs :
|
||||
Services :
|
||||
|
||||
- **Dispatcher** — service systemd cosma-qc-dispatcher.
|
||||
Boucle principale qui dispatch les jobs aux workers GPU.
|
||||
- **Dispatcher** — ``cosma-qc-dispatcher.service`` (systemd)
|
||||
Lance les extractions ffmpeg sur les workers GPU.
|
||||
|
||||
- **Dashboard FastAPI** — conteneur Docker exposé sur le port **3849**.
|
||||
Interface web de monitoring des jobs.
|
||||
- **Dashboard FastAPI** — conteneur Docker ``cosma-qc-app``, port **3849**.
|
||||
Accès via Caddy sur core : ``http://192.168.0.82/cosma-qc``
|
||||
|
||||
- **Gitea** — dépôt source floppyrj45/cosma-qc.
|
||||
|
||||
- **Grafana / InfluxDB** — monitoring infrastructure.
|
||||
- **OpenVPN** — ``openvpn-server@cosma.service``, port 1194 UDP.
|
||||
Permet aux collègues daccéder au LAN depuis lextérieur.
|
||||
|
||||
Commandes utiles :
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# SSH vers cosma-vm
|
||||
ssh cosma@192.168.0.83
|
||||
|
||||
# Statut dispatcher
|
||||
sudo systemctl status cosma-qc-dispatcher
|
||||
|
||||
# Logs dispatcher temps réel
|
||||
sudo journalctl -u cosma-qc-dispatcher -f
|
||||
# Logs dispatcher
|
||||
tail -f /home/cosma/cosma-qc-data/dispatcher.log
|
||||
|
||||
# Dashboard
|
||||
http://192.168.0.82:3849
|
||||
# Dashboard local
|
||||
http://192.168.0.83:3849
|
||||
|
||||
|
||||
Nœud core (.82)
|
||||
----------------
|
||||
|
||||
**Rôle :** routeur et services communs.
|
||||
|
||||
Services actifs :
|
||||
|
||||
- **Caddy** — reverse proxy HTTPS, route ``/cosma-qc`` vers .83:3849.
|
||||
- **Gitea** — dépôt source ``floppyrj45/cosma-qc``.
|
||||
- **Grafana / InfluxDB** — monitoring infrastructure.
|
||||
|
||||
|
||||
Nœuds GPU workers (.84 et .87)
|
||||
@@ -59,7 +78,7 @@ Nœuds GPU workers (.84 et .87)
|
||||
:widths: 15 25 20 40
|
||||
|
||||
* - IP
|
||||
- Nom
|
||||
- Nom SSH
|
||||
- GPU
|
||||
- VRAM
|
||||
* - .84
|
||||
@@ -71,83 +90,46 @@ Nœuds GPU workers (.84 et .87)
|
||||
- RTX 3060
|
||||
- 12 GB
|
||||
|
||||
**Rôle :** exécution de ffmpeg (extraction frames) et lingbot-map (reconstruction 3D).
|
||||
|
||||
Répertoire de travail sur chaque worker :
|
||||
**Rôle :** exécution de ffmpeg et lingbot-map.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
/cosma-qc-frames/
|
||||
├── job_1/
|
||||
│ ├── frame_000001.jpg … frame_NNNNNN.jpg
|
||||
│ ├── .video_0.done
|
||||
│ ├── reconstruction.ply
|
||||
│ └── reconstruction.glb (généré à la demande)
|
||||
├── job_2/
|
||||
│ └── …
|
||||
└── stitch_1.ply
|
||||
└── job_N/
|
||||
├── frame_NNNNNN.jpg
|
||||
├── .video_V.done
|
||||
├── reconstruction.ply
|
||||
└── reconstruction.glb
|
||||
|
||||
|
||||
Nœud z620 (.168)
|
||||
-----------------
|
||||
Nœud z620 (.168) — stockage MP4
|
||||
----------------------------------
|
||||
|
||||
**Rôle :** stockage des MP4 bruts GoPro.
|
||||
HP Z620 (Proxmox). SSD /mnt/portablessd avec les MP4 GoPro bruts.
|
||||
Les MP4 **ne quittent jamais** z620 — ffmpeg sy exécute via SSH-relay.
|
||||
|
||||
- Proxmox host HP Z620.
|
||||
- SSD monté sur /mnt/portablessd.
|
||||
- Les MP4 **ne quittent jamais** z620 — ffmpeg s'y exécute via SSH.
|
||||
|
||||
Accès SSH depuis core :
|
||||
SSH relay (depuis cosma-vm ou core) :
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ssh floppyrj45@192.168.0.168
|
||||
ssh ml-stack "ssh z620 \\"commande\\""
|
||||
|
||||
|
||||
Service systemd dispatcher
|
||||
---------------------------
|
||||
OpenVPN — accès collègues
|
||||
--------------------------
|
||||
|
||||
Fichier de service : /etc/systemd/system/cosma-qc-dispatcher.service
|
||||
Serveur OpenVPN sur cosma-vm (.83:1194 UDP).
|
||||
Adresse externe : ``laboratoire.freeboxos.fr:1194``
|
||||
|
||||
.. code-block:: ini
|
||||
Profil client : ``cosma-qc-collegue1.ovpn`` (Nextcloud sync).
|
||||
|
||||
[Unit]
|
||||
Description=COSMA QC Dispatcher
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
User=floppyrj45
|
||||
WorkingDirectory=/home/floppyrj45/docker/cosma-qc
|
||||
ExecStart=/usr/bin/python3 app/dispatcher.py
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
Commandes de gestion :
|
||||
Génération dun nouveau client :
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl start cosma-qc-dispatcher
|
||||
sudo systemctl stop cosma-qc-dispatcher
|
||||
sudo systemctl restart cosma-qc-dispatcher
|
||||
sudo systemctl enable cosma-qc-dispatcher # démarrage auto
|
||||
|
||||
|
||||
Conteneur Docker dashboard
|
||||
---------------------------
|
||||
|
||||
Le dashboard FastAPI tourne dans un conteneur Docker.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cd /home/floppyrj45/docker/cosma-qc
|
||||
docker compose up -d # démarrer
|
||||
docker compose down # arrêter
|
||||
docker compose logs -f # logs
|
||||
|
||||
Accès : http://192.168.0.82:3849
|
||||
cd /home/cosma/openvpn-ca/easyrsa3
|
||||
./easyrsa gen-req NOMCLIENT nopass
|
||||
./easyrsa sign-req client NOMCLIENT
|
||||
|
||||
|
||||
Ports réseau récapitulatifs
|
||||
@@ -160,12 +142,16 @@ Ports réseau récapitulatifs
|
||||
* - Host
|
||||
- Port
|
||||
- Service
|
||||
* - .82
|
||||
* - .83
|
||||
- 3849
|
||||
- Dashboard FastAPI cosma-qc
|
||||
* - .83
|
||||
- 1194/udp
|
||||
- OpenVPN server
|
||||
* - .84 / .87
|
||||
- 8100+N
|
||||
- Viser viewer (reconstruction job N)
|
||||
- Viser viewer (job N)
|
||||
* - .84 / .87
|
||||
- 8300
|
||||
- HTTP server GLB export
|
||||
- HTTP server GLB/PLY export
|
||||
|
||||
|
||||
6
docs/_build/html/index.html
vendored
6
docs/_build/html/index.html
vendored
@@ -93,11 +93,11 @@
|
||||
</li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="infrastructure.html">Infrastructure</a><ul>
|
||||
<li class="toctree-l2"><a class="reference internal" href="infrastructure.html#reseau-lan-192-168-0-0-24">Réseau LAN — 192.168.0.0/24</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="infrastructure.html#noeud-cosma-vm-83-serveur-principal-cosma-qc">Nœud cosma-vm (.83) — serveur principal cosma-qc</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="infrastructure.html#noeud-core-82">Nœud core (.82)</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="infrastructure.html#noeuds-gpu-workers-84-et-87">Nœuds GPU workers (.84 et .87)</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="infrastructure.html#noeud-z620-168">Nœud z620 (.168)</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="infrastructure.html#service-systemd-dispatcher">Service systemd dispatcher</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="infrastructure.html#conteneur-docker-dashboard">Conteneur Docker dashboard</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="infrastructure.html#noeud-z620-168-stockage-mp4">Nœud z620 (.168) — stockage MP4</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="infrastructure.html#openvpn-acces-collegues">OpenVPN — accès collègues</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="infrastructure.html#ports-reseau-recapitulatifs">Ports réseau récapitulatifs</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
|
||||
173
docs/_build/html/infrastructure.html
vendored
173
docs/_build/html/infrastructure.html
vendored
@@ -47,11 +47,11 @@
|
||||
<li class="toctree-l1"><a class="reference internal" href="pipeline.html">Pipeline cosma-qc</a></li>
|
||||
<li class="toctree-l1 current"><a class="current reference internal" href="#">Infrastructure</a><ul>
|
||||
<li class="toctree-l2"><a class="reference internal" href="#reseau-lan-192-168-0-0-24">Réseau LAN — 192.168.0.0/24</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="#noeud-cosma-vm-83-serveur-principal-cosma-qc">Nœud cosma-vm (.83) — serveur principal cosma-qc</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="#noeud-core-82">Nœud core (.82)</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="#noeuds-gpu-workers-84-et-87">Nœuds GPU workers (.84 et .87)</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="#noeud-z620-168">Nœud z620 (.168)</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="#service-systemd-dispatcher">Service systemd dispatcher</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="#conteneur-docker-dashboard">Conteneur Docker dashboard</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="#noeud-z620-168-stockage-mp4">Nœud z620 (.168) — stockage MP4</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="#openvpn-acces-collegues">OpenVPN — accès collègues</a></li>
|
||||
<li class="toctree-l2"><a class="reference internal" href="#ports-reseau-recapitulatifs">Ports réseau récapitulatifs</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
@@ -87,44 +87,62 @@
|
||||
<h1>Infrastructure<a class="headerlink" href="#infrastructure" title="Link to this heading"></a></h1>
|
||||
<section id="reseau-lan-192-168-0-0-24">
|
||||
<h2>Réseau LAN — 192.168.0.0/24<a class="headerlink" href="#reseau-lan-192-168-0-0-24" title="Link to this heading"></a></h2>
|
||||
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>┌─────────────────────────────────────────────────────────┐
|
||||
│ LAN 192.168.0.0/24 │
|
||||
│ │
|
||||
│ .82 CORE Dispatcher (systemd) + FastAPI :3849 │
|
||||
│ Gitea + Grafana + InfluxDB + Caddy │
|
||||
│ │
|
||||
│ .84 ml-stack GPU worker RTX 3090 24GB │
|
||||
│ .87 gpu GPU worker RTX 3060 12GB │
|
||||
│ │
|
||||
│ .168 z620 Proxmox host HP Z620 │
|
||||
│ SSD → /mnt/portablessd (MP4 bruts) │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>┌───────────────────────────────────────────────────────────────┐
|
||||
│ LAN 192.168.0.0/24 │
|
||||
│ │
|
||||
│ .82 CORE Gitea + Grafana + InfluxDB + Caddy │
|
||||
│ Reverse-proxy → cosma-vm pour /cosma-qc │
|
||||
│ │
|
||||
│ .83 cosma-vm Dispatcher cosma-qc (systemd) │
|
||||
│ Dashboard FastAPI :3849 │
|
||||
│ OpenVPN server :1194 UDP │
|
||||
│ │
|
||||
│ .84 ml-stack GPU worker RTX 3090 24GB │
|
||||
│ .87 gpu GPU worker RTX 3060 12GB │
|
||||
│ │
|
||||
│ .168 z620 Proxmox host HP Z620 — SSD MP4 bruts │
|
||||
│ .101 poxML Proxmox hyperviseur — héberge cosma-vm (VM 202)│
|
||||
└───────────────────────────────────────────────────────────────┘
|
||||
</pre></div>
|
||||
</div>
|
||||
</section>
|
||||
<section id="noeud-cosma-vm-83-serveur-principal-cosma-qc">
|
||||
<h2>Nœud cosma-vm (.83) — serveur principal cosma-qc<a class="headerlink" href="#noeud-cosma-vm-83-serveur-principal-cosma-qc" title="Link to this heading"></a></h2>
|
||||
<p>VM #202 sur Proxmox poxML (.101). 4 vCPU, 4 GB RAM, 40 GB SSD.
|
||||
User : <code class="docutils literal notranslate"><span class="pre">cosma</span></code> (avec docker group et sudo).</p>
|
||||
<p>Services :</p>
|
||||
<ul class="simple">
|
||||
<li><p><strong>Dispatcher</strong> — <code class="docutils literal notranslate"><span class="pre">cosma-qc-dispatcher.service</span></code> (systemd)
|
||||
Lance les extractions ffmpeg sur les workers GPU.</p></li>
|
||||
<li><p><strong>Dashboard FastAPI</strong> — conteneur Docker <code class="docutils literal notranslate"><span class="pre">cosma-qc-app</span></code>, port <strong>3849</strong>.
|
||||
Accès via Caddy sur core : <code class="docutils literal notranslate"><span class="pre">http://192.168.0.82/cosma-qc</span></code></p></li>
|
||||
<li><p><strong>OpenVPN</strong> — <code class="docutils literal notranslate"><span class="pre">openvpn-server@cosma.service</span></code>, port 1194 UDP.
|
||||
Permet aux collègues daccéder au LAN depuis lextérieur.</p></li>
|
||||
</ul>
|
||||
<p>Commandes utiles :</p>
|
||||
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># SSH vers cosma-vm</span>
|
||||
ssh<span class="w"> </span>cosma@192.168.0.83
|
||||
|
||||
<span class="c1"># Statut dispatcher</span>
|
||||
sudo<span class="w"> </span>systemctl<span class="w"> </span>status<span class="w"> </span>cosma-qc-dispatcher
|
||||
|
||||
<span class="c1"># Logs dispatcher</span>
|
||||
tail<span class="w"> </span>-f<span class="w"> </span>/home/cosma/cosma-qc-data/dispatcher.log
|
||||
|
||||
<span class="c1"># Dashboard local</span>
|
||||
http://192.168.0.83:3849
|
||||
</pre></div>
|
||||
</div>
|
||||
</section>
|
||||
<section id="noeud-core-82">
|
||||
<h2>Nœud core (.82)<a class="headerlink" href="#noeud-core-82" title="Link to this heading"></a></h2>
|
||||
<p><strong>Rôle :</strong> orchestrateur central du pipeline.</p>
|
||||
<p><strong>Rôle :</strong> routeur et services communs.</p>
|
||||
<p>Services actifs :</p>
|
||||
<ul class="simple">
|
||||
<li><p><strong>Dispatcher</strong> — service systemd cosma-qc-dispatcher.
|
||||
Boucle principale qui dispatch les jobs aux workers GPU.</p></li>
|
||||
<li><p><strong>Dashboard FastAPI</strong> — conteneur Docker exposé sur le port <strong>3849</strong>.
|
||||
Interface web de monitoring des jobs.</p></li>
|
||||
<li><p><strong>Gitea</strong> — dépôt source floppyrj45/cosma-qc.</p></li>
|
||||
<li><p><strong>Caddy</strong> — reverse proxy HTTPS, route <code class="docutils literal notranslate"><span class="pre">/cosma-qc</span></code> vers .83:3849.</p></li>
|
||||
<li><p><strong>Gitea</strong> — dépôt source <code class="docutils literal notranslate"><span class="pre">floppyrj45/cosma-qc</span></code>.</p></li>
|
||||
<li><p><strong>Grafana / InfluxDB</strong> — monitoring infrastructure.</p></li>
|
||||
</ul>
|
||||
<p>Commandes utiles :</p>
|
||||
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Statut dispatcher</span>
|
||||
sudo<span class="w"> </span>systemctl<span class="w"> </span>status<span class="w"> </span>cosma-qc-dispatcher
|
||||
|
||||
<span class="c1"># Logs dispatcher temps réel</span>
|
||||
sudo<span class="w"> </span>journalctl<span class="w"> </span>-u<span class="w"> </span>cosma-qc-dispatcher<span class="w"> </span>-f
|
||||
|
||||
<span class="c1"># Dashboard</span>
|
||||
http://192.168.0.82:3849
|
||||
</pre></div>
|
||||
</div>
|
||||
</section>
|
||||
<section id="noeuds-gpu-workers-84-et-87">
|
||||
<h2>Nœuds GPU workers (.84 et .87)<a class="headerlink" href="#noeuds-gpu-workers-84-et-87" title="Link to this heading"></a></h2>
|
||||
@@ -137,7 +155,7 @@ http://192.168.0.82:3849
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr class="row-odd"><th class="head"><p>IP</p></th>
|
||||
<th class="head"><p>Nom</p></th>
|
||||
<th class="head"><p>Nom SSH</p></th>
|
||||
<th class="head"><p>GPU</p></th>
|
||||
<th class="head"><p>VRAM</p></th>
|
||||
</tr>
|
||||
@@ -155,69 +173,36 @@ http://192.168.0.82:3849
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><strong>Rôle :</strong> exécution de ffmpeg (extraction frames) et lingbot-map (reconstruction 3D).</p>
|
||||
<p>Répertoire de travail sur chaque worker :</p>
|
||||
<p><strong>Rôle :</strong> exécution de ffmpeg et lingbot-map.</p>
|
||||
<div class="highlight-text notranslate"><div class="highlight"><pre><span></span>/cosma-qc-frames/
|
||||
├── job_1/
|
||||
│ ├── frame_000001.jpg … frame_NNNNNN.jpg
|
||||
│ ├── .video_0.done
|
||||
│ ├── reconstruction.ply
|
||||
│ └── reconstruction.glb (généré à la demande)
|
||||
├── job_2/
|
||||
│ └── …
|
||||
└── stitch_1.ply
|
||||
└── job_N/
|
||||
├── frame_NNNNNN.jpg
|
||||
├── .video_V.done
|
||||
├── reconstruction.ply
|
||||
└── reconstruction.glb
|
||||
</pre></div>
|
||||
</div>
|
||||
</section>
|
||||
<section id="noeud-z620-168">
|
||||
<h2>Nœud z620 (.168)<a class="headerlink" href="#noeud-z620-168" title="Link to this heading"></a></h2>
|
||||
<p><strong>Rôle :</strong> stockage des MP4 bruts GoPro.</p>
|
||||
<ul class="simple">
|
||||
<li><p>Proxmox host HP Z620.</p></li>
|
||||
<li><p>SSD monté sur /mnt/portablessd.</p></li>
|
||||
<li><p>Les MP4 <strong>ne quittent jamais</strong> z620 — ffmpeg s’y exécute via SSH.</p></li>
|
||||
</ul>
|
||||
<p>Accès SSH depuis core :</p>
|
||||
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>ssh<span class="w"> </span>floppyrj45@192.168.0.168
|
||||
<section id="noeud-z620-168-stockage-mp4">
|
||||
<h2>Nœud z620 (.168) — stockage MP4<a class="headerlink" href="#noeud-z620-168-stockage-mp4" title="Link to this heading"></a></h2>
|
||||
<p>HP Z620 (Proxmox). SSD /mnt/portablessd avec les MP4 GoPro bruts.
|
||||
Les MP4 <strong>ne quittent jamais</strong> z620 — ffmpeg sy exécute via SSH-relay.</p>
|
||||
<p>SSH relay (depuis cosma-vm ou core) :</p>
|
||||
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>ssh<span class="w"> </span>ml-stack<span class="w"> </span><span class="s2">"ssh z620 \\"</span>commande<span class="se">\\</span><span class="s2">""</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</section>
|
||||
<section id="service-systemd-dispatcher">
|
||||
<h2>Service systemd dispatcher<a class="headerlink" href="#service-systemd-dispatcher" title="Link to this heading"></a></h2>
|
||||
<p>Fichier de service : /etc/systemd/system/cosma-qc-dispatcher.service</p>
|
||||
<div class="highlight-ini notranslate"><div class="highlight"><pre><span></span><span class="k">[Unit]</span>
|
||||
<span class="na">Description</span><span class="o">=</span><span class="s">COSMA QC Dispatcher</span>
|
||||
<span class="na">After</span><span class="o">=</span><span class="s">network.target</span>
|
||||
|
||||
<span class="k">[Service]</span>
|
||||
<span class="na">User</span><span class="o">=</span><span class="s">floppyrj45</span>
|
||||
<span class="na">WorkingDirectory</span><span class="o">=</span><span class="s">/home/floppyrj45/docker/cosma-qc</span>
|
||||
<span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/bin/python3 app/dispatcher.py</span>
|
||||
<span class="na">Restart</span><span class="o">=</span><span class="s">on-failure</span>
|
||||
<span class="na">RestartSec</span><span class="o">=</span><span class="s">10</span>
|
||||
|
||||
<span class="k">[Install]</span>
|
||||
<span class="na">WantedBy</span><span class="o">=</span><span class="s">multi-user.target</span>
|
||||
<section id="openvpn-acces-collegues">
|
||||
<h2>OpenVPN — accès collègues<a class="headerlink" href="#openvpn-acces-collegues" title="Link to this heading"></a></h2>
|
||||
<p>Serveur OpenVPN sur cosma-vm (.83:1194 UDP).
|
||||
Adresse externe : <code class="docutils literal notranslate"><span class="pre">laboratoire.freeboxos.fr:1194</span></code></p>
|
||||
<p>Profil client : <code class="docutils literal notranslate"><span class="pre">cosma-qc-collegue1.ovpn</span></code> (Nextcloud sync).</p>
|
||||
<p>Génération dun nouveau client :</p>
|
||||
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span><span class="w"> </span>/home/cosma/openvpn-ca/easyrsa3
|
||||
./easyrsa<span class="w"> </span>gen-req<span class="w"> </span>NOMCLIENT<span class="w"> </span>nopass
|
||||
./easyrsa<span class="w"> </span>sign-req<span class="w"> </span>client<span class="w"> </span>NOMCLIENT
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>Commandes de gestion :</p>
|
||||
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>sudo<span class="w"> </span>systemctl<span class="w"> </span>start<span class="w"> </span>cosma-qc-dispatcher
|
||||
sudo<span class="w"> </span>systemctl<span class="w"> </span>stop<span class="w"> </span>cosma-qc-dispatcher
|
||||
sudo<span class="w"> </span>systemctl<span class="w"> </span>restart<span class="w"> </span>cosma-qc-dispatcher
|
||||
sudo<span class="w"> </span>systemctl<span class="w"> </span><span class="nb">enable</span><span class="w"> </span>cosma-qc-dispatcher<span class="w"> </span><span class="c1"># démarrage auto</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</section>
|
||||
<section id="conteneur-docker-dashboard">
|
||||
<h2>Conteneur Docker dashboard<a class="headerlink" href="#conteneur-docker-dashboard" title="Link to this heading"></a></h2>
|
||||
<p>Le dashboard FastAPI tourne dans un conteneur Docker.</p>
|
||||
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span><span class="w"> </span>/home/floppyrj45/docker/cosma-qc
|
||||
docker<span class="w"> </span>compose<span class="w"> </span>up<span class="w"> </span>-d<span class="w"> </span><span class="c1"># démarrer</span>
|
||||
docker<span class="w"> </span>compose<span class="w"> </span>down<span class="w"> </span><span class="c1"># arrêter</span>
|
||||
docker<span class="w"> </span>compose<span class="w"> </span>logs<span class="w"> </span>-f<span class="w"> </span><span class="c1"># logs</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
<p>Accès : <a class="reference external" href="http://192.168.0.82:3849">http://192.168.0.82:3849</a></p>
|
||||
</section>
|
||||
<section id="ports-reseau-recapitulatifs">
|
||||
<h2>Ports réseau récapitulatifs<a class="headerlink" href="#ports-reseau-recapitulatifs" title="Link to this heading"></a></h2>
|
||||
@@ -234,17 +219,21 @@ docker<span class="w"> </span>compose<span class="w"> </span>logs<span class="w"
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr class="row-even"><td><p>.82</p></td>
|
||||
<tr class="row-even"><td><p>.83</p></td>
|
||||
<td><p>3849</p></td>
|
||||
<td><p>Dashboard FastAPI cosma-qc</p></td>
|
||||
</tr>
|
||||
<tr class="row-odd"><td><p>.84 / .87</p></td>
|
||||
<td><p>8100+N</p></td>
|
||||
<td><p>Viser viewer (reconstruction job N)</p></td>
|
||||
<tr class="row-odd"><td><p>.83</p></td>
|
||||
<td><p>1194/udp</p></td>
|
||||
<td><p>OpenVPN server</p></td>
|
||||
</tr>
|
||||
<tr class="row-even"><td><p>.84 / .87</p></td>
|
||||
<td><p>8100+N</p></td>
|
||||
<td><p>Viser viewer (job N)</p></td>
|
||||
</tr>
|
||||
<tr class="row-odd"><td><p>.84 / .87</p></td>
|
||||
<td><p>8300</p></td>
|
||||
<td><p>HTTP server GLB export</p></td>
|
||||
<td><p>HTTP server GLB/PLY export</p></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
2
docs/_build/html/searchindex.js
vendored
2
docs/_build/html/searchindex.js
vendored
File diff suppressed because one or more lines are too long
@@ -6,49 +6,68 @@ Réseau LAN — 192.168.0.0/24
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ LAN 192.168.0.0/24 │
|
||||
│ │
|
||||
│ .82 CORE Dispatcher (systemd) + FastAPI :3849 │
|
||||
│ Gitea + Grafana + InfluxDB + Caddy │
|
||||
│ │
|
||||
│ .84 ml-stack GPU worker RTX 3090 24GB │
|
||||
│ .87 gpu GPU worker RTX 3060 12GB │
|
||||
│ │
|
||||
│ .168 z620 Proxmox host HP Z620 │
|
||||
│ SSD → /mnt/portablessd (MP4 bruts) │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
┌───────────────────────────────────────────────────────────────┐
|
||||
│ LAN 192.168.0.0/24 │
|
||||
│ │
|
||||
│ .82 CORE Gitea + Grafana + InfluxDB + Caddy │
|
||||
│ Reverse-proxy → cosma-vm pour /cosma-qc │
|
||||
│ │
|
||||
│ .83 cosma-vm Dispatcher cosma-qc (systemd) │
|
||||
│ Dashboard FastAPI :3849 │
|
||||
│ OpenVPN server :1194 UDP │
|
||||
│ │
|
||||
│ .84 ml-stack GPU worker RTX 3090 24GB │
|
||||
│ .87 gpu GPU worker RTX 3060 12GB │
|
||||
│ │
|
||||
│ .168 z620 Proxmox host HP Z620 — SSD MP4 bruts │
|
||||
│ .101 poxML Proxmox hyperviseur — héberge cosma-vm (VM 202)│
|
||||
└───────────────────────────────────────────────────────────────┘
|
||||
|
||||
|
||||
Nœud core (.82)
|
||||
----------------
|
||||
Nœud cosma-vm (.83) — serveur principal cosma-qc
|
||||
--------------------------------------------------
|
||||
|
||||
**Rôle :** orchestrateur central du pipeline.
|
||||
VM #202 sur Proxmox poxML (.101). 4 vCPU, 4 GB RAM, 40 GB SSD.
|
||||
User : ``cosma`` (avec docker group et sudo).
|
||||
|
||||
Services actifs :
|
||||
Services :
|
||||
|
||||
- **Dispatcher** — service systemd cosma-qc-dispatcher.
|
||||
Boucle principale qui dispatch les jobs aux workers GPU.
|
||||
- **Dispatcher** — ``cosma-qc-dispatcher.service`` (systemd)
|
||||
Lance les extractions ffmpeg sur les workers GPU.
|
||||
|
||||
- **Dashboard FastAPI** — conteneur Docker exposé sur le port **3849**.
|
||||
Interface web de monitoring des jobs.
|
||||
- **Dashboard FastAPI** — conteneur Docker ``cosma-qc-app``, port **3849**.
|
||||
Accès via Caddy sur core : ``http://192.168.0.82/cosma-qc``
|
||||
|
||||
- **Gitea** — dépôt source floppyrj45/cosma-qc.
|
||||
|
||||
- **Grafana / InfluxDB** — monitoring infrastructure.
|
||||
- **OpenVPN** — ``openvpn-server@cosma.service``, port 1194 UDP.
|
||||
Permet aux collègues daccéder au LAN depuis lextérieur.
|
||||
|
||||
Commandes utiles :
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# SSH vers cosma-vm
|
||||
ssh cosma@192.168.0.83
|
||||
|
||||
# Statut dispatcher
|
||||
sudo systemctl status cosma-qc-dispatcher
|
||||
|
||||
# Logs dispatcher temps réel
|
||||
sudo journalctl -u cosma-qc-dispatcher -f
|
||||
# Logs dispatcher
|
||||
tail -f /home/cosma/cosma-qc-data/dispatcher.log
|
||||
|
||||
# Dashboard
|
||||
http://192.168.0.82:3849
|
||||
# Dashboard local
|
||||
http://192.168.0.83:3849
|
||||
|
||||
|
||||
Nœud core (.82)
|
||||
----------------
|
||||
|
||||
**Rôle :** routeur et services communs.
|
||||
|
||||
Services actifs :
|
||||
|
||||
- **Caddy** — reverse proxy HTTPS, route ``/cosma-qc`` vers .83:3849.
|
||||
- **Gitea** — dépôt source ``floppyrj45/cosma-qc``.
|
||||
- **Grafana / InfluxDB** — monitoring infrastructure.
|
||||
|
||||
|
||||
Nœuds GPU workers (.84 et .87)
|
||||
@@ -59,7 +78,7 @@ Nœuds GPU workers (.84 et .87)
|
||||
:widths: 15 25 20 40
|
||||
|
||||
* - IP
|
||||
- Nom
|
||||
- Nom SSH
|
||||
- GPU
|
||||
- VRAM
|
||||
* - .84
|
||||
@@ -71,83 +90,46 @@ Nœuds GPU workers (.84 et .87)
|
||||
- RTX 3060
|
||||
- 12 GB
|
||||
|
||||
**Rôle :** exécution de ffmpeg (extraction frames) et lingbot-map (reconstruction 3D).
|
||||
|
||||
Répertoire de travail sur chaque worker :
|
||||
**Rôle :** exécution de ffmpeg et lingbot-map.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
/cosma-qc-frames/
|
||||
├── job_1/
|
||||
│ ├── frame_000001.jpg … frame_NNNNNN.jpg
|
||||
│ ├── .video_0.done
|
||||
│ ├── reconstruction.ply
|
||||
│ └── reconstruction.glb (généré à la demande)
|
||||
├── job_2/
|
||||
│ └── …
|
||||
└── stitch_1.ply
|
||||
└── job_N/
|
||||
├── frame_NNNNNN.jpg
|
||||
├── .video_V.done
|
||||
├── reconstruction.ply
|
||||
└── reconstruction.glb
|
||||
|
||||
|
||||
Nœud z620 (.168)
|
||||
-----------------
|
||||
Nœud z620 (.168) — stockage MP4
|
||||
----------------------------------
|
||||
|
||||
**Rôle :** stockage des MP4 bruts GoPro.
|
||||
HP Z620 (Proxmox). SSD /mnt/portablessd avec les MP4 GoPro bruts.
|
||||
Les MP4 **ne quittent jamais** z620 — ffmpeg sy exécute via SSH-relay.
|
||||
|
||||
- Proxmox host HP Z620.
|
||||
- SSD monté sur /mnt/portablessd.
|
||||
- Les MP4 **ne quittent jamais** z620 — ffmpeg s'y exécute via SSH.
|
||||
|
||||
Accès SSH depuis core :
|
||||
SSH relay (depuis cosma-vm ou core) :
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
ssh floppyrj45@192.168.0.168
|
||||
ssh ml-stack "ssh z620 \\"commande\\""
|
||||
|
||||
|
||||
Service systemd dispatcher
|
||||
---------------------------
|
||||
OpenVPN — accès collègues
|
||||
--------------------------
|
||||
|
||||
Fichier de service : /etc/systemd/system/cosma-qc-dispatcher.service
|
||||
Serveur OpenVPN sur cosma-vm (.83:1194 UDP).
|
||||
Adresse externe : ``laboratoire.freeboxos.fr:1194``
|
||||
|
||||
.. code-block:: ini
|
||||
Profil client : ``cosma-qc-collegue1.ovpn`` (Nextcloud sync).
|
||||
|
||||
[Unit]
|
||||
Description=COSMA QC Dispatcher
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
User=floppyrj45
|
||||
WorkingDirectory=/home/floppyrj45/docker/cosma-qc
|
||||
ExecStart=/usr/bin/python3 app/dispatcher.py
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
|
||||
Commandes de gestion :
|
||||
Génération dun nouveau client :
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
sudo systemctl start cosma-qc-dispatcher
|
||||
sudo systemctl stop cosma-qc-dispatcher
|
||||
sudo systemctl restart cosma-qc-dispatcher
|
||||
sudo systemctl enable cosma-qc-dispatcher # démarrage auto
|
||||
|
||||
|
||||
Conteneur Docker dashboard
|
||||
---------------------------
|
||||
|
||||
Le dashboard FastAPI tourne dans un conteneur Docker.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
cd /home/floppyrj45/docker/cosma-qc
|
||||
docker compose up -d # démarrer
|
||||
docker compose down # arrêter
|
||||
docker compose logs -f # logs
|
||||
|
||||
Accès : http://192.168.0.82:3849
|
||||
cd /home/cosma/openvpn-ca/easyrsa3
|
||||
./easyrsa gen-req NOMCLIENT nopass
|
||||
./easyrsa sign-req client NOMCLIENT
|
||||
|
||||
|
||||
Ports réseau récapitulatifs
|
||||
@@ -160,12 +142,16 @@ Ports réseau récapitulatifs
|
||||
* - Host
|
||||
- Port
|
||||
- Service
|
||||
* - .82
|
||||
* - .83
|
||||
- 3849
|
||||
- Dashboard FastAPI cosma-qc
|
||||
* - .83
|
||||
- 1194/udp
|
||||
- OpenVPN server
|
||||
* - .84 / .87
|
||||
- 8100+N
|
||||
- Viser viewer (reconstruction job N)
|
||||
- Viser viewer (job N)
|
||||
* - .84 / .87
|
||||
- 8300
|
||||
- HTTP server GLB export
|
||||
- HTTP server GLB/PLY export
|
||||
|
||||
|
||||
Binary file not shown.
Reference in New Issue
Block a user