More CI TODOs

[troyfeng116]

• Instead of configuring env vars locally on lada, configure from upstream (Github, repo?)
• Read Github Actions secret management documentation, secret env var handling (configure with repo owner permissions?)
• Run Actions in isolated VM to avoid executing malicious programs on lab servers
• Dynamically assign GPUs based on lada workloads (write script to parse nvidia-smi output?)

[troyfeng116]

NOTE: for reproducibility and local testing, environment variables need to be present on local machine to run local tests if tests depend on (use) environment variables.

Question: how to remotely/centrally manage environment variables used in testing, without exposing them in repository, so that 1) GH actions have access (i.e. env vars must be configured in Troy's ziva user space, where the self-hosted runner lives) and 2) all users who wish to run tests locally before pushing to a PR can do so?

If test files live on ziva (as they currently do), then tests will only pass if run locally on ziva (lab servers have distributed disk? so any lab server). However, developers on other machines will not be able to pass tests that depend on local data files, etc.

One solution: push all data files needed for tests to separate repo/registry, configure tests in isolated VM or docker container that pulls data files from registry. Then configure env vars upstream from Github?

[niansong1996]

lab servers have distributed disk? so any lab server

Yes, lab servers have shared storage

I think we can leave this as low-pri for now (as marked in labels), I would say #42 is definitely higher priority