merge master

Author: zpoint

CONTRIBUTING.md

@@ -283,18 +283,23 @@ By convention, we define API payloads in `sky/server/api/payloads.py` and there
 - When receiving a payload from an older version without the new field, the default value is used for the missing new field.
 - When receiving a payload from a newer version with a new field, the value of the new field is ignored.
 
-However, when the value of the new field is taken from an user input (e.g. CLI flag), we should add a warning message to inform the user that the new field is ignored. An API version bump is required in this case. For example:
+However, when the value of the new field is taken from user input (e.g. CLI flag), we should warn (or throw an error) to inform the user that the new field is not supported on the current api server version. Calling `versions.get_remote_api_version()` in `sky/client/cli/command.py` will return `None` until we check the api server status which can be triggered by adding a `@server_common.check_server_healthy_or_start` decorator around the cli entry point. For example:
+
 
 ```python
 from sky.server import versions
 
 @click.option('--newflag', default=None)
+# Must have this or the version will be None!
+@server_common.check_server_healthy_or_start
 def cli_entry_point(newflag: Optional[str] = None):
     # The new flag is set but the server does not support the new field yet
     if newflag is not None and versions.get_remote_api_version() < 12:
         logger.warning('The new flag is ignored because the server does not support it yet.')
 ```
 
+We can also just check for the unsupported field in the sdk and surface the error in the cli.
+
 We should also be careful when adding new fields that are not directly visible in
 `sky/server/api/payloads.py`, but is also being sent from the client to the server. This
 is mainly for validating objects from the client.

README.md

@@ -39,6 +39,7 @@
 ----
 
 :fire: *News* :fire:
+- [Oct 2025] Run **RL training for LLMs** with SkyRL on your Kubernetes or clouds: [**example**](./llm/skyrl/)
 - [Oct 2025] Train and serve [Andrej Karpathy's](https://x.com/karpathy/status/1977755427569111362) **nanochat** - the best ChatGPT that $100 can buy: [**example**](./llm/nanochat)
 - [Oct 2025] Run large-scale **LLM training with TorchTitan** on any AI infra: [**example**](./examples/training/torchtitan)
 - [Sep 2025] Scaling AI infrastructure at Abridge - **10x faster development** with SkyPilot: [**blog**](https://blog.skypilot.co/abridge/)
@@ -106,8 +107,7 @@ pip install "skypilot-nightly[kubernetes,aws,gcp,azure,oci,nebius,lambda,runpod,
 </p>
 
 Current supported infra: Kubernetes, AWS, GCP, Azure, OCI, Nebius, Lambda Cloud, RunPod, Fluidstack,
-Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai,
-VMware vSphere, Seeweb.
+Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai, VMware vSphere, Seeweb, Prime Intellect.
 <p align="center">
   <picture>
     <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/skypilot-org/skypilot/master/docs/source/images/cloud-logos-dark.png">

charts/skypilot/templates/grafana-ingress.yaml

@@ -25,7 +25,10 @@ metadata:
 spec:
   ingressClassName: {{ .Values.grafana.ingress.ingressClassName }}
   rules:
-  - http:
+  - {{- if .Values.ingress.host }}
+    host: {{ .Values.ingress.host | quote }}
+    {{- end }}
+    http:
       paths:
       - backend:
           service:

charts/skypilot/values.schema.json

@@ -384,8 +384,17 @@
                         "config": {
                             "type": "object",
                             "properties": {
+                                "gzip-level": {
+                                    "type": "integer"
+                                },
+                                "gzip-min-length": {
+                                    "type": "integer"
+                                },
                                 "http-snippet": {
                                     "type": "string"
+                                },
+                                "use-gzip": {
+                                    "type": "string"
                                 }
                             }
                         },

charts/skypilot/values.yaml

@@ -350,6 +350,9 @@ ingress-nginx:
         service.beta.kubernetes.io/port_443_health-probe_protocol: "TCP"
         service.beta.kubernetes.io/port_80_health-probe_protocol: "TCP"
     config:
+      use-gzip: "true"
+      gzip-level: 5
+      gzip-min-length: 1000
       http-snippet: |
         map $http_upgrade $connection_upgrade {
             default upgrade;
@@ -570,7 +573,10 @@ grafana:
     enableAuthedIngress: true
     path: "/grafana"
     ingressClassName: nginx
-    hosts: null
+    # @schema type: [array]; item: string
+    # If you set hosts to null, the Grafana Helm chart will use its default value ([chart-example.local]).
+    # To match all hosts, set hosts to an empty array ([]).
+    hosts: []
   grafana.ini:
     server:
       domain: localhost

docs/source/docs/index.rst

@@ -206,8 +206,7 @@ SkyPilot **cuts your cloud costs & maximizes GPU availability**:
 SkyPilot supports your existing GPU, TPU, and CPU workloads, with no code changes.
 
 Current supported infra: Kubernetes, AWS, GCP, Azure, OCI, Nebius, Lambda Cloud, RunPod, Fluidstack,
-Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai,
-VMware vSphere.
+Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai, VMware vSphere, Seeweb, Prime Intellect.
 
 .. raw:: html
 

docs/source/examples/managed-jobs.rst

@@ -149,7 +149,7 @@ Work with managed jobs
 
 For a list of all commands and options, run :code:`sky jobs --help` or read the :ref:`CLI reference <cli>`.
 
-See a list of all managed jobs:
+See a list of managed jobs:
 
 .. code-block:: console
 
@@ -163,6 +163,8 @@ See a list of all managed jobs:
   2  roberta  1x [A100:8][Spot]   2 hrs ago   2h 47m 18s      2h 36m 18s     0            RUNNING
   1  bert-qa  1x [V100:1][Spot]   4 hrs ago   4h 24m 26s      4h 17m 54s     0            RUNNING
 
+This command shows 50 managed jobs by default, use ``--limit <num>`` to show more jobs or use ``--all`` to show all jobs.
+
 Stream the logs of a running managed job:
 
 .. code-block:: console
@@ -615,6 +617,7 @@ Submit multiple jobs at once
 
 Pools support a :code:`--num-jobs` flag to conveniently submit multiple jobs at once.
 Each job will be assigned a unique environment variable :code:`$SKYPILOT_JOB_RANK`, which can be used to determine the job partition.
+Additionally, the :code:`$SKYPILOT_NUM_JOBS` environment variable will be set to the total number of jobs submitted.
 
 For example, if you have 1000 prompts to evaluate, each job can process prompts with sequence numbers
 :code:`$SKYPILOT_JOB_RANK * 100` to :code:`($SKYPILOT_JOB_RANK + 1) * 100`.
@@ -630,7 +633,7 @@ Here is a simple example:
     accelerators: {H100:1, H200:1}
 
   run: |
-    echo "Job rank: $SKYPILOT_JOB_RANK"
+    echo "Job rank: $SKYPILOT_JOB_RANK out of $SKYPILOT_NUM_JOBS"
     echo "Processing prompts from $(($SKYPILOT_JOB_RANK * 100)) to $((($SKYPILOT_JOB_RANK + 1) * 100))"
     # Actual business logic here...
     echo "Job $SKYPILOT_JOB_RANK finished"
@@ -766,7 +769,7 @@ For managed jobs, SkyPilot uses an intermediate bucket to store files used in th
 
 If you do not configure a bucket, SkyPilot will automatically create a temporary bucket named :code:`skypilot-filemounts-{username}-{run_id}` for each job launch. SkyPilot automatically deletes the bucket after the job completes.
 
-**Object store access is not necessary to use managed jobs.** If cloud object storage is not available (e.g., Kubernetes deployments), SkyPilot automatically falls back to a two-hop upload that copies files to the jobs controller and then downloads them to the jobs. 
+**Object store access is not necessary to use managed jobs.** If cloud object storage is not available (e.g., Kubernetes deployments), SkyPilot automatically falls back to a two-hop upload that copies files to the jobs controller and then downloads them to the jobs.
 
 .. tip::
 
@@ -1015,4 +1018,4 @@ For absolute maximum parallelism, the following per-cloud configurations are rec
 .. note::
   Remember to tear down your controller to apply these changes, as described above.
 
-With this configuration, you can launch up to 512 jobs at once. Once the jobs are launched, up to 2000 jobs can be running in parallel.
+With this configuration, you can launch up to 512 jobs at once. Once the jobs are launched, up to 2000 jobs can be running in parallel.

docs/source/examples/training/index.rst

@@ -20,4 +20,5 @@ Training
    Training on TPUs <tpu.md>
    Unsloth <unsloth.md>
    Verl (RLHF) <verl.md>
+   SkyRL <skyrl.md>
    Vertex AI <https://medium.com/google-cloud/streamline-ai-ml-model-development-on-gke-with-skypilot-and-vertex-ai-workbench-453729a8897c>

docs/source/examples/training/skyrl.md

@@ -0,0 +1 @@
+../../generated-examples/skyrl.md

docs/source/reference/api-server/examples/api-server-metrics-setup.rst

@@ -35,6 +35,21 @@ chart deploy everything for you with a single command:
       --set prometheus.enabled=true \
       --set grafana.enabled=true
 
+.. dropdown:: Turn off GPU metrics scraping
+
+   The above command also configures Prometheus to scrape the SkyPilot API server's ``/gpu-metrics`` endpoint. To disable scraping of ``/gpu-metrics``, append ``--set prometheus.extraScrapeConfigs=""`` to the Helm command:
+
+   .. code-block:: bash
+
+       helm upgrade --install skypilot skypilot/skypilot-nightly --devel \
+         --namespace skypilot \
+         --create-namespace \
+         --reuse-values \
+         --set apiService.metrics.enabled=true \
+         --set prometheus.enabled=true \
+         --set prometheus.extraScrapeConfigs="" \
+         --set grafana.enabled=true
+
 You can access Grafana at the ``/grafana`` endpoint:
 
 .. code-block:: bash