[Storage] Add coreweave object store (#6519)

* Add coreweave

* Format

* Format

* format

* Add conf

* Add conf

* Add conf

* Add conf

* format

* Add check for createbucket

* Add check for createbucket

* Add check for createbucket

* Add check for createbucket

* Add check for createbucket

* Add check for createbucket

* Change coreweave to be like cloudflare

* format

* Address comments

* support cw object storage

* fix storage delete command

* remove useless code

* disable bucket creation

* add ut

* update doc

* address comments

---------

Co-authored-by: DanielZhang <[email protected]>

Author: Maknee

docs/source/getting-started/installation.rst

@@ -694,6 +694,55 @@ Seeweb |community-badge|
     api_key = <your-api-token>
 
 
+CoreWeave
+~~~~~~~~~
+
+`CoreWeave <https://www.coreweave.com/>`__ provides S3-compatible object storage that can be used with SkyPilot for storing and accessing data in your workloads.
+
+**Step 1: Configure CoreWeave credentials**
+
+SkyPilot uses separate configuration files for CoreWeave to avoid conflicts with your AWS credentials. Run the following command to configure your CoreWeave access credentials:
+
+.. code-block:: shell
+
+  AWS_SHARED_CREDENTIALS_FILE=~/.coreweave/cw.credentials aws configure --profile cw
+
+When prompted, enter your CoreWeave Object Storage credentials:
+
+.. code-block:: text
+
+  AWS Access Key ID [None]: <your_access_key_id>
+  AWS Secret Access Key [None]: <your_secret_access_key>
+  Default region name [None]:
+  Default output format [None]: json
+
+**Step 2: Configure the S3 endpoint**
+
+Next, configure the endpoint URL and addressing style for CoreWeave Object Storage. This tells AWS CLI how to connect to CoreWeave's S3-compatible service:
+
+.. code-block:: shell
+
+  # For external access (outside CoreWeave CKS clusters)
+  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set endpoint_url https://cwobject.com --profile cw
+  AWS_CONFIG_FILE=~/.coreweave/cw.config aws configure set s3.addressing_style virtual --profile cw
+
+.. note::
+
+  **Choosing the right endpoint**:
+
+  - **External access (recommended)**: Use ``https://cwobject.com`` when launching SkyPilot clusters in non-CoreWeave CKS clusters. This endpoint is accessible from anywhere and uses secure HTTPS.
+
+  - **Internal access (advanced)**: Use ``http://cwlota.com`` only if you are launching SkyPilot clusters inside CoreWeave CKS clusters and do not need to upload local data to the bucket. The LOTA endpoint provides faster access within CoreWeave's network but only supports HTTP and is not accessible externally. Refer to `LOTA documentation <https://docs.coreweave.com/docs/products/storage/object-storage/lota/about>`_ for more details.
+
+**Obtaining your credentials**
+
+To get your CoreWeave Object Storage Access Key ID and Secret Access Key:
+
+1. Log into your `CoreWeave Cloud console <https://cloud.coreweave.com/>`__.
+2. Navigate to **Object Storage** → **Keys** in the left sidebar.
+3. Generate a new key pair.
+
+
 Request quotas for first time users
 --------------------------------------
 

docs/source/reference/storage.rst

@@ -3,7 +3,7 @@
 Cloud Buckets
 ==============
 
-SkyPilot tasks can access data from buckets in cloud object storages such as AWS S3, Google Cloud Storage (GCS), Cloudflare R2, OCI Object Storage or IBM COS.
+SkyPilot tasks can access data from buckets in cloud object storages such as AWS S3, Google Cloud Storage (GCS), Cloudflare R2, CoreWeave Object Storage, OCI Object Storage or IBM COS.
 
 Buckets are made available to each task at a local path on the remote VM, so
 the task can access bucket objects as if they were local files.
@@ -28,7 +28,7 @@ Object storages are specified using the :code:`file_mounts` field in a SkyPilot
           # Mount an existing S3 bucket
           file_mounts:
             /my_data:
-              source: s3://my-bucket/  # or gs://, https://<azure_storage_account>.blob.core.windows.net/<container>, r2://, cos://<region>/<bucket>, oci://<bucket_name>
+              source: s3://my-bucket/  # or gs://, https://<azure_storage_account>.blob.core.windows.net/<container>, r2://, cw://, cos://<region>/<bucket>, oci://<bucket_name>
               mode: MOUNT  # MOUNT or COPY or MOUNT_CACHED. Defaults to MOUNT. Optional.
 
         This will `mount <storage-mounting-modes_>`__ the contents of the bucket at ``s3://my-bucket/`` to the remote VM at ``/my_data``.
@@ -45,7 +45,7 @@ Object storages are specified using the :code:`file_mounts` field in a SkyPilot
           file_mounts:
             /my_data:
               name: my-sky-bucket
-              store: gcs  # Optional: either of s3, gcs, azure, r2, ibm, oci
+              store: gcs  # Optional: either of s3, gcs, azure, r2, coreweave, ibm, oci
 
         SkyPilot will create an empty GCS bucket called ``my-sky-bucket`` and mount it at ``/my_data``.
         This bucket can be used to write checkpoints, logs or other outputs directly to the cloud.
@@ -68,7 +68,7 @@ Object storages are specified using the :code:`file_mounts` field in a SkyPilot
             /my_data:
               name: my-sky-bucket
               source: ~/dataset  # Optional: path to local data to upload to the bucket
-              store: s3  # Optional: either of s3, gcs, azure, r2, ibm, oci
+              store: s3  # Optional: either of s3, gcs, azure, r2, coreweave, ibm, oci
               mode: MOUNT  # Optional: either MOUNT or COPY. Defaults to MOUNT.
 
         SkyPilot will create a S3 bucket called ``my-sky-bucket`` and upload the
@@ -162,11 +162,16 @@ its performance requirements and size of the data.
     The targets must be separately mounted or else the symlinks may break.
 
 .. note::
-    **Architecture compatibility**: S3 storage mounting (including S3-compatible services like 
-    Cloudflare R2 and Nebius) works on all architectures including ARM64 (e.g., Apple Silicon, 
-    AWS Graviton). SkyPilot automatically uses the optimal mounting tool for each architecture: 
+    **Architecture compatibility**: S3 storage mounting (including S3-compatible services like
+    Cloudflare R2, CoreWeave Object Storage, and Nebius) works on all architectures including ARM64 (e.g., Apple Silicon,
+    AWS Graviton). SkyPilot automatically uses the optimal mounting tool for each architecture:
     goofys for x86_64 and rclone for ARM64.
 
+.. note::
+    CoreWeave buckets may take a long time to become accessible after creation.
+    Therefore, SkyPilot does not automatically create them.
+    Please manually create your CoreWeave bucket and verify its accessibility before using it.
+
 .. _mount_cached_mode_in_detail:
 
 MOUNT_CACHED mode in detail
@@ -344,14 +349,15 @@ Storage YAML reference
             - gs://<bucket_name>
             - https://<azure_storage_account>.blob.core.windows.net/<container_name>
             - r2://<bucket_name>
+            - cw://<bucket_name>
             - cos://<region_name>/<bucket_name>
             - oci://<bucket_name>@<region>
 
           If the source is local, data is uploaded to the cloud to an appropriate
-          bucket (s3, gcs, azure, r2, oci, or ibm). If source is bucket URI,
+          bucket (s3, gcs, azure, r2, coreweave, oci, or ibm). If source is bucket URI,
           the data is copied or mounted directly (see mode flag below).
 
-        store: str; either of 's3', 'gcs', 'azure', 'r2', 'ibm', 'oci'
+        store: str; either of 's3', 'gcs', 'azure', 'r2', 'coreweave','ibm', 'oci'
           If you wish to force sky.Storage to be backed by a specific cloud object
           storage, you can specify it here. If not specified, SkyPilot chooses the
           appropriate object storage based on the source path and task's cloud provider.