diff --git a/docs/docs.json b/docs/docs.json index 5d37ed1..2db3ae6 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -172,6 +172,7 @@ "group": "Deployment & scaling", "pages": [ "geneva/deployment/index", + "geneva/deployment/helm", "geneva/deployment/troubleshooting" ] }, diff --git a/docs/geneva/deployment/helm.mdx b/docs/geneva/deployment/helm.mdx new file mode 100644 index 0000000..bcf93d3 --- /dev/null +++ b/docs/geneva/deployment/helm.mdx @@ -0,0 +1,86 @@ +--- +title: Deploy Geneva using Helm +sidebarTitle: Helm Deployment +description: Learn how to deploy Geneva on Kubernetes using the Geneva Helm Chart +icon: cogs +--- + + +**Feature Engineering is deployed automatically in LanceDB Enterprise** + +In self-managed environments, Geneva can be installed into existing Kubernetes clusters using Helm. Please [contact LanceDB](https://lancedb.com/contact/) for access to the Helm Chart and related resources. + + +## Pre-requisites + +- An existing Kubernetes cluster +- An existing node pool(s) for Geneva workloads. By default, Geneva uses node selector +`{"geneva.lancedb.com/ray-head": "true"}` for Ray head nodes, and +`{"geneva.lancedb.com/ray-worker-cpu": "true"}` and `{"geneva.lancedb.com/ray-worker-gpu": "true"}` +for Ray CPU worker and Ray GPU worker nodes respectively. This can be overridden in the Geneva client. +- Geneva Helm chart. Please [contact LanceDB](https://lancedb.com/contact/) for access to the Helm Chart and related resources. + +For more information on deploying the required cloud resources, see the [manual deployment instructions](/geneva/deployment/). + +## Geneva Helm Chart + +The Helm chart includes resources required for running [Geneva](https://lancedb.com/docs/geneva/) in Kubernetes. + +It includes services, service accounts, RBAC roles, etc. that are used by the Geneva client to manage resources. + +## Install + +1. Authenticate with Kubernetes cluster, i.e. update kubeconfig +2. Configure Helm chart values + +In values.yaml, configure the service account, node selectors, and cloud resources, if applicable. + +``` +geneva: + # Object storage root URI + rootUri: + value: "s3://my-data-bucket" + + serviceAccount: + # Service account for Geneva worker pods and services + annotations: + # Set per-CSP annotations to provide access to CSP resources, i.e. + # eks.amazonaws.com/role-arn: arn:aws:iam::0123456789:role/geneva_service_role + # iam.gke.io/gcp-service-account: geneva-service-account@my-project.iam.gserviceaccount.com + + gcp: + # GCP service account email for the Geneva client. + # It should have access to the GKS cluster and "roles/storage.objectUser" + # permissions on the object storage bucket. + # e.g., geneva-client-sa@project-id.iam.gserviceaccount.com + clientServiceAccount: "" + + aws: + # AWS IAM role ARN to be assumed by the Geneva client. + # This role should have an access entry to the cluster with username matching the role ARN. + # It should also have r/w access to the object storage bucket. + # e.g., arn:aws:iam::123456789012:role/geneva-client-role + clientRoleArn: "" +``` + +3. Install kuberay operator +```bash +export NAMESPACE=geneva + +helm repo add kuberay https://ray-project.github.io/kuberay-helm/ +helm repo update +helm install kuberay-operator kuberay/kuberay-operator -n $NAMESPACE --create-namespace +``` +4. Install NVIDIA device plugin (if using GPU nodes) + +For GPU support, the NVIDIA device plugin must be installed in your EKS cluster: + +```bash +curl https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.0/deployments/static/nvidia-device-plugin.yml > nvidia-device-plugin.yml +kubectl apply -f nvidia-device-plugin.yml +``` + +5. Install Geneva Helm chart +```bash +helm install geneva ./geneva -n $NAMESPACE --create-namespace +``` \ No newline at end of file diff --git a/docs/geneva/deployment/index.mdx b/docs/geneva/deployment/index.mdx index f391292..14644a9 100644 --- a/docs/geneva/deployment/index.mdx +++ b/docs/geneva/deployment/index.mdx @@ -1,12 +1,12 @@ --- -title: Geneva on Kubernetes Deployments -sidebarTitle: Deployment +title: Manual Deployment on Kubernetes +sidebarTitle: Manual Deployment description: Learn how to deploy Geneva on Kubernetes using KubeRay for distributed feature engineering workflows on GKE and EKS. icon: cogs --- -**Feature Engineering is deployed automatically in LanceDB Enteprise** +**Feature Engineering is deployed automatically in LanceDB Enterprise** Feature Engineering is deployed automatically as part of [LanceDB Enterprise](/enterprise/). For manual installation in self-managed environments, follow the instructions below. @@ -23,6 +23,11 @@ See below for installation instructions for: ## Basic Kubernetes Setup + +Kubernetes resources can be deployed automatically via [Helm](/geneva/deployment/helm/) or manually +via the instructions below. + + In the following sections we'll use these variables: ```bash