How to install and deploy AIBrix on a single server?

[Alan-D-Chen]

How to install and deploy AIBrix on a single server?

[Alan-D-Chen]

This is the original question. #1396 (comment)
I kindly request your help. Thank you. @Jeffwan @googs1025

[Alan-D-Chen]

Thank you very much for your reply. However, for beginners who are new to LLM AIBrix, the first installation and use can be difficult. Is there a more clear and straightforward installation and usage tutorial? I have a server with 8 L40S GPUs (I have the Qwen3-32B model files here, and of course other models as well, like Llama3.2 3B).

step 1 https://aibrix.readthedocs.io/latest/getting_started/installation/lambda.html

I encountered some problems during the installation process. First, do I need to create a Docker container on the server and then install AIBrix in this container? According to the tutorial you provided, I tried to install Lambda Cloud. But on the Lambda docs official website (https://docs.lambda.ai/), I don't know how to select and install it.

step 2 I get into https://docs.lambda.ai/ go to the PUBLIC CLOUD

I get these: https://docs.lambda.ai/education/ ,

Large language models (LLMs)# 大型语言模型（LLMs）#
Deploying a Llama 3 inference endpoint
部署Llama 3推理端点
Deploying Llama 3.2 3B in a Kubernetes (K8s) cluster
在Kubernetes（K8s）集群中部署Llama 3.2 3B
Using KubeAI to deploy Nous Research's Hermes 3 and other LLMs
使用KubeAI部署Nous Research的Hermes 3和其他大语言模型
Serving Llama 3.1 405B on a Lambda 1-Click Cluster
在Lambda一键式集群上运行Llama 3.1 405B
Serving the Llama 3.1 8B and 70B models using Lambda Cloud on-demand instances
使用Lambda Cloud按需实例部署Llama 3.1 8B和70B模型
Running DeepSeek-R1 70B using Ollama
使用Ollama运行DeepSeek-R1 70B

Which one should I choose？ Whatever，I get into https://docs.lambda.ai/education/large-language-models/k8s-ollama-llama-3-2/.

IS all right ?

[Alan-D-Chen]

Is there a contradiction between creating a Docker container on the server and using "Deploying Llama 3.2 3B in a Kubernetes (K8s) cluster"? If there is no contradiction, should we first create a Docker container and then deploy Llama 3.2 3B in the Kubernetes (K8s) cluster, or first deploy Llama 3.2 3B in the Kubernetes (K8s) cluster and then create a Docker container? I am very confused.

@Jeffwan Please forgive my forwardness. I am eager to deploy AIBrix on a single server, but I have indeed encountered many problems. Thank you very much for your patience and attention.

[Alan-D-Chen]

@Jeffwan At the very beginning, I followed this tutorial to perform the operations. https://aibrix.readthedocs.io/latest/getting_started/installation/installation.html#install-aibrix-in-testing-environments
First, I used the Dockerfile to create an image, then used this image to create a Docker container,

# 基于现有NVIDIA PyTorch镜像（自带GPU/CUDA支持）
FROM nvcr.io/nvidia/pytorch:24.09-py3

# 关闭交互模式，加速系统包安装
ENV DEBIAN_FRONTEND=noninteractive

# 安装系统依赖工具并清理缓存
RUN apt-get update && apt-get install -y \
    sudo \
    apt-transport-https \
    ca-certificates \
    software-properties-common \
    systemd \
    curl \
    && rm -rf /var/lib/apt/lists/*

# 安装kubectl（通过官方GitHub release，使用国内加速代理）
RUN set -eux; \
    # 使用GitHub镜像加速下载v1.28.0版本kubectl（避免404）
    curl -fL --retry 3 "https://ghproxy.com/https://github.com/kubernetes/kubernetes/releases/download/v1.28.0/kubectl-linux-amd64" -o kubectl \
    && chmod +x kubectl \
    && mv kubectl /usr/local/bin/ \
    && kubectl version --client  # 验证安装

# 安装K3s单节点集群（国内源加速，添加启动验证）
RUN set -eux; \
    curl -sfL --http1.1 https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh | sh -s - \
        --disable traefik \
        --write-kubeconfig-mode 644 \
        --kubeconfig /etc/rancher/k3s/k3s.yaml \
        --data-dir /var/lib/k3s \
        --system-default-registry "registry.cn-hangzhou.aliyuncs.com"; \
    # 等待K3s服务启动（最多等待60秒）
    count=0; \
    until systemctl is-active --quiet k3s; do \
        echo "等待K3s服务启动..."; \
        sleep 5; \
        if [ $((++count)) -ge 12 ]; then \
            echo "K3s启动超时！"; \
            exit 1; \
        fi; \
    done

# 配置kubectl默认连接本地K3s集群
RUN mkdir -p /root/.kube && ln -s /etc/rancher/k3s/k3s.yaml /root/.kube/config

# 复制AIBrix安装脚本并赋予执行权限
COPY install-aibrix.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/install-aibrix.sh

# 容器启动后自动执行AIBrix安装脚本
CMD ["install-aibrix.sh"]

and after that, installed the AIBrix environment according to this file -- install-aibrix.sh .

docker run -itd --name aibrix-alanchen --privileged --gpus all --network host --restart always aibrix-ready:v0.4.1

#!/bin/bash
set -e

# 等待K3s集群就绪（最多30秒）
echo "=== 等待K3s集群启动 ==="
for i in {1..30}; do
    if kubectl get nodes &> /dev/null; then
        echo "K3s集群就绪！"
        break
    fi
    sleep 1
done

# 安装AIBrix依赖组件（Envoy Gateway + KubeRay）
echo -e "\n=== 安装AIBrix依赖 ==="
kubectl apply -f https://github.com/vllm-project/aibrix/releases/download/v0.4.1/aibrix-dependency-v0.4.1.yaml --server-side

# 等待依赖组件启动（超时5分钟）
kubectl wait --for=condition=available deployment --all -n aibrix-system --timeout=300s || true
kubectl wait --for=condition=available deployment --all -n envoy-gateway-system --timeout=300s || true

# 安装AIBrix核心组件
echo -e "\n=== 安装AIBrix核心 ==="
kubectl apply -f https://github.com/vllm-project/aibrix/releases/download/v0.4.1/aibrix-core-v0.4.1.yaml

# 验证安装结果
echo -e "\n=== 安装完成！当前AIBrix组件状态 ==="
kubectl get pods -n aibrix-system
kubectl get pods -n envoy-gateway-system

# 保持容器运行（监控K3s日志）
tail -f /var/log/k3s/k3s.log

[Alan-D-Chen]

But it ultimately failed. Could you please tell me if there is a quick and effective installation tutorial? If possible, I am willing to write a single-server version of the AIBrix installation tutorial based on my actual experience, hoping to make my contribution to the AIBrix community. @Jeffwan Thanks again.

[Alan-D-Chen]

@googs1025 🧑🏻‍💻Thank you very much for your reminder. It is dedicated and professional contributors like you who have been continuously promoting the progress and development of the community. I will follow in your footsteps. Could you please help me solve this problem? #1690 Thank you so much.

[Jeffwan]

@Alan-D-Chen I think you just need to follow this guidance. this page gives you everything you need. what's you followed like lambda cloud llama installation is not helpful. Seems you tried some unrelated guidances.

https://aibrix.readthedocs.io/latest/getting_started/installation/lambda.html

Let me know if you encounter other issues.

[Alan-D-Chen]

hi，goodmoring @Jeffwan .
Hello, I'm sorry, but I've encountered a tricky problem here. When proceeding to this step (https://aibrix.readthedocs.io/latest/getting_started/installation/lambda.html),

I ran into an issue:

kubectl get pods -n aibrix-system

these six Pods cannot be obtained.

I got five of them (or similar versions) from other sources.

I need to modify the tag of kuberay/operator:nightly to meet the requirements of kubectl get pods -n aibrix-system.

However, aibrix/gpu-optimizer:v0.4.1 cannot be found at all with the server and the local PC.

And I change the docker daemon.json:

final, these do not work at all. Can you do me a favor? thank you !

[Alan-D-Chen]

AND, I can not get good help from issues:
#1682
#1539