Quickstart¶
1. Check the cluster¶
Confirm your cluster meets the requirements:
kubectl auth can-i '*' '*'
kubectl get nodes
kubectl describe node <gpu-node> | grep -A2 "nvidia.com/gpu"
kubectl get storageclass
2. Set the environment variables¶
| Variable | Required | Purpose |
|---|---|---|
PULUMI_CONFIG_PASSPHRASE |
Yes | Encrypts installer state. Reuse the same value on every run |
HF_TOKEN |
Yes | Downloads model weights from Hugging Face |
GHCR_TOKEN |
No | Only needed if using private images |
PRIVATE_KEY_PATH |
On-prem | Path inside the container to the SSH key used for Harbor image preload |
CLOUDSDK_CONFIG |
No | Lets the GKE auth plugin work inside the container |
export PULUMI_CONFIG_PASSPHRASE="<passphrase>"
export HF_TOKEN="<token>"
Store the passphrase with your other platform secrets. Without it the installer cannot read its previous state on upgrades.
3. Run the installer¶
The installer ships with everything it deploys: Pulumi projects, charts, CRDs, the image list and the supported models are baked into the image. You choose which models to serve in the installer, after it connects to your cluster.
STORAGE_PATH=<storage-path>
mkdir -p "${STORAGE_PATH}"
docker run --rm -it \
--pull always \
--network host \
-e PULUMI_CONFIG_PASSPHRASE \
-e HF_TOKEN \
-v "$HOME/.kube/config:/.kube/config:ro" \
--mount "type=bind,src=${STORAGE_PATH},dst=/var/shaide-installer" \
ghcr.io/axem-solutions/shaide/installer:v1.0.0
The installer prompts for configuration and deploys the platform. Installation can take some time, because model weights must be uploaded to the internal Harbor registry and pulled onto GPU nodes.
Full walkthrough: Installer guide.
4. Verify¶
curl https://<endpoint>/v1/models -H "Authorization: Bearer <key>"
Returns the models currently served. More checks in Verify installation.
5. First request¶
from openai import OpenAI
client = OpenAI(base_url="https://<endpoint>/v1", api_key="<key>")
response = client.chat.completions.create(
model="<model-id>",
messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)