Skip to content

Quickstart

Get a model serving in five minutes. This page covers the fast path: deploy the pack, apply one LLMModel, and call the API.

For a full production-grade setup (air-gapped clusters, HuggingFace token secrets, monitoring, and all values), see Installation and Configuration.

Prerequisites

Section titled “Prerequisites”

Deploy the pack

Section titled “Deploy the pack”

The pack is deployed as an ArgoCD Application. A multi-source setup lets you keep model definitions in a separate Git repo:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: nebari-llm-serving
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "7"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: foundational            # adjust to your ArgoCD project


  sources:
    # Source 1: LLM serving pack Helm chart
    - repoURL: https://github.com/nebari-dev/llm-serving-pack.git
      targetRevision: v0.1.0-alpha.9
      path: charts/nebari-llm-serving
      helm:
        releaseName: nebari-llm-serving
        values: |
          platform:
            baseDomain: "your-cluster.example.com"
            # Gateway names below must match the Envoy Gateways in your cluster.
            # This example points both endpoints at one shared gateway; the chart
            # default for the internal gateway is "nebari-internal-gateway".
            gateway:
              external:
                name: nebari-gateway
                namespace: envoy-gateway-system
              internal:
                name: nebari-gateway
                namespace: envoy-gateway-system
              manageSharedListeners: true
            tls:
              clusterIssuer: letsencrypt-production


          defaults:
            storage:
              storageClassName: efs-sc  # or gp3, longhorn, etc.


          auth:
            oidc:
              issuerURL: "https://keycloak.your-cluster.example.com/realms/nebari"
              groupsClaim: groups


          keyManager:
            enabled: true


    # Source 2: LLMModel CRs from your cluster config repo
    - repoURL: https://github.com/your-org/your-cluster-config.git
      targetRevision: main
      path: clusters/your-cluster/manifests/llm-models


  destination:
    server: https://kubernetes.default.svc
    namespace: nebari-llm-serving-system


  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
      - SkipDryRunOnMissingResource=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

For all available Helm values, see Configuration.

Deploy a model

Section titled “Deploy a model”

Add an LLMModel resource to your cluster config repo (the path referenced by Source 2 above):

apiVersion: llm.nebari.dev/v1alpha1
kind: LLMModel
metadata:
  name: qwen3-5-35b-a3b-gptq-int4
  namespace: nebari-llm-serving-system
spec:
  model:
    name: "Qwen/Qwen3.5-35B-A3B-GPTQ-Int4"
    source: huggingface
    storage:
      type: pvc
      size: "30Gi"
      # storageClassName: efs-sc  # optional, overrides the pack default
  resources:
    gpu:
      count: 1
      type: nvidia
    requests:
      cpu: "2"
      memory: "8Gi"
    limits:
      cpu: "4"
      memory: "12Gi"
  serving:
    replicas: 1
    tensorParallelism: 1
    vllmArgs:
      - "--quantization"
      - "gptq_marlin"
      - "--max-model-len"
      - "8192"
  access:
    public: false
    groups:
      - "llm"
  endpoints:
    external:
      enabled: true
    internal:
      enabled: true

For gated HuggingFace models, create a Secret holding your token, then reference it from the model spec:

Terminal window
kubectl create secret generic hf-token -n nebari-llm-serving-system \
  --from-literal=HF_TOKEN=hf_your_token_here
spec:
  model:
    authSecretName: hf-token  # Secret with key "HF_TOKEN"

The operator handles the rest: model download, vLLM pods, InferencePool, routing, and auth. Watch progress with:

Terminal window
kubectl -n nebari-llm-serving-system get llmmodels -w

For the full CRD reference including all spec fields, see Configuration.

Use the model

Section titled “Use the model”

All models on the cluster share one hostname pair. Clients select a model via the model field in the request body, matching the OpenAI API convention.

External access (API key)

Section titled “External access (API key)”

Generate a key via the key manager UI, served at the hostname you set in keyManager.nebariApp.hostname (e.g. https://keys.llm.<baseDomain>/). Then:

Terminal window
curl https://llm.your-cluster.example.com/v1/chat/completions \
  -H "Authorization: Bearer sk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3.5-35B-A3B-GPTQ-Int4", "messages": [{"role": "user", "content": "Hello"}]}'

You can also mint a key from the command line instead of the UI. The key-manager exposes a REST API under the same hostname (keyManager.nebariApp.hostname, e.g. keys.llm.<baseDomain>):

Terminal window
# Mint a key via the API instead of the UI (bearer token = a Keycloak JWT):
curl -X POST https://keys.llm.<baseDomain>/api/keys \
  -H "Authorization: Bearer <your-jwt>" \
  -H "Content-Type: application/json" \
  -d '{"modelName": "Qwen/Qwen3.5-35B-A3B-GPTQ-Int4"}'

To revoke a key, delete it in the key manager UI, or call DELETE /api/keys/{namespace}/{model}/{clientID} with the same bearer token (the namespace, model, and clientID come from the mint response, or from GET /api/keys). A revoked key immediately returns 401 on the external route.

Internal access (JWT from JupyterLab or in-cluster service)

Section titled “Internal access (JWT from JupyterLab or in-cluster service)”
import os
from openai import OpenAI


client = OpenAI(
    base_url="https://llm-internal.your-cluster.example.com/v1",
    api_key=os.environ["JUPYTERHUB_API_TOKEN"],  # JWT from Nebari
)
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-35B-A3B-GPTQ-Int4",
    messages=[{"role": "user", "content": "Hello"}],
)

Route to an external provider

Section titled “Route to an external provider”

Instead of serving a model locally, a PassthroughModel points the shared endpoints at an external OpenAI-compatible provider (OpenRouter, api.openai.com, a remote vLLM). Users get the same two auth layers and mint keys in the same UI; the platform’s provider key is injected upstream and never exposed.

First create a Secret holding the platform’s provider key (the apiKey field), in the operator namespace:

Terminal window
kubectl create secret generic openrouter-api-key -n nebari-llm-serving-system \
  --from-literal=apiKey='sk-or-v1-...'

Then apply the PassthroughModel:

examples/passthrough-openrouter.yaml
# PassthroughModel routing the shared LLM endpoints to OpenRouter.
#
# Prerequisites:
#   - The pack installed with the AI Gateway (see the Installation guide)
#   - A Secret holding the platform's OpenRouter API key:
#       kubectl create secret generic openrouter-api-key -n nebari-llm-serving-system \
#         --from-literal=apiKey='sk-or-v1-...'
#
# After it reconciles, any OpenRouter model id works against
# https://llm.<baseDomain>/v1 with a key minted in the key-manager UI
# (or against llm-internal.<baseDomain> with a Keycloak bearer token):
#
#   curl https://llm.<baseDomain>/v1/chat/completions \
#     -H "Authorization: Bearer <your-key>" -H "Content-Type: application/json" \
#     -d '{"model":"openai/gpt-5.2","messages":[{"role":"user","content":"hi"}]}'
apiVersion: llm.nebari.dev/v1alpha1
kind: PassthroughModel
metadata:
  name: openrouter
  namespace: nebari-llm-serving-system
spec:
  provider:
    hostname: openrouter.ai
    # OpenRouter serves the OpenAI API under /api/v1 (api.openai.com would be "v1").
    schemaVersion: api/v1
    credentialSecretName: openrouter-api-key
  models:
    # Route every model id not claimed by a locally served LLMModel.
    catchAll: true
    # Declared ids also show up in /v1/models so OpenAI-compatible UIs
    # (OpenWebUI etc.) can render a model picker.
    declared:
      - openai/gpt-5.2
      - anthropic/claude-opus-4.6
      - meta-llama/llama-4-maverick
  access:
    # Same semantics as LLMModel: public: true, or one or more Keycloak groups.
    groups:
      - llm

Once it reconciles, any provider model id works against the shared external endpoint with a minted key (or against llm-internal.<baseDomain> with a Keycloak JWT):

Terminal window
curl https://llm.your-cluster.example.com/v1/chat/completions \
  -H "Authorization: Bearer sk-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{"model": "openai/gpt-5.2", "messages": [{"role": "user", "content": "Hello"}]}'

See the PassthroughModel CRD reference for every field.

Built with Nebari - nebari.dev© 2026 Nebari