This document is relevant for: Trn2, Trn3

Quickstart: Launch an OpenAI-compatible API server with vLLM Neuron

This quickstart guides you through launching an OpenAI-compatible API server with vLLM Neuron and validating it with your first request.

Prerequisites

• Completed the setup guide (vLLM Neuron installed and verified).

• Permission to pull the model you plan to serve (for example, an accepted Hugging Face license).

• Familiarity with basic shell and HTTP clients such as curl or the OpenAI Python SDK.

Step 1: Launch the API server

In this step, you start an OpenAI-compatible endpoint serving GPT-OSS 20B.

First, set compilation and execution timeouts. GPT-OSS compilation can exceed the default 600-second budget:

export VLLM_NEURON_COMPILATION_TIMEOUT=1200
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1200

Then launch the server:

vllm serve openai/gpt-oss-20b \
  --tensor-parallel-size 8 \
  --max-model-len 4096 \
  --max-num-batched-tokens 2048 \
  --max-num-seqs 4 \
  --port 8000 \
  --hf-overrides '{"quantization_config": {}}' \
  --additional-config '{
    "neuron_config": {
      "on_device_sampling_config": {"all_greedy": "true"}
    }
  }'

What each argument does:

• --tensor-parallel-size 8 — shards the model across 8 NeuronCores (fits a 20B model comfortably on Trn2).

• --max-model-len 4096 — maximum sequence length (prompt + generation) the server will accept.

• --max-num-seqs 4 — maximum concurrent requests batched together.

• --additional-config — passes Neuron-specific configs; here enabling greedy on-device sampling.

The first launch compiles the model for Neuron, which takes several minutes. Once ready you will see:

INFO:     Started server process [12345]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Step 2: Verify the endpoint with curl

In this step, you confirm the server is healthy and responding to chat completion requests.

First, check the health endpoint:

curl http://localhost:8000/health

You should receive an empty 200 OK response. Next, send a chat completion request:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "openai/gpt-oss-20b",
        "messages": [
          {"role": "user", "content": "What is the capital of France? Reply in one sentence."}
        ],
        "max_tokens": 30,
        "temperature": 0
      }' | python -m json.tool

Expected output (generation text will vary):

{
    "id": "chatcmpl-abc123",
    "object": "chat.completion",
    "created": 1715000000,
    "model": "openai/gpt-oss-20b",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "The capital of France is Paris."
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 18,
        "completion_tokens": 7,
        "total_tokens": 25
    }
}

If you see a JSON response with a choices array containing generated text, the server is working correctly.

Step 3: Call the API with the OpenAI Python SDK

In this step, you confirm that standard OpenAI client libraries work unchanged against the Neuron endpoint.

Install the SDK if you haven’t already:

pip install openai

Then run the following script:

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
)

response = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[
        {"role": "user", "content": "Explain what Neuron cores are in two sentences."}
    ],
    max_tokens=60,
    temperature=0,
)

print(response.choices[0].message.content)

Expected output:

Neuron cores are custom-designed machine learning accelerators built by AWS,
optimized for high-throughput inference and training workloads. They are the
processing units inside AWS Inferentia and Trainium chips.

If the script prints a coherent completion, you have confirmed end-to-end OpenAI SDK compatibility.

Confirmation

You have launched a vLLM server on Neuron, validated it over HTTP with curl, and called it from the OpenAI Python SDK. Any application or framework that targets the OpenAI chat completions API can now point to http://<your-instance-ip>:8000/v1 and use Neuron-accelerated inference without code changes.

Common issues

• Server fails to start: Verify the Neuron SDK and vLLM Neuron plugin versions match. See setup guide.

• Model pull fails: Confirm your Hugging Face token is set and you have access to the model.

• Out-of-memory at startup: Reduce --max-num-seqs or --max-model-len, or use a smaller model for the quickstart.

Clean up

Stop the server with Ctrl+C in the terminal where it’s running.

Next steps

• Quickstart: Offline serving — Offline batch inference with vLLM Neuron.

• Features guide — Detailed feature coverage.

This document is relevant for: Trn2, Trn3