This document is relevant for: Trn2, Trn3

vLLM Metrics Integration#

This document describes how vLLM Neuron publishes custom Prometheus metrics alongside vLLM’s built-in metrics at the /metrics endpoint.

Overview#

vLLM exposes a /metrics HTTP endpoint that returns all registered metrics in the Prometheus text exposition format. vLLM Neuron extends this endpoint with Neuron-specific metrics.

Architecture#

vLLM uses prometheus_client with a global default registry. In the default single-server configuration, vLLM runs metrics collection in the API server process and uses IPC (SchedulerStats via EngineCoreOutputs) to transport stats from the EngineCore process.

vLLM Neuron cannot use this IPC path because SchedulerStats is not extensible by plugins. Instead, vLLM Neuron enables prometheus_client multiprocess mode so that metrics observed in the EngineCore and Worker processes are written to shared mmap files and aggregated by the API server’s /metrics endpoint.

[!NOTE] A future improvement would be to propose an upstream vLLM API for plugin metrics that uses the existing IPC path.

Multiprocess Prometheus Setup#

PROMETHEUS_MULTIPROC_DIR is set at module level in vllm_neuron/__init__.py before prometheus_client is imported anywhere. This ensures all processes (API server, EngineCore, workers) use multiprocess mode for metric storage.

# vllm_neuron/__init__.py (top of file)
if "PROMETHEUS_MULTIPROC_DIR" not in os.environ:
    os.environ["PROMETHEUS_MULTIPROC_DIR"] = tempfile.mkdtemp(
        prefix="vllm_neuron_prometheus_"
    )

Users can override this by setting PROMETHEUS_MULTIPROC_DIR before starting the server. The directory must exist and should be wiped between server restarts to avoid stale metrics.

vLLM Neuron Metrics#

vLLM Neuron reports Neuron-specific metrics under the vllm_neuron: prefix. Metrics are defined at module scope in vllm_neuron/metrics.py so they auto-register with the prometheus_client global default registry.

vLLM Neuron uses labels to characterize metrics according to common dimensions like model. Common labels include:

model_name – The name of the model for that server process. Present on all metrics.
bucket_name – The name of the relevant bucket, such as for compilation or execution metrics. The bucket name follows a format similar to prefill_s1024.
rank_id – The process local rank ID. This rank ID is the local rank (i.e. relative to start rank) rather than the core ID.

General Metrics#

Metric Name	Type	Description
`vllm_neuron:num_seqs_padding`	Histogram	Number of padded batch lines processed by the model, labeled by `model_name` and `bucket_name`.
`vllm_neuron:num_batched_tokens_padding`	Histogram	Number of padded sequence lengths processed by the model, labeled by `model_name` and `bucket_name`.
`vllm_neuron:neff_execution_count`	Counter	Number of NEFF executions, labeled by `model_name` and `bucket_name`.

Server Startup Metrics#

Metric Name	Type	Description
`vllm_neuron:startup_time_seconds`	Gauge	Total server startup time from worker spawn to ready, labeled by `model_name`.
`vllm_neuron:compilation_time_seconds`	Gauge	Time spent compiling Neuron graphs (FX trace + neuronxcc compile). Includes compile cache hits. Labeled by `model_name` and `bucket_name`.
`vllm_neuron:model_load_time_seconds`	Gauge	Time spent loading model weights to device (host to HBM transfer), labeled by `model_name`.
`vllm_neuron:model_load_size_bytes`	Gauge	Size of model weights transferred to device (host to HBM transfer), labeled by `model_name`.

Adding New Metrics#

Add the metric definition to vllm_neuron/metrics.py.
Use the vllm_neuron: prefix.
Include a model_name label for consistency with vLLM.
Update the metric from the relevant code path.
Add unit tests and integration tests.