This document is relevant for: Trn2, Trn3

Multimodal#

Design documentation for multimodal inference. For configuration guidance, see Multimodal support in the features guide.

Topic

Description

Block Packing Attention

FFD block packing for multi-image attention efficiency

On-Device Encoder Cache

Block-based on-device cache for encoder outputs

M-RoPE

Spatial position embeddings for VLMs

Encoder Cache Metrics

Cache layers and their Prometheus metrics

This document is relevant for: Trn2, Trn3