This document is relevant for: Trn1, Trn2, Trn3

NxD Training

NxD Training is a PyTorch library for end-to-end distributed training.

NxD Training Overview & Setup

• Overview

• Setup

API Reference Guide

• YAML Configuration Settings

Developer Guide

• Integrating a New Model

• Integrating a new dataset/dataloader

• Registering an optimizer and LR scheduler

• Migrating from Neuron-NeMo-Megatron to Neuronx Distributed Training

• NxD Training Compatibility with NeMo

Tutorials

• Megatron GPT Pretraining

• HuggingFace Llama3.1/Llama3-8B Pretraining

• HuggingFace Llama3.1/Llama3-8B Supervised Fine-tuning

• HuggingFace Llama3.1/Llama3-8B Efficient Supervised Fine-tuning with LoRA (Beta)

• HF Llama3.1/Llama3-8B Direct Preference Optimization (DPO) and Odds Ratio Preference Optimization (ORPO) based Fine-tuning (Beta)

• HuggingFace Llama3.1/Llama3-70B Pretraining

• Checkpoint Conversion

App Notes

• Introducing NxD Training

• Tensor Parallelism Overview

• Pipeline Parallelism Overview

• Activation Memory Reduction

Misc

• Known Issues and Workarounds

This document is relevant for: Trn1, Trn2, Trn3