This document is relevant for: Trn1, Trn2

Training Tutorials

• Training with Tensor Parallelism

• Training GPT-NeoX 6.9B with Tensor Parallelism and ZeRO-1 Optimizer

• Training GPT-NeoX 20B with Tensor Parallelism and ZeRO-1 Optimizer

• Training Llama-2-7B/13B/70B using Tensor Parallelism and Pipeline Parallelism with Neuron PyTorch-Lightning

• Fine-tuning Llama2 7B with tensor parallelism and ZeRO-1 optimizer using Neuron PyTorch-Lightning

• Fine-tuning Llama3 8B with tensor parallelism and LoRA using Neuron PyTorch-Lightning

This document is relevant for: Trn1, Trn2