This document is relevant for: Inf2, Trn1, Trn2

Training Tutorials

• Training with Tensor Parallelism

• Training GPT-NeoX 6.9B with Tensor Parallelism and ZeRO-1 Optimizer

• Training GPT-NeoX 20B with Tensor Parallelism and ZeRO-1 Optimizer

• Training Llama3.1-8B, Llama3-8B and Llama2-7B with Tensor Parallelism and ZeRO-1 Optimizer

• Training Llama-3.1-70B, Llama-3-70B or Llama-2-13B/70B with Tensor Parallelism and Pipeline Parallelism

• Training Llama-2-7B/13B/70B using Tensor Parallelism and Pipeline Parallelism with Neuron PyTorch-Lightning

• Fine-tuning Llama2 7B with tensor parallelism and ZeRO-1 optimizer using Neuron PyTorch-Lightning

• Fine-tuning Llama3 8B with tensor parallelism and LoRA using Neuron PyTorch-Lightning

This document is relevant for: Inf2, Trn1, Trn2