Not established from the available sources.
META · MODEL RELEASE TRACKER
Llama 3 405B
Llama 3 405B is a specific model in the Llama 3 family. Supplied excerpts report that during its pre-training Meta ran a 54-day pre-training period on up to 16K NVIDIA H100 GPUs and attributed a small number of unexpected job interruptions to silent data corruption in hardware.CURRENT SNAPSHOT0/5 DIMENSIONS WITH DATA
The dimensions that change the decision.
Not established from the available sources.
Not established from the available sources.
Not established from the available sources.
Not established from the available sources.
VERIFIABLE FACTS
Every value stays attached to a source and date.
CAPABILITIES · Pre-training compute and durationDEVELOPER CLAIM
Pre-trained over a 54-day period on up to 16,000 NVIDIA H100 GPUs (distributed training).
LIMITATIONS · Silent data corruption interruptions during pre-trainingDEVELOPER CLAIM
During that pre-training, Meta attributed 6 of 419 unexpected job interruptions to silent data corruption; those were only the cases that both interrupted the job and were traced to silent data corruption.
LIMITATIONS · Silent data corruption caused unexpected job interruptions during pre-trainingDEVELOPER CLAIM
During a 54-day period of Llama 3 405B pre-training on up to 16K NVIDIA H100 GPUs, Meta attributed 6 of 419 unexpected job interruptions to silent data corruption.
WHAT CHANGED
Stored passport versions, without reconstructed history.
LIMITATIONS · Silent data corruption caused unexpected job interruptions during pre-training+ During a 54-day period of Llama 3 405B pre-training on up to 16K NVIDIA H100 GPUs, Meta attributed 6 of 419 unexpected job interruptions to silent data corruption.