Megatron LM
Megatron LM has a captured commit titled Fix MFSDP v2 backward with shared-layer MTP.
Megatron LM: Ongoing research training transformer models at scale.

Source-linked text edition ·
The written, source-linked counterpart to this video.
Published edition1 story