The idea, in brief
In this talk, I describe CARD, our approach to improving MoE training on Ascend NPU clusters. We use training logs and profiling to find where waiting and repeated computation slow a training step.
A closer look
Four optimization areas
-
Communication
Coordinate communication and computation
CARD schedules the full MoE layer in one framework, covering token dispatch, expert computation, the return of expert outputs, shared experts, and the backward pass. This reduces communication exposed on the critical path.
-
Expert computation
Streamline expert computation and backpropagation
For grouped matrix multiplication, activations, and routing probability calculations inside MoE experts, CARD reduces the intermediate tensors saved by standard autograd and its backward scheduling overhead. Selected paths use direct gradient computation and accumulation to reduce redundant computation and device memory use.
-
Output & loss
Optimize the output layer and loss
For the LM head and loss, CARD combines chunked loss computation, output-layer gradient accumulation, and matrix accumulation on the NPU. Together, they preserve the loss result and reduce intermediate buffers and repeated matrix multiplications in the backward pass of large matrix operations.
-
Pipeline & memory
Balance memory and recomputation
CARD chooses activation storage and recomputation policies for each pipeline stage to balance memory use and recomputation. Automatic policy generation reduces manual configuration and makes the policies easier to adapt to different layer counts and parallelism configurations.
The original talk
Seminar announcement
This English overview is based on Guangming Laboratory’s Chinese announcement and abstract for Communication-Aware Scheduling Optimization for Large-Scale MoE Model Training on Ascend NPUs. The original includes the event poster and speaker biography.