Densifying Assumed-sparse Tensors: Improving Memory Efficiency and MPI Collective Performance during Tensor Accumulation for Parallelized Training of Neural Machine Translation Models
Event Type
Research Paper
AI/Machine Learning/Deep Learning
Parallel Algorithms
Performance Analysis and Optimization
TimeTuesday, June 18th9am - 9:30am CEST
LocationAnalog 1, 2
DescriptionNeural machine translation - using neural networks to translate human language - is an area of active research exploring new neuron types and network topologies with the goal of dramatically improving machine translation performance. Current state-of-the-art approaches, such as the multi-head attention-based transformer, require very large translation corpuses and many epochs to produce models of reasonable quality. Recent attempts to parallelize the official TensorFlow "Transformer" model across multiple nodes have hit roadblocks due to excessive memory use and resulting out of memory errors when performing MPI collectives.

This paper describes modifications made to the Horovod MPI-based distributed training framework to reduce memory usage for transformer models by converting assumed-sparse tensors to dense tensors, and subsequently replacing sparse gradient gather with dense gradient reduction. The result is a dramatic increase in scale-out capability, with CPU-only scaling tests achieving 91% weak scaling efficiency up to 1200 MPI processes (300 nodes), and up to 65% strong scaling efficiency up to 400 MPI processes (200 nodes) using the Stampede2 supercomputer.
Software Development Engineer
Applied Scientist
Principal Engineer
Senior Engineer
Systems Development Engineer, Deep Learning
Senior Manager
AI Software Principal Engineer
Artificial Intelligence Research
Principal Systems Development Engineer
Data Scientist