Deep learning systems have become vital tools across many fields, but the increasing model
sizes mean that training must be accelerated to maintain such systems' utility. Current
systems like Tensorflow and MXNet focus on one specific parallelization strategy, data
parallelism, which requires large training batch sizes in order to scale. We cast the problem
of finding the best parallelization strategy as the problem of finding the best tiling to partition
tensors with the least overall communication. We propose an algorithm that can find the …