PyTorch: LinkedIn sped up the DuaLip solver 75x on GPUs

The PyTorch Foundation published a case study on how LinkedIn rebuilt its distributed linear-programming solver DuaLip on PyTorch to solve extreme-scale optimization problems. The original tool relied on a CPU-bound Scala/Spark stack, and the new GPU-accelerated version is named DuaLip-GPU.

What was linear programming solving at LinkedIn?

Linear programming (LP) is a mathematical method for finding the best decision under given constraints. At LinkedIn it answers the question: out of millions or billions of options, which is the best set of actions under constraints? Concrete examples include matching jobs with potential candidates, balancing multiple business metrics in ranking and recommendations, and optimizing email send volume. Such problems span hundreds of millions of users and up to trillions of decision variables.

How was the speedup achieved?

The PyTorch implementation relies on three techniques: sparse tensor operations to handle billions to trillions of variables, distributed optimization by partitioning variables across GPUs with synchronization through collective communication (all-reduce, broadcast), and convergence improvements such as row normalization and accelerated gradient descent (AGD/FISTA variants).

What are the results?

The study reports up to 75x faster iterations in wall-clock time on 8 GPUs compared to the original Scala implementation, with an order-of-magnitude speedup over CPU systems and near-linear scaling across multiple GPUs. Accuracy was preserved in the process, which enables LinkedIn “production-scale optimization at scales that were previously infeasible.” The implementation is available as open source.