580 California St., Suite 400
San Francisco, CA, 94104
Key research themes
1. How can nested and irregular data-parallelism be implemented efficiently for algorithms on complex data structures?
This research theme addresses the challenge of supporting efficient data-parallel computations on irregular and nested data structures such as graphs, sparse matrices, trees, and complex objects within parallel computing frameworks and languages. Efficient execution models and programming abstractions are needed that can represent irregular data and enable parallel traversals or computations without excessive overhead, while preserving portability and performance across different hardware architectures.
Key finding: The paper presents NESL, the first portable nested data-parallel language supporting nested data structures and nested data-parallel functions, enabling concise expression of parallel algorithms over irregular data like... Read more
Key finding: The study proposes an intermediate language and runtime scheduling method to map independent traversals of multiple irregular pointer-based data structures (like forests of decision trees and regular expressions) onto SIMD... Read more
Key finding: This work investigates intra- and inter-object parallelism for query processing on complex, nested objects such as 3D models or molecules in non-standard databases. It proposes a layered architecture using nested transactions... Read more
2. What strategies enable efficient hybrid parallelism for training large-scale deep neural networks beyond conventional data or model parallelism?
As deep learning models grow in size and complexity, single-model data parallelism or model parallelism alone often become insufficient for memory capacity or communication efficiency. This research theme explores techniques combining or extending data, model, pipeline, and operator-level parallelisms to efficiently train huge networks on distributed systems. Key questions include how to split models and data, partition operators, optimize communication, and schedule tasks for maximal scalability and throughput while maintaining training accuracy.
Key finding: RaNNC middleware automatically partitions PyTorch models for hybrid parallelism by identifying atomic subcomponents and grouping them into blocks, then using dynamic programming to find balanced pipeline partitions fitting... Read more
Key finding: Alpa framework unifies data, operator (intra-operator), and pipeline (inter-operator) parallelisms into a hierarchical execution plan space and automatically derives efficient distributed training plans. By mapping... Read more
Key finding: SplitBrain introduces hybrid parallelism for distributed CNN training by combining data parallelism and model parallelism with layer-specific partitioning. Compute-intensive convolution layers are co-located while... Read more
Key finding: This work extends data parallelism by incorporating spatial parallelism to partition single samples in large 3D CNNs, enabling strong scaling beyond mini-batch size limitations. Implemented within LBANN for CosmoFlow and 3D... Read more
3. How can task-parallel pipeline programming models and asynchronous execution improve performance and composability in parallel algorithms?
Pipeline parallelism is a fundamental pattern capturing sequences of task stages with dependencies, common in streaming and hierarchical computations. Current pipeline frameworks focus on data-centric abstractions which are convenient but can be inefficient and inflexible for purely task-parallel pipeline algorithms. This theme investigates programming models, scheduling algorithms, and runtime techniques that separate data abstraction from pipeline task scheduling, enhance composability with other parallel paradigms, and enable efficient dynamic load balancing and resource utilization.
Key finding: Pipeflow is a novel C++ task-parallel pipeline programming framework built atop the Taskflow system that decouples pipeline scheduling from data abstractions. It provides a composable interface enabling users to explicitly... Read more
Key finding: The Encore programming language integrates active object parallelism with unshared local heaps and capabilities to guarantee race-free concurrency without complex synchronization. Combining message-based concurrency with... Read more