580 California St., Suite 400
San Francisco, CA, 94104
Key research themes
1. How can communication-efficient strategies mitigate bandwidth and latency constraints in distributed and federated learning systems?
This research area focuses on developing algorithms and frameworks that reduce communication overhead in distributed learning settings, such as federated learning, where bandwidth limitations and communication latency pose critical bottlenecks. Efficient communication protocols aim to compress, sparsify, or coordinate information exchange to maintain model performance while minimizing the amount and frequency of data transmitted between clients and servers or peers. This is crucial for scalability, privacy preservation, and operational feasibility in heterogeneous and resource-constrained environments.
Key finding: This work generalizes the DRIVE algorithm to support adjustable bandwidth constraints for gradient compression, utilizing random rotations and Kashin's representation to achieve low normalized mean squared error (NMSE) in... Read more
Key finding: The authors propose Sufficient Factor Broadcasting (SFB), which transmits low-rank updates (outer products of two vectors) rather than full parameter matrices during distributed model training. This reduces communication cost... Read more
Key finding: This paper introduces a distributed event-triggered stochastic gradient descent (DETSGRAD) algorithm that leverages a communication-triggering mechanism allowing agents to update model parameters aperiodically, rather than at... Read more
Key finding: The authors present Federated Stochastic Block Coordinate Descent (FedBCD), a collaborative learning framework for vertically partitioned (feature-distributed) data that enables multiple local updates before communication... Read more
Key finding: Through an extensive simulation campaign comparing centralized federated learning (CFL), gossip federated learning (GFL), and blockchain-enabled federated learning, this study quantifies communication overhead, convergence... Read more
2. What algorithmic frameworks and optimization techniques enable scalable, provably convergent distributed training of nonconvex models such as neural networks?
Training neural networks in distributed environments inherently involves solving nonconvex optimization problems with data partitioned across agents connected by possibly dynamic network topologies. Designing algorithms that guarantee convergence to stationary points, handle nonconvexity, and efficiently utilize computation and communication resources is nontrivial. Research in this area develops frameworks based on successive convex approximation, primal convexification, dynamic consensus, and gradient coding to provide scalable, general solutions with convergence guarantees and support for parallelism and robustness to stragglers.
Key finding: This work proposes a general distributed framework for training neural networks where data is partitioned among agents connected over sparse, time-varying networks, formulating the learning as a nonconvex social cost... Read more
Key finding: The authors develop a theoretical framework that leverages successive convex approximation and dynamic consensus to train neural networks in distributed settings without centralized coordination. The approach handles general... Read more
Key finding: To mitigate stragglers in synchronous distributed gradient descent, this paper develops a dynamic gradient coding (GC) approach with dynamic clustering (GC-DC), where redundant data and computations are assigned adaptively... Read more
Key finding: The study introduces an age-aware dynamic encoding strategy for coded computation with partial recovery in distributed gradient descent, which reorders codewords and computation priorities over time based on the age of... Read more
3. How can distributed learning be designed to effectively handle data heterogeneity, non-IIDness, and decentralized data partitions in federated and collaborative learning?
Data heterogeneity and non-identically distributed (non-IID) characteristics of clients' local datasets pose significant challenges to federated and collaborative learning. Approaches addressing vertical (feature-partitioned) and horizontal (sample-partitioned) data distributions focus on enabling collaborative training without sharing raw data or model parameters, preserving privacy, and ensuring effective convergence. Research targets clustering clients by data similarity, multiple local update methods, multi-agent systems for dynamic node management, and ensemble architectures for continual learning on streaming nonstationary data—seeking personalized and robust global models in realistic distributed environments.
Key finding: This paper tackles collaborative learning with vertically partitioned features across parties, proposing Federated Stochastic Block Coordinate Descent (FedBCD) that allows multiple local updates before communication. Parties... Read more
Key finding: Proposing a distributed training algorithm for Random Vector Functional-Link (RVFL) networks in vertically partitioned (feature-distributed) settings, this work uses the Alternating Direction Method of Multipliers (ADMM) and... Read more
Key finding: The authors design a federated learning framework implemented through a multi-agent system (MAS) using the SPADE platform, where agents dynamically join or leave as training nodes. Model training occurs locally at each agent... Read more
Key finding: FLIS dynamically clusters clients without accessing private data by measuring inference similarity of local models, enabling grouped training on similar data distributions under non-IID settings. Unlike prior approaches... Read more
Key finding: Introducing Ensemble and Continual Federated Learning (ECFL), this approach leverages ensemble techniques to aggregate diverse local models trained on data streams subject to concept drift, enabling continual adaptation in... Read more