Improving Collaboration among Heterogeneous Models through Knowledge Transfer and Multi-Model Inference
This dissertation investigates how to make collaboration among heterogeneous models more effective and efficient by determining which models should collaborate, what knowledge should be transferred, and when additional interaction is worthwhile.
Overview
Modern machine learning increasingly relies on collections of models trained on different data, specialized for different tasks, and operating under different constraints. These models often possess complementary knowledge and capabilities, creating opportunities for collaboration. However, collaboration is not always beneficial. During training, knowledge transfer can introduce negative transfer or forgetting; during inference, interaction with other models can correct incorrect predictions but also corrupt initially correct ones.
First, we systematically investigate knowledge distillation (KD) between pre-trained models, examining how distillation methods, hyperparameters, transfer data, and teacher–student relationships influence learning outcomes. Our findings provide practical guidance for improving knowledge transfer and demonstrate how KD-based model consolidation can accelerate federated learning.
Building on these insights, we introduce Query-based Knowledge Transfer (QKT), a framework that enables models to acquire specific knowledge from relevant peers without exchanging raw data. Through targeted teacher selection and query-focused distillation, QKT enables efficient knowledge acquisition while mitigating interference and forgetting. Across standard and clinical image-classification benchmarks, QKT substantially outperforms existing collaborative learning approaches.
Finally, we investigate collaboration among large language models (LLMs) at inference time. We demonstrate that cross-model collaboration can recover failures beyond the capabilities of individual models but may also introduce harmful changes. To address this trade-off, we introduce COMED, a framework that selectively determines whether to accept an initial answer, verify it, or escalate to cross-model collaboration. Across diverse reasoning benchmarks and model pools, COMED consistently improves accuracy while reducing unnecessary model interactions and inference costs.
Together, these contributions demonstrate that effective collaboration among heterogeneous models requires selective knowledge exchange that balances potential performance improvements against the risks and costs of interaction.
Presenters
Brief Biography
Norah Alballa is a Ph.D. student in Computer Science at King Abdullah University of Science and Technology (KAUST), advised by Professor Marco Canini. She received her M.Sc. in Computer Science from Portland State University, USA, and her B.Sc. in Information Technology from King Saud University, graduating with First-Class Honors in both degrees. During her Ph.D. studies, she received the KAUST CEMSE Division's Dean’s List Award. She has also been involved in teaching and mentoring activities.