Recent progress in artificial intelligence, particularly with large language models (LLMs) and other deep learning models, such as recommender and classification systems, has been driven by scaling data and computational resources. While effective, this approach is increasingly unsustainable; demanding significant energy and hardware investment, along with its environmental impacts, concentrating AI capabilities within a few well-funded organizations, and limiting accessibility and innovation.
My research focuses on the development of resource-efficient, sustainable, and widely accessible methods for training and deploying large-scale deep learning models, with a focus on LLMs and tasks involving large output spaces. Specifically, I will explore techniques such as speculative decoding (to reduce inference latency), dynamic sparsity training (to optimize computational use), and quantization to reduce memory footprint while maintaining accuracy at extremely low bit levels (e.g., 1-bit, 1.58-bits, 4-bits).
Overall, my research combines theoretical analysis with algorithmic design and development on modern accelerators, utilizing hardware-friendly kernels where possible. Evaluation will use public datasets, measuring performance across key metrics: training/inference speed, memory, model size, and task accuracy, along with estimating energy use to quantify environmental impact. The expected outcomes are faster, cheaper, and more sustainable AI systems, enabling broad participation and deployment on edge and resource-limited devices, while remaining competitive with the state-of-the-art models.
Efficient AI, Large Language Models, Large Output Space, Sparse Neural Network, Optimization, Deep Learning, Machine Learning, Generative AI, Computer Vision, Number Theory, Cryptography
MSc in Machine Learning and Artificial Intelligence, Stellenbosch University, South Africa
MSc in Mathematics and Applications, University of Fianarantsoa, Madagascar
MSc in Mathematical Science, University of Buea, African Institute for Mathematical Sciences (AIMS), Cameroon
BSc in Mathematics and Applications, University of Fianarantsoa, Madagascar
Dr Rohit Babbar
Dr Vinay Namboodiri
