Eligibility: this project is only available to students on the online MSc Statistics and Sustainability programme. If interested please contact Cangxiong Chen at Cangxiong.Chen@tcd.ie (personal webpage https://cangxiongchen.github.io/).
Knowledge distillation is a machine learning technique in which a smaller student model learns to reproduce information from a larger teacher model, typically using the teacher’s outputs as training targets. Knowledge distillation has been applied to transfer capabilities from Large Language Models to smaller models, for example, in the training of DeepSeek R1. This project will investigate whether a smaller student model can retain the predictive performance of an overparameterised teacher while reducing the overall computational cost of deployment. The student will train a teacher and compare a smaller model trained from scratch with a student trained using knowledge distillation from the teacher. The analysis will vary one factor, such as the distillation temperature or transfer-set size, and measure prediction quality, calibration, model size, training energy, inference energy and estimated carbon emissions.