Implemetation of simple example of knowledge distillation. Knowledge distillation is a process in machine learning where a small, more efficient model is trained to mimic the behavior of a larger, more complex model. The goal is to transfer the knowledge learned by the larger model to the smaller one, while reducing its size and computational requirements. This technique is often used to improve the performance of resource-limited devices or to accelerate the inference time of models in production.
- Reference: paper
- Dataset: custom dataset for classification task
- Teacher model: Res-Net152
- Student model: Conv-Net with tree convolution blocks and two fully-connected
- black line - with teacher
- blue line - withous teacher
