Fair Feature Distillation Using Teacher Models of Larger Architecture

Sangwon Jung, Taesup Moon

http://doi.org/10.5626/JOK.2021.48.11.1176

Achieving algorithmic fairness is becoming increasingly essential for various vision applications. Although a state-of-the-art fairness method, dubbed as MMD-based Fair feature Distillation (MFD), significantly improved accuracy and fairness via feature distillation based on Maximum Mean Discrepancy (MMD) compared to previous works, MFD could be limitedly applied into when a teacher model has the same architecture as a student model. In this paper, based on MFD, we propose a systematic approach that mitigates unfair biases via feature distillation of a teacher model of larger architecture, dubbed as MMD-based Fair feature Distillation with a regressor (MFD-R). Throughout the extensive experiments, we showed that our MFD-R benefits from the use of the larger teacher compared to MFD as well as other baseline methods.