MatchDistill-Net: knowledge distillation of global context from transformers to convolutional networks for monocular depth estimation

Transformer-based architectures excel in monocular depth estimation by capturing global context but suffer from high computational complexity. Conversely, Convolutional Neural Networks (CNNs) offer efficiency but lack long-range dependency modeling. To bridge this gap, we propose MatchDistill-Net, a framework that transfers global representations from a Transformer teacher to a lightweight CNN student via adaptive, feature-alignment-based knowledge distillation. This strategy removes the teacher Swin Transformer at inference, leaving only the efficient window-based attention of the decoder. Our key innovations include the ConvMatcher block for cross-domain feature alignment and a Shifted-Window Cross-Attention Memory decoder. Additionally, a novel Soft Delta Loss enhances boundary precision and outlier robustness. Evaluated on NYU Depth V2 and KITTI, MatchDistill-Net achieves accuracy comparable to that of substantially larger Transformer-based models while maintaining only 39.1 M inference parameters and 73.3 ms latency (13.64 FPS). Our objective is to enable near real-time, high-accuracy monocular depth estimation on resource-constrained devices by eliminating teacher-Transformer computation at inference while preserving the global context learned during training. Our code is publicly available at https://github.com/KrisnaPinasthika/MatchDistill-Net.