Memory Efficient Quantization-Aware Fine-Tuning Diffusion Models through Implementation of L4Q in EfficientDM Framework

Rapid evolution of large-scale machine learning models has created unprecedented demands on computational resources, particularly GPU memory, which has emerged as a critical bottleneck in both research and production environments. While recent advances in memory-efficient training techniques have shown promise in addressing efficiency concerns, the fundamental GPU memory bottleneck during training remains unresolved, limiting its applicability in resource-constrained environments and hindering exploration of efficient diffusion models for complex tasks such as image generation. This work proposes an integration of Lowrank Adaptive Learning Quantization for LLMs (L4Q) with Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models (EfficientDM), a quantization-aware training framework that integrates the immediate gradient flushing mechanism with the temporal calibration strategy. The integration aims to make efficient diffusion model training accessible to researchers with limited computational resources and democratize access to state-of-the-art generative modeling capabilities, enabling broader exploration of diffusion models across diverse application domains. Our experiments demonstrate that this integration yields substantial improvements in both training efficiency and generative quality. Specifically, the proposed method achieves a speed increase of 1.38x in training time, a reduction of 37% in GPU memory usage peak, while also improving FID by 9-19 points. Code is accessible at https://github.com/FirH/L4Q-on-EfficientDM.