Compress Clean Signal from Noisy Raw Image: A Self-Supervised Approach

Zhihao Li* Yufei Wang* Alex Kot Bihan Wen
zhihao.li [AT] ntu.edu.sg yufei001 [AT] e.ntu.edu.sg eackot [AT] ntu.edu.sg bihan.wen [AT] ntu.edu.sg

ICML 2024 Paper



TL;DR: Noisy Raw (14.5M 40dB) → Clean Raw (0.1M 50dB) without the need of the noise-clean paired data.


Abstract: Raw images offer unique advantages in many lowlevel visual tasks due to their unprocessed nature. However, this unprocessed state accentuates noise, making raw images challenging to compress effectively. Current compression methods often overlook the ubiquitous noise in raw space, leading to increased bitrates and reduced quality. In this paper, we propose a novel raw image compression scheme that selectively compresses the noise-free component of the input, while discarding its real noise using a self-supervised approach. By excluding noise from the bitstream, both the coding efficiency and reconstruction quality are significantly enhanced. We curate a full-day dataset of raw images with calibrated noise parameters and reference images to evaluate the performance of models under a wide range of input signalnoise ratios. Experimental results demonstrate that our method surpasses existing compression techniques, achieving a more advantageous ratedistortion balance with improvements ranging from +2 to +10dB and yielding a bit saving of 2 to 50 times.

< Rendering Results >

Sony a7S II:   Ours  vs.  RAW

Original Image RAW
Modified Image Ours
Original Image RAW
Modified Image Ours

Redmi Note12 Turbo:   Ours  vs.  RAW

Original Image RAW
Modified Image CBUnet
Original Image RAW
Modified Image CBUnet

< Method  >

The proposed framework for self-supervised raw image denoising and compression without reliance on paired clean images. Distinct from typical learning-based compressors, our approach first subtracts fixed pattern noise \( n_{fp} \) from the noise input \( \tilde{x} \) in the compressor encoder. Then, it compresses the predicted clean signal \( \hat{x} \), constrained by \( \tilde{x} - F_n(\tilde{x}; \Omega) \), using a compressor with an integrated hyperprior module. To regularize the predicted noise \( \hat{n} = F_n(\tilde{x}; \Omega) \), we use a bijective mapping based on the physical-based noise model to map the complicated noise distribution to a latent space where the distribution is known. Besides, a covariance loss is used to enhance the spatial independence of the disentangled noise \( \hat{n} \).

< Full Day Raw Image Compression Dataset >


Existing raw image denoising datasets mainly focus on lowlight or nighttime scenes, e.g., SID (Chen et al., 2018) captured under around 5 lux conditions and ELD (Wei et al., 2020) is even below 0.3 lux as shown in Fig. 4a. However, the demand for image compression is not only at night but throughout the whole day. Besides, noise is also prevalent in daylight raw images, which possess considerably higher SNR. Due to the significant gap among input SNR, compressors trained solely on low SNR data are less effective in higher SNR scenarios. Therefore, the existing datasets cannot well meet the needs of training and performance evaluation.


To address this limitation, we develop a full-day raw image compression (FDRIC) dataset that covers a wide range of SNR, ensuring comprehensive training and evaluation. We collected our dataset using the Redmi Note12 Turbo smartphone, with an OV64B sensor of a resolution of 4624×3472. Our dataset includes 549 noisy images for training and 32 noise-clean image pairs for evaluation. Our dataset contains indoor and outdoor scenes with illumination ranging from 0.1 to more than 100,000 lux.


< Code >


PyTorch implementation and pre-trained models can be found here

< Citation >

@inproceedings{
  Cleans,
  title=Compress Clean Signal from Noisy Raw Image: A Self-Supervised Approach,
  author={Li, Zhihao and Wang, Yufei and Kot, Alex and Wen, Bihan},
  booktitle={Forty-first International Conference on Machine Learning}
}

*The theme of the website was borrowed from PyNET.