SIGGRAPH 2022

Face Deblurring using Dual Camera Fusion on Mobile Phones

Wei-Sheng Lai, YiChang Shih, Lun-Cheng Chu, Xiaotong Wu, Sung-Fang Tsai, Michael Krainin, Deqing Sun, Chia-Kai Liang
Google

Abstract

Motion blur of fast-moving subjects is a longstanding problem in photography and very common on mobile phones due to limited light collection efficiency, particularly in low-light conditions. While we have witnessed great progress in image deblurring in recent years, most methods require significant computational power and have limitations in processing high-resolution photos with severe local motions. To this end, we develop a novel face deblurring system based on the dual camera fusion technique for mobile phones. The system detects subject motion to dynamically enable a reference camera, e.g., ultrawide angle camera commonly available on recent premium phones, and captures an auxiliary photo with faster shutter settings. While the main shot is low noise but blurry, the reference shot is sharp but noisy. We learn ML models to align and fuse these two shots and output a clear photo without motion blur. Our algorithm runs efficiently on Google Pixel 6, which takes 463 ms overhead per shot. Our experiments demonstrate the advantage and robustness of our system against alternative single-image, multi-frame, face-specific, and video deblurring algorithms as well as commercial products. To the best of our knowledge, our work is the first mobile solution for face motion deblurring that works reliably and robustly over thousands of images in diverse motion and lighting conditions.
Paper

ACM TOG (SIGGRAPH 2022)
Arxiv (with supplementary materials)
@article{lai2022face,
    author    = {Lai, Wei-Sheng and Shih, YiChang and Chu, Lun-Cheng and Wu, Xiaotong and Tsai, Sung-Fang and Krainin, Michael and Sun, Deqing and Liang, Chia-Kai}, 
    title     = {Face Deblurring using Dual Camera Fusion on Mobile Phones}, 
    journal   = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH)},
    year      = {2022}
}

Videos

Full presentation
Fast forward
Supplementary Materials

Comparisons to academic works (cropped faces with reference images)
Comparisons to commercial products (cropped faces with reference images)
Our full-resolution results (no reference images)
Our intermediate results (with reference images)
Press Coverage

Verge Now the Pixel 6 also has the ultrawide grab a fast, sharper image to capture that detail.
WIRED The features of the child's face became clearer.
Gizmodo Pixel 6 was able to convert the photo from one that might end up in the recycle bin to something you’d actually want to keep.
Android Police The Pixel 6 camera will try to magically unblur people's faces in your photos.
Android headline The Pixel 6 and Pixel 6 Pro camera will try to fix facial blur in images.
Engadget The tools that have a greater impact on your photos are Magic Eraser and Face Unblur, and despite some quirks they’re both quite effective.