Toward Multi-Plane Image Reconstruction from a Casually Captured Focal Stack 📷

Shiori Ueda1, Hideo Saito1, and Shohei Mori2, 1

1Keio University, 2Graz University of Technology

Int. Conf. on Computer Vision Theory and Applications (VISAPP) 2024

🎖️ Selected High Quality Paper

Pipeline of MPI generation from a casual focal stack with a manually controlled camera. (a) The user takes a focal stack either by rotating the focus ring at once in a sequence (continuous-rotation) or by repeating rotating and stopping the focus ring for a short interval (delta-rotation). We discuss which approach is better regarding MPI rendering quality in our experiment. (b) A vision technique corrects spatial misalignment over the focal stack images. (c) A fixed number of focal stack images is selected. (d) A U-Net that is trained by analysis by synthesis generates MPI. (e) Applications include depth rendering, per-layer defocus for occlusion-aware depth of field effect, and de-fencing.

Abstract 3D imaging combining a focal stack and multi-plane images (MPI) facilitates various real-time applications, including view synthesis, 3D scene editing, and augmented and virtual reality. Building upon the foundation of MPI, originally derived from multi-view images, we introduce a novel pipeline for reconstructing MPI by casually capturing a focal stack optically using a handheld camera with a manually modulated focus ring. We hypothesized two distinct strategies for focus ring modulation that users could employ, to sample defocus images along the front-facing axis uniformly. Our quantitative analysis using a synthetic dataset suggests tendencies in possible simulated errors in focus modulations, while qualitative results illustrate visual differences. We further showcase applications utilizing the resultant MPI, including depth rendering, occlusion-aware defocus filtering, and de-fencing.

    author = {Ueda, Shiori and Saito, Hideo and Mori, Shohei},
    title = {Toward Multi-Plane Image Reconstruction from a Casually Captured Focal Stack},
    booktitle = {Int. Joint Conf. on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP},    
    year = {2024}