Toward Multi-Plane Image Reconstruction from a Casually Captured Focal Stack 📷

Shiori Ueda1, Hideo Saito1, and Shohei Mori2, 1

1Keio University, 2Graz University of Technology

Int. Conf. on Computer Vision Theory and Applications (VISAPP) 2024

🎖️ Selected High Quality Paper

ueda_visapp24.png
Pipeline of MPI generation from a casual focal stack with a manually controlled camera. (a) The user takes a focal stack either by rotating the focus ring at once in a sequence (continuous-rotation) or by repeating rotating and stopping the focus ring for a short interval (delta-rotation). We discuss which approach is better regarding MPI rendering quality in our experiment. (b) A vision technique corrects spatial misalignment over the focal stack images. (c) A fixed number of focal stack images is selected. (d) A U-Net that is trained by analysis by synthesis generates MPI. (e) Applications include depth rendering, per-layer defocus for occlusion-aware depth of field effect, and de-fencing.

Abstract 3D imaging combining a focal stack and multi-plane images (MPI) facilitates various real-time applications, including view synthesis, 3D scene editing, and augmented and virtual reality. Building upon the foundation of MPI, originally derived from multi-view images, we introduce a novel pipeline for reconstructing MPI by casually capturing a focal stack optically using a handheld camera with a manually modulated focus ring. We hypothesized two distinct strategies for focus ring modulation that users could employ, to sample defocus images along the front-facing axis uniformly. Our quantitative analysis using a synthetic dataset suggests tendencies in possible simulated errors in focus modulations, while qualitative results illustrate visual differences. We further showcase applications utilizing the resultant MPI, including depth rendering, occlusion-aware defocus filtering, and de-fencing.

SCITEPRESS PDF
BibTex 
@inproceedings{ueda_visapp2024,
    author = {Ueda, Shiori and Saito, Hideo and Mori, Shohei},
    title = {Toward Multi-Plane Image Reconstruction from a Casually Captured Focal Stack},
    booktitle = {Int. Joint Conf. on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP},    
    pages={72--82},
    publisher={SciTePress},
    organization={INSTICC},
    doi={10.5220/0012438700003660},
    isbn={978-989-758-679-8},
    issn={2184-4321},
    year = {2024}
}