Expert Review
This review article comprehensively and deeply introduces the principle framework of Fourier ptychographic imaging, including hardware systems, imaging models, and image processing algorithms. It systematically discusses the impact of various parameter errors on the quality of Fourier ptychographic imaging and their corresponding correction methods. Another highlight of the article is the classification summary and discussion of the performance and expansion methods of the imaging system, and it introduces various strategies and methods to improve performance.
This review can provide useful references for researchers related to ptychographic imaging and computational optics. Depending on different application scenarios, researchers can choose and debug the corresponding Fourier ptychographic imaging system based on this review article. It is noteworthy that the Nature Reviews series journal Nature Reviews Physics recently published a review article titled “Concept, implementations and applications of Fourier ptychography.” The content of this article forms an effective complement to it, especially regarding the detailed summary and analysis of system parameter correction and phase recovery algorithms.
Guoan Zheng
University of Connecticut
Zhang Shaohui, Zhou Guocheng, Cui Baiqi, Hu Yao, Hao Qun. Fourier Ptychographic Microscopy: Models, Algorithms, and System Research Overview[J]. Laser and Optoelectronics Progress, 2021, 58(14): 1400001
Cover Interpretation
This cover image illustrates the basic principles, typical architecture, and imaging effects of Fourier ptychographic microscopy, illuminating the sample from multiple angles so that the light entering the imaging system carries different spatial frequency components of the sample, and then performing phase recovery and aperture synthesis in the frequency domain with the low-resolution images corresponding to each illumination angle, ultimately achieving large field of view and high spatial resolution imaging effects.
01
Background
Optical microscopy has played a very important and irreplaceable role in life sciences, medicine, and industrial inspection, but its performance still struggles to meet the higher cognitive demands of microscopic structures in life sciences and artificial intelligence technology. Among them, issues related to large spatial bandwidth product (SBP) imaging and transparent sample imaging are particularly prominent.
In microscopy, the mainstream strategy to improve SBP currently is super-resolution microscopy, represented by stimulated emission depletion microscopy (STED), photoactivated localization microscopy (PALM), stochastic optical reconstruction microscopy (STORM), and structured illumination microscopy (SIM); the mainstream imaging technologies for transparent/semi-transparent samples include fluorescence labeling and phase measurement techniques such as digital holographic microscopy (DHM), coherent diffraction imaging (CDI), and transport of intensity equation (TIE).
Fourier ptychographic microscopy (FPM) is a significant computational imaging technique proposed in 2013 by Guoan Zheng and others at the California Institute of Technology[1]. FPM achieves large field of view, high spatial resolution, and quantitative phase imaging by varying the illumination direction of the sample, capturing corresponding low-resolution images, and performing phase recovery and aperture synthesis in the frequency domain. Since its inception, FPM has received widespread attention and research. This paper reviews, analyzes, and discusses the research and improvements in the FPM theoretical model, algorithm framework, and imaging system.
02
Technical Principles
A typical FPM system uses a programmable LED array board as a light source, sequentially illuminating different LED units to provide illumination from different directions while capturing a series of low-resolution images, relying on back-and-forth iterations and constraints in both spatial and frequency domains to obtain the optimal solution of complex amplitude that simultaneously satisfies the spatial “amplitude constraint” and frequency domain “support domain constraint,” as shown in the algorithm flow in Figure 2.
Figure 2: Phase recovery flowchart of Fourier ptychographic imaging[1]
In addition to large field of view and high spatial resolution imaging, FPM can also achieve label-free quantitative phase imaging, as shown in the imaging results in Figure 3. It can be seen that structures that are difficult to manifest in intensity images can be presented and enhanced in the phase images reconstructed by FPM.
03
Technical Development
Correction of Theoretical Model
The initial FPM theoretical framework has many approximations and assumptions. During the development of FPM technology, researchers have conducted extensive studies on various parameter deviations in the system construction process to achieve better imaging results.
Among them, the geometric pose deviation of the LED board significantly affects the FPM imaging results, and modeling and correction of it is representative work in the research of system parameter correction. Such work can be divided into hardware-based and software algorithm-based corrections. The LED board model construction based on software algorithms is shown in Figure 4[3].
Figure 4: LED array board pose parameter error model[3]
The deviation of the imaging lens group from the ideal model also affects the FPM imaging results. Figure 5 shows the EPRY-FPM imaging results proposed by Xiaoze Ou and others at the California Institute of Technology. It can be seen that after correcting the system pupil function, the FPM imaging results have been significantly improved.
Figure 5: EPRY-FPM frequency domain “probe function” recovery results[4]
Improvement of Optimization Algorithms
The FPM algorithm framework has multiple construction methods. Under the interactive projection framework, the algorithm can be divided into global update type and sequential update type based on the number of low-resolution images used in each iteration update; under the optimization framework, the FPM algorithm can be roughly divided into first-order gradient-related methods, second-order gradient-related methods, adaptive step size optimization methods, and deep learning-related algorithms based on the differences in loss function construction methods and update strategies.
The FPM algorithm is basically consistent with its dual form, the Ptychography Iterative Engine (PIE) algorithm. Therefore, different types of PIE algorithms also represent the improvement process of the FPM algorithm. Literature [5] provides a detailed comparison and analysis of several representative PIE algorithm sample update equation forms.
Machine learning, as a computational framework that has received much attention in recent years, has also been applied in FPM. The FPM schemes using machine learning can be divided into two categories:
(1) Constructing neural networks based on the phase recovery process to obtain high-resolution images from low-resolution images (Figure 6);
(2) Constructing the sample function to be recovered as a layer of the network, with its parameters representing the complex amplitude parameters of the sample (Figure 7).
FPM System Performance Improvement
FPM technology can achieve large spatial bandwidth product and quantitative phase imaging, but specific sample models and imaging methods also limit the application of FPM. This article mainly compares and analyzes the related research on improving FPM’s three-dimensional imaging capabilities and enhancing FPM imaging speed.
FPM imaging extends from two-dimensional to three-dimensional mainly through two technologies: digital refocusing and tomographic imaging. The former still assumes that the sample conforms to the “thin object assumption” in the local area, achieving three-dimensional imaging through digital refocusing of the local area; the latter extends the FPM model to three-dimensional k-vector space and directly solves the three-dimensional distribution of the sample. Figures 8-9 show the results of digital refocusing and tomographic imaging, respectively.
The long original image acquisition process of FPM limits its temporal resolution. Based on its imaging characteristics, improving the speed of FPM imaging requires reducing the number of LED illuminations and high dynamic range data acquisition. Representative results include illumination/imaging parameter reuse (multi-modal) and channel synthesis. Figures 10-11 show representative schemes/results of multi-modal FPM and high dynamic range FPM, respectively.
Figure 10: Different modal illumination schematic diagram[10]. (a) Single-modal FPM; (b) Multi-LED position modality; (c) Multi-LED wavelength modality
Expansion of System Construction Methods
The improvement and expansion of FPM system construction methods aim at two directions: one is to enhance imaging performance (Figure 12); the other is to save system construction costs (Figure 13), including spatial costs and economic costs.
The representative work for improving imaging performance in system construction methods is the design and construction of reflective and long-distance FPM systems (Figure 12).
Figure 12: Reflective and long-distance FPM framework [12,13]. (a) Reflective FPM system structure schematic; (b) Long-distance aperture scanning macro FPM system
Figure 13: Improved FPM system construction[14-17]. (a-b) FPM system design scheme based on mobile phone lens; (c) FPM system design scheme based on Raspberry Pi and its development modules; (d) FPM system design scheme based on camera array multi-aperture; (e) FPM system design scheme based on industrial cameras and industrial telecentric lenses
04
Summary and Outlook
Thanks to its excellent imaging performance and typical computational imaging framework characteristics, FPM has received extensive attention and research since its inception, achieving significant progress in theoretical models, optimization algorithms, as well as system construction, expansion, and application. However, this does not mean that research in this direction has become complete or saturated. Currently, FPM still faces issues such as complex system adjustment processes, sensitivity to ambient light interference, low temporal resolution, and long reconstruction times. Whether in theory, algorithms, or applications, FPM continues to attract considerable attention and research to continue improving FPM’s models, algorithms, and system design and construction.
References
Research Group Introduction
The research group is based on the Measurement and Imaging Laboratory of Beijing Institute of Technology, focusing on research in computational imaging theoretical models, system construction, and optimization algorithms.
Zhang Shaohui, Associate Researcher at the School of Optoelectronics, Beijing Institute of Technology, PhD supervisor. Youth editorial board member of Optical Technology, Youth member of the Laser Application Branch of the Chinese Optical Society. Research directions: computational imaging, three-dimensional measurement, laser precision measurement, etc. In 2020, he received the “Jin Guofan Youth Scholar Award” from the Instrumentation Society. As the first/corresponding author, he has published 15 papers in authoritative journals in the field of optics such as Optics Letters, Optics Express, and has made invited/oral presentations at domestic and international academic conferences multiple times. He has hosted or participated as a technical backbone in multiple scientific research projects such as the National Natural Science Foundation, National Major Projects, and Postdoctoral First-Class Funding.
Hao Qun, Professor and PhD supervisor at the School of Optoelectronics, Beijing Institute of Technology. Executive director of the Chinese Optical Society, chairman of the Optoelectronics Committee, director of the Chinese Instrument and Instrumentation Society, executive vice chairman of the Optomechanics and System Integration Committee, director of the Measurement and Testing Society, member of the Military Commission Science and Technology Committee, and Equipment Development Professional Group member. He serves as the deputy editor of Defense Technology magazine and the deputy director of the editorial board of Journal of Weapon Equipment Engineering. He has long been engaged in teaching and scientific research in the field of new optoelectronic imaging sensing technology and optoelectronic precision testing technology, with research directions including computational imaging, new optoelectronic imaging technology, bionic optoelectronic sensing technology, vibration-resistant interference measurement technology, and instruments. He has hosted over thirty scientific research projects, including key projects of the National Natural Science Foundation, major scientific instrument and equipment development projects, key projects of the basic strengthening plan, common technology projects for equipment development, and key defense basic research projects; he has published over 118 SCI-indexed papers in international journals such as Optics Express, Optics Letters; authored three monographs; and has made over twenty invited presentations at international conferences; he has been granted over 120 national invention patents.
End
First release of cutting-edge optical achievements, broadcasting fresh optical activities
If you need to reprint, please leave a message directly.
Business cooperation: Mr. Zhu 13918384218
Disclaimer
All images appearing in this article are reproduced. If there are any copyright issues, please contact the authors within 20 working days, and we will coordinate to address them (pay manuscript fees or delete according to regulations).
The final interpretation rights belong to the Chinese Lasers magazine.