1.

Introduction

Fourier ptychographic microscopy (FPM)^1–4 is a recently proposed computational imaging technique. By recording multiple low-resolution (LR) intensity images of the sample from angle-varied illumination and iteratively stitching these different LR intensity images together in the Fourier space, FPM recovers a high-resolution (HR) complex amplitude image of the sample with a large field of view (FOV), which overcomes the physical space-bandwidth-product limit of a low numerical aperture (NA) imaging system. The final reconstruction resolution is determined by the sum of the objective lens and illumination NAs.⁵ Due to its flexible setup, good performance, and rich redundancy of the acquired data, FPM has been widely applied in three-dimensional imaging,^6,7 fluorescence imaging,^8,9 multiplexing imaging,^10–12 etc.

In current FPM imaging platforms, systematic error sources mainly come from aberrations, light-emitting diode (LED) intensity fluctuation, parameter imperfections, and noise, all of which may severely degrade the reconstruction results with similar artifacts despite the different generation mechanisms.^13–17 Therefore, it is hard to say which kind of error is going to be blamed for image degradation, and which kind of mechanism-targeted algorithm^13–16 will be applied. In this case, the apparent way to improve recovery quality may be a successive attempt at different algorithms. However, as these algorithms are pointed at a specific error mechanism, they take little effect to calibrate the mixed systematic errors.

But fortunately, they all solve parts of problems and share the same root with alternating projection (AP) method,^18,19 which offers great flexibilities to be adapted to more complicated mathematical models for many advanced applications. Therefore, we set up a comprehensive mathematical model to explain all the error mechanisms and propose a system calibration procedure, termed SC-FPM, to calibrate the mixed systematic errors simultaneously from an overall perspective. Four modules, numbered from 1 to 4, are involved in our procedure to address parameter imperfections, LED intensity fluctuation and noise, which are based on the simulated annealing (SA) algorithm¹⁴ and nonlinear regression process,¹⁴ LED intensity correction method,¹⁵ and adaptive step-size strategy,¹⁶ respectively. At first, a number of initial iterations (around 10 iterations) for bright-field (BF) images with low illumination NAs are implemented by modules 1 and 2 to correct those low-frequency apertures’ parameters, and module 3 to calibrate their intensity measurements, enabling the BF images suffering less from systematic noise and thus obtaining more precise initial parameters. After the correction of BF images, all the captured raw images are empirically iterated only once by modules 3 for intensity updating. Finally, the updated images are iterated several times by modules 1 and 2 to optimize global parameters, together with module 4 to resist the fluctuation of final reconstructions influenced by noise. Note that the parameters re-estimation in module 2 is from a global perspective to enhance the iterative accuracy, and the updating of coherent transfer function (CTF) is included in module 3 to offset aberrations,¹³ which differs from that in original LED intensity corrections.¹⁵ As validated in simulations and experiments, the proposed system calibration procedure improves the robustness of FPM, relaxes the experiment conditions, and does not require any prior knowledge, making the FPM more practical. Due to the universality and flexibility of the AP method, our work can be further combined with many excellent algorithms such as motion deblurring²⁰ and multiplexing imaging,^10–12 producing a better performance of the FP approach. Once the system errors have been calibrated, some prior knowledge can be added into models and then using some noise suppression methods^21,22 may further improve the results. Source code is released in Ref. 23 for noncommercial use.

2.

Method

2.2.

Conflict Between Aberration Estimation and LED Intensity Correction

The SA algorithm,¹⁴ nonlinear regression process,¹⁴ and adaptive step-size strategy¹⁶ could easily be combined with a few modifications in step-size and update order. But there exists strong conflict between aberration estimation and LED intensity correction, which are in disagreement with the CTF updating. More specifically, if the sample spectrum as well as the CTF is simultaneously updated in the process of LED intensity correction, it is unlikely to obtain a satisfactory recovery quality due to the mutual transformation between the error of aberration and LED intensity, which also degrades the convergence properties of iterative algorithms.

If there exists LED intensity fluctuation, then Eq. (1) needs to be accordingly modified to

Eq. (7)

$$I_{m,n}^u(x,y)=c_{m,n}·{|{\mathcal{F}}^{-1}[O(u-u_{m,n},v-v_{m,n})·P(u,v)]|}^2,$$

where $c_{m,n}$ is defined as¹⁵

Eq. (8)

$$c_{m,n}=\frac{\sum _{x,y}{|{\varphi }_{m,n}^e(x,y)|}^2}{\sum _{x,y}{I}_{m,n}^c(x,y)}.$$

Then, the captured intensity images are updated by

Eq. (9)

$$I_{m,n}^u(x,y)=c_{m,n}·I_{m,n}^c(x,y).$$

An inverted light microscope, illuminated by a $15\times 15$ LED matrix with the excitation wavelength of 632 nm, equipped with a $4\times /0.1\mathrm{NA}$ objective and an image sensor with the pixel size of $6.5\mu \mathrm{m}$, is modeled in our simulations. Figure 1 indicates the conflict between aberration estimation and LED intensity correction. Figures 1(a) and 1(b) show the HR input intensity and phase profiles within a small segment of $128\times 128\text{pixels}$, which are served as the ground truth of the complex sample. The distance between the sample and LED array is $h=86\mathrm{mm}$. The intensity and phase reconstruction accuracy is evaluated by root-mean-square error (RMSE). For illustration purpose only, 200% intensity fluctuation is artificially introduced by multiplying each raw image with a random constant ranging from zero to two. In general, different types of aberrations can be quantified as different Zernike modes at the pupil plane.¹³ Here, we introduce $Z_3^1$(coma) as an example shown in Fig. 1(d). Figures 1(a1)–1(c1) show the reconstructed images by original FPM algorithm at six iterations with only Eq. (3), which are blurred by the introduced intensity fluctuation, and the spectral artifacts could be clearly observed in Fig. 1(c1). The effeteness of LED intensity correction method could be observed in Figs. 1(a2)–1(c2), but the CTF updating process is not available in this method. Figures 1(a3)–1(d3) and 1(a4)–1(d4) show the recovered results by PIE-based intensity correction at first iteration and 30 iterations, respectively, which introduces the CTF updating in the intensity correction method. Unexpectedly, the reconstructions are less satisfactory, showing a strong conflict between LED intensity correction and CTF updating (aberration correction), which may be attributed to the mutual transformation of different errors. In addition, the qualities degrade with the increase of iterations, and the best results appear at the first iteration. Note that even with original LED intensity correction method, the recovery results are extremely unstable with sharp oscillation as shown by red line in Figs. 1(e1) and 1(e2), as Eq. (9) is not from an overall perspective, where the intensity of each raw image is updated by different coefficients. It is worth mentioning that if multiplying each raw image with the same constant, the constant can be ignored in the FPM model that will have no effect on the final reconstruction. For addressing this issue, the updating operation needs to run at the same standard by introducing a unified standard intensity. To reduce the conflict between aberration correction and intensity correction, the PIE-based intensity correction method would be employed only once. So, the modified solution is as follows.

Fig. 1

Conflict between aberration estimation and LED intensity correction. Groups (a–d) show the recovery results of intensity, phase, spectrum, and aberrations, respectively, with different algorithms. (e1) and (e2) present the intensity and phase reconstruction accuracy versus iteration times for different algorithms.

First, calculate the ratio $c_{m,n}$ after the first iteration and update the raw images for the second iteration. After that, without the intensity correction process, calibrate the aberration only through Eqs. (3) and (4) for the rest iterations. Here, the center LED is set as the reference that is supposed to be free of intensity fluctuation. Then, Eq. (8) needs to be modified to

Eq. (10)

$$c_{m,n}=\frac{\sum _{x,y}{|{\varphi }_{m,n}^e(x,y)|}^2}{c_{\mathrm{0,0}}·\sum _{x,y}{I}_{m,n}^c(x,y)}(m,n\ne 0),$$

where $c_{\mathrm{0,0}}=\sum _{x,y}{|{\varphi }_{\mathrm{0,0}}^e(x,y)|}^2/\sum _{x,y}{I}_{\mathrm{0,0}}^c(x,y)$ and the update operation is

Eq. (11)

$$I_{m,n}^u(x,y)=\{\begin{array}{ll}{c}_{m,n}·I_{m,n}^c(x,y)& (m,n\ne 0)\\ I_{\mathrm{0,0}}^c(x,y)& (m=n=0)\end{array}.$$

The reconstructions by our modified LED intensity correction method are shown as in Figs. 1(a5)–1(d5) at nine iterations, which enable one to obtain aberration while correcting intensity. In addition, the modified solution features a better stability and stronger robustness over original intensity correction method, which is demonstrated by pink line in Figs. 1(e1) and 2(e2).

Fig. 2

Flow chart of SC-FPM method.

3.

Simulations

Before employed in the experiments, the effectiveness of SC-FPM is validated by several groups of simulations, and the system parameters are the same as Sec. 2. The systematic errors, such as aberration, LED intensity fluctuation, parameter imperfections, and the noise are deliberately exaggerated in our simulations to better demonstrate the robustness and superiorities of SC-FPM over other existing algorithms. Positional misalignment is introduced by four positional factors with random values. Here for illustration, we set $\theta =5\mathrm{deg}$, $\mathrm{\Delta }x=1\mathrm{mm}$, $\mathrm{\Delta }y=1\mathrm{mm}$, $h=87\mathrm{mm}$ as the real situation, while $\theta =0\mathrm{deg}$, $\mathrm{\Delta }x=0\mathrm{mm}$, $\mathrm{\Delta }y=0\mathrm{mm}$, $h=86\mathrm{mm}$ as the ideal condition. The noise is artificially introduced by corrupting each LR image with 40% Gaussian noise with different variances. The noise level is quantified by the average mean absolute error (MAE),¹⁶ defined as $\mathrm{AMAE}=⟨|I_n-I|⟩/⟨I⟩$, where $⟨I⟩$ is the mean value of all noise-free DF intensity images and $⟨|I_n-I|⟩$ is the averaged MAE of the corresponding noisy images.

Figure 3 shows the performance of the original PIE-based algorithm with different systematic error sources, each with six iterations. Figures 3(a1)–3(c1) show the recovery results only with aberrations. Reasonably, the original PIE-based algorithm has the capability to compensate the aberrations by simultaneously updating the sample spectrum and the CTF with Eqs. (3) and (4). As a result, the reconstructions are free of artifacts and quite approximate to the ground truth as shown in Figs. 3(a) and 3(b). However, the other three errors respectively present some particular features in extreme conditions. Figures 3(a2)–3(c2) present recovery images with 200% LED intensity fluctuation. Compared with Fig. 3(c1), the recovered sample spectrum has been severely blurred and filled with patches, which are the symbol of artifacts caused by LED intensity fluctuation. The reconstructions with parameter imperfections present obvious wrinkles as shown in Figs. 3(a3) and 3(b3), and the upper-left of central bright spot in the spectrum is somewhat distorted as shown in Fig. 3(c3), featured by different characteristics from that in Figs. 3(a2)–3(c2). Figures 3(a4)–3(c4) present the recovery results with 40% Gaussian noise, and synthetically, Figs. 3(a5)–3(c5) show the reconstructions with mixed four errors. Although different errors are fused together whether in the spatial domain or frequency domain, their respective features are retained in the final results.

Fig. 3

The performance of the original PIE-based algorithm with different systematic error sources. (a) HR input intensity and (b) phase profiles serve as the ground truth of the simulated complex sample. Groups (a–c) show the recovery results of intensity, phase, and spectrum, respectively, with the original PIE-based algorithm.

Figure 4 indicates the experimental results by different algorithms with mixed systematic errors. Groups (a), (b), and (c) show the recovered intensity, phase, and spectrum, respectively. The recovery images in Figs. 4(a1)–4(c1), 4(a2)–4(c2), and 4(a3)–4(c3) are reconstructed by original PIE-based algorithm, LED intensity correction method, and adaptive FPM, respectively, which could be found little improvements of recovery quality by these specific algorithms. Figures 4(a4)–4(c4) present the recovery results by PC-FPM for parameter corrections, however, it fails to retrieval the complex sample either and the spectrum rotates clockwise as shown in Fig. 4(c4). In fact, despite the similarities between SC-FPM and PC-FPM in the parts of SA algorithm and nonlinear regression process, SC-FPM still produces a stronger robustness as indicated in Figs. 4(a5)–4(c5), demonstrating its effectiveness under such extreme conditions.

Fig. 4

Experimental results by different algorithms with mixed systematic errors. Groups (a), (b), and (c) show the recovery results of intensity, phase, and spectrum, respectively.

Figure 5 shows the detailed results of each iteration in SC-FPM. As indicated in Fig. 5(a), the recovery results fluctuate within the first 10 iterations but tend to stabilize after 15 iterations. The four positional factors, rotation factor $\theta$, shift factors ($\mathrm{\Delta }x,\mathrm{\Delta }y$), and height factor $h$, tend to converge on 5 deg, (1 mm, 1 mm), and 87 mm, respectively, which are demonstrated in Figs. 5(b)–5(d). Figure 5(e) presents the central position of each aperture corresponding to different illuminations in the frequency domain, where the ideal, real, and corrected positions are denoted by red triangles, green dots, and blue diamonds, respectively. Finally, the corrected positions accurately converge on $\theta =4.97\mathrm{deg}$, $\mathrm{\Delta }x=0.922\mathrm{mm}$, $\mathrm{\Delta }y=0.974\mathrm{mm}$, $h=86.799\mathrm{mm}$, which approximate to the real parameters introduced above, validating the strong robustness and great adaptability of SC-FPM in the real situation.

Fig. 5

The results of each iteration of SC-FPM in detail. (a) The RMSE of intensity and phase images. (b–d) The recovered four positional factors, rotation factor $\theta$, shift factors $\mathrm{\Delta }x$, $\mathrm{\Delta }y$, and height factor $h$, respectively. (e) The central position of each aperture corresponding to different illuminations in the frequency domain, where the ideal, real, and corrected positions are denoted by red triangles, green dots, and blue diamonds, respectively.

4.

Experiments

In order to validate the effectiveness of SC-FPM experimentally, we first compare the recovered intensity and phase distributions of one segment ($90\times 90\text{pixels}$) in a USAF target with different algorithms. Figure 6 shows the schematic diagram of the experiments. All LR images are captured with a $4\times /0.1\mathrm{NA}$ objective and a CCD camera with pixel pitch $3.75\mu \mathrm{m}$ (DMK23G445, Imaging Source Inc., Germany). A programmable $32\times 32$ RGB LED array with 4 mm spacing, controlled by an Arduino is placed at 86 mm above the sample. The central $15\times 15$ red LEDs (with the central wavelength of 631.13 and 20 nm bandwidths) are employed to provide angle-varied illuminations, resulting in a final synthetic NA of 0.5 theoretically.

Fig. 6

Schematic diagram of experiments. (a) The enlargement of $32\times 32$ RGB LED array. (b) The enlargement of microscope with light path diagram. MO, microscope objective; TL, tube lens; M1 and M2, mirrors; BS, beam splitter.

Figure 7 shows the experimental results of one segment ($90\times 90\text{pixels}$) in a USAF target by different algorithms. Figure 7(a) presents the FOV of a USAF target, whereas (a1) shows the enlargement of a subregion of (a), which is low-pass filtered by the low NA of the employed objective. Groups (b), (c), and (d) show the recovered intensity, phase, and spectrum, respectively, with different algorithms. The recovery images in Figs. 7(b1)–7(d1), 7(b2)–7(d2), and 7(b3)–7(d3) are reconstructed by an original PIE-based algorithm at 6 iterations, intensity correction method at 6 iterations, and adaptive FPM at 18 iterations, respectively. Through a comparison of these reconstructions, it could be inferred that the systematic errors mainly come from the noise and parameter imperfections that are featured by obvious wrinkles in the recovered results. In addition, the intensity image in Fig. 7(b3) is greatly improved over (b1) and (b2), demonstrating the advancement of adaptive FPM in noise suppression. Figures 7(b4)–7(d4) show the recovery results by PC-FPM at 12 iterations, where the obvious wrinkles are eliminated and the positional parameters are converged on $\theta =-1.8\mathrm{deg}$, $\mathrm{\Delta }x=1.274\mathrm{mm}$, $\mathrm{\Delta }y=1.270\mathrm{mm}$, $h=94.399\mathrm{mm}$. But the intensity image is less satisfactory as the group 8 elements 4, 5, 6 cannot be clearly resolved. Figures 7(b5)–7(d5) indicate the reconstructions by SC-FPM at 28 iterations with the final corrected parameters of $\theta =-1.6\mathrm{deg}$, $\mathrm{\Delta }x=1.188\mathrm{mm}$, $\mathrm{\Delta }y=1.003\mathrm{mm}$, $h=89.733\mathrm{mm}$. Compared with other algorithms, SC-FPM produces a better performance with higher contrast and improved resolution in the final reconstructions, where each line pair is clearly resolved with a uniformly distributed background, demonstrating its great adaptability and strong robustness to unknown systematic errors. As for the corrected positional parameters, only the height factor presents obvious distinctions between PC-FPM and SC-FPM. Actually, 94 mm by PC-FPM far deviated from the practical measurements of $h=86\mathrm{mm}$ while 89 mm by SC-FPM seems more reasonable. In addition, the performance of original FPM algorithm could be significantly enhanced by an offset with the parameters obtained by SC-FPM, which could be found from the comparisons between Figs. 7(b6)–7(d6) and 7(b1)–7(d1), further validating the reliability of SC-FPM.

Fig. 7

Experimental results of one segment ($90\times 90\text{pixels}$) in a USAF target recovered by different algorithms. (a) The FOV captured with a $4\times /0.1\mathrm{NA}$ objective. (a1) The enlargement of a subregion of (a). Groups (b–d) show the recovery results of intensity, phase, and spectrum, respectively.

In addition, we also test our method with a biological sample (stem transection of dicotyledon) by different algorithms as shown in Fig. 8. The LED array is placed at 85.9 mm above the sample with central red $9\times 9$ LEDs providing the angle-varied illuminations, resulting in a final synthetic NA of 0.35 theoretically. But the results are quite different from above due to the different composition of systematic errors. The recovery images in Figs. 8(b1)–8(d1), 8(b2)–8(d2), and 8(b3)–8(d3) are reconstructed by the original FPM algorithm at 6 iterations, intensity correction method at 6 iterations, and adaptive FPM at 18 iterations, respectively, all of which show a failure in reconstructions either with poorly visible intensities or uneven-distributed phases. Figures 8(b4)–8(d4) present the recovery results by PC-FPM at 12 iterations with the final corrected parameters of $\theta =-4.9\mathrm{deg}$, $\mathrm{\Delta }x=-0.104\mathrm{mm}$, $\mathrm{\Delta }y=-0.384\mathrm{mm}$, and $h=84.522\mathrm{mm}$. The recovered phase image is quite better than that in Figs. 8(c1) and 8(c3), but the contrast of the intensity image still remains to be improved. Figures 8(b5)–8(d5) indicate the reconstructions by SC-FPM at 34 iterations with the final corrected parameters of $\theta =-3.4\mathrm{deg}$, $\mathrm{\Delta }x=-0.427\mathrm{mm}$, $\mathrm{\Delta }y=-0.287\mathrm{mm}$, $h=84.733\mathrm{mm}$. The superiority of SC-FPM over other algorithms could be observed by the higher contrast and remarkable resolution in final reconstructions as indicated in Figs. 8(b5) and 8(c5), demonstrating its effectiveness and adaptability in practical applications.

Fig. 8

Experimental results of one segment ($200\times 200\text{pixels}$) in a biological sample (stem transection of dicotyledon) recovered by different algorithms. (a) The FOV captured with a $4\times /0.1\mathrm{NA}$ objective. (a1) The enlargement of a subregion of (a). Groups (b–d) show the recovery results of intensity, phase, and spectrum, respectively.

5.

Conclusions

In this paper, we theoretically and experimentally report a system calibration procedure, termed SC-FPM, based on the SA algorithm, LED intensity correction method, nonlinear regression process, and adaptive step-size strategy. SC-FPM can retrieve a high-quality, noise-robust complex object in the case of extreme multiple errors, including aberrations, LED intensity fluctuation, parameter imperfections, and noise in a variety of forms. The effectiveness and robustness of SC-FPM are demonstrated and the great performance has been achieved in both simulations and experiments.

Note that the reasonably selected parameters, such as initial iteration times and step length of SA algorithm, would contribute a lot to the accuracy and efficiency of SC-FPM. Therefore, they need to be carefully estimated according to the actual situations. Here, we mainly focus on the quality of reconstructions, and the aberration is introduced with a simple Zernike polynomial. The quantitative research on aberration is another noteworthy issue especially under such mixed system errors. Whether the aberration is real or the result of different errors it still needs to be analyzed theoretically and experimentally, which may be the subject of future work.