1.1.

Overview

Determining the height profile of a target is useful in many applications, such as industrial production where rapid feature recognition is key to high fidelity and low cost. One common noncontact method to produce height profiles uses structured light illumination (SLI),^1,2 in which a pattern of light from a projector illuminates the object and camera images the scene. The pattern varies spatially and is further distorted by the sample’s three-dimensional (3D) shape. Image processing techniques then reconstruct the 3D image and produce the desired height maps.³ As with most optical systems, accurate optical alignment of SLI systems is key to the reconstruction resolution and performance.

Recent advances in computer power and algorithms have enabled the predesign of optical systems. Côté et al.⁴ created a deep-learning enabled framework called LensNet that extracts features of successful lens designs and then recombines the features to create new layouts. Ivanov et al.⁵ published a method to determine the number of optical surfaces necessary to achieve diffraction-limited performance prior to any actual optical design. Oleszko and Gross⁶ used a numerical model to study freeform optical systems surface-by-surface and correct for aberrations beyond the fourth-order.

This paper applies the predesign process to projector alignment in custom SLI systems. The projector is aligned with the quad target method (QTM),⁷ a simple height reconstruction technique that produces an estimate of the projector’s focal position using only a single camera image. The presented predesign method uses the native optical design file to include the effect of residual aberrations, a key step for accurate simulation of the experimental behavior. The goal here is to quickly move from the design to the lab by enabling a user to completely design the alignment process variables prior to any lab work, even tailoring the responses with a trade space analysis. The simulated alignment process is compared with experimental measurements on a custom SLI system to determine the accuracy of the predesign method. Results suggest that the predesign method can accurately model the experimental behavior.

This paper is organized into five sections. The remainder of Sec. 1 briefly describes SLI and the QTM. Then, Sec. 2 displays the test system since the optical design must be well-established prior to utilizing the presented predesign method. Section 3 presents the predesign process including the coordinate system and a simulation flowchart. Section 4 describes the experimental methods and displays and compares the experimental and simulated results. Conclusions are found in Sec. 5.

1.2.

Structured Light Illumination

One common method to perform 3D profilometry is structured light illumination (SLI). A schematic of a general SLI system is shown in Fig. 1.² Here, the structured light projector images a particular pattern onto the target. Many patterns are found in the literature, such as binary,⁸ gray-level,⁹ and colored bar patterns.¹⁰ A popular subset of SLI is fringe projection profilometry (FPP) where the projected pattern is sinusoidal fringes.^11–13 The rectangular target distorts the color bars in Fig. 1 due to its height above the substrate. A camera then images the entire scene. The combination of the camera image and a processing method enables the reconstruction of the target’s 3D shape.^1,2

Fig. 1

Schematic of a general structured light illumination system. Notice the projected color bars are distorted by the target’s shape. Reprinted with permission from Ref. 2 © The Optical Society.

1.3.

Quad Target Method

This paper utilizes the quad target method (QTM),⁷ an alignment technique that features a linearized response to focus the projector of an SLI or FPP system. The linearization is achieved by sampling the field at two different locations along the optical axis using the quad target. The quad target is a $2\times 2$ array of flat surfaces in pairs of quadrants separated by a step height $h_{\text{step}}$ as seen in Fig. 2. Then, focus is efficiently linearized by measuring the difference in the projected fringe frequency contrasts between the two quad target heights. QTM can produce an estimate of the focus position using only a single image, a $3\times$ improvement in speed over the common three-phase reconstruction technique.³ Thus, QTM provides linear feedback with a clear zero-crossing, to minimize any focus ambiguity, and fast positional feedback, to minimize adjustment lag.

Fig. 2

Cross-section showing a finite conjugate SLI system and two steps of the quad target. Notice that the blue field is focused on the front step and the red field is defocused on the back step. Other aspects: the thick black line is the optical axis, the digital micromirror device (DMD) produces the SLI pattern and is tilted to meet the Scheimpflug condition, distance $Z_1$ is the camera’s target-space working distance, and distance $Z_2$ is the projector’s back working distance.⁷

QTM is governed by two input variables: the frequency of the projected fringes ${\xi }_{\mathrm{project}}$ and the height of the quad target step $h_{\text{step}}$; it outputs two quantities: the slope of the linearized values $m_L$ and an estimate of the current focus position $Z_{\text{focus}}$. A schematic of QTM is shown in Fig. 2. Two field points (red and blue) of sinusoidal fringes at frequency ${\xi }_{\text{project}}$ are imaged by the projector optics onto the quad target, a $2\times 2$ array of flat surfaces physically separated along the optical axis, with two of the four steps shown in profile in Fig. 4. Notice that due to the quad target step height $h_{\text{step}}$ (greatly exaggerated for viewing ease), the blue field is in focus and the red defocused. A camera images the entire scene with the camera’s front working distance labeled $Z_1$. The distance $Z_2$ is the projector’s back working distance which can be physically altered to shift the projector’s focus position during alignment.

Figure 3 shows an example output of QTM. The raw and fitted fringe contrast values from the front step are in blue with the same curves for the back step are in red. The raw data is fit to a fifth-order polynomial (dotted lines) for noise reduction and robustness with a curve shape. Note that in this paper the fitted data curves for the front and back steps are called the $F$ and $B$ curves, respectively. Next, the linearized response $L$ (orange asterisk) is calculated from the $F$ and $B$ curves and also fit to fifth-order polynomial (orange dotted line) for the same reasons. Then, the nonlinear coefficients of the fit are set to zero to ensure that $L$ is linear near the calibrated focus position at $Z=0$. Finally, the black dotted line shows the current focus position of $Z=6.2\mu \mathrm{m}$, i.e., where $L=0$ by definition of QTM.⁷

Fig. 3

Experimental curves QTM including the raw data and fitted curves. Reprinted from Ref. 7.

As stated above, the raw $F,B$, and $L$ curves in Fig. 3 are all individually fit to fifth-order polynomials. For the given input variables, the curves in Fig. 3 are well-approximated by a second-order polynomial leading to the potential for errors caused by over-fitting. However, Ref. 7 showed that fifth-order fits do not lead to over-fitting for a wide range of QTM input variables. Additionally, proper choice of the projected light level and the large averaging described in Sec. 3.1 should ensure no over-fitting of the curves.

2.1.

Layout

This paper begins with the test system, since the design must be established before the alignment method can be predesigned. The custom SLI sensor¹⁴ shown in Fig. 4 has two cameras and four projectors, meaning that there are eight total SLI channels. Throughout this paper, however, only a single projector/camera combination was examined for brevity. A listing of the optical attributes of the sensor is found in Table 1. Note that the optical design is outside the scope of this paper.

Fig. 4

The SLI system under test. The single projector/camera combination examined is labeled.

Table 1

List of optical attributes for the custom SLI sensor.

In Table 1, two fields of view (FOVs) are listed: 3D and raw. The 3D FOV is the lateral region where the sensor can measure and generate 3D height maps, whereas the raw FOV is the full FOV of the camera. The raw FOV is slightly larger than the alignment FOV to ensure good alignment across the 3D FOV and to reduce edge effects during height reconstruction.

2.2.

Through Focus Performance

As will be described in Sec. 4, an important aspect in predicting the alignment performance is the through-focus response of the test system. Figure 5 shows the through-focus modulation transfer function (MTF) contrasts for three spatial frequencies using the native optical design. These specific frequencies will be tested in Sec. 4. Notice that the highest spatial frequency of 13.7 cy/mm in Fig. 5(a) has a much lower depth of field than the lowest frequency of 3.4 cy/mm in (c), and that the contrast dynamic range for 13.7 cy/mm is much larger than 3.4 cy/mm (where “dynamic range” is the difference between the maximum and minimum values). Thus, the tradeoff with ${\xi }_{\mathrm{project}}$ is higher spatial frequencies resulting in more sensitivity but narrower Z range for adjustment.

Fig. 5

Through focus contrasts for the three tested spatial frequencies.

3.1.

Coordinate System

The alignment predesign is performed via simulation. The coordinate system for the simulation method is shown in Fig. 6. The alignment FOV (red box) is slightly larger than the system FOV (green box), as described in Sec. 2.1. The black boxes show the experimental regions of interest (ROIs), i.e., the ROIs utilized during the projector alignment. The experimental ROIs are set as large as possible to reduce noise while avoiding the edges of the quad target step. To match the experimental ROIs as much as possible, the simulated field points are placed at the center of the four ROIs, thereby assuming that the aberrations across the ROI are roughly balanced at the center.

Fig. 6

Layout of coordinate plane for predesign and experiment: blue asterisks are the simulated field points; black boxes are the experimental regions of interest; the green box is the test system’s field of view; the red box is the alignment field of view.

3.2.

Simulation Flowchart

The simulation method uses the native optical design file. With this approach, the residual aberrations are accurately modeled, and their effects are included in the predicted response. Therefore, simulating with the native file is a key step for accurate simulation of the experimental behavior.

The simulation raytracing was performed in Zemax OpticStudio;¹⁵ using the ZOS-API, a MATLAB¹⁶ script managed the data flow and graphed the results. Figure 7 shows a flowchart of the simulation code. The QTM variables of $h_{\text{step}}$ and ${\xi }_{\mathrm{project}}$ are set to the desired values. Then, the target plane is shifted by $\pm h_{\text{step}}$ to simulate the front and back surfaces of the quad target, respectively. Next, the contrasts are determined for the four field points shown in Fig. 6 using the operand MTFT since the triangulation axis and tangential axis are parallel. The $F$ and $B$ contrasts are determined by multiplying the contrasts from the projector and camera,¹⁷ specifically

Fig. 7

Flowchart for simulation of the test system using OpticStudio¹⁵ and MATLAB.¹⁶ The steps with blue, dotted lines were performed in Zemax, while the black outlines were in MATLAB.

Distortion is of particular concern to SLI systems because it causes the projected fringe frequency to change across the FOV and, in turn, alters the measured fringe contrast. Since the native optical design file is used in the simulation method, the distortion’s effect is already included in the prediction; therefore, there is no need to correct the camera’s distortion for the presented simulation method. If additional distortion is found experimentally, the unmodeled distortion can be mitigated by decreasing the ROIs in Fig. 6 to avoid the FOV edges where distortion is at its largest magnitude, or by increasing the width of the bandpass filter used to determine the contrast to ensure that the altered frequency is measured.

3.3.

Simulation Outputs

With this method, the test system’s response can be simulated. Figure 8 displays the $F,B$, and $L$ curves through focus for $h_{\text{step}}=500\mu \mathrm{m}$ and ${\xi }_{\mathrm{project}}=6.9\mathrm{cy}/\mathrm{mm}$. This graph shows the $\mathrm{Z}$ position along the $X$ axis; the normalized contrast on the left-hand $Y$ axis; and the linearized value $L$ on the right-hand $Y$ axis. The main two outputs of QTM are garnered from Fig. 8: the slope of the linearized values $m_L$ and the current projector focus position $Z_{\text{focus}}$. The slope $m_L$ affects the resolution and “feel” of the adjustment for the user.

Fig. 8

Simulated data for $h_{\text{step}}=500\mu \mathrm{m}$ and ${\xi }_{\mathrm{project}}=6.9\mathrm{cy}/\mathrm{mm}$ to show $F,B$, and $L$ curves.

The current focus position $Z_{\text{focus}}$ is estimated using the linear fit of $L$. By definition of QTM, the current focus position is where $L=0$, so the focus position $Z_{\text{focus}}$ is estimated as

3.4.

Trade Space Analysis

We can probe the effect of $h_{\text{step}}$ and ${\xi }_{\text{project}}$ with a large-scale trade space analysis.¹⁸ Here, both variables are altered and the predesign method determines $m_L$ and $Z_{\text{focus}}$. This process has a unique benefit for the user: it allows the user to tailor the alignment method for their preferences. For example, if a user desires maximum adjustment sensitivity, the user might employ the highest projected fringe frequency to maximize $m_L$.

4.1.

Experimental Process

To test the accuracy of the predesign method presented in Sec. 3, experimental measurements of $m_L$ and $Z_{\text{focus}}$ were taken using the test system in Sec. 2. As shown in Table 2, seven different combinations of step heights and spatial frequencies were examined to sample the two variables across a wide range. Specifically, $h_{\text{step}}$ covers $1\times$, $3\times$, and $5\times$ of the camera’s depth of field and ${\xi }_{\text{project}}$ covers $0.24\times$, $0.49\times$, $1\times$ of the projector’s maximum spatial frequency (referring to Table 1). Notice that combinations of low ${\xi }_{\text{project}}$ and low and mid $h_{\text{step}}$ are not included since Ref. 7 showed that these variable combinations do not produce accurate estimates of focus position over $Z$ ranges of $\sim 1000\mu \mathrm{m}$.

Table 2

Table of hstep and ξproject combinations tested both experimentally and in simulation.

Figure 9 shows two of the three tested quad targets resting below the test sensor. The targets are displaced in $Z$ to change the distance $Z_1$ in Fig. 2. For all experimental and simulated results in Sec. 7, the range of $Z$ values was $\pm 500\mu \mathrm{m}$ with a $50\mu \mathrm{m}$ step size. The stage is a Newport GTS30V and is controlled via a MATLAB script. Finally, the test sensor is rigidly mounted to a granite frame to reduce vibrations, as seen in Fig. 4.

Fig. 9

Picture of quad target mounted on $Z$ stage.

4.2.

Single Comparison

As an initial comparison between the experiment and the predesign method, Fig. 10 shows the experimental curves from Run #3 with the simulated curves from Fig. 8. The error bars in Fig. 10 are $\pm 2\sigma$ from four experimental trials. Overall, the predesign method seems to predict the experimental curves well, as evidenced by the good visual agreement and high $R^2$ values in Table 3. Figure 10 also shows the most straightforward method to demonstrate the simulated and experimental agreement since it only uses one set of input variables.

Fig. 10

Comparison of experimental and simulation results. Labels – “Exp” is the experimental curve, “Sim” is the simulation curve, and the error bars are normalized contrast values of $\pm 2\sigma$ from four experimental trials.

Table 3

Table R2 values from a linear regression of the experimental and simulated curves for Run #3.

4.3.

Trade Space Analysis

As described in Sec. 3.3, the predesign method can perform a trade space analysis of the variable space. The analysis studied ${\xi }_{\text{project}}$ and $h_{\text{step}}$ over $5\times$ ranges, with ${\xi }_{\text{project}}$ starting from the maximum fringe frequency and decreasing $5\times$, and $h_{\text{step}}$ starting from $1\times$ the camera’s depth of field and increasing. Then, to further examine the predesign method’s accuracy, we compared the experimental and simulated values for $m_L$ and $Z_{\text{focus}}$.

4.3.1.

Linearized slope

One of the two outputs of QTM is the slope of the $L$ curve $m_L$. Figure 11(a) shows the simulated slope $m_L$ and (b) the localized gradient of the surface in (a). The mean angle of the gradient is 59.0 deg meaning that ${\xi }_{\text{project}}$ has a slightly stronger effect on $m_L$ than $h_{\text{step}}$. Thus, if a user desires the alignment to be more sensitive, i.e., a higher $m_L$, increasing the fringe frequency or step height will have roughly the same effect.

Fig. 11

Results from the simulated trade space analysis for linearized slopes.

The seven combinations in Table 2 were tested experimentally using the test system in Sec. 2. The results are displayed graphically in Fig. 12 with summary statistics in Table 4. The graph exhibits very good linear regression (the dotted line) both visually and with $R^2=0.99$. Furthermore, the root mean square (RMS) difference for all runs is $0.02{\mathrm{mm}}^{-1}$ and the regressed slope are very close to ideal at 1.04. These results suggest that the predesign method can accurately predict the linearized slope $m_L$ and seems insensitive to the ${\xi }_{\text{project}}$ and $h_{\text{step}}$ combination.

Fig. 12

Comparison of simulated and experimental results for linearized slope. The error bars are experimental slopes of $\pm 2\sigma$ from four experimental trials.

Table 4

Table of linearized slope numerical results from trade space analysis.

Summary measure	Value
Regressed slope	1.04
Regressed bias (${\mathrm{mm}}^{-1}$)	$-0.01$
Regressed $R^2$	0.99
RMS difference (${\mathrm{mm}}^{-1}$)	0.02

Table 5

Table of best focus numerical results from trade space analysis.

Summary measure	Value
Regressed slope	1.01
Regressed bias ($\mu \mathrm{m}$)	$-0.9$
Regressed $R^2$	0.50
RMS difference ($\mu \mathrm{m}$)	1.6

4.3.2.

Focus location

The other output from QTM is the estimate of the current focus location $Z_{\text{focus}}$. Figure 13 shows the output of the tradespace analysis for $Z_{\text{focus}}$ and Fig. 16(b) the localized gradient of the surface. Unlike linearized slope, the $Z_{\text{focus}}$ location changes very little for the variable ranges except for the lower right-hand corner, i.e., low ${\xi }_{\text{project}}$ and high $h_{\text{step}}$. For the gradient, ${\xi }_{\text{project}}$ very strongly affects the focus position with a mean gradient angle of $-76.9\mathrm{deg}$. Furthermore, most of the vectors are nearly 90 deg, meaning that ${\xi }_{\text{project}}$ is essentially the only effect on $Z_{\text{focus}}$. This is likely caused by the projector’s MTF having different responses for different ${\xi }_{\text{project}}$, as seen in Fig. 5.

The dynamic range of the $Z_{\text{focus}}$ values in Fig. 13(a) is $20.9\mu \mathrm{m}$. While only 10.4% of the camera’s depth of field, that range may be too large for SLI systems that require micron-focus precision. Looking back to the through-focus curves in Fig. 5, we find that focus positions for ${\xi }_{\text{project}}=13.7$ and 3.4 cy/mm are not equal and in fact are separated by $27.0\mu \mathrm{m}$. Thus, the range of $Z_{\text{focus}}$ in Fig. 13(a) test system is due to the test system’s frequency-dependent focus position and not an error within the predesign method.

Fig. 13

Results from the simulated trade space analysis for best focus position.

As in Sec. 4.3.1, $Z_{\text{focus}}$ was measured for the seven combinations using the same experimental procedure. $Z_{\text{focus}}$ was determined with Eq. (2) with the results shown graphically in Fig. 14 and numerically in Table 5. Figure 14 displays a noticeable correlation between experimental and simulated focus locations, with the regressed slope at 1.05. However, the correlation is not as strong as $m_L$ with $R^2=0.55$ likely due to experimental noise. The predicted accuracy of $Z_{\text{focus}}$ is very high with an RMS difference of $1.6\mu \mathrm{m}$ over the seven combinations, thereby suggesting that the predesign method can accurately estimate the current focus location.

4.4.

Alignment Design Recommendation

The presented results suggest that the alignment process can be simulated. Thus, we can now predesign the QTM variables using the results in Figs. 12 and 14. For the tested SLI system, maximum adjustment sensitivity is desired more than the usable $Z$ range due to the relatively small lateral resolutions of $5.0\mu \mathrm{m};$ thus, $h_{\text{step}}=1000\mu \mathrm{m}$ and ${\xi }_{\mathrm{project}}=13.7\mathrm{cy}/\mathrm{mm}$.

Fig. 14

Comparison of simulated and experimental results for best focus. Note that the error bars are $\pm 2\sigma$ from four experimental trials.