1.

Introduction

Multi- and hyperspectral imaging (HSI) are important technologies for a variety of applications. The first setups were realized as early as 1869 by Janssen¹ for the observation of the solar corona. However, the technology was largely ignored until the 1980s due to a lack of detectors. Since then, the number of applications and technical approaches grew steadily.^2,3

In addition to the well-established scanning line spectrometers, the so-called pushbroom systems, different single-shot or snapshot sensors have been proposed. In these sensors, a hyperspectral image of a two-dimensional (2D) scene is captured using a single exposure of an image sensor. A good review was given by Hagen and Kudenov.²

A commercially available approach is a spectral resolving filter array (SRFA). However, these detectors are rather expensive due to their complex fabrication.^4,5 They are also particularly susceptible to fabrication tolerances.^6,7 Another approach, proposed by Bacon⁸ and Courtes et al.,⁹ relies on spatial sampling using a microlens array in an intermediate image plane. This allows one to generate and capture a spectrum for each aperture on a conventional image sensor. Computed tomographic imaging uses (CTIS) the diffractive elements to generate multiple, overlapping, and spectrally separated copies of a scene. Numerical reconstruction, e.g., using expectation maximization algorithms, then leads to the final hyperspectral image.¹⁰

In this contribution, we propose a new method for snapshot HSI. The approach also relies on diffraction, but we use microgratings located in an intermediate image plane to achieve HSI. This leads to a relatively easy application-specific implementation. One of the main advantages of the proposed sensing principle is a quite large freedom of choice concerning spatial and spectral sampling in arbitrary image patches. Furthermore, compared with the IMEC SRFA sensor investigated by Hahn et al.,^6,7 the method is superior in the detection of spectral shifts, is less sensitive to manufacturing tolerances, does not necessarily need a pixelwise calibration, is more robust to varying ray angles, and does not require any further spectral filters for proper operation. However, the proposed method leads to broader spectral channels at a given spatial resolution.

The method could be of interest in food quality control,¹¹ irrigation control in agriculture,¹² monitoring of vital parameters in medicine,¹³ determination of human tissue,¹⁴ or optical surface metrology.¹⁵

In Sec. 2, we present the basic principle and derive a signal model that leads to an uncertainty relation for the spatial and spectral resolutions in Sec. 3. In Sec. 4, we introduce our laboratory setup for a first experimental demonstration of the sensor principle before showing measurements in Sec. 5. A conclusion is given in Sec. 6.

2.

Sensor Concept

The working principle of the method is shown in Fig. 1. The scene to be analyzed is imaged onto an intermediate image plane using an image-sided telecentric imaging system in which an application-specific pattern of microgratings is located. This pattern is realized as a lithographically fabricated diffractive optical element (DOE). The local micrograting period $d$ then determines the deflection angle of the ray bundles for a given wavelength $\lambda$. The chief ray is deflected according to the grating equation $\sin \varphi =\lambda /d$. The intermediate image is imaged onto the image sensor by the HSI unit, which is realized by a double-sided telecentric system. In principle, object-sided telecentricity is sufficient, but for most image sensors, it is advantageous if the angle of incidence of the light is small. The aperture stop $D_2$ is used for spectral filtering.

Fig. 1

Principle of the hyperspectral detection. Microgratings in an intermediate image plane deflect the light toward a spectral filtering iris. Each micrograting is optimized for an individual application-dependent wavelength.

As shown in Fig. 1, only parts of the spectrum can pass the aperture (green solid line). The blocked wavelength (red dashed line) can pass the aperture from other field positions, if the microgratings at these positions result in the suitable diffraction angle for the wavelength (red solid line). All microgratings with identical grating periods lead to the same angular deviation of the ray bundles and, therefore, the same spectral band being able to pass the aperture. On the monochrome image sensor, only the areas of the image with a wavelength that matches the design wavelength of the grating appear bright.

Of course, the width of the aperture stop determines the bandwidth for the spectral filtering. On the other hand, it also determines the lateral resolution of the imaging of the intermediate image onto the image sensor. In addition, the exit pupil diameter of the first imaging (object to intermediate image plane) should exactly fit the entrance pupil diameter of the HSI system. In other words, the aperture stop $D_1$ of the first imaging system has to be imaged onto the aperture stop of the HSI system.

3.

Signal Model

In the following, we derive a one-dimensional (1D) estimation of the resulting spectral response for imaging a point $(X,Y)$ in the object plane. We assume that, at a certain wavelength ${\lambda }_0$, the light will pass the iris of the second imaging system (see Fig. 1).

A shift of the wavelength $\mathrm{\Delta }\lambda$ leads to a shift $\mathrm{\Delta }x$ of the ray bundle on the filter plane. $\mathrm{\Delta }x$ is approximated as

For simplicity, we first describe the system in a 1D model. In this model, we can describe the filtering aperture with diameter $D_2$ by a rect function with width $D_2$. The amplitude distribution due to a certain object point also has a width of $D_2$ for one wavelength ${\lambda }_0$.

At a certain wavelength $\lambda ={\lambda }_0+\mathrm{\Delta }\lambda$, the corresponding rectangular amplitude distribution is shifted (dispersion at the grating) by $\mathrm{\Delta }x$. If we are interested in the spectral full-width half-maximum (FWHM), the allowed shift is given as

The FWHM corresponds to $\pm \mathrm{\Delta }\lambda$; therefore,

For an arbitrary $\mathrm{\Delta }\lambda$, the amplitude distribution $\mathrm{\Phi }(\mathrm{\Delta }\lambda )$ after the filter is

With $x\prime :=-x/D_2$, we obtain

By considering the $y$-axis symmetry of the rect function $\mathrm{rect}(-x)=\mathrm{rect}(x)$, it follows that

With the 1D lateral resolution, corresponding to the well-known resolution limit $0.61·\lambda /\mathrm{NA}$ for circular apertures, the resolution in the object plane is

We then find the uncertainty relation for spectral and lateral resolution

For a lateral resolution of $\mathrm{\Delta }X=50\hspace{0.17em}\mu \mathrm{m}$ and a grating period $d$ of $2.5\hspace{0.17em}\mu \mathrm{m}$, we obtain an FWHM of 30 nm at $\lambda =600$ nm.

One has to be careful about the interpretation of $\mathrm{\Delta }X$. This is the physical resolution defined by the optical system. However, if we want to use, e.g., nine ($3\times 3$) spectral channels, the spatial resolution would be effectively—as with the typical Bayer-like mosaics—reduced, in this example by a factor of at least three.

For a 2D imaging model, one has to replace the rect functions by circ functions. The triang result of Eq. (9) then becomes an arccos function, similar to the optical transfer function for aberration-free incoherent imaging for a circular pupil.¹⁶ The difference, however, is negligible, so in the following we continue using Eq. (9). The 1D model also stays valid if slit apertures perpendicular to the dispersion direction are used in the setup. This will result in having the resolution as described in Eq. (12) along one sensor axis, while the other axis stays unaffected by the filtering.

The result of Eq. (9) corresponds to the case for an extended grating. However, if we use a hologram consisting of a pixelated pattern of microgratings, additional complications arise at the transition from one grating to the next. Due to the limited spatial resolution of the imaging, a point in the image plane receives light from an extended patch in the object plane. As shown in Fig. 2, this patch might fall onto the transition between two microgratings. The point receives light from two different gratings and, therefore [Eq. (9)], with two different spectral distributions.

Fig. 2

Imaging a point close to the transition between two microgratings. Due to the limited spatial resolution, a spectral and spatial blur occurs.

As a result, we have the weighted sum $\mathrm{\Gamma }$ of two spectral distributions. The weight is proportional to the corresponding (incoherent) point spread function of the imaging:

This leads to a noteworthy broadening of the spectral response only, if the grating periods are considerably different and the gratings are small. For larger gratings (compared with the PSF extension in object space), the broadening is only visible at the position of the transitions.

4.

Experimental Verification

We realized an exemplary setup with nine spectral channels designed for a wavelength range between 510 and 700 nm. The DOE consists of a repeating stripe pattern (microgratings) as shown in Fig. 3. A block of nine microgratings is combined into the so-called macrograting.

Fig. 3

Schematic drawing of the DOE. A so-called macrograting, which consists of nine microgratings, is repeated along one spatial axis. Each micrograting defines the spectral range to be detected by the pixel corresponding to that micrograting.

An optics simulation was performed using Zemax®. The numerical aperture was set to 0.005. For the simulation, a grating with a constant frequency of $0.425\hspace{0.17em}\mathrm{lines}/\mu \mathrm{m}$ was used, leading to a diffraction angle of 14.79 deg for a wavelength of 600 nm. For collimation, an achromatic lens with a focal length of 200 mm (Thorlabs AC254-200-A) was inserted, leading to a beam diameter of 2.00 mm, which defines the size of the spectral filtering aperture $D_2$. The last lens, a 100-mm achromatic lens (Thorlabs AC254-100-A), is used to focus the light onto the image sensor. Based on the simulation, a spectral resolution of ~21 nm FWHM for the system is expected.

According to the Rayleigh criterion

The width of the microgratings is set to $70\hspace{0.17em}\mu \mathrm{m}$ in our laboratory setup, which is close to the spatial resolution limit for the central wavelength. The DOE size of $10\mathrm{mm}\times 10\mathrm{mm}$ leads to 142 microgratings and 15 macrogratings.

A monochrome image sensor (Ximea MQ013MG-E2) with an active area of $6.9\mathrm{mm}\times 5.5\mathrm{mm}$ and a pixel size of $5.3\mu \mathrm{m}$ was used. The response of the used sensor is shown in Fig. 4.

Fig. 4

Response of the used Ximea camera.¹⁷

The imaging system generating the intermediate image is set up by a 40-mm lens and a 100-mm lens. The aperture is placed in the object-sided focal plane of the second lens to generate image-sided telecentric imaging. The diameter of the aperture $D_1$ is adjusted to fit the targeted NA of the HSI-system, such that the filtering aperture fits the beam diameter.

5.

Sensor Validation

To characterize the system, a USAF target (Thorlabs R1DS1P) was illuminated by a fiber-coupled broadband halogen lamp (Helmut Hund GmbH, FLQ 150) in combination with different bandpass filters. Figure 5(a) shows a measurement without any spectral filter. This and all other measurements shown in this paper consist of 50 averaged individual measurements to reduce noise. Due to the broadband illumination, the complete scene can be seen. The illumination spectrum is shown in Fig. 5(b) by the solid red line.

Fig. 5

(a) HSI measurement taken with a broadband illumination and (b) signal along the marked lines in (a) depicted together with the reference spectra.

The visible fringes in the measurement [Fig. 5(a)] originate from the spectral intensity distribution of the light source, which decreases significantly with increasing wavelength. Figure 5(b) shows the signals along the marked lines in Fig. 5(a) as well as the spectrum of the light source captured by a spectrometer. All signals were normalized and show a similar behavior. The differences between the spectrum and the measurement are, on the one hand, caused by manufacturing irregularities of the gratings, resulting in different diffraction efficiencies for the gratings. On the other hand, the spectral resolution of the system is not high enough to resolve the small details of the illumination spectrum.

The smallest resolvable element of the USAF target has a spatial frequency of 28.50 lp/mm. Taking into account the magnification of the imaging system, the resolution is $87\hspace{0.17em}\mu \mathrm{m}$. This is close to the theoretical resolution of $85\hspace{0.17em}\mu \mathrm{m}$ for 700 nm [Eq. (15)].

Figure 6 shows the same scene as shown in Fig. 5(a) but with a spectrally filtered illumination. The employed bandpass filter has a central wavelength of 620 nm and a bandwidth of 10 nm. In Fig. 6, only the areas where the microgratings match the wavelength of the illumination appear bright.

Fig. 6

(a) HSI measurement captured with an inserted bandpass filter with a central wavelength of 620 nm and a bandwidth of 10 nm. (b) The illumination spectrum.

For the evaluation of the wavelength sensitivity, the bandpass filter was successively tilted, such that the center of mass (COM) of the spectrum changed from 623.0 to 622.5, 621.4, and 619.0 nm. The shift was observed with a spectrometer. The illumination spectra are shown in Fig. 7(a).

Fig. 7

(a) Illumination spectra for the different signals shown in (b) and (c). The example measurements in (b) and (c) were performed along the $y$ axis at a single position inside the (b) green and (c) red marked areas in Fig. 6, respectively. The different curves were obtained by bandpass filtering of a halogen light source controlled by a commercial spectrometer.

Figures 7(b) and 7(c) show typical signals acquired along the y axis of the sensor on an example position in the green and red marked areas given in Fig. 6(a), respectively. As can be seen, the peak location changes for the different wavelengths.

Figure 8 shows the variation of the COM for the different illumination spectra with respect to the spectrum with a COM of 623.0 nm. The threshold for the values considered for the calculation of the COM was set to 50% of the maximum of each evaluated column. In Fig. 6, the evaluated areas for Figs. 8(a) and 8(b) are marked in green and red, respectively. Each area consists of 200 columns along the $x$ axis. As can be seen, the amount of displacement in both the red and green highlighted areas is overlaid by a periodic structure. The amplitude of this structure increases with stronger COM displacement. The origin of this effect is unclear and needs to be further researched. In addition, a different trend can be observed in the two plots. This could be caused by possible aberrations of the optical system or by the DOE itself. Both the periodic pattern and the different trend should be easily corrected by calibration.

Fig. 8

Change of the COM for 200 pixel columns inside the (a) green and (b) red marked areas in Fig. 6(a).

Table 1 shows the average change of the COM as well as the standard deviation for the change inside the red and green areas, respectively.

Table 1

Average shift and standard deviation for the COM shift for different wavelength shifts inside the red and green marked areas in Fig. 6(a).

For the evaluation of the spectral resolution and for the validation of Eq. (12), the system was equipped with a grating that has a period of $2\mu \mathrm{m}$ over the whole field. The spectral resolution defining the minimal distinguishable distance between two wavelengths is described by the half-width of the signal under monochromatic illumination. The spectral resolution should not be confused with the detection of a spectral shift studied in the previous section. For the illumination of the system, a monochromator that allows a spectral tunable illumination from 450 to 550 nm was used. The half-width of the illumination spectrum is about 10 nm. The illumination was systematically tuned in 1 nm steps.

As shown in Fig. 9, the signal varies with illumination as expected. The half-width is $\sim 18\mathrm{nm}$, which is our spectral resolution $\mathrm{\Delta }\lambda$. Using Eq. (12), a theoretical lateral resolution $\mathrm{\Delta }X$ of $55.5\mu \mathrm{m}$ in the intermediate image plane can be expected for a wavelength $\lambda =500\mathrm{nm}$. By looking at a USAF target with an illumination of 500 nm, a structure with 40.3 lp/mm can be resolved. This corresponds to a resolution of $62\mu \mathrm{m}$ in the intermediate image plane.

Fig. 9

Sensor response of a single pixel on the image sensor as a function of the illumination wavelength. A grating with a constant period over the whole field was used.

6.

Conclusion

We have shown a new approach for hyperspectral snapshot imaging based on microgratings in an intermediate image plane. An individually manufactured DOE in combination with an aperture performs the spectral filtering. Compared with conventional Fabry–Perot-based mosaic filters, fabrication is simpler, and a stable filter response is automatically achieved. Most important is the possibility of choosing arbitrary spatiospectral patterns for a given application.

These advantages are achieved at the cost of a coupling of the spatial and spectral resolutions along the direction of the spectral separation. However, it should be mentioned that the spectral resolution is not always the relevant parameter. For some applications, e.g., chromatic confocal microscopy, the exact determination of the center of gravity of a broad spectral distribution is the relevant specification. For such applications, we could demonstrate measurements of spectral shifts as small as 0.5 nm for our proof-of-principle system covering a wavelength range from 510 to 700 nm with nine channels. This is analogous to human color vision, in which quite broad spectral channels are used and still excellent color discrimination in the yellow-green spectral range can be achieved. Therefore, the proposed method is especially suited for such applications in which broad and overlapping spectral channels can be employed or are advantageous.

We also want to note that for many applications some kind of restoration of spatial resolution might be achieved by postprocessing using common demosaicing algorithms. However, since the limit of spatial resolution in the proposed method is still coupled to spectral resolution, the achievable resolution enhancement probably will be strongly limited.