2.1.

System Configuration

The MGIS is a modified version of the mobile mapping system introduced in Ref. 7. In the current version of the system, a NovAtel Span CPT GNSS/INS unit replaced the Gladiator LandMark 20 GPS/INS and Topcon Hiper Lite used in the previous system version. The hyperspectral sensor of the MGIS comprises a V10E Spectrograph integrated with an Imperx industrial camera. The spectrograph has a spectral range of 400 to 1000 nm and a 2.8-nm spectral resolution. The Imperx camera offers a resolution of $1920\times 1080\text{pixels}$. The hyperspectral sensor produces a final hyperspectral image cube with 800 spatial pixels (samples) and configurable (270 or 135) bands through 2, 4, or 8 pixel binning configuration.⁷ Also part of the MGIS is a Nikon D 300 DSLR camera synchronized to GPS time via GPS timing receivers. The hyperspectral and digital cameras were fitted inside a custom-designed aluminum housing. The GNSS/INS unit was installed on top of the housing, along with two GPS timing antennas that provide GPS time tags for the captured imagery, allowing synchronization with the trajectory information through GPS time. Figure 1 shows a schematic diagram of the system configuration.

Fig. 1

MGIS data acquisition components.

The whole system is composed of off-the-shelf components, as described in the study of Abd-Elrahman et al.⁷ The hyperspectral camera and the NovAtel Span CPT GNSS/INS comprise the most expensive components of the system. Each of these systems costs around $22,000. The remaining system components cost another $6000 to 10,000. It should be mentioned that some of these estimates are for academic use and do not include the cost associated with the image acquisition software and synchronization mechanism developed in-house.

During the field data collection, the DSLR camera was set to take pictures at configurable acquisition rate through a custom-developed application, which could be used in future studies for digital elevation model (DEM) generation and/or updates to the trajectory. Two custom-made interface boxes were developed to trigger and time stamp the hyperspectral and DSLR camera imagery with GPS time using two Trimble Accutime Gold GPS receivers. Triggering signals are generated and controlled through custom applications developed using National Instruments Labview software for the hyperspectral sensor and the DSLR camera. Each GPS receiver is capable of timing events at an acquisition rate of 5 Hz or less. Considering the higher acquisition rate for the hyperspectral imagery (e.g., 40 Hz), the system is only capable of providing time stamps at a frequency that is less than the hyperspectral image acquisition rate. For example, if the hyperspectral image is acquired at 40 Hz ($40\text{frames}/\text{second}$), only one in every eight acquired lines are tagged with GPS time. One of the subobjectives of this research is to test the georeferencing accuracy of the system characterized by this limitation.

Each time a hyperspectral image frame is acquired, the data are transferred to the computer and the acquisition software reformats the individual frames to produce a hyperspectral image cube in a Binary Interlaced by Line (BIL) format. This image cube is then saved on the computer running the acquisition application. The DSLR camera imagery is stored on the camera flash drive to be retrieved after the imaging session. One of the main functionalities of the image acquisition software, in addition to sensor/camera trigger and saving the hyperspectral image, is to generate the time stamp log for the hyperspectral image and the DSLR camera frames.

2.2.

Study Site

Field measurements in the form of three MGIS scans took place at the University of Florida-Gulf Coast Research and Education Center (GCREC) in Wimauma, FL, $\sim 27°{45}^{\prime }{40}^{\prime \prime }\mathrm{N}$ and 82°13′40′′ W (Fig. 2). Two 100-m long stretches over pavement and agricultural beds were selected within the GCREC area. The MGIS was installed on a forklift tractor and set to capture data as it moved (Fig. 3). The first scan was collected over the paved road stretch [Fig. 2(a)] extending approximately in the east–west direction at 2:15 PM on June 4, 2014 (31°C and 43% humidity). Another two scans (scans A and B) were performed over a 100-m stretch of cucumbers and watermelon agricultural beds extending in the north–south direction [Fig. 2(b)] at 10:35 AM on July 1, 2014 (32°C and 45% humidity). DSLR images over the pavement lacked prominent features to support automated image matching necessary for processing, and thus were excluded from the analysis. It should be noted that choosing the scanned stretches to extend in east–west or north–south directions helped in debugging the results of the navigation data processing and image georeferencing steps, especially with the control point configuration extended along and off the scan trajectory line, as shown in Fig. 2.

Fig. 2

Field measurements conducted on (a) the pavement site and (b) within the agricultural site.

Fig. 3

The MGIS mounted on elevated platform (forklift tractor). Images collected over: (a) pavement road surface and (b) agricultural beds.

2.3.

Control Points and Lidar Data Acquisition

Several GCPs were established within each data collection site. Three GNSS receivers were set to collect static data for more than 4 h on control points labeled GCP_A, GCP_B, and GCP_C, established near the pavement road surface, as shown in Fig. 4(a). The same GNSS receivers were also set to collect static measurements exceeding 4 h on the control points labeled GCP_D, GCP_E, and GCP_F in the vicinity of the agricultural beds [Fig. 4(b)]. Sixteen control points were established on the pavement road surface, as shown in Fig. 5(a). The control point coordinates were surveyed using a total station by occupying GCP_C and backsighting GCP_A. Similarly, the total station was used to occupy GCP_D and backsight GCP_E to survey another 16 control points within the agricultural beds [Fig. 5(b)]. The GCP coordinates were in UTM zone 17 N (NAD83). Grid scale factor correction was applied to the total station measurements to account for the distortion associated with the map projection.

Fig. 4

Control points established using the GNSS receivers (a) near the pavement road surface and (b) within the agriculture beds. (Background images captured from Google Earth in 2014.)

Fig. 5

Ground control points established using total station: (a) over the paved road surface and (b) within the agricultural beds.

A terrestrial laser scanner (lidar) was used to collect point cloud data for the paved road. The point cloud was used to create a DEM for image georeferencing. When performing the lidar scans, several spherical targets were placed on top of control points to allow georeferencing of the point clouds. The control points utilized in producing the processed lidar point cloud were measured independently from the total station GCP measurements, using GNSS receivers in static observation mode (1- to 3-h occupation time).

3.1.

Global Navigation Satellite System/Inertial Navigation System Trajectory Postprocessing

The MGIS hyperspectral sensor functions similar to most pushbroom sensors used in airborne and spaceborne remote sensing mapping applications, except this system is designed to capture ground-based images from an elevated mount on moving farm equipment. In order to process the GNSS/INS data correctly, three-dimensional (3-D) distance offsets (lever arm offsets) were measured to relate each sensor to the IMU origin. Offsets measured from the center of the IMU to the GNSS antenna were input to the GNSS/INS software in order to relate the GNSS antenna measurements to the IMU axis frame. Offsets were also measured from the center of the IMU to the hyperspectral sensor and DSLR camera. This allowed translation of the navigation data to the actual imaging sensor locations. Figure 6 shows the offsets from the IMU to the GNSS antenna and hyperspectral sensor.

Fig. 6

The MGIS sensor orientation and lever arm offsets.

Two GNSS receivers were used to collect data at the GCP_A and GCP_B locations (independent of the observations used to generate the GCPs) while the MGIS scans were performed on the paved surface to enable differential correction. Similarly, two GNSS receivers were also set to collect measurements on the GCP_E and GCP_F control points located near the cucumber and watermelon agricultural beds site. The GNSS/INS data were processed using NovAtel Inertial Explorer. Output trajectories were produced at 100 Hz for each MGIS scan containing GPS time, latitude, longitude, pitch, roll, heading, and ellipsoidal heights. A custom script was developed to interpolate position and attitude angles for each captured DSLR camera and hyperspectral line image using the time stamp logs produced by the image acquisition software and the trajectory information exported from the Inertial Explorer software.

3.2.

Digital Elevation Models

Constructing a DEM is an essential step in georeferencing the collected hyperspectral images. The DEM used to process the hyperspectral imagery collected in the pavement stretch was constructed based on the lidar point cloud collected in the field. A simple DEM for the agricultural field was produced using control point elevations observed at the beginning and end of the beds. This configuration seemed reasonable given that bed preparation is conducted using machinery that employs GPS technology for leveling. In the future, we envision obtaining an accurate DEM as an output of the field preparation step conducted at the beginning of the crop cycle, mostly using GNSS-assisted machinery, such as tractors.

4.1.

Hyperspectral Image Georeferencing

The Parge²⁰ orthorectification add-on to EXELIS ENVI 4.8 was utilized to georeference the hyperspectral images. The georeferencing process starts with importing the raw image cube and defining the DEM. Trajectory information is input as an ASCII file containing position (latitude, longitude, and height) and attitude angles (pitch, roll, and heading) of each line in the hyperspectral cube. GCP coordinates are used to compute the boresight angle offsets, defining the angular attitude of the sensor relative to the IMU coordinate system. Alternatively, georeferencing can be performed using predetermined boresight offsets without the need for GCPs.

The equations describing scanning geometry have been derived by Ref. 21 and further refined by Ref. 22. As described in Ref. 20, the geometric model applied in Parge starts with the scan vector $L_{\mathrm{o}}^{\prime }$, as shown in Fig. 7. The same figure shows the initial coordinate system, which corresponds directly to image coordinate system, having the $x^{\prime }$; (sample) direction across track and the $y^{\prime }$; (line) direction along track. The initial scan vector for each pixel can be derived from the pixel scan angle assuming the across track vectors lie in a coplanar system. This is shown in the three equations listed below:²⁰

Fig. 7

The geometric sensor model applied in Parge.

To get the real length of the scanning vector $L_0$, the scan vector length has to be related to the altitude of the MHIS as follows:

Parge uses a statistical approach based on GCPs to calculate individual offsets for roll and pitch. For every GCP, the difference between known and estimated GCP position is calculated using the collinearity condition. By rotating the coordinate system back to the heading direction, the offsets for roll angle and pitch angle can be calculated for every GCP.²⁰ Within the software main processor algorithm, the image geometry is calculated and the nearest neighbor interpolation option is applied to resample the output georeferenced hyperspectral image to the DEM grid.²³

4.2.

DSLR Image Processing

Agisoft PhotoScan was used to process the DSLR camera data. Following the collection and organization of camera data, processing consists of four essential components: (1) initial image matching and orientation (alignment), (2) triangulation via bundle adjustment, (3) DSM generation, and (4) orthorectification. The initial image matching phase determines the location of conjugate points between multiple stereo images, from which the relative positions and angular orientation of all photos can be estimated. GCP coordinates and/or GNSS/INS observations associated with the camera are then used to establish the absolute position and angular orientation of the camera exposure stations via bundle adjustment. In order to obtain the best accuracy, camera calibration can also be included as unknowns in the triangulation/bundle adjustment phase and resolved in the solution.

Once the absolute image geometry and camera calibration parameters are established from triangulation, fine-scale point clouds can be created using this information and a second denser image matching process. From the dense point cloud data, a DSM can be formed. Finally, orthophoto mosaics, dependent on both the absolute image geometry and DSM from previous steps, are produced via backprojection from the DSM to the images, a process called orthorectification. Orthorectified, mosaics consist of combined images corrected for the effects of camera tilt and distortions caused by topographical relief. In contrast with “rubber-sheeting,” using simple 2-D transformations to approximate map geometry, orthophoto mosaics are formed using statistically rigorous photogrammetric methods. They can be considered spatially accurate maps and reliably used to obtain planimetric measurements.

A total of 85 DSLR camera images were processed to produce an orthorectified mosaic of the cucumber beds. As indicated earlier, the DSLR camera was mounted with the hyperspectral and navigating sensors on the forklift tractor at about 4 m above the agricultural bed. The coordinates of each camera exposure position were included in the photogrammetric solution. Two solutions and orthorectified mosaics were produced. One solution was generated using all 16 GCPs, and the other using only two control points at each end of the beds (a total of four control points). Using control points at the start and end of the beds, where minimal farming operations occur, is feasible in typical farming operations. The accuracy of these solutions based on checkpoints is presented in Sec. 5.

5.1.

Boresight Calibration of the Hyperspectral Images

Failing to account for the pitch and roll boresight offsets in the hyperspectral image, georeferencing process has a great effect on the image positional accuracy. Table 1 presents the horizontal positional accuracy evaluated using the georeferenced hyperspectral images captured at the pavement site before and after performing the boresight angle calibration. The table shows the difference between the GNSS/total-station surveyed control point coordinates and the ones extracted from the georeferenced images. Root mean square error (RMSE) values of 0.083 and 0.142 m were calculated for the easting and northing coordinates, respectively, without boresight calibration. In this case, individual coordinate differences in the easting (approximately along track direction) are consistently negative with small standard deviation. Given the flat surface nature of the study site, this may be attributable to systematic error in image synchronization or pitch angle boresight offset. However, the effect of image synchronization offset may not be significant though due to the low speed of the moving platform (about 0.3 to $0.5\mathrm{m}/\mathrm{s}$).

Table 1

Horizontal positional accuracy assessment of the pavement hyperspectral image before and after applying the boresight calibration.

The errors in the northing (across track) direction before applying the boresight calibration are also consistently negative. However, they vary among the three groups of control points located approximately north and south of the trajectory line. For each of these groups, the error values are similar in the easting direction, while the magnitudes of the northing errors differ from one group to the other. This observation highlights the boresight error in the across track direction and potential errors in the platform or DEM.

Similarly, Tables 2 and 3 show the horizontal positional accuracy for the hyperspectral images captured in the first and second scans of the agricultural field with and without boresight calibration. The RMSE shown in Table 2 of the without-boresight calibration results is 0.123 and 0.115 m in the easting and northing coordinates, while Table 3 shows RMSE values of 0.101 and 0.097 m for the same component directions. The results shown in Tables 2 and 3 for the positional accuracy without applying the boresight calibration indicate consistently positive values for the calculated coordinate differences in the easting direction while the coordinate differences calculated in the northing direction are shown to be consistently negative. This pattern matches the one observed for the pavement scan, considering the prominently east–west direction of the pavement scan line compared with the north–south scan direction of the agricultural fields and highlights the existence of significant boresight angle offsets.

Table 2

Horizontal positional accuracy assessment of the first hyperspectral image in the agricultural bed before and after applying the boresight calibration.

Table 3

Horizontal positional accuracy assessment of the second hyperspectral image in the agricultural bed before and after applying the boresight calibration.

The Parge software was used to calculate the boresight offsets using image and trajectory data in addition to ground control coordinates. These offsets have to be estimated and set once per flight or at least once per mission.²³ The platform carrying the MGIS had nominally linear trajectories, following the same general direction from the beginning until the end of each scanning session. The effect of the heading boresight angles within each scan was determined to be very small. Accordingly, the effect of heading offset was considered negligible in this study. In fact, when accounting for the heading angle offset, the calculated accuracy was almost identical to that obtained when using only the roll and pitch boresight offsets. The roll and pitch offsets computed at each of the 16 control points in the pavement stretch are shown in Table 4. The table displays average roll and pitch offset values of 2.0 and $-1.0\mathrm{deg}$, respectively. These values were calculated by averaging the roll and pitch offset determined for each control point listed in the table.

Table 4

Roll and pitch boresight offsets calculated for the pavement surface using all 16 control points, along trajectory centerline control points, and two control points only.

Comparing the roll boresight offsets estimated for the point groups north and south of the trajectory line, we noted that the roll offset values of each group were similar, having standard deviations that did not exceed 0.2 deg. Similarly, roll and pitch boresight offsets were calculated using only the control points located along the trajectory centerline and using only two control points (ECP and MCP) located at the eastern and western sides of the pavement scan to explore the potential of using fewer control points instead of the laborious 16 points used earlier. Table 4 shows roll offset angles of 1.8 and 2.0 deg and pitch offset values of $-1.0$ and $-1.2\mathrm{deg}$ for the two cases. The roll and pitch values shown in the table indicates a 0.2-deg difference, which can result in 0.014-m error in the easting and northing directions at 4-m sensor elevation above ground.

For the two MGIS hyperspectral images collected in the agriculture field, two of the control points established at the beginning and ending of the agriculture beds were utilized in the offset calculations for each scan. Table 5 displays the roll and pitch offset values of 1.7 and 1.2 deg, respectively, calculated for the data collected in the first agriculture field scan. These values were obtained by averaging the roll and pitch offsets at control points 2-M-3 and 15-M-3, as shown in Table 5. The table also shows roll and pitch offset values of 1.5 and $-0.7\mathrm{deg}$, respectively, calculated for the second MGIS scan in the agriculture field, which were obtained by averaging the roll and pitch offsets calculated using control points 6-M-1 and 15-M-3. It should be noted that control point 6-M-1 was used instead of 2-M-3 in the roll and pitch offset calculations for the second scan line since control points 1-E-3, 2-M-3 and 3-W-3 were not visible in the hyperspectral images of the second scan line of the agriculture field due to inadvertent termination of the image acquisition process at the end of the scan line.

Table 5

Boresight offsets calculated using two control points for the two images collected in the agriculture field.

5.2.

Horizontal Positional Accuracy Assessment of the Georeferenced Hyperspectral Images after Including the Boresight Offsets

The horizontal positional accuracy of the georeferenced hyperspectral images acquired on the pavement stretch after applying the boresight calibration is shown in Table 1. These results were achieved by including the roll and pitch boresight calibration offsets estimated using control points MCP and ECP, as discussed in the previous section. The MCP and ECP control point coordinates were excluded from the accuracy calculations, as shown in Table 1. The table shows RMSE value of 0.015 m calculated for the easting coordinates and 0.031 m RMSE for the northing direction. The higher errors in the northing (across track) direction may be attributed to systematic errors in the sensor height above ground. This can lead to magnified error levels in the north–south (across track) direction compared to the errors in the east–west (along track) direction due to the existence of control points to the north and south of the trajectory line and the possibility that this error was absorbed in the pitch boresight offset determination process. This is evidenced by the consistently negative and positive values of the errors in the northing coordinates of the points located to the north and to the south of the trajectory, respectively.

Table 2 shows the horizontal positional accuracy evaluated within the hyperspectral images captured in the first scan of the agricultural beds when performing the boresight calibration. RMSE values of 0.024 and 0.027 m are shown for the easting and northing coordinates, respectively. Similarly, Table 3 shows the horizontal positional accuracy evaluated for the hyperspectral images captured in the second scan of the agricultural field when implementing the boresight calibration. Calculated RMSE values of 0.023 and 0.025 m are shown for the easting and northing coordinates, respectively, which are comparable to those obtained for the first agricultural beds scan.

The similarities between the error values calculated for the agricultural field scans are expected since the same DEM was utilized to process both datasets. However, these two scans were acquired using two independent trajectories and image data collection sessions. Such consistency increases the confidence in the obtained results given the harsh environment, exemplified by the tractor vibration and the rough ride, encountered during the agricultural field data collection sessions. Figure 8 shows two subsets of the georeferenced hyperspectral image for the road and first agricultural bed image. The images were produced (resampled) at 1-cm spatial resolution and some of the control points and land features (e.g., manhole on the asphalt in the pavement image) are visible.

Fig. 8

(a) Subset of the georeferenced hyperspectral image taken for the paved road (grayscale image) and (b and c) two subsets of the first scan over the agricultural field (false color images).

5.3.

Horizontal Positional Accuracy Assessment of the Georeferenced Digital SLR Images

The results of the bundle adjustment of the DSLR camera images are presented in Table 6. The table shows the results of the solution using all 16 control points and using only two points at each end of the beds. A RMSE value of 0.004 m was obtained in both easting and northing directions in the former case. The RMSE values for the four-control-points case (0.016 and 0.026 m in the easting and northing directions, respectively) were significantly more than the case when all 16 control points were used. However, as stated before, using several control points along an agricultural bed is not practical, especially when human labor and machinery are active. The proposed points can be established at the beginning of the crop season and used throughout the season for all imaging sessions.

Table 6

Horizontal accuracy assessment of the orthorectified image created from the DSLR images.

Our experimentation to solve the bundle adjustment using only direct georeferencing (no GCPs) resulted in a poorly resolved camera distortion model. This problem can be somewhat mitigated if multiple overlapped imaging lines were used. Nevertheless, special consideration should be taken in the image planning process as the shadow of the platform can consistently show in the images and causes the automated image matching to fail.