1.

Introduction

The spread of the coronavirus pandemic has impacted our daily life and continues to impose changes in our everyday activities and behaviors in society. As protective measures increase against the virus, it is an emerging and inevitable necessity to transform our close-contact day-to-day routines to remote and touchless. At the time of COVID-19, biometric systems were required to verify the identity of individuals and validate their credentials at a distance without requiring contacting a surface or removing a face mask. Compared with other biometric modalities such as fingerprint, palm, and face, iris comes into prominence as being a nonintrusive and contactless authentication method in addition to its superior accuracy, unique distinctiveness, enhanced security, and unbreakable reliability.¹ However, its dependence on image acquisition conditions limits its accuracy and hinders its potential usage in different applications. Traditional iris recognition systems can recognize individuals in a well-controlled environment, so they require capturing high-quality frontal images. Therefore, existing systems show degraded performance for images captured at nonideal conditions including the gaze angle, iris occlusion, and pupil dilation.²

The development of standoff biometric systems enables the identification of both cooperative and noncooperative individuals in a less constrained environment.³ Compared with existing systems, subjects have more freedom during data acquisition, and the standoff system can still capture their images while subjects are moving around at-a-distance. Since subjects are not required to stand up and look at the camera in standoff biometric systems, captured images will likely be nonideal, including off-angle. Figure 1 shows an off-angle image in which the camera axis and subject’s gaze do not overlap. When an off-angle image is compared with the frontal images of the same person, traditional iris recognition systems calculate their similarity score close to images from different subjects.

Fig. 1

An off-angle image at the NIR spectrum covering the ocular and periocular structures including eyebrows, eyelashes, iris, pupil, sclera, eyelids, and skin.

Traditional iris recognition systems generally adopted a similar methodology to Daugman’s initial design.¹ After the image acquisition, they follow four steps for iris recognition: (1) segmenting inner and outer iris boundaries, (2) normalizing the segmented image into dimensionless rectangular polar coordinates, (3) encoding the binary features from the normalized images to generate iris codes, and (4) comparing iris codes with the enrolled ones using Hamming distances (HDs). However, existing systems depend on image quality, are less flexible to a change in iris texture, and mostly require adjusting parameters for new datasets. In addition, off-angle iris images also introduce several additional challenges to the biometric community including the refraction of light at the cornea, the appearance of three-dimensional (3D) iris textures, blur due to the depth of field loss, accommodation of lens, and limbus occlusion at iris texture.⁴ Therefore, off-angle iris biometrics is an emerging research area that addresses these challenging issues.

As initial attempts, elliptical unwrapping and perspective transformation methods have been proposed to improve the recognition performance using traditional iris frameworks.^1,5 Since these approaches aim to remove the perspective distortion, they show poor recognition performance for images with $>30\text{-}\mathrm{deg}$ gaze differences. Figure 2 shows examples of off-angle images captured at the near-infrared (NIR) spectrum from different gaze angles and their normalized iris images using the elliptical unwrapping method. The distortion differences at the normalized iris textures become more visible as gaze angles increase from the frontal image in Fig. 2(a) to the 50-deg off-angle image in Fig. 2(f). Existing algorithms could not authenticate the identity of these two images being the same individual. Therefore, addressing the challenging degradation factors in off-angle images is required for designing an accurate standoff biometric recognition system.

Fig. 2

Examples of biometrics images with their normalized iris images from (a) 0 deg, (b) 10 deg, (c) 20 deg, (d) 30 deg, (e) 40 deg, (f) 50 deg in gaze angles.

The development of convolutional neural networks (CNNs) allows researchers in different disciplines to pursue solutions for their difficult problems using deep learning methods. Related to iris biometrics, there are several deep learning-based studies focusing on iris segmentation,⁶ spoofing attack detection,⁷ and recognition performance improvement.^8,9 Although these CNN-based works investigated several feature extraction methods from iris texture and achieved performance improvement for nonideal frontal images, they do not address the challenging issues in off-angle images, and they do not report any detailed results about the off-angle biometrics.

Instead of focusing on correction of iris deformations in off-angle images, another alternative approach to improving the recognition performance is to collaborate with other available information in a captured image, such as pupil, sclera, eyelash, skin, and even eyebrows. There exist several studies on other biometric modalities using ocular and periocular structures including dynamics of eye movements,^10,11 shape-based geometric features,¹² conjunctival vasculature structures,¹³ local binary patterns (LBPs),¹⁴ and different filter models to extract features.^15,16 These studies have shown that the fusion of multiple biometric modalities improves the overall recognition performance of biometric systems. In addition, deep-PRWIS¹⁷ suggested using the CNN-based AlexNet architecture for visible spectrum periocular images for better performance by including the periocular structures and ignoring the ocular components. Even if these methods improved the recognition performance for nonideal frontal images, they did not include any results for off-angle images.

This paper presents deep learning frameworks to improve the recognition performance of iris, ocular, and periocular biometric modalities for off-angle images. Our contribution is fourfold: (a) we utilized three off-the-shelf CNN-based frameworks for iris/ocular/periocular biometric modalities using transfer learning, (b) we study the effect of the gaze angle difference on iris/ocular/periocular recognition, (c) we discuss the information fusion capability of CNNs for different off-angle images, and (d) we compare our results with Gabor-based traditional iris recognition frameworks. The novelty of the paper is it being the first work investigating the recognition performance of the off-angle iris/ocular/periocular systems with respect to their gaze angle to the best of our knowledge. This study helps to design standoff biometric systems that capture images from a distance and angle that can be used by a variety of applications ranging from passport control to hospital check-in. It may give the opportunity to develop fast and secure standoff biometric systems.

This paper is organized as follows: Sec. 2 summarizes works related to off-angle iris recognition and information fusion approaches for ocular and periocular biometric modalities. The adapted CNN-based deep network architectures are presented in Sec. 3. Section 4 describes the details of the data collection and dataset preparation protocols and presents the baseline iris recognition algorithm to compare with the proposed methods. Section 5 shows the experimental results and discusses the important findings. Finally, we conclude in Sec. 6.

2.

Related Works

The biometrics literature includes several studies related to iris recognition. They have mostly contributed to designing new methods to improve the recognition performance and have proposed solutions for occlusion, refraction, variations in illumination, and blur problems in frontal images. The majority of the recent research ignores the challenging issues in standoff biometrics and off-angle images. Although some address problems on nonideal images, they are mainly frontal images, and only a few studies of off-angle images exist.

For off-angle iris recognition, Daugman revised his initial circle detection method using the integrodifferential operator with an active contour algorithm in which an ellipse is fitted to both the pupil and iris boundaries.¹ Then, the geometric distortion of the off-angle iris images is corrected using affine transformation. Since two-dimensional (2D) affine transformation is inadequate to project 3D iris texture, it offers a limited improvement for off-angle iris images. For distortion compensation in off-angle images, a brute-force method⁵ was proposed to project off-angle images into their frontal perspective for every possible angle. Instead of frontal projection of off-angle images, the elliptical unwrapping method can be used to normalize off-angle iris images directly from the elliptical segmentation of pupil and iris boundaries.¹⁸ While both the perspective projection and the elliptical normalization methods correct some degree of geometric deformations in off-angle images and increase the identification accuracy, their improvement is limited beyond 30 deg in angle because they do not address the challenging issues at off-angle iris images such as refraction of light at the cornea and iris occlusion due to the limbus.

To address these degradation factors of off-angle images, there are quite a few works proposing solutions in the traditional iris recognition pipeline. For example, Santos-Villalobos et al.¹⁹ presented a frontal iris reconstruction method using raytracing in a biometric eye model to compensate for the effect of corneal refraction. Although it proved that corneal refraction elimination is possible at synthetic off-angle images, its correction at real off-angle iris images is limited due to other challenges in off-angle images such as limbus occlusion. Karakaya et al.²⁰ reported the “limbus effect” on off-angle iris images in which the semitransparent limbus occludes iris texture at the sclera-iris boundaries and causes segmentation differences between frontal and off-angle images. Another study on off-angle iris images²¹ showed that lens accommodation affects iris surface curvature and results in a performance drop. Since all of these efforts addressed each challenging degradation factor separately, their recognition improvements on the real off-angle image were limited, and how to eliminate the challenges together with a comprehensive solution is still an open question.

With the recent improvements in the area of computation and neural networks, deep learning architectures, especially CNNs,²² get attention from different research communities for utilizing deep networks in different applications, such as image classification tasks. In terms of CNN-based iris recognition, DeepIrisNet⁸ was an early work in which deep networks were used for feature extraction from iris texture. Instead of converting grayscale iris texture to binary iris codes using Gabor filters, the CNN framework extracted features from a square-sized normalized iris image. For iris matching, features at the last fully connected layers were compared using the Euclidean distance to other subjects. Another DeepIris network²³ was proposed to improve the heterogeneous iris verification; it consists of nine layers to learn pairwise filters from different sources. Minaee et al.⁹ utilized the VGG-Net framework to extract features from iris texture and used principal component analysis for dimensionality reduction. They fed the raw image as an input to the deep network without segmenting and unwrapping the iris. Nguyen et al.²⁴ first applied transfer learning method to adapt pretrained CNNs to extract features from segmented and normalized iris images and classify the feature vectors using multiclass support vector machines. These existing CNN-based papers focused on iris recognition for only frontal images and do not deal with the problems in off-angle images.

Data acquisition platforms usually capture the eye structures surrounding the iris such as pupil, sclera, eyelash, skin, and even eyebrows. Traditional iris recognition algorithms discard the information in the ocular and periocular components during the segmentation step. However, there might be useful information in ocular and periocular areas for developing a more reliable biometric system. Therefore, combining iris with ocular and periocular biometrics is a growing research area. The fusion of multiple biometric modalities can be performed in different ways for performance improvement in nonideal images. There are numerous studies in the biometrics literature for information fusion of iris texture with features in ocular and periocular structures such eye movement dynamics,^10,11 shape-based geometric features,¹² conjunctival vasculature structures,¹³ LBPs,¹⁴ and different filter models to extract features.^15,16 Since our paper is focused on iris, ocular, and periocular biometrics to improve the recognition performance for off-angle images, information fusion with other modalities is out of our scope in this paper.

The initial studies showed the feasibility of using periocular structures as a biometrics measure. After aligning the images using the iris center, Park et al.²⁵ extracted several global and local features from the periocular region using gradient orientation histogram, LBP, and scale-invariant feature transform (SIFT) descriptors. Scores were calculated using Euclidean distance for each descriptor, and they were fused at the score level using the weighted sum. Their results showed that including additional periocular structures in decision-making produced outcomes with higher accuracy. Its promising results attracted other researchers^26–29 to investigate ocular and periocular biometric modalities using various filters for feature extraction and matching fusion methods for their final scores. For example, Tan and Kumar³⁰ exploited SIFT, GIST, LBP, HoG, and LMF filters for feature extraction from the periocular region and combined their matching scores linearly with traditional iris recognition. Proenca³¹ proposed a score-level fusion of separate modalities for images captured in visible light with one exploring the iris texture and another extracting shape parameters from eyelids, eyelashes, and skin.

In addition to these feature extraction-based solutions, some deep learning-based approaches exist in the biometrics literature. The deep-PRWIS¹⁷ framework is proposed for periocular biometrics in visible spectrum images using an AlexNet-based network. For the visible spectrum images, they point out several degradation factors for ocular structures, including corneal reflections on iris, motion blur, and eyelid and eyelash occlusions on iris texture, and suggested disregarding ocular structures (i.e., iris and sclera) from the periocular input image using multiclass artificial sampling to get more accurate results. Zhang et al.³² proposed a deep feature fusion network to fuse the discriminative features of iris and periocular biometrics for a better identification performance on mobile devices. Zhao and Kumar³³ proposed semantics-assisted CNN in which several deep networks were trained separately to learn additional semantic information, such as gender and ethnicity from the periocular region, and their output features were combined to improve the recognition performance. These works on ocular and periocular structures show some degree of improvement in recognition performance for frontal images, but they do not include any results for off-angle images.

To address the performance degradation at off-angle images, our previous study³⁴ initially investigated AlexNet-based deep networks following the traditional and nontraditional iris recognition frameworks. We also presented the effects of gaze angle on iris recognition and the feasibility of information fusion for various off-angle images. This paper extends our initial work in four aspects: (i) including additional CNN architectures and comparing their performance, (ii) studying the network behavior for untrained gaze angles, (iii) investigating the effect of the gaze angle and number of sample images per angle from each subject in the training set, and (iv) comparing the recognition performance of CNN-based iris, ocular, and periocular biometrics with a traditional state-of-the-art iris recognition algorithm. As a result, this paper allows us to investigate the effect of ocular and periocular structures on iris recognition using different input types including cropped, masked, and normalized images.

3.

Methods

This section provides an overall structure of the adopted CNN-based deep learning frameworks. The CNN architecture extended the application areas of neural networks from one-dimensional signals to 2D images, and its popularity has boomed in various research communities. For iris recognition, CNN first extracts essential features from input images using acquired knowledge by convolutions in the training process. Then, it classifies the input into one of the classes defined in training without requiring segmentation, normalization, and encoding steps. The most critical part of a deep network is the training process, which requires thousands of images to adjust the network parameters. However, there are no publicly available biometrics datasets that have enough off-angle images for training. In addition to demanding a large dataset, training deep networks from scratch also requires extensive computational time and resources. Therefore, we adapted several pretrained networks and applied the transfer learning approach to retraining them for iris, ocular, and periocular recognition. Figure 3 shows the flowchart of the major steps in the proposed deep learning-based recognition methods.

Fig. 3

Flow chart of the proposed deep learning-based biometrics recognition method using transfer learning.

Since CNN is one of the extensively used deep learning architectures in image classification applications, we adopted the most common CNN architectures: AlexNet,³⁵ GoogLeNet,³⁶ and ResNet50.³⁷ These deep networks were trained with millions of images in the ImageNet dataset.³⁸ The original task of these networks is to classify an input image into thousands of different classes such as cars, animals, and tools with high accuracy. MATLAB constructed AlexNet, GoogLeNet, and ResNet50 architectures using 25, 144, and 177 layers, respectively.

The main steps of these networks are the feature extraction, starting from low-level features at shallow layers, gradually increasing the complexity at deeper layers, and basing the final decision on the last fully connected layer. Compared with others, AlexNet is the smallest network with five convolutional layers. To minimize the number of parameter spaces for low-level feature extraction, each convolutional layer is followed by activation and max-pooling layers. GoogLeNet is a deeper network than AlexNet in which nine inception modules are stacked upon each other. The block encapsulation enables the extraction of low-level and high-level spatial features using different sizes of filters. The adapted deepest network in this work is ResNet50. The skip connection strategy allows the features to be fed from the previous layers to the next layers without changing them. The computational complexity reduction in GoogLeNet and ResNet50 is achieved using $1\times 1$ convolutional layers, in whichthe number of inputs is reduced from one layer to another. This trick shrinks the size of the model and drops the number of parameters from 138 million to 4 million while allowing for an increased network depth with more layers.

Instead of training AlexNet, GoogLeNet, and ResNet50 architectures from scratch, we utilized the transfer learning approach to adapt their initial weights and parameters. Several different transfer learning approaches for adapting the weights from the base network exist. Yosinski et al.³⁹ proposed copying the weights of the initial layers of the base network as is and letting them update freely in the retraining process with a new dataset. Instead of copying the weights, Chaabouni et al.⁴⁰ proposed adopting the weight from the base network using a velocity function with weight decay and momentum coefficients for saliency map generation. Since the initial layers of CNNs extract similar low-level Gabor-based features from the datasets, we adapted the first approach to copy the initial weights from the base networks. As shown in Fig. 3, the transfer learning drops the last three layers of these networks and replaces them with new fully connected, softmax, and classification layers. The new fully connected layer extracts the final features from the previous layers and collects them to feed them into the softmax layer. The softmax layer calculates the likelihood function for an input belonging to a certain class. Since training images are captured from $n$ many subjects, the classification layer has $n$ classes. Based on the highest probability, the classification layer gives the final decision and selects the output class. After training, each test image is compared with $n$ classes for identity verification. Based on the probability value, a decision will be made for a match or nonmatch to the compared class. During the retraining process, the parameters and weights in each layer are updated for each image minibatch. The new network is retrained with the off-angle images using the stochastic gradient descent algorithm with several learning rates ranging from 0.0001 to 0.01 and a minibatch size of 4, 8, and 16 images. Its number of max epochs was set to 8, 16, and 32 based on the training dataset.

4.

Experimental Setup and Dataset

For our experiments, we used an off-angle biometric dataset that contains 5687 frontal and off-angle images from 52 different subjects. We used NIR-sensitive IDS-UI-3240ML-NIR cameras for data acquisition. The off-angle camera was attached to the moving arm to capture images from $-50\mathrm{deg}$ to $+50\mathrm{deg}$ in angle with a 10-deg step-size. Frontal iris images are captured at a 0-deg gaze angle while subjects look at the camera. For off-angle images, the camera is moved to the left for capturing positive angles and to the right for negative angles while subjects are still looking at a fixation point at 0 deg. Navitar Zoom 7000 lenses are attached to the cameras and are set at 40-mm focal length with a 5.6 f-stop. The periocular region is illuminated with a 780-nm power LED. To increase the image quality, we attached a 720-nm high-pass filter to the lens.

Figure 4 shows the data acquisition setup for off-angle biometrics. The off-angle camera captures 10 images at each gaze angle, and in total it provides 10 frontal and 100 off-angle images for each subject. Example images captured from each gaze angle are shown in Fig. 5. In an original frontal image, the iris diameter is around 410 pixels. For more details about the off-angle dataset, readers can look at our previous paper.⁴¹ Each image includes textures from the iris, sclera, eyelashes, eyelids, eyebrows, and skin. For dataset requests, readers can contact the author. The captured images using our data acquisition platform are at the NIR spectrum, so our experiments only include images at the NIR spectrum. There is no available off-angle image dataset captured at the visible spectrum following the same procedures from $-50\mathrm{deg}$ to $+50\mathrm{deg}$ in angle with a 10-deg step-size. Although existing datasets at the visible spectrum^42–44 have some off-angle images, they do not provide the ground truth gaze angles. Therefore, we could not include them in our study. However, we believe that this study can be extended for images at the visible spectrum.

Fig. 4

Experimental setup of off-angle biometrics data acquisition.

Fig. 5

Examples of off-angle images from different gaze angles as (a) $-50\mathrm{deg}$, (b) $-40\mathrm{deg}$, (c) $-30\mathrm{deg}$, (d) $-20\mathrm{deg}$, (e) $-10\mathrm{deg}$, (f) 0 deg, (g) $+10\mathrm{deg}$, (h) $+20\mathrm{deg}$, (i) $+30\mathrm{deg}$, (j) $+40\mathrm{deg}$, (k) $+50\mathrm{deg}$ in angle.

4.1.

Dataset Preparation

To investigate the recognition performance of CNNs for iris, ocular, and periocular biometric modalities, we designed several experiments with cropped, masked, and normalized images from the original off-angle images. Each CNN architecture is trained and tested with different types of biometric images that may include different ocular and periocular components. Although there are several interpretations for the ocular region, the definition of periocular is clear and common. Periocular is defined as the part of the facial region surrounding the eye including eyelids, lashes, and eyebrows. The term “periocular” comes from the prefix, “peri-,” meaning “around or about,” and the root word, “ocular,” meaning “of or relating to the eye.”^45,46 Although some papers describe the ocular regions bigger than periocular,⁴⁷ we use the general definition in which the periocular region surrounds the ocular region. Figure 6 shows the generated seven types of biometric images from each original off-angle image in our dataset for the experiments: (a) Periocular: periocular image with pupil, iris, sclera, eyelids, eyelash, and skin textures; (b) PeriocularIris: periocular image with only iris texture; (c) PeriocularNoIris: periocular image without iris; (d) Ocular: ocular image; (e) OcularIris: ocular image with only iris; (f) OcularNoIris: ocular image without iris; and (g) NormalizedIris: normalized iris image (Table 1).

Fig. 6

Input images for deep learning architectures: (a) periocular image, (b) periocular image with only iris, (c) periocular image without iris, (d) ocular image, (e) ocular image with only iris, (f) ocular image without iris, and (g) normalized iris image.

Table 1

Training dataset groups.

Name	Type	# of images	Image per angle	Angles
F	Frontal	520	10	0 deg
O	Off-angle	5167	10	ALL
O-s1	Off-angle	520	1	ALL
O-s2	Off-angle	1032	2	ALL
O-s3	Off-angle	1541	3	ALL
O-o3	Off-angle	1538	10	−30 deg, 0 deg, 30 deg
O-o4	Off-angle	2565	10	−40 deg, −20 deg, 0 deg, 20 deg, 40 deg
O-o5	Off-angle	2567	10	−50 deg, −30 deg, 0 deg, 30 deg, 50 deg

The original resolution of our off-angle biometric dataset is $1280\times 1024\text{pixels}$ with a single grayscale channel. However, AlexNet, GoogLeNet, and ResNet50 require a square-sized color image as an input. The input size of AlexNet is $227\times 227\times 3$. GoogLeNet and ResNet50 require $224\times 224\times 3$. Therefore, we cropped, resized, and masked the images. To generate masked and normalized images, iris and pupil boundaries were segmented as ellipses $(C_x,C_y,R_x,R_y,\theta )$ using an off-angle iris segmentation algorithm.⁴⁸ The segmentation results for the iris-pupil (inner) boundary are almost perfect for all images except a few cases. However, due to changes in eyelid occlusions and gaze angle differences, iris-sclera (outer) boundary results are not consistent at the frontal and off-angle images. Therefore, the segmentation parameters are checked using a manual ground-truth tool and corrected if needed to have error-free and consistent segmentation results. The ellipse equation for iris and pupil boundaries is defined as

Eq. (1)

$${\left(\frac{(\mathrm{x}-C_x)\cos \theta +(\mathrm{y}-C_y)\sin \theta }{R_x}\right)}^2+{\left(\frac{(\mathrm{x}-C_x)\sin \theta -(\mathrm{y}-C_y)\cos \theta }{R_y}\right)}^2=1,$$

where $(C_x,C_y)$ is the ellipse center, $(R_x,R_y)$ are the major and minor axes, and $\theta$ is the orientation angle.

Using the ellipse parameters, we first create elliptical disks for the pupil and iris regions where equality (=) in Eq. (1) is changed to less than ($<$). To exclude the inside of the ellipse, equality is changed to greater than ($>$). Then, we multiply these elliptical disks with Periocular and Ocular images to generate the masked image subsets including PeriocularIris, PeriocularNoIris, OcularIris, and OcularNoIris as shown in Fig. 6. To generate seven types of image subsets, we performed the following steps:

• (i) For periocular images in Figs. 6(a)–6(c), we first cropped 128 pixels from the left and right sides of each image.

• (ii) For ocular images in Figs. 6(d)–6(f), we cropped 454 pixels around the iris center.

• (iii) For PeriocularIris and OcularIris images in Figs. 6(b) and 6(e), we excluded the inside of the pupil and the outside of the iris by masking their elliptical disk of pupil and iris.

• (iv) For PeriocularNoIris and OcularNoIris images in Figs. 6(c) and 6(f), we excluded the inside of the iris-sclera by masking the elliptical disk of the iris.

• (v) For NormalizedIris in Fig. 6(g), we sampled the iris texture using elliptical normalization. We selected 227 pixels from 227 angles starting from 3 o-clock at CWD.

• (vi) For all images, we downsampled them into the required size.

• (vii) Finally, to convert the grayscale images to three-channel RGB images, we copied and pasted them into every channel.

4.3.

Baseline Iris Recognition Algorithm

To compare the recognition performance of CNN-based iris, ocular, and periocular recognition, we adopted a well-known traditional Gabor-based state-of-the-art iris recognition algorithm.¹ After elliptical iris segmentation and normalization, iris codes were generated using OSIRIS phase-quadrant demodulation⁵⁰ to calculate the HD. Every image in our off-angle dataset is compared with the others, and over 26.7 million comparisons were performed for the entire dataset. The intraclass histogram shows the comparisons between images from the same subjects. The comparisons between two different subjects are shown in the interclass histogram. Figure 7 shows the intraclass and interclass distributions of our off-angle datasets. The intraclass HD changes from 0.01 to 0.48 with an average of 0.2735 and a standard deviation of 0.0960. The interclass comparisons range from 0.35 to 0.59 with a mean value of 0.4764 and a standard deviation of 0.0225. We observed that the traditional recognition methods can measure the HD scores in the intraclass distribution better than interclass, which generates errors such as false matches and mismatches. The main reason for the performance degradation is the comparison of the images from different angles with them changing from $-50\mathrm{deg}$ to $+50\mathrm{deg}$ in our off-angle iris dataset.

Fig. 7

Histograms of HD of off-angle images. The left plot represents intraclass comparisons and the right one represents interclass comparisons.

To visualize the effect of the gaze angle differences on the traditional iris recognition, we calculated the HD scores of irises with different gaze angles from the same subject as shown in Fig. 8. For example, there is no gaze angle difference between images from $-50\mathrm{deg}$ to $-50\mathrm{deg}$, 0 deg to 0 deg, 50 deg to 50 deg, and so on. The result for no gaze angle difference is shown as a green solid line with an o marker. Its HD score changes from 0.01 to 0.30 due to the dilation differences and segmentation variations between images. This distance can be tolerated in traditional iris recognition systems since their data capture scenario is based on the comparison of frontal images. However, the HD score increases as the gaze angle difference between images increases, as shown in Fig. 8. For example, images at $-50\mathrm{deg}$ to $-40\mathrm{deg}$, 0 deg to 10 deg, and 40 deg to 50 deg have a 10-deg gaze angle difference. Their HD scores change from 0.09 to 0.38. Images at $-50\mathrm{deg}$ have a 100-deg gaze angle difference to images at 50 deg, and their HD is between 0.34 and 0.49. When the gaze difference between images is $>30\mathrm{deg}$, the HD score may fall into a false reject region in traditional systems. The main reason for this performance drop is the challenging issues in off-angle images including the corneal refraction, limbus occlusion, depth of field blur, and 3D structure of the iris texture.

Fig. 8

Histograms of intraclass HD distributions of off-angle iris images with different gaze angle differences. Each histogram represents comparisons of specific angle differences from 0 deg to 100 deg.

5.

Results and Discussion

In this section, we first evaluate the recognition performance of the adopted pretrained deep networks with the periocular images. Second, we compare the recognition results of the ResNet50 framework for off-angle images from Periocular, PeriocularIris, PeriocularNoIris, Ocular, OcularIris, OcularNoIris, and Normalized datasets when the network is trained with only frontal images. Third, we study the effect of the gaze angle difference on the iris, ocular, and periocular recognition and discuss the information fusion capability for CNN networks training with multiple off-angle images. Fourth, we investigate the network behavior for untrained gaze angles, the effect of the number of sample images per angle, and the effect of the number of subjects in the training set. Finally, the recognition performance of CNN-based frameworks is compared with a well-known Gabor-based traditional iris recognition algorithm.

Receiver operating characteristic (ROC) curve, equal error rate (EER), and area under the curve (AUC) are calculated as metrics for the performance analysis of biometric recognitions. ROC curves plot the true-positive rate (TPR) and the false-positive rate (FPR) for different thresholds at the final probability scores of the networks. AUC shows the 2D area under each ROC curve. EER gives the error rate in which false-positive and false-negative values are equal in an ROC curve. Curves close to the upper left corner, with lower EER scores, and with higher AUC values refer to higher overall accuracy for compared experiments.

5.1.

Comparison of AlexNet, GoogLeNet, and ResNet50

In our first set of experiments, we investigate the recognition performance of AlexNet, GoogLeNet, and ResNet50 architectures. We first retrained these off-the-shelf deep networks with periocular images in the frontal subset (F) and tested with their off-angle images (O-ALL). Then, we repeat the same experiments with periocular images in the off-angle training subset (O-s1). ROC curves, shown in Fig. 9, evaluate the recognition performance of these experiments. It is first observed that the deep networks showed better performance for training on off-angle images (O-s1) compared with frontal image training (F). When deep networks are trained with only frontal images (F) and tested with off-angle images, GoogLeNet showed the lowest recognition performance with an EER of 15.12%, compared with AlexNet and ResNet50 with EER scores of 9.93% and 7.24%, respectively. However, GoogLeNet performed better than AlexNet for training with off-angle images (O-s1). Even if AlexNet showed better performance for frontal images, its performance could not catch up to the deeper networks for off-angle training. Since AlexNet is the shallowest network, its depth may not be good enough to learn all of the required information from multiple off-angle images. In addition, ResNet50 showed the best performance for both frontal and off-angle image subsets since it is the deepest network with more layers compared with AlexNet and GoogLeNet. Table 2 summarizes the EER and AUC scores calculated in our frontal and off-angle biometrics experiments for AlexNet, GoogLeNet, and ResNet50. Since ResNet50 showed better performances compared with others and proved its effectiveness and efficiency with frontal and off-angle images, it was used for the rest of our experiments.

Fig. 9

Performance analysis using ROC for original images. AlexNet, GoogLeNet, and ResNet50 are trained with frontal and off-angle subsets (F and O-s1) and tested with off-angle image subset (O-ALL).

Table 2

Comparison of networks for frontal and off-angle trainings using EER and AUC.

Pretrained networks	Frontal training (F versus O-ALL)		Off-angle training (O-s1 versus O-ALL)
	AUC	EER (%)	AUC	EER (%)
AlexNet	0.9758	9.93	0.9998	0.181
GoogLeNet	0.9382	15.12	0.9999	0.032
ResNet50	0.9864	7.24	0.9999	0.023

5.2.

Results for Frontal Training Images

In our second set of experiments, we first train the ResNet50 framework with seven different types of frontal (F) periocular, ocular, and iris image datasets: (1) Periocular, (2) PeriocularIris, (3) PeriocularNoIris, (4) Ocular, (5) OcularIris, (6) OcularNoIris, and (7) NormalizedIris. After training, we test each network with their corresponding off-angle image subsets (O-ALL) to compare their recognition performance and to investigate the effect of the existence of iris, ocular, and periocular structures on off-angle biometrics recognition, including the pupil, iris, sclera, eyelashes, eyelids, and skin. Note that, in each experiment, training and testing images come from one of seven biometric datasets.

Figure 10(a) shows the recognition performance analysis using ROC curves for training with frontal images from seven periocular, ocular, and iris image datasets. Since only frontal image subset (F) is used for network training, the recognition performance for off-angle images is lower than the desirable level ($\mathrm{TPR}>0.99$ and $\mathrm{FPR}<0.01$). The main reason for low accuracy is the gaze angle difference between the training and testing images. Table 3 shows the EERs of these seven experiments. Based on the EER and ROC results, ocular and iris images (Ocular, OcularIris, and NormalizedIris) deliver very similar results, with their EERs being 8.25%, 7.97%, and 8.92%, respectively. We observed that segmentation and normalization of iris texture do not help for recognition for training with frontal images. Compared with ocular results, Periocular shows slightly lower performance with a 9.61% EER score; here, having additional periocular structures in the frontal training images did not offer additional discriminative features. In addition, additional experiments were conducted with the periocular image with only iris texture (PeriocularIris) and Periocular image without iris (PeriocularNoIris) datasets. PeriocularIris shows better performance than PeriocularNoIris, and it is very close to Periocular images with an EER score of 10.68%. It is seen that removing the iris texture from the periocular image does not change the accuracy too much because the iris area covers $<10\%$ of the periocular image. We also observed that the performance of PeriocularIris is lower than other OcularIris due to the lower resolution of the iris texture in the input image. Finally, OcularNoIris showed the worst accuracy with 17.78% in EER; here, iris texture is masked out from the ocular image. Based on the experimental results and our observations, we can conclude that the iris texture information provides more distinctive features when only frontal images are available for deep network training. Iris is more reliable biometrics than ocular and periocular structures because sclera, eyelids, and skin may appear more different than iris texture if there exists a gaze angle difference between training and testing images. In addition, the iris texture resolution is directly proportional to accuracy in the masked images.

Fig. 10

Performance analysis using ROC curves for periocular, ocular, and iris images: periocular image (Periocular), periocular image with only iris (PeriocularIris), periocular image without iris (PeriocularNoIris), ocular image (Ocular), ocular image with only iris (OcularIris), ocular image without iris (OcularNoIris), and normalized iris images (NormalizedIris). (a) CNNs are trained with frontal image subsets (F) and tested with off-angle image subsets (O-ALL), (b) CNNs are trained with off-angle image subset (O-s1) and tested with off-angle image subset (O-ALL).

Table 3

EER of different image subsets for frontal and off-angle trainings.

Image subsets	Frontal training (F versus O-ALL) (%)	Off-angle training (O-s1 versus O-ALL) (%)
Periocular	9.61	0.012
PeriocularIris	13.54	0.120
PeriocularNoIris	10.68	0.016
Ocular	8.25	0.043
Ocular Iris	7.97	0.053
OcularNoIris	17.78	0.287
NormalizedIris	8.92	0.091

For further investigation on the effect of the gaze angle difference on biometrics recognition, Fig. 11 shows more in-depth results for images in the ocular dataset (Ocular). First, CNNs are trained with frontal ocular image subset (F). Then, we tested the images from the ocular off-angle subset (O-ALL) at different gaze angles ranging from $-50\mathrm{deg}$ to 50 deg, separately. Since the network is trained with only frontal images, its recognition performance drastically decreases as the gaze angle of the test images increases. The EER scores for each gaze angle ranging from 50 deg to 50 deg in angle with a 10-deg step-size are 16.56%, 9.22%, 5.35%, 3.09%, 0.42%, 0.01%, 0.78%, 3.01%, 6.76%, 13.96%, and 22.91%, respectively. We also observe that results for positive angles are not symmetrical to negative angles because of the anatomical 5-deg angle difference between the visual axis and the optical axis, aka “angle alpha.” For example, the result for gaze 50 deg, marked with the dotted purple ROC curve, is worse than images from gaze $-50\mathrm{deg}$, marked with a blue curve.

Fig. 11

Comparison of recognition performances for ocular images (Ocular) with different gaze angles. CNNs are trained with frontal ocular images (F) and tested with ocular off-angle images from each gaze angle ranging from $-50\mathrm{deg}$ to 50 deg.

5.4.

Training with Limited Angles, Multiple Samples, and More Subjects

In our fourth set of experiments, we investigate the network behavior for untrained gaze angles with ocular images (Ocular). We trained the network with images in the Ocular dataset from each frontal and limited off-angles image subset, F, O-o3, O-o4, and O-o5, separately and tested with the rest of the off-angle images (O-ALL). Figure 12(a) presents ROC curves for these experiments, and their EERs are shown in Table 4. It is observed that, when the network is trained with only the frontal images subset, F, it shows the worst performance results, as marked with a solid red line with 8.25% EER. When additional off-angle images from $-30\mathrm{deg}$ and 30 deg gaze angles are included in the training set (O-o3), the network performance is improved, as shown in a solid blue line with 0.40% EER. The best network performance is shown for training with the O-o5 subset (images from $-50\mathrm{deg}$, $-30\mathrm{deg}$, 0 deg, 30 deg, and 50 deg) with 0.04% EER. Deep learning networks can learn from off-angle images even if their training does not include the images from every angle. We also observe that the results for O-o4 are not as good as the results of O-o5 (EER: 0.22%) because it only includes images from $-40\mathrm{deg}$, $-20\mathrm{deg}$, 0 deg, 20 deg, and 40 deg gaze angles. Since the images with $-50\mathrm{deg}$ gaze angle are the most distorted and are not included in the training, the network performance for O-o4 is lower than the experiment with O-o5. To improve the recognition performance for off-angle images, it is enough to train the network with steep angles, and it is not required to train it with every gaze angle because CNN can handle small variations in the gaze angles.

Fig. 12

Performance analysis using ROC for ocular image (Ocular). (a) CNNs are trained with frontal and limited off-angles image subsets (F, O-o3, O-o4, and O-o5) and tested with off-angle image subset (O-ALL), (b) CNNs are trained with multiple sample off-angle image subsets O-s1, O-s2, and O-s3 and tested with off-angle (O-ALL).

Table 4

EER of deep learning frameworks and traditional iris recognition.

Training image subsets	Deep learning framework (%)	Traditional iris recognition (%)51
Frontal	8.25	1.59
Off-angle (O-o3)	0.40	0.41
Off-angle (O-o4)	0.22	0.34
Off-angle (O-o5)	0.04	0.14

We also studied the effect of the number of sample images per angle from each subject in the training set. We used the images in the ocular dataset (Ocular) from multiple sample image subsets O-s1, O-s2, and O-s3, with 1, 2, and 3 sample images per angle from each subject being included in the training set, respectively. As shown in the ROC curves in Fig. 12(b), the performance was improved as the number of sample images per angle from each subject was increased. In addition, EERs decreased to 0.043%, 0.024%, and 0.015% for subsets O-s1, O-s2, and O-s3, respectively. Even if we included off-angle images from every gaze angle in the training set, there might be a difference in pupil dilations. Therefore, including more samples per angle from each subject in the network training helps to improve the performance of deep learning-based iris recognition algorithms.

To investigate the effect of the number of subjects in the dataset on the recognition performance of CNNs, we perform additional experiments using images in the ocular dataset (Ocular) from frontal (F) and off-angle (O-s1) subsets. We train our networks with images from various numbers of subjects including 50, 75, and 90 subjects. Then, we test the networks with the rest of the images in their off-angle subset (O-ALL) to show the network behavior for a larger-scale evaluation. Figure 13 shows the ROC curves for the performance evaluations of training the network with frontal (F) and off-angle (O-s1) subsets, respectively. Although ROC curves showed that the recognition performance of the deep networks slightly changes for different numbers of subjects in the dataset, there are no significant and consistent performance changes based on the number of subjects. For frontal training of the network, EERs were 8.25%, 9.19%, and 9.02% for 50, 75, and 90 subjects, respectively. In off-angle training, EERs decreased to 0.047%, 0.039%, and 0.035% for 50, 75, and 90 subjects, respectively. Therefore, the proposed method can be applied to larger datasets with more subjects without any significant decrement in the recognition performance.

Fig. 13

Performance analysis using ROC for ocular image (Ocular) for different number of subjects. (a) CNNs are trained with frontal image subset (F) and tested with off-angle image subset (O-ALL), (b) CNNs are trained with off-angle image subsets O-s1 and tested with off-angle (O-ALL).

5.5.

Comparison with State-of-the-Art Algorithm

Last, we present the recognition performance of the Gabor-based traditional iris recognition algorithm as a state-of-the-art algorithm to compare with our proposed deep learning frameworks. Traditional recognition systems store only one frontal iris image for each subject in its image gallery and compare them with incoming image probes. Therefore, we first included the iris codes from the frontal image subsets in our gallery and tested with off-angle (O-ALL) images. Then, we implemented the gallery approach⁵¹ in which multiple off-angle images of the same subject are included in the gallery, and the verification decision is made based on the minimum HD of probe and gallery images. We used O-o3, O-o4, and O-o5 subsets as the gallery and tested with the rest of the off-angle (O-ALL) images. Figure 14 presents ROC curves, with the frontal image gallery, F, giving the worst performance compared with the other gallery approaches. As shown in Table 4, the EER scores are 1.59%, 0.41%, 0.34%, and 0.14% for subsets F, O-o3, O-o4, and O-o5, respectively. As expected, while the number of multiple off-angle images increases in the gallery, the recognition performance improves. However, multiple images per subject in the gallery multiplies the number of comparisons required for recognition, which causes a longer decision time for traditional iris recognition algorithms.

Fig. 14

Performance analysis using ROC for traditional iris recognition with frontal (F) and multiple off-angle image subsets (O-o3, O-o4, and O-o5) included in gallery and tested with off-angle (O-ALL) images.

Compared with deep learning frameworks, the accuracy of the traditional method is only better for frontal images. Even if we include multiple off-angle images in the gallery, the traditional method could not reach the performance of deep learning frameworks for off-angle iris images. In addition, the traditional method requires iris segmentation with eyelid detection that is an error-prone process, and faulty segmentation may drop the accuracy. Eyelid detection also helps to mask the occluded iris texture in the traditional method. Therefore, deep learning frameworks show superior performance for the off-angle images compared with the traditional iris recognition methods in which they also do not need iris segmentation and eyelid detection.

6.

Conclusions

This study compared CNN-based AlexNet, GoogLeNet, and ResNet50 architectures to improve the recognition performance of iris, ocular, and periocular biometric modalities for off-angle images. It also investigated the effect of the gaze angle difference between training and testing images, the network behavior for untrained gaze angles, and the information fusion capability of CNN networks at multiple off-angle images.

Based on our experimental results, the important conclusions are as follows: first, ResNet50 shows the best performance for both frontal and off-angle training scenarios compared with AlexNet and GoogLeNet since it is the deepest network with more layers. Second, the network accuracy significantly decreases as the difference between gaze angles of training and test images increases. Third, even though iris texture provides more reliable information at frontal training; iris, ocular, and periocular structures can be used together for improving the recognition performance at off-angle images. Fourth, CNN-based deep learning frameworks showed data fusion capability in which they can easily fuse the information from multiple off-angle images. Since CNN can handle small gaze angle variations, training with steep angles is enough, and there is no need for every angle. As the main conclusion, compared with the Gabor-based traditional iris recognition algorithm, CNN-based deep learning architectures improve the recognition performance for off-angle images when the network is trained with multiple off-angle images.