3.1

Multi-scale feature extraction

Different levels of the convolutional neural network will produce different spatial resolution feature maps, and the feature maps obtained through different convolution layers contain different information. High-level features focus more on semantic information and less on image details., while the low-level features may contain more details and chaotic background information. Therefore, most researchers combine the features of multiple scales to complement the features of different levels in this simple and effective way. The pedestrian re-identification network structure designed in our paper removes the last full connection layer of the ResNet50 network, plus the average pooling layer and the max pooling layer, as shown in Figure 1. Avg (m) and Max (m) represent average pooling and max pooling respectively, and a feature mAP with width and height of m is obtained. In this paper, the output features of layer 3 in the ResNet50 network are global average pooling and global max pooling respectively, and Pavg1 and Pmax1 feature maps with output dimensions of 1×1×1024 are obtained. Similarly, the output features of layer4 in the ResNet50 network are global average and global max respectively, and Pavg2 and Pmax2 feature maps with output dimensions of 1×1×2048 are obtained. To reduce the impact of pooling on information loss, the stripes of average pooling and max pooling are adjusted to obtain richer feature information. Pavg3 and Pmax3 feature maps with output dimensions of 2×2×2048 are obtained. The multi-scale features Pavg1, Pmax1, Pavg2, Pmax2, Pavg3, and Pmax3 of the pedestrian images are sent to the residual pooling module to obtain the corresponding residual features Pcont1, Pcont2, and Pcont3, which are transformed into a unified dimension for fusion, the fused features are sent to the classifier for classification, and finally, the pedestrian re-identification results are obtained.

3.2

Residual pooling module

Figure 2 shows the average pooling and max pooling commonly used in convolutional neural networks. Average pooling is to average the features in the neighborhood, and max pooling is to maximize the features in the neighborhood. Although the average pooling features can transfer the global information of the image more completely, their calculation methods are easily affected by background clutter and occlusion, and cannot highlight the difference between pedestrians and the background. Compared with the average pooling, the max pooling can reduce the impact of background clutter, but the max pooling pays more attention to extracting the local salient features of pedestrian images and the contour information of pedestrians. The pooling features cannot completely contain the whole body information of pedestrians. In the actual recognition environment, due to the change of camera angle and external illumination, it is necessary to remove the influence of background clutter while preserving the whole body information of pedestrians and highlighting the difference between pedestrians and background. On this basis, this paper proposes a residual pooling module, as shown in Figure 3. By combining the advantages of max pooling and average pooling, this module can make up for the shortcomings of average pooling and max pooling, and on the basis of preserving the whole body information, highlight the pedestrian contour and focus on the difference between the pedestrian and the background, to make the final feature expression of pedestrian images more comprehensive and discriminatory and improve the accuracy of pedestrian re-identification.

Figure 2.

Schematic diagram of pooling layer.

Figure 3.

Structure of residual pooling module.

As can be seen from Figure 3, the residual pooling module subtracts the multi-scale feature Pavg of a pedestrian image extracted by Resnet50 from Pmax and uses a convolutional kernel of 1×1 to obtain the difference between Pavg and Pmax. The Pmax feature obtained through max pooling also uses a convolutional kernel of 1×1, and adds the difference feature between Pavg and Pmax to obtain the residual feature Pcont. The residual pooling module integrates the merits of max pooling and average pooling. The obtained residual feature Pcont not only covers the whole body of pedestrians and deepens the contour of pedestrians, but also reduces the impact of background clutter, and pays more attention to the difference between pedestrians and background. The residual feature Pcont is calculated as shown in equation (1):

where: Pavg and Pmax are the features obtained after average pooling and max pooling respectively; δ_1×1(x) are used for 1×1.

3.3.

Loss Function

We use the joint optimization model of triple margin loss and cross-entropy loss to train the model. The loss function is shown in equation (2):

where: L_{total_loss} is a total loss. is triple loss. is the cross-entropy loss. Triple loss makes the distance between positive sample pairs shorter and the distance between positive and negative samples larger. Cross entropy loss focuses on the closeness between the actual output and the expected output, where, i∈[1,6] in , i represents the ith feature among the six basic pedestrian image features Pavg1, Pmax1, Pavg2, Pmax2, Pavg3, and Pmax3 extracted after passing through layer3 and layer4 of the ResNet50 network. j represents the jth feature among the three comparison features Pcont1, Pcont2 and Pcont3 extracted through the residual pooling module. The loss function designed in this paper adopts the joint optimization model of triple loss and cross-entropy loss. The convergence of the model is accelerated by calculating multiple losses.

4.1

Experimental dataset

This paper’s comparative laboratory is based on the market1501¹⁶ and Duke MTMC Reid¹⁷ datasets.

Market1501¹⁶ includs 1501 pedestrians captured by 6 cameras. Training set: 12936 images of 751 pedestrians. test set: 19732 images of an additional 750 pedestrians. Test set includes: query set and gallery set. MTMC-ReID¹⁷ dataset includes 36411 images of 1404 pedestrians captured by 8 cameras. Training set:16522 images of 702 pedestrians. Test set: 17661 images of an additional 702 pedestrians. Test set includes: query set and gallery set.

4.2

Experimental preparation

The algorithm is implemented on PyTorch framework. We are experimenting with a computing platform with a GPU model NVIDIA GTX1080Ti. CPU model Intel® Core™ i7-7700k @ 4.20 GHz, the memory is 32GB. When training the model, the resolution of the input pedestrian image is set to 288×144 pixels, 32 training batches, and 220 iterations in total. We use SGD optimizer, set the learning rate to 0.03 and the weight decay rate 0.0005. When iteration ordinal number gradually increases to 0.003, it then drops to 0.003, 0.0003, and 0.0003 at 40, 110, and 150 iterations, respectively.

4.3

Experimental evaluation criteria

In this experiment, Rank-1, Rank-5, Rank-10 and mean Average Precision (mAP) in the cumulative matching features (CMC) curve are used as evaluation indicators. During the test, take a query image from the query set and measure the similarity between all the images in the test set and the query image. CMC refers to the probability of successful matching with the query image in the first K candidate images. The values of Rank-1, Rank-5, and Rank-10 are the corresponding accuracy when K=1, 5, and 10 in CMC (K). the mAP is the average of the areas under the accuracy recall curve for all samples.

4.4

Analysis of experimental results

To validate the proposed method, a comparative experiment is carried out on Market1501¹⁶ and DukeMTMC-reID¹⁷ datasets. Figure 4 shows the re-identification results. The first column of each row is the query image, the last 10 columns are the top 10 query results. In Figure 4, the solid line border represents the correct query results, and the dotted line border represents the wrong query results.

Figure 4.

Person re-identification results of the proposed method.

The AVG method is to pool the basic features of images by using our pedestrian re-identification network structure to obtain the feature maps. Max method is to maximize the pool and get the feature graph. Figure 5 shows the comparison of accuracy and loss value between avg method, max method, and residual pool module (residual) method on the Market1501¹⁶ dataset. The accuracy of our method is higher than avg method and max method, and the loss value decreases faster. It is verified that the residual pooling feature obtained based on the residual pooling module can reach better performance in pedestrian re-identification tasks.

Figure 5.

Accuracy and loss values comparison among different methods.

In this paper, Grad-CAM avg is used to visualize avg, max method, and residual methods. Max method and residual method are visualized by using the Grad-CAM class activation thermodynamic diagram. Figure 6 shows that the AVG method performs well for the acquisition of whole body information, but it is also easily affected by the background clutter; the max method focuses on to the local pedestrian contour but does not include the whole pedestrian body. Our method combines the advantages of both methods, which can include the whole body of pedestrians and reduce the impact of background clutter.

Figure 6.

Visualization results of different methods. (a): input images; (b): avg method; (c): max method; (d): residual method.

Table 2 shows the comparison of several mainstream methods’ indicators on the Market150¹⁶ and DukeMTMC-reID¹⁷ datasets. Our residual +re ranking method optimizes the network performance by re-ranking¹⁸ technology, ResNet50_baseline method directly utilizes the features of layer 4 output of the ResNet50 network to optimize the network performance. The mAP of the residual +re-ranking methodon Market1501 and DukeMTMC-reID datasets is 94.52% and 89.30% respectively. This method’s indicators can be significantly improved with resnet50 as the backbone network.

Table 2.

Performance indicators comparison 1 among different methods (%).

Table 3 shows the indicators of Residual, Residual +re-ranking and the representative methods (SVDNet, GLAD⁸, PCB⁹ PCB+RPP⁹ BEF, etc.), The indicators of the comparison methods are all quoted from the original text, where “—” means that there is no such experimental result in the original text. At the same time, the indicators after using re-ranking technology to optimize our method are given. Rank1 and mAP of Residual +re-ranking method are better than the current advanced methods, especially in the mAP indicators. On the Market1501¹⁶ dataset, after reordering, the proposed method (residual + re-ranking) is 2.61% higher than the PCB+RPP method based on local features in Rank1 and about 13% higher in mAP. Compared with the BEF method, Rank1 and mAP are increased by 1.1% and 7.8%, respectively; Compared with the DG-Net method, Rank1 and mAP are increased by 1.61% and 8.52%, respectively; It is 2.2% higher than that of CtF method in Rank1 and 9.62% points higher in mAP. On the DukeMTMC-reID¹⁷ dataset, compared with the BEF, the Rank1 of this method (residual) is lower, but the mAP is increased by 1.7%. After reordering, the Rank1 and mAP of Residual +re-ranking are higher than the representative methods.

Table 3.

Performance indicators comparison 1 among different methods (%).

Table 4 shows the comparison of parameter quantity, calculation quantity, and inference time between this method and the comparison method. Compared with most representative methods, the method in Residual has more model Parameters, but the method in Residual has less computation (FLOPs) and consumes less time for reasoning in a single image.

Table 4.

Parameters, calculation, and inference time comparison among different methods.