Deep capsule networks have more capsule layers, which makes their performance better on complex images. However, with the increase of layers, overfitting will become more serious. Image reconstruction is an effective regularization method for capsule networks. To improve it, we propose an adversarial decoder that introduces the generative adversarial network framework into the reconstruction process to implement learnable reconstruction losses. This architecture consists of three parts: a deep capsule network, a decoder, and a discriminator. The deep capsule network extracts feature capsules from input images, which are then reconstructed by the decoder. The discriminator is the learnable reconstruction loss function that evaluates the similarity between reconstructed images and input images. Minimizing this learnable reconstruction loss and mean square error of images provides a regularization effect for the deep capsule network. Experimental results show that our models have a competitive performance of regularization on CIFAR10, CIFAR100, and FMNIST.

Estimating three-dimensional (3D) human poses from the positions of two-dimensional (2D) joints has shown promising results. However, using 2D joint coordinates as input loses more information than image-based approaches and results in ambiguity. To overcome this problem, we combine bone length and camera parameters with 2D joint coordinates for input. This combination is more discriminative than the 2D joint coordinates in that it can improve the accuracy of the model’s prediction depth and alleviate the ambiguity that comes from projecting 3D coordinates into 2D space. Furthermore, we introduce direction constraints, which can better measure the difference between the ground truth and the output of the proposed model. The experimental results on the Human3.6M show that the method performed better than other state-of-the-art 3D human pose estimation approaches. The code is available at: https://github.com/XTU-PR-LAB/ExtraPose/.

Guest editors Ofer Hadar and Touradj Ebrahimi introduce the Special Section on Image and Video Compression Using Deep Neural Networks.

Compressed sensing (CS) is a signal processing framework for effectively reconstructing signal from a small number of measurements obtained by linear projections of the signal. It is an ongoing challenge for the real-time image reconstruction of the computational imaging, including single pixel imaging based on CS. We built a block-based CS (BCS) image reconstruction framework via a deep learning network with smoothed projected Landweber (SPL). A fully connected network performs both BCS linear sensing and non-linear reconstruction stages, and SPL removes the blocking artifacts due to incorporate Wiener filtering into projected Landweber (PL) method at each iteration. The sensing matrix and nonlinear prediction operator are jointly optimized, and the smoothing filtering is coalesced into the PL framework for eliminating high-frequency oscillatory blocking artifact. Experimental results reveal that the optimized scheme outperforms the approach only based on deep neural network. The reconstruction quality can be improved while being only slightly slower, especially the gain of structural similarity is significantly better than peak signal-to-noise ratio, and the reconstruction image texture details are vivid and natural. At 10% sensing rate, the structural similarity maximum (minimum) gain reaches 0.098 (0.021). The proposed approach is not only far superior to other state-of-the-art CS algorithms in terms of reconstruction time and quality but also comparable with up-to-date deep learning methods.

Hyperspectral images (HSIs) must be preprocessed by a compression model, which reduces the pressure of storing and transmitting huge data in applications. Whereas most of the existing methods consider only the compression or reconstruction requirements, an end-to-end optimization would simultaneously improve the performance of both requirements. We propose a three-dimensional convolutional autoencoder that precisely achieves end-to-end joint spectral–spatial compression and reconstruction of HSIs. In an experimental evaluation, the proposed method improved the spectral angle mapper, peak signal-to-noise ratio, and structural similarity index measurement of the reconstructed HSIs by 20.8% to 33.1%, 0.9% to 11.5%, and 0.5% to 2.2%, respectively, relative to competitive methods.

The recent convolutional neural network based studies on the compression artifact reduction (CAR) task have made great progress. However, most of these CAR methods still have some inadequacies. They are limited on the network capability due to treating extracted features equally and generate unpleasant visual results due to using the pixel-wise loss (e.g., L1/L2 loss) in training. Therefore, to address these issues, we propose a progressive multi-scale attention network (PMANet) for image CAR task and further introduce a PMANet-based generative adversarial network (PMAGAN) for visual quality improvement. Specifically, the key idea and the basic component of the PMANet is a multi-scale attention dense block, which effectively incorporates the multi-scale information to the model with the attention mechanism and thus enhances the network’s representation ability. The PMANet can be further improved with the designed progressive restoration structure. In addition, PMAGAN takes the PMANet as the generator and brings a generative adversarial networks framework with the adversarial training strategy. Experiments show that PMANet performs better than the state-of-the-art CAR methods, and PMAGAN can further achieve better visual quality with more natural and sharper textures.

The advanced three-dimensional extension of high-efficiency video coding (3D-HEVC) is the latest coding standard for 3D video. The coding of the depth map for 3D-HEVC is very time-consuming. With the development of deep learning, it has become feasible to employ convolutional neural networks (CNNs) to predict the coding unit (CU) division of the depth map. However, there are three types of CU sizes: 64, 32, and 16, which makes it difficult to unify the model. The features of the depth map are very different from the texture map. In view of the aforementioned problems, we propose an adaptive CU size CNNs for fast 3D-HEVC depth map intracoding. We first employ spatial pyramid pooling to fully extract the features of the three types of CUs. Then, we apply the nonlocal self-attention mechanism to make it suitable for depth maps. Compared with the 3D-HEVC reference algorithm, the proposed network reduces the coding time by an average of 35.7%, while the quality degradation of the synthesized virtual view is negligible.

Hyperspectral (HS) images hold both spatial and spectral information of an imaged scene. This allows one to take advantage of the distinct spectral signatures of materials to perform classification tasks. Since HS data are also typically very large and redundant, it is appealing to utilize compressive sensing (CS) techniques for HS acquisition. CS avoids the need for postacquisition digital compression, as the compression is inherently performed electrooptically during acquisition. We research the performance of deep learning classification applied directly on the compressive measurements. We show that by using a spectral CS technique we previously developed, it is possible to reduce the captured data by an order of magnitude without significant loss in the classification performance.

The stationarity of a texture can be considered a fundamental property of images, although the property of stationarity is difficult to define precisely. We propose a stationarity test based on multiscale, locally stationary, 2D wavelets. Three separate experiments were performed to evaluate the capabilities and the limitations of this test. The experiments comprised a chessboard stationarity analysis, two classification tasks, and a psychophysical experiment. The classification tasks were performed on 110 texture images from a texture database. In one subtask, five texture feature vectors were extracted from each image and the classification accuracy of two classical methods compared, whereas in the second subtask, the classification accuracy of several methods was compared to the descriptors defined for each image within the database. In the psychophysical experiment, the correlation between the classification results and observer judgements of texture similarity were determined. It was found that a combination of wavelet shrinkage and rotation-invariant local binary pattern best predicted the observer response. The results show that the proposed stationarity test is able to provide relevant information for texture analysis.

The detection of the optic disk (OD) and fovea is crucial to the automatic diagnosis based on fundus images. This task is very challenging, especially when varieties of lesions exist. Traditional handcrafted feature-based methods are inaccurate, and deep learning based methods fail easily in abnormal cases. We propose a framework that simultaneously detects the OD and fovea based on deep convolutional neural networks. The original image is first preprocessed and then followed by pseudo label generation. These labels are then fed into a fully convolutional neural network with residual modules for localization of the OD and fovea. Polar transformation is then introduced to the segmentation of the OD. The proposed algorithm achieves a relatively high success rate for OD localization and a 100% success rate for fovea localization on several public datasets. For the segmentation of the OD, the proposed algorithm achieves a low overlapping error on several public datasets. Compared with previous work, the proposed method achieves promising accuracy and robustness, and it is useful for practical applications since it detects the OD and fovea simultaneously and completely.

Infrared and visible image fusion is a hot research direction in the fields of computer vision and image processing, and it is a common multimodal image fusion. An effective image fusion algorithm via convolutional sparse representation (CSR) and guided filtering is proposed for fusing infrared and visible images in this paper. First, a series of dictionary filters are trained by the CSR strategy, and the smooth image component and detailed image component are obtained by classifying those filters into high-pass filters and low-pass filters. Then two rules are designed to fuse the smooth image component and detailed image component, respectively. For the detailed image component, a weight construction method based on guided filtering is designed to get the weight maps, and the smooth image component is fused by applying the “choose-max” strategy to the corresponding sparse coefficients. Finally, the fused image is obtained by combining the fused smooth image component and detailed image component. Experimental results show that the proposed algorithm achieves good fusion results and exhibits advantages over comparison image fusion methods.

The automatic detection of defects is an essential part of the printed circuit board (PCB) production process. In recent years, while great progress has been made in the detection of PCB defects, there are still various problems in traditional defect detection methods, for example, over-reliance on the perfect template, difficult to achieve precise image registration, and highly vulnerable to environmental factors such as light, noise, and reflectivity. We propose a fast defect detection network. On one hand, this algorithm solved the problems of traditional methods. On the other hand, this algorithm solved the problems of large model size and poor real-time of existing deep learning methods. First of all, the k-means clustering algorithm is used to obtain more reasonable anchors boxes; second, an improved MobileNetV2 is used as the backbone network; after the feature extraction network, the spatial pyramid pooling (SPP) structure is introduced to increase the receptive field of the image; then, we use complete intersection over union to optimize the loss function; finally, we build an enhanced feature extraction network based on the feature pyramid network for multi-scale feature fusion. The experimental results show that this method has small model size, good real-time, and good portability, which is suitable for practical production.

To fully develop the complementary advantages of different visual features and to improve the robustness of multi-feature fusions, we propose a robust correlation filter tracker with adaptive multi-complementary features fusion based on game theory. By combining the complementary features selected from handcrafted features and convolution features, our method constructs two robust combined features in the tracking framework of discriminative correlation filters (DCFs). In addition, by utilizing game theory, the two combined features are regarded as two sides of the game, achieving the best balance through continuous gaming throughout the tracking process and thus obtaining a more robust fused feature. The experimental results obtained on the OTB2015 benchmark dataset demonstrate that our tracker improves the robustness of object tracking in complex scenarios, such as occlusion and deformation, and performs favorably against eight state-of-the-art methods.

Scene text recognition (STR) is a challenging computer vision task. Recent progress has been made on developing a complex network to increase recognition accuracy. Most STR algorithms focus on improving the network structure and correcting slanted text to improve the accuracy of text recognition. Inspired by the concept of curriculum learning, we applied this method to the field of text recognition. We propose an easy-to-implement method that improves the accuracy of text recognition using the concept of curriculum learning. Taking into account the specific characteristics of text, we propose defining the difficulty of scene images from both the human perspective and the machine perspective. The key idea of the proposed method is to guide the training process to begin with training simple samples and progressively increase the complexity of the training samples. Experimental results demonstrate that the proposed method effectively accelerates the convergence and improves the accuracy of text recognition.

When deep convolutional neural networks perform feature extraction, the features computed at each layer express different abstractions of visual information. The earlier layers extract highly compact low-level features such as bandpass and directional primitives, whereas deeper layers extract structural features of increasing abstraction, similar to contours, shapes, and edges, becoming less effable as the depth increases. We propose a different kind of end-to-end no-reference (NR) image quality assessment (IQA) model, which is defined as a multi-depth output convolutional neural network (MoNET). It accomplishes this by mapping both shallow and deep features to perceived quality. MoNET delivers three outputs that express shallow (lower-level) and deep (high-level) features, and maps them to subjective quality scores. The multiple outputs are combined into a single, final quality score. MoNET does this by combining the responses of three learning machines, so it may be viewed as a form of ensemble learning. The experimental results on three public image quality databases show that our proposed model achieves better performance than other state-of-the-art NR IQA algorithms.

Suppose the current three-dimensional point cloud splicing method is well applicable to point clouds with insufficient features, it will benefit the noncontact surface quality detection and reverse design of objects with insufficient surface features. However, unfortunately, because the point cloud features are not rich, the current method is not applicable. We propose a point cloud splicing method based on spatial reference objects to solve the problem. Also, we describe the data processing method in detail. We first preprocess the point cloud and segment the spherical reference point cloud. Then, fit the spherical center coordinates of the spherical point cloud. Next, we calculate the relative spatial relationship between the target point cloud and the test point cloud and achieve coarse registration. Finally, we use the iterative closest point algorithm to complete the point cloud pair’s precise registration. From the experimental results, we can find that the proposed method has a splicing root mean square error of <0.03  mm for point clouds with insufficient features. When the point clouds’ initial relative spatial positions are significantly different, the average distance between the registered point clouds is <0.04  mm. In addition, experiments have proved that the arrangement of reference objects, the point cloud’s initial relative position, and the cumulative error have little effect on point cloud splicing accuracy. The proposed method does not rely on point cloud features and can achieve high-precision splicing of point clouds with insufficient features.

Violence behavior recognition is an important research scenario in behavior recognition and has broad application prospects in the field of network information review and intelligent security. Inspired by the long-short-term memory network, we estimate that temporal shift module (TSM) may have more room for improvement in the feature extraction ability of long-term information. In order to verify the above conjecture, we explored based on TSM. After many attempts, it was finally proposed to connect the two TSMs in a cascaded manner, which can expand the receptive field of the model. In addition, an efficient channel attention module was introduced at the front end of the network, which strengthened the model’s spatial feature extraction capabilities. At the same time due to behavior recognition prone to over-fitting, we extended and processed on the basis of some open-source datasets to form a larger violence dataset and solved the problem of over-fitting. The final experimental results show that the algorithm proposed can improve the model’s feature extraction ability of violent behavior in the space and temporal dimension and realize the recognition of violent behavior, which verified the above point of view.

Existing 3D single object trackers (SOTs) of a point cloud all apply downscaling when extracting features from points. This operation leads to a loss of spatial and structural information, degrading tracking performance of sparsely distributed and small-scale objects. To address this problem, a structure aware SOT of a point cloud is proposed. Specifically, the backbone network is combined with the auxiliary network to learn point-wise representations. During the training stage, the subsidiary network is used to perform additional tasks and supervisions, which guides the backbone network to extract discriminative structural features. During the inference stage, this network part is detached to meet a real-time requirement as well as to ensure the tracking accuracy. In addition, the impacts of the quantity setting of the input point cloud and re-initiation strategy are discussed; these are significant to the performance but have been ignored by former works. The experimental results show that the proposed method has a distinct improvement even if the tracked object is sparse and small scale.

Correlation filters (CF)-based tracking methods have drawn extensive attention due to their impressive performance in visual object tracking (VOT) challenges. However, the existing CF-based trackers incur the inherent problems of boundary effect and filter degradation. Meanwhile, the scale variants of targets result in tracking-drift problems, which degrade the tracking accuracy significantly. To address the aforementioned problems, a tracking model named spatiotemporal regularization and scale-aware correlation filters (STR-SACF) is proposed. The STR-SACF model consists of two CFs, namely translation filter and scale filter (SF). The translation filter improves the accuracy of target localization, whereas the SF achieves the optimal and fast scale estimation. The proposed tracking model can improve the tracking accuracy while reducing computational complexity. Moreover, a spatiotemporal regularization term is introduced into the translation filter and SF to suppress boundary effect and filter degradation. The proposed model is effectively optimized based on the alternating direction method of multipliers algorithm. Experimental results on four challenging tracking benchmarks, i.e., OTB2013, OTB2015, TC128, and VOT2016, demonstrate the superiority of the proposed tracker compared with more than 20 state-of-the-art trackers.

Principal component analysis (PCA) has been successfully employed for face recognition. However, if the training process occurs frequently, owing to the update or downdate of the face images used for training, batch PCA becomes prohibitively expensive to recalculate. To overcome this limitation, incremental principal component analysis (IPCA) and decremental principal component analysis (DPCA) can be utilized as a good alternative to PCA because it reuses their previous results for its updates. Many IPCA or DPCA algorithms have been proposed; however, inaccurate tracking of the mean values of the face image data accumulates decomposition errors, which results in poor performance compared with batch PCA. We proposed faster and more accurate algorithms for IPCA and DPCA that maintain accurate decomposition results. The experimental results reveal that the proposed algorithms produce eigenvectors that are significantly close to the eigenvectors of batch PCA and exhibit faster execution speed for face recognition.

Small object detection is always a hot research direction in the field of computer vision, and widely used in traffic, production, industry, and other fields. A neural network for detecting small objects based on original Cascade RCNN is proposed. First, we modify the traditional feature pyramid network and introduce multi-branch dilated convolutions to enhance the feature information of the small target. Second, the feature extraction networks in original Cascade RCNN are replaced with multi-layer deformable convolution networks, which can better adapt to the geometric variations of detection target and huge size span of the objects in the same scene. Finally, Soft non-maximum suppression is also integrated into the network to avoid problems in dense object detection. To verify the practicability of our proposed network, we apply it to the garbage detection system. The experimental results show that compared with original Cascade RCNN and other commonly used target detection networks, our proposed method, with a recall rate >0.98 as well as mean average precision up to 0.964, performs better not only in small object detection but also in industrial applications.

The study of image segmentation is a crucial impression in image processing and computer vision appliances, which segments an input image into distinct non-overlapping homogenous divisions and helps to depict the image more conveniently. To improve the performance of the segmentation of gray-level images by introducing a normalized graph cut measure as a thresholding principle to separate an object from the background based on the neutrosophic membership function. We proposed an innovative neutrosophic approach combining a set of features of some efficient algorithms. The implementation of the proposed algorithm is known as the neutrosophic normalized graph cut (NNGC) method. In the gray image segmentation, the problem of wrongly segmentation and segmentation with low accuracy can identify by using this approach. This proposed algorithm is compared with the fuzzy entropy method, neutrosophic graph cut (NGC), classical graphcut, Otsu, and Kittler method. Moreover, we examine that in most cases, our algorithm gives the lowest absolute error, misclassification error, and higher values of signal to noise ratio values that improve the segmentation process of gray images. Finally, we analyze the change of different parameter values in the NNGC and the effect of the substitutes. Also, we discuss the computational complexity of neutrosophic weight matrix results with a weight matrix (classical) results.

We describe a method for symbol spotting in born-digital architectural floor plans, which are a particularly challenging application domain of graphics recognition. We propose a hybrid method that capitalizes on strengths of both vector-based and pixel-based symbol spotting techniques. In the description phase, the salient geometric constituents of a symbol are extracted by a variety of vectorization techniques, including a proposed voting-based algorithm for finding partial ellipses. This enables us to better handle local shape irregularities and boundary discontinuities, as well as partial occlusion and overlap. In the matching phase, the spatial relationship between the geometric primitives is encoded via a primitive-aware proximity graph. A statistical approach is then used to rapidly yield a coarse localization of symbols within the plan. Localization is further refined with a pixel-based step implementing a modified cross-correlation function. Experimental results on the public SESYD synthetic dataset demonstrate that our approach clearly outperforms other popular symbol spotting approaches. Moreover, our approach yields promising results on real-world images, which are significantly more challenging in terms of overlap, occlusion, and presence of non-symbolic graphical data.

In the task of interactive image segmentation, the user initially indicates some pixels as target seeds and background seeds and propagates the seeds’ labels to the rest of the image. We propose an equidistant seed generation method for interactive image segmentation that combines saliency detection and random walk (RW) with restart with optimal restarting probability. Our method includes two stages: first, we utilize the saliency detection method to obtain the initial target region and generate the initial seeds. Then we relabel the seeds using a Gaussian mixture model with limited user interaction to obtain accurate seeds. The relabeled seeds can limit the range of object; therefore, in the second stage, we can select an optimal restarting probability by calculating the area of the RW that exceeds the limited range of the object seeds. Our method can generate effective seeds and thus reduce the dependence on user interaction. Furthermore, using the optimal restarting probability, we can obtain higher segmentation accuracy. Extensive experiments on the Berkeley segmentation dataset, GrabCut dataset, and MSRA1000 dataset demonstrate that our method can reduce the dependence on user interaction and achieve better performance than the latest interactive image segmentation methods, such as RW, RW with restart, normalized RW, sub-Markov RW, and label propagation through complex networks.

The complex distribution, mutual occlusion, and scale difference greatly increase the difficulty of cotton detection in the wild. To reduce the omission ratio and raise the detection accuracy of cotton, a dual-path feature extraction (DPFE) cotton detection algorithm is proposed. It consists of a DPFE convolutional neural network, a multi-path feature fusion module, and a multi-scale prediction module. First, the algorithm uses the Darknet network as the main path for feature extraction. At the same time, the double downsampling feature map of the main path is enhanced by a proposed feature enhancement module—spatial pyramid convolution. Then a four-layer convolutional neural structure is designed as the auxiliary path for feature extraction. Finally, multiple feature information is incorporated to locate and recognize cotton with a higher accuracy. In addition, we collected and labeled a cotton dataset with 168 high-resolution images, including 4922 cotton instances for research. The experimental results demonstrate that the DPFE algorithm increases the average detection precision by 9.55% and the recall rate by 13.69%, compared with the traditional algorithm.

Due to the high efficiency of discriminative correlation filter (DCF), it has attracted widespread attention in the field of UAV object tracking. To handle the problem of filter degradation, many trackers usually introduce temporal regularization to enhance the discriminative power of the filter. However, these temporal regularization methods only utilize the limited information between two consecutive frames, which are susceptible interference by previous corrupted information. Besides, regularization terms with predefined hyperparameters can not well adapt to the variations of the target across sequent frames, which may cause the model degradation or drift. We propose a tracker based on DCF framework to fully exploit the information hidden in the historical response map, namely adaptive weighted response consistency-based DCF tracking. Specifically, carefully selected historical response maps with fixed weight distribution are introduced in training phase to increase the robustness of the filter. Further, we present a unified loss for jointly learning the filter and the weight distribution, which can be solved by the alternate convex search method. The joint loss guarantees that reliable response maps contribute more to filter learning, leading to a more discriminative and adaptive filter for tracking the target. Extensive experiments show that the proposed method achieves state-of-the-art results on two datasets.

Monocular depth estimation, which provides a critical method for understanding 3D scene geometry, is an ill-posed problem. Recent research studies have achieved significant progress by reliable network architecture and optimized constraints, such as spatial propagation network and depth metrics. We propose an effective generative adversarial network for fast and accurate monocular depth estimation. Our approach demonstrates the feasibility of applying a dense-connected UNet for reducing information transmission loss and then fine-tuning the blur depth by the high-order convolutional spatial propagation network (CSPN) that used a modified loss function of discriminator. Furthermore, we modify the loss function of discriminator by adding the correlation loss that is used to measure the similarity of real and fake labels. Compared with the original CSPN, the high-order CSPN reduces the computation complexity and accelerates the convergence of the generator network by increasing the order of kernel, which emphasizes the weight of kernels in the update formula. With these modifications, our generative adversarial second-order convolutional spatial propagation network (GA-CSPN) achieves more accurate results against state-of-the-art methods in both indoor and outdoor scenes on Make3D, KITTI 2015, and NYUv2 datasets.

Tone mapping is one of the main techniques to convert high-dynamic range (HDR) images into low-dynamic range (LDR) images. We propose to use a variant of generative adversarial networks to adaptively tone map images. We designed a conditional adversarial generative network composed of a U-Net generator and patchGAN discriminator to adaptively convert HDR images into LDR images. We extended previous work to include additional metrics such as tone-mapped image quality index (TMQI), structural similarity index measure, Fréchet inception distance, and perceptual path length. In addition, we applied face detection on the Kalantari dataset and showed that our proposed adversarial tone mapping operator generates the best LDR image for the detection of faces. One of our training schemes, trained via 256  ×  256 resolution HDR–LDR image pairs, results in a model that can generate high TMQI low-resolution 256  ×  256 and high-resolution 1024  ×  2048 LDR images. Given 1024  ×  2048 resolution HDR images, the TMQI of the generated LDR images reaches a value of 0.90, which outperforms all other contemporary tone mapping operators.

We present an improved Criminisi image inpainting algorithm through kriging pretreatment and facet model. We propose three improvement strategies. First, we propose an improved priority. A priority used in Criminisi algorithm consists of confidence term and data term. As iteration progresses, the confidence term is updated and plays a key role in the confidence term of the next edge block. At the same time, the confidence term gradually tends to 0, leading to a weakening of the role of priority, and ultimately affecting the repair effect. We propose an improved priority, which is represented by a piecewise function. Importantly, this improved priority reduces the risk of being weakened during iteration. Second, Criminisi algorithm uses fixed-size sample blocks to repair damaged images, regardless of whether the image content is a textured area or a structure area. We introduce an adaptive method to select sample block size based on the facet model. The size of the sample block is adaptively adjusted for different image contents, thereby improving the quality of the repaired image. Third, we show a weighted sum of squares differences matching principle based on the facet model. The matching formula is determined by the pixel gray value of a sample block and the structure value of four directions, which improves matching accuracy between target block and optimal matching block. Finally, experimental results show that the proposed algorithm is competitive with some state-of-the-art inpainting techniques in terms of both objective metrics and subjective visual inspection.

Affine-scale invariant feature transform (ASIFT) has performed very well for perspective images, but the intrinsic nonlinear distortion makes it difficult to utilize ASIFT directly for fisheye images to find a large amount of correspondences. We reuse ASIFT and propose a new feature matching method for uncorrected fisheye images. First, we employ the Gaussian hemi-image to divide fisheye images into patches. In this process, we introduce Plücker coordinates and side operator to establish the correspondences between the hemispherical projection model, Gaussian hemi-image, and fisheye images. Simultaneously, maximum stable extreme region algorithm is used to detect the target regions. After that, each patch in these regions is simulated at different orientation parameters in ASIFT, and SIFT algorithm is applied to all simulated patches. Experiments on real-world images show that the proposed method can achieve good performance: the numbers of the point correspondences increase greatly with satisfactory accuracy, reliability, and efficiency.

Person re-identification (re-ID) is the task of matching the same individuals across multiple cameras, and its performance is greatly influenced by background clutter. Most re-ID methods remove background clutter using hard manners, such as the use of segmentation algorithms. However, the hard manner may damage the structure information and smoothness of original images. In this work, we propose a unidirectional information-interaction network (UI²N) that consists of a global stream (G-Stream) and a background-graying stream (BGg-Stream). The G-Stream and BGg-Stream carry out unidirectional information interaction such that their features are complementary. We further propose a soft manner with the UI²N to weaken background clutter by background-graying. The soft manner can help the UI²N filter out background interference and retain some informative background cues. Extensive evaluations demonstrate that our method significantly outperforms many state-of-the-art approaches in the challenging Market-1501, DukeMTMC-reID, and CUHK03-NP datasets.

Convolutional neural networks (CNNs) are effective for image classification, and deeper CNNs are being used to improve classification performance. Indeed, as needs increase for searchability of vast printed document image collections, powerful CNNs have been used in place of conventional image processing. However, better performances of deep CNNs come at the expense of computational complexity. Are the additional training efforts required by deeper CNNs worth the improvement in performance? Or could a shallow CNN coupled with conventional image processing (e.g., binarization and consolidation) outperform deeper CNN-based solutions? We investigate performance gaps among shallow (LeNet-5, -7, and -9), deep (ResNet-18), and very deep (ResNet-152, MobileNetV2, and EfficientNet) CNNs for noisy printed document images, e.g., historical newspapers and document images in the RVL-CDIP repository. Our investigation considers two different classification tasks: (1) identifying poems in historical newspapers and (2) classifying 16 document types in document images. Empirical results show that a shallow CNN coupled with computationally inexpensive preprocessing can have a robust response with significantly reduced training samples; deep CNNs coupled with preprocessing can outperform very deep CNNs effectively and efficiently; and aggressive preprocessing is not helpful as it could remove potentially useful information in document images.

A block color image encryption algorithm based on chaotic system, elementary cellular automata (ECA), and deoxyribonucleic acid (DNA) sequence operations is proposed. The architecture is comprised of forward-diffusion, permutation, and backward-diffusion. First, the red, green, and blue components of a color image are converted into three DNA matrices by DNA encoding rule. Exclusive OR (XOR) operation is applied between the DNA coding chaotic sequence generated by the improved Henon map and three DNA matrices. Second, the large DNA matrix, composed of three DNA matrices, is divided into several small blocks with unequal sizes. ECA is employed to scramble the positions of these blocks. Finally, DNA computations are introduced to diffuse three permutated DNA matrices. In addition, initial values and system parameters of the chaotic system are calculated from the secure hash algorithm (SHA-256) hash value of the plain image and the given parameters. Both DNA encoding and ECA order rules rely on plaintext information. Therefore, the proposed algorithm is highly sensitive to plain image. Simulation results and security analyses demonstrate that our algorithm achieves excellent security performance and acceptable running time.

Grayscale images are fundamental to many image processing applications, such as data compression, feature extraction, printing, and tone mapping. However, some image information is lost when converting from color to grayscale. We propose a lightweight and high-speed image decolorization method based on human perception of color temperatures. Chromatic aberration results from differential refraction of light depending on its wavelength. It causes some rays corresponding to cooler colors (such as blue, green) to converge before the warmer colors (such as red and orange). This phenomenon creates a perception of warm colors “advancing” toward the eye, whereas the cool colors to be “receding” away. In this proposed color-to-gray conversion model, we implement a weighted blending function to combine red (perceived warm) and blue (perceived cool) channels. Our main contribution is threefold. First, we implement a high-speed color processing method using exact pixel-by-pixel processing, and we report a 5.7  ×   speed up compared with other new algorithms. Second, our optimal color conversion method produces luminance in images that are comparable to other state-of-the-art methods that we quantified using the objective metrics (E-score and C2G-SSIM) and subjective user studies (decolorization and tone mapping). Third, we demonstrate that an effective luminance distribution can be achieved using our algorithm using global and local tone mapping applications.

Crowd counting is still a challenging task due to the variability of the distance scale, crowd occlusion, and complex background information. However, the deep convolution neural network has been proved to be effective in solving these problems. By loading input images, the network generates predicted density maps, and the average absolute error between the predicted density maps and given ground truth (GT) maps is a solid standard for evaluating the quality of the network. We propose a mask-based generative adversarial network (MBGAN) structure to generate accurate predicted density maps. The network consists of two parts: the generator and the discriminator. In the generator, we embed a fundamental feature extracting module, multiple level dilated convolution blocks, a predicted mask, and shortcut connection operations. The discriminator is mainly used to distinguish whether the density map comes from the generator or the GT and urges the generator to produce the density map that can confuse itself. The training of the proposed MBGAN model is through the joint action of density loss and adversarial loss. In the training strategy, we use the cross training of the generator and discriminator. Through experiments on five available datasets, the MBGAN achieved state-of-the-art performances that outperform other advanced methods.

Person re-identification (Re-ID) aims to retrieve specific individuals across non-overlapping camera views. In recent years, attention-based models contribute to many computer vision tasks due to their great ability for learning discriminative features. We propose the deep progressive attention (DPA) in a more natural manner for person Re-ID. Similar to human visual attention mechanism, the proposed DPA progressively selects the most discriminative parts of a specific individual and formulates feature representation for comparison purpose. Concretely, on the one hand, the proposed DPA uses a long-term reward to optimize the discriminative feature selection. On the other hand, a deep convolutional architecture is integrated into a recurrent model for feature representation learning. Extensive experiments on three person Re-ID benchmarks Market-1501, DukeMTMC-reID, and CUHK03-NP demonstrate the proposed DPA is on par with the state-of-the-art. Moreover, the experiments on partial person Re-ID datasets indicate the proposed DPA is competitive with the specially designed partial person Re-ID methods.