Guest Editors Igor Jovančević and Jean-José Orteu introduce the Special Section on Quality Control by Artificial Vision VII.

“Flying spot” laser infrared thermography (FST) is a non destructive testing technique able to detect defects by scanning surfaces with a laser heat source. Defects, such as cracks on metallic parts, are revealed by the disturbance of heat propagation measured by an infrared camera. Deep learning approaches are now very efficient to automatically analyze and use contextual information from data and can be used for crack detection. However, in the literature only few works deal with the use of deep learning for the crack detection in FST. Indeed obtaining a large amount of data from FST examinations can be expensive and time-consuming. This work presents an open-access database for “flying spot” laser thermography, annotated for crack-type defect localization. It is followed by a benchmark of several state-of-the-art machine learning architectures. The benefits of this database for transfer to the detection and localization of cracks in coated metallic samples are highlighted, with very reduced amounts of data. Finally, this work gives a preliminary exploration on the use of individual thermal images, yielding to detection and localization performance comparable to the previously studied reconstructed thermal maps, which consists in the normalized mean of all the thermal images on a region of interest, after registration.

Visual inspection plays a pivotal role in numerous industrial production processes, and the pursuit of automation has surged with the rise of deep learning and convolutional neural networks (CNNs). Therein, the deployment of visual inspection CNNs on resource-constrained edge devices stands as a critical problem as these devices are the most affordable and well-suited for many industrial applications, e.g., production chains. Nonetheless, it faces challenges in meeting the computational demands of deep CNN models. Consequently, optimizing these models for efficient operation in such settings is imperative. Visual inspection tasks are often highly specialized, differing significantly from general computer vision tasks. As a result, state-of-the-art CNNs can be excessively large for achieving high accuracy on these specific datasets. To address this challenge, this paper introduces a novel approach utilizing neural architecture search (NAS) and hyperparameter optimization. We present the generic toolkit for NAS (GT-NAS), an open-source toolkit available for public use on GitLab ( https://gitlab.com/pmf5/open-source/generic-toolkit-for-neural-architecture-search). We showcase the results of applying our methodology to two established state-of-the-art CNN models designed for surface defect detection, a problem that encompasses binary classification and segmentation of images.Our approach yields significantly smaller models relative to baselines, but with accuracy in line with the current state-of-the-art results, demonstrating the potential for enhanced efficiency in industrial visual inspection systems. In one experimental setting (optimizing the Mixed Supervision model on the KolektorSDD2 dataset), GT-NAS produced an architecture that is 6.2 times faster than the baseline while sacrificing only 0.25% of its average precision for binary classification. In another batch of experiments (optimizing the TriNet model on the SensumSODF dataset), GT-NAS also achieved remarkable results. It found a TriNet architecture five times smaller than the baseline, at a small cost of a 0.25% drop in the ROC-AUC classification score on the capsule subset of the SensumSODF dataset. Furthermore, on the softgel subset of the same dataset, GT-NAS produced a model that was 2.7 times smaller than the baseline, yet 0.19% more precise.

Precision characterization is fundamental to achieve expected performance in semiconductors where Moore’s law pushes the boundaries to miniaturize components. To measure these attributes, deep learning models are used, which require manual annotation of several objects captured via electron microscopy. However, this annotation can be laborious and time-consuming. We propose a semi-automated method for annotating items in electron microscopy images, in an effort to be innovative, efficient, and reliable. Our approach involves identifying objects, enhancing boundaries with use of a unique loss function incorporating physical aspects from electron microscopy images. It greatly reduces the need for users to undertake the annotation model’s training process. It also minimizes post-inference processing by delivering a ready-to-use model. The constrained dynamic match loss (C-DML) incorporates dynamic matching with horizontal/vertical symmetry constraints to address the distinct challenges presented by manufactured objects acquired by microscopy imaging. Metrology metrics from the contour predictions obtained with C-DML obtain a mean relative error (MRE) of <10% and a correlation coefficient surpassing 90% when compared with ground truth corresponding to manual annotations. Our experimental results demonstrate the superior performance of C-DML over both classical DML and state-of-the-art deep annotation models. An extensive investigation demonstrates the effectiveness of our approach on heterogeneous datasets, including diverse objects of different materials and shapes, leading to state-of-the-art measurement results. Additionally, we show with the experiments that we can obtain better performances with better hyperparameters and data augmentation. Furthermore, this investigation presents a technique for annotating electron microscopy images efficiently and sheds light on the essential elements that dictate the approach’s overall efficacy.

Even though neural network methodologies have been established for a long time, only recently have they achieved exceptional efficacy in practical deployments, predominantly due to improvements in hardware computational capacity and the large amounts of available data for learning. Nonetheless, substantial challenges remain in utilizing deep learning in many domains, mainly because of the lack of large amounts of labeled data that are versatile enough for deep learning models to learn useful information. For instance, in mechanical assembly inspection, annotating data for each type of mechanical part to train a deep learning model can be very labor-intensive. Additionally, it is required to annotate data after each modification of mechanical part specification. Also, the system for inspection is typically not available until the first few samples are built to collect data. This paper proposes a solution for these challenges in case of the visual mechanical assembly inspection by processing point cloud data acquired via a three-dimensional (3D) scanner. To reduce the necessity for manually labeling large amounts of data, we employed synthetically generated data for both training and validation purposes, reserving the real sensor data exclusively for the testing phase. Our approach reduces the need for large amounts of labeled data by using synthetically generated point clouds from computer-aided design models for neural network training. Domain gap is a significant challenge for the usage of synthetically generated data. To reduce the domain gap, we used different preprocessing techniques, as well as a neural network architecture that focuses more on shared features that will not significantly change between synthetically generated data and real data from the 3D sensor.

Producing near net shape parts with complex geometries using wire-laser additive manufacturing (AM) often requires a mastered and optimized process. Differences between the constructed and nominal geometries of the manufactured entities demand an in-situ defects measurement to complete the production of the entire part successfully. A contactless measuring system is needed to evaluate geometrical deviations without requiring complex post-processing operations. To overcome this challenge and validate a measuring tool that serves the manufacturing purpose, a global stereocorrelation approach is used to measure defects in wire-laser additively manufactured parts. This method relies on the cameras’ self-calibration phase that uses the part substrate’s nominal model. Then, a modal basis is defined to model and evaluate the surface dimensional and shape defects. Hence, an analysis of the texture obtained in AM is conducted to assess whether or not it is sufficient for image correlation and defect measurement. Finally, natural and pattern textures are compared to highlight their influence on the measurement results.

Anomaly detection (AD) in an unsupervised manner has become the go-to approach in applications where data labeling proves problematic. However, these approaches are not completely unsupervised, since they rely on the weak knowledge of the dataset distribution into anomalous and anomaly-free subsets and typically require post-training threshold calibration in order to perform AD. Yet, they do not take advantage of available positive samples during training. In contrast, fully supervised approaches have proven to be more accurate and more efficient; however, they require a sufficient number of anomalous images to be labeled on a per-pixel level, which represents a labor-intensive task. In this article, we propose a hybrid approach that utilizes the best of both worlds. We use an unsupervised approach to build a model for generating pseudo labels, followed by a supervised approach to increase the robustness of AD. Moreover, we extend this approach with an active learning schema that results in learning with mixed supervision. We achieve several improvements, i.e., the utilization of available positive image samples, improved AD performance, and the retention of real-time performance. The proposed approach yields results that are comparable to the fully supervised approach, and at the very least, reduces the number of required labeled anomalous samples.

This paper aims to delineate a comprehensive method that integrates machine vision and deep learning for quality control within an industrial setting. The proposed innovative approach leverages a microservice architecture that ensures adaptability and flexibility to different scenarios while focusing on the employment of affordable, compact hardware, and it achieves exceptionally high accuracy in performing the quality control task and keeping a minimal computation time. Consequently, the developed system operates entirely on a portable smart camera, eliminating the need for additional sensors such as photocells and external computation, which simplifies the setup and commissioning phases and reduces the overall impact on the production line. By leveraging the integration of the embedded system with the machinery, this approach offers real-time monitoring and analysis capabilities, facilitating the swift detection of defects and deviations from desired standards. Moreover, the low-cost nature of the solution makes it accessible to a wider range of manufacturing enterprises, democratizing quality processes in Industry 5.0. The system was successfully implemented and is fully operational in a real industrial environment, and the experimental results obtained from this implementation are presented in this work.

Electrical cables consist of numerous wires, the three-dimensional (3D) shape of which significantly impacts the cables’ overall properties, such as bending stiffness. Although X-ray computed tomography (CT) provides a non-destructive method to assess these properties, accurately determining the 3D shape of individual wires from CT images is challenging due to the large number of wires, low image resolution, and indistinguishable appearance of the wires. Previous research lacked quantitative evaluation for wire tracking, and its overall accuracy heavily relied on the accuracy of wire detection. In this study, we present a long short-term memory-based approach for wire tracking that improves robustness against detection errors. The proposed method predicts wire positions in subsequent frames based on previous frames. We evaluate the performance of the proposed method using both actual annotated cables and artificially noised annotations. Our method exhibits greater tracking accuracy and robustness to detection errors compared with the previous method.

The advent of deep learning methods has led to impressive advances in computer vision tasks over the past decades, largely due to their ability to extract non-linear features that are well adapted to the task at hand. For supervised approaches, data labeling is essential to achieve a high level of performance; however, this task can be so fastidious or even troublesome in difficult contexts (e.g., specific defect detection, unconventional data annotations, etc.) that experts can sometimes erroneously provide the wrong ground truth label. Considering classification problems, this paper addresses the issue of handling noisy labels in datasets. Specifically, we first detect the noisy samples of a dataset using set-valued labels and then improve their classification using Venn–Abers predictors. The obtained results reach more than 0.99 and 0.90 accuracy for noisified versions of two widely used image classification datasets, digit MNIST and CIFAR-10 respectively with a 40% two-class pair-flip noise ratio and 0.87 accuracy for CIFAR-10 with 10-class uniform 40% noise ratio.

To enhance control and monitoring of industrial crystallization processes, we propose an innovative nondestructive imaging method utilizing in situ 2D vision sensors. This approach enables the acquisition of 2D videos depicting crystal aggregates throughout the batch crystallization process. Our approach is built upon experimental observations, specifically regarding the process dynamics and sensor fouling. It involves dynamic segmentation of observed aggregates, from which quantitative analyses are derived. Notably, our method allows for tracking the evolution of the particle size distribution of crystal aggregates over time and the determination of the growth kinetics of crystals that agglomerate at the sensor air gap. This enables the detection of key stages in the crystallization process and the geometric characterization of crystal aggregate production.

Throughout the steel production process, a variety of surface defects inevitably occur. These defects can impair the quality of steel products and reduce manufacturing efficiency. Therefore, it is crucial to study and categorize the multiple defects on the surface of steel strips. Vision transformer (ViT) is a unique neural network model based on a self-attention mechanism that is widely used in many different disciplines. Conventional ViT ignores the specifics of brain signaling and instead uses activation functions to simulate genuine neurons. One of the fundamental building blocks of a spiking neural network is leaky integration and fire (LIF), which has biodynamic characteristics akin to those of a genuine neuron. LIF neurons work in an event-driven manner such that higher performance can be achieved with less power. The goal of this work is to integrate ViT and LIF neurons to build and train an end-to-end hybrid network architecture, spiking vision transformer (S-ViT), for the classification of steel surface defects. The framework relies on the ViT architecture by replacing the activation functions used in ViT with LIF neurons, constructing a global spike feature fusion module spiking transformer encoder as well as a spiking-MLP classification head for implementing the classification functionality and using it as a basic building block of S-ViT. Based on the experimental results, our method has demonstrated outstanding classification performance across all metrics. The overall test accuracies of S-ViT are 99.41%, 99.65%, 99.54%, and 99.77% on NEU-CLSs, and 95.70%, 95.93%, 96.94%, and 97.19% on XSDD. S-ViT achieves superior classification performance compared to convolutional neural networks and recent findings. Its performance is also improved relative to the original ViT model. Furthermore, the robustness test results of S-ViT show that S-ViT still maintains reliable accuracy when recognizing images that contain Gaussian noise.

We introduce a method for efficient and expressive high-resolution image synthesis, harnessing the power of variational autoencoders (VAEs) and transformers with sparse attention (SA) mechanisms. By utilizing VAEs, we can establish a context-rich vocabulary of image constituents, thereby capturing intricate image features in a superior manner compared with traditional techniques. Subsequently, we employ SA mechanisms within our transformer model, improving computational efficiency while dealing with long sequences inherent to high-resolution images. Extending beyond traditional conditional synthesis, our model successfully integrates both nonspatial and spatial information while also incorporating temporal dynamics, enabling sequential image synthesis. Through rigorous experiments, we demonstrate our method’s effectiveness in semantically guided synthesis of megapixel images. Our findings substantiate this method as a significant contribution to the field of high-resolution image synthesis.

Image watermarking technology plays a key role in the protection of intellectual property rights. In addition to digital watermarking, optical watermarking has also been widely considered. A watermarking scheme based on Gerchberg–Saxton (GS) algorithm and integer wavelet transform (IWT) is proposed for image encryption. The scheme uses the unique phase reconstruction characteristics of GS algorithm, which makes it able to deal with a variety of complex local attacks in the process of protection and effectively restore the original image information. The obfuscation of position and numerical value information is realized by means of variable step Joseph space scrambling and pixel value bit processing. The carrier image is decomposed into subbands with different frequencies using IWT, all the information of the secret image is embedded bit by bit, which realizes the hiding of the image information. In addition, the SHA-256 function, the RSA algorithm, and the hyperchaotic system are combined to generate the cipher stream. The experimental results show that the algorithm has good imperceptibility and security, as well as strong robustness to cutting and salt and pepper noise attacks, and can restore the image quality well.

In the realm of low-resolution (LR) to high-resolution (HR) image reconstruction, denoising diffusion probabilistic models (DDPMs) are recognized for their superior perceptual quality over other generative models, attributed to their adept handling of various degradation factors in LR images, such as noise and blur. However, DDPMs predominantly focus on a single modality in the super-resolution (SR) image reconstruction from LR images, thus overlooking the rich potential information in multimodal data. This lack of integration and comprehensive processing of multimodal data can impede the full utilization of the complementary characteristics of different data types, limiting their effectiveness across a broad range of applications. Moreover, DDPMs require thousands of evaluations to reconstruct high-quality SR images, which significantly impacts their efficiency. In response to these challenges, a novel multimodal DDPM based on structured knowledge distillation (MKDDPM) is introduced. This approach features a multimodal-based DDPM that effectively leverages sparse prior information from another modality, integrated into the MKDDPM network architecture to optimize the solution space and detail features of the reconstructed image. Furthermore, a structured knowledge distillation method is proposed, leveraging a well-trained DDPM and iteratively learning a new DDPM, with each iteration requiring only half the original sampling steps. This method significantly reduces the number of model sampling steps without compromising on sampling quality. Experimental results demonstrate that MKDDPM, even with a substantially reduced number of diffusion steps, still achieves superior performance, providing a novel solution for single-image SR tasks.

In this article, we show how to extend the multi-view stereo technique when the object to be reconstructed is inside a transparent but refractive medium, which causes distortions in the images. We provide a theoretical formulation of the problem accounting for a general non-planar shape of the refractive interface, and then a discrete solving method. We also present a pipeline to recover precisely the geometry of the refractive interface, considered as a convex polyhedral object. It is based on the extraction of visible polyhedron vertices from silhouette images and matching across a sequence of images acquired under circular camera motion. These contributions are validated by tests on synthetic and real data.

In the field of three-dimensional (3D) reconstruction, neural radiation fields (NeRF) can implicitly represent high-quality 3D scenes. However, traditional neural radiation fields place very high demands on the quality of the input images. When motion blurred images are input, the requirement of NeRF for multi-view consistency cannot be met, which results in a significant degradation in the quality of the 3D reconstruction. To address this problem, we propose KT-NeRF that extends NeRF to motion blur scenes. Based on the principle of motion blur, the method is derived from two-dimensional (2D) motion blurred images to 3D space. Then, Gaussian process regression model is introduced to estimate the motion trajectory of the camera for each motion blurred image, with the aim of learning accurate camera poses at key time stamps during the exposure time. The camera poses at the key time stamps are used as inputs to the NeRF in order to allow the NeRF to learn the blur information embedded in the images. Finally, the parameters of the Gaussian process regression model and the NeRF are jointly optimized to achieve multi-view anti-motion blur. The experiment shows that KT-NeRF achieved a peak signal-to-noise ratio of 29.4 and a structural similarity index of 0.85, an increase of 3.5% and 2.4%, respectively, over existing advanced methods. The learned perceptual image patch similarity was also reduced by 7.1% to 0.13.

Underwater environments have characteristics such as unclear imaging and complex backgrounds that lead to poor performance when applying mainstream object detection models directly. To improve the accuracy of underwater object detection, we propose an object detection model, RF-YOLO, which uses a receptive field enhancement (RFE) module in the backbone network to finish RFE and extract more effective features. We design the free-channel iterative attention feature fusion module to reconstruct the neck network and fuse different scales of feature layers to achieve cross-channel attention feature fusion. We use Scylla-intersection over union (SIoU) as the loss function of the model, which makes the model converge to the optimal direction of training through the angle cost, distance cost, shape cost, and IoU cost. The network parameters increase after adding modules, and the model is not easy to converge to the optimal state, so we propose a training method that effectively mines the performance of the detection network. Experiments show that the proposed RF-YOLO achieves a mean average precision of 87.56% and 86.39% on the URPC2019 and URPC2020 datasets, respectively. Through comparative experiments and ablation experiments, it was verified that the proposed network model has a higher detection accuracy in complex underwater environments.

Grasp detection within unstructured environments encounters challenges that lead to a reduced success rate in grasping attempts, attributable to factors including object uncertainty, random positions, and differences in perspective. This work proposes a grasp detection algorithm framework, Swin-transNet, which adopts a hypothesis treating graspable objects as a generalized category and distinguishing between graspable and non-graspable objects. The utilization of the Swin transformer module in this framework augments the feature extraction process, enabling the capture of global relationships within images. Subsequently, the integration of a decoupled head with attention mechanisms further refines the channel and spatial representation of features. This strategic combination markedly improves the system’s adaptability to uncertain object categories and random positions, culminating in the precise output of grasping information. Moreover, we elucidate their roles in grasping tasks. We evaluate the grasp detection framework using the Cornell grasp dataset, which is divided into image and object levels. The experiment indicated a detection accuracy of 98.1% and a detection speed of 52 ms. Swin-transNet shows robust generalization on the Jacquard dataset, attaining a detection accuracy of 95.2%. It demonstrates an 87.8% success rate in real-world grasping testing on a visual grasping system, confirming its effectiveness for robotic grasping tasks.

The primary focus of key frame extraction lies in extracting changes in the motion state from surveillance videos and considering them to be crucial content. However, existing key frame evaluation indicators cannot accurately assess whether the algorithm can capture them. Hence, key frame extraction methods are assessed from the viewpoint of target trajectory reconstruction. The motion trajectory reconstruction degree (MTRD), a key frame selection criterion based on maintaining target global and local motion information, is then put forth. Initially, this evaluation indicator extracts key frames using various key frame extraction methods and reconstructs the motion trajectory based on these key frames using a linear interpolation algorithm. Then, the original motion trajectories of the target are quantified and compared with the reconstructed set of motion trajectories. The more minor the MTRD discrepancy is, the better the trajectory overlap is, and the more accurate the key frames extracted with this method will be for the description of the video content. Finally, inspired by the novel MTRD criterion, we develop an MTRD-oriented key frame extraction method for the surveillance video. The outcomes of the simulations demonstrate that MTRD can more accurately capture the variations in the global and local motion states and is more compatible with the human visual perception.

The absorption by a water body and the scattering of suspended particles cause blurring of object features, which results in a reduced accuracy of underwater salient object detection (SOD). Thus, we propose a polarization spatial and semantic learning lightweight network for underwater SOD. The proposed method is based on a lightweight MobileNetV2 network. Because lightweight networks are not as capable as deep networks in capturing and learning features of complex objects, we build specific feature extraction and fusion modules at different depth stages of backbone network feature extraction to enhance the feature learning capability of the lightweight backbone network. Specifically, we embed a structural feature learning module in the low-level feature extraction stage and a semantic feature learning module in the high-level feature extraction stage to maintain the spatial consistency of low-level features and the semantic commonality of high-level features. We acquired polarized images of underwater objects in natural underwater scenes and constructed a polarized object detection dataset (PODD) for object detection in the underwater environment. Experimental results show that the detection effect of the proposed method on the PODD is better than other SOD methods. Also, we conduct comparative experiments on the RGB-thermal (RGB-T) and RGB-depth (RGB-D) datasets to verify the generalization of the proposed method.

The deblurring of flotation froth images significantly aids in the characterization of coal flotation and fault diagnosis. Images of froth acquired at a flotation site contain considerable noise and blurring, making feature extraction and segmentation processing difficult. We present an effective method for deblurring froth images based on disentangled representations. Disentangled representation is achieved by separating the content and blur features in the blurred image using a content encoder and a blur encoder. Then, the separated feature vectors are embedded into a deblurring framework to deblur the froth image. The experimental results show that this method achieves a superior deblurring effect on froth images under various conditions, which lays the foundation for the intelligent adjustment of parameters to guide the flotation site.

Test-time adaptation (TTA) aims to address potential differences in data distribution between the training and testing phases by modifying a pretrained model based on each specific test sample. This process is especially crucial for deep learning models, as they often encounter frequent changes in the testing environment. Currently, popular TTA methods rely primarily on pseudo-labels (PLs) as supervision signals and fine-tune the model through backpropagation. Consequently, the success of the model’s adaptation depends directly on the quality of the PLs. High-quality PLs can enhance the model’s performance, whereas low-quality ones may lead to poor adaptation results. Intuitively, if the PLs predicted by the model for a given sample remain consistent in both the current and future states, it suggests a higher confidence in that prediction. Using such consistent PLs as supervision signals can greatly benefit long-term adaptation. Nevertheless, this approach may induce overconfidence in the model’s predictions. To counter this, we introduce a regularization term that penalizes overly confident predictions. Our proposed method is highly versatile and can be seamlessly integrated with various TTA strategies, making it immensely practical. We investigate different TTA methods on three widely used datasets (CIFAR10C, CIFAR100C, and ImageNetC) with different scenarios and show that our method achieves competitive or state-of-the-art accuracies on all of them.

The challenge in fine-grained image classification tasks lies in distinguishing subtle differences among fine-grained images. Existing image classification methods often only explore information in isolated regions without considering the relationships among these parts, resulting in incomplete information and a tendency to focus on individual parts. Posture information is hidden among these parts, so it plays a crucial role in differentiating among similar categories. Therefore, we propose a posture-guided part learning framework capable of extracting hidden posture information among regions. In this framework, the dual-branch feature enhancement module (DBFEM) highlights discriminative information related to fine-grained objects by extracting attention information between the feature space and channels. The part selection module selects multiple discriminative parts based on the attention information from DBFEM. Building upon this, the posture feature fusion module extracts semantic features from discriminative parts and constructs posture features among different parts based on these semantic features. Finally, by fusing part semantic features with posture features, a comprehensive representation of fine-grained object features is obtained, aiding in differentiating among similar categories. Extensive evaluations on three benchmark datasets demonstrate the competitiveness of the proposed framework compared with state-of-the-art methods.

The visual characteristics of greenhouse-grown tomatoes undergo significant alterations under artificial lighting, presenting substantial challenges in accurately detecting targets. To address the diverse appearances of targets, we propose an improved You Only Look Once Version 5 (YOLOv5) model named BiSR_YOLOv5, incorporating the single-point and regional feature fusion module (SRFM) and the bidirectional spatial pyramid pooling fast (Bi-SPPF) module. In addition, the model adopts SCYLLA-intersection over union loss instead of complete intersection over union loss. Experimental results reveal that the BiSR_YOLOv5 model achieves F1 and mAP@0.5 scores of 0.867 and 0.894, respectively, for detecting truss tomatoes. These scores are 2.36 and 1.82 percentage points higher than those achieved by the baseline YOLOv5 algorithm. Notably, the model maintains a size of 13.8M and achieves real-time performance at 35.1 frames per second. Analysis of detection results for both large and small objects indicates that the Bi-SPPF module, which emphasizes finer feature details, is better suited for detecting small-sized targets. Conversely, the SRFM module, with a larger receptive field, is better suited for detecting larger targets. In summary, the BiSR YOLOv5 test results validate the positive impact of accurate identification on subsequent agricultural operations, such as yield estimation or harvest. This is achieved through the implementation of a simple maturity algorithm that utilizes the process of “finding flaws.”

In recent years, various segmentation models have been developed successively. However, due to the limited availability of nighttime datasets and the complexity of nighttime scenes, there remains a scarcity of high-performance nighttime semantic segmentation models. Analysis of nighttime scenes has revealed that the primary challenges encountered are overexposure and underexposure. In view of this, our proposed Histogram Multi-scale Retinex with Color Restoration and No-Exposure Semantic Segmentation Network model is based on semantic segmentation of nighttime scenes and consists of three modules and a multi-head decoder. The three modules—Histogram, Multi-Scale Retinex with Color Restoration (MSRCR), and No Exposure (N-EX)—aim to enhance the robustness of image segmentation under different lighting conditions. The Histogram module prevents over-fitting to well-lit images, and the MSRCR module enhances images with insufficient lighting, improving object recognition and facilitating segmentation. The N-EX module uses a dark channel prior method to remove excess light covering the surface of an object. Extensive experiments show that the three modules are suitable for different network models and can be inserted and used at will. They significantly improve the model’s segmentation ability for nighttime images while having good generalization ability. When added to the multi-head decoder network, mean intersection over union increases by 6.2% on the nighttime dataset Rebecca and 1.5% on the daytime dataset CamVid.

The existing iris recognition methods offer excellent recognition performance for known classes, but they do not perform well when faced with unknown classes. The process of identifying unknown classes is referred to as open-set recognition. To improve the robustness of iris recognition system, this work integrates a hash center to construct a deep metric learning method for open-set iris recognition, called central similarity based deep hash. It first maps each iris category into defined hash centers using a generation hash center algorithm. Then, OiNet is trained to each iris texture to cluster around the corresponding hash center. For testing, cosine similarity is calculated for each pair of iris textures to estimate their similarity. Based on experiments conducted on public datasets, along with evaluations of performance within the dataset and across different datasets, our method demonstrates substantial performance advantages compared with other algorithms for open-set iris recognition.

Existing style transfer methods need more texture structure of style images with aesthetic guidance, resulting in the loss of a large amount of texture details, which affects the visual effect. We propose a style transfer model based on multi-adaptive generative adversarial networks (MA-GANs) to address this. Specifically, the aesthetic ability learned by the discriminator is used for feature extraction, and the obtained more generalized features are passed into the multi-attention aesthetic module, which includes a collaborative self-attention (CSA) module and a self-attention normalization (SAN) module. The CSA calculates the correlation between the entangled style features and aesthetic features, capturing texture details and style geometric structures. The SAN balances the content semantic structure with the style geometric structure, harmoniously integrating the style pattern into the content image, thus achieving a more practical style transfer. Extensive qualitative and quantitative experimental studies demonstrate the superiority of MA-GAN in visual quality, enabling the synthesis of art images with smooth brushstrokes and rich colors.

Due to the particularity of remote sensing images, their target detection performance is much lower than that of natural images. In this article, we designed a new model to improve object detection performance in remote sensing images significantly. First, we redesigned the feature extraction network. We deepen the network to obtain feature maps of more sizes and increase the number of detection heads, making the prediction anchors more precise and able to adapt to detection tasks with large target scale spans. Second, to avoid excessive information loss caused by a deep network, we designed a three-level feature fusion network to supplement as much original information as possible into the output feature map. Third, we have introduced a transformer module in the last layer of the backbone, which can compensate for the convolutional network’s weak global information extraction ability without increasing too much computational complexity. In addition, we replaced the original filter with soft-non-maximum suppression (soft-NMS) to solve the problem of missed detections caused by small target clustering in remote sensing images. Experimental results on the DIOR (optical remote sensing image detection) dataset have shown that our model performs well when there are significant differences in object size and small target clustering. Compared with the original network, the mean average precision has improved by 4.8%. We have expanded the DIOR dataset to enhance the model’s generalization ability and explore the network’s potential. The model trained using the expanded dataset is more robust and can work effectively under various interferences. The mean average precision can reach 76.2%. Our model can achieve good results with a small amount of computing resources.

This work proposes efficient multi-scale object detection model with space-to-depth convolution and BiFPN combined with FasterNet (ES-BiCF-YOLOv8), a deep learning method, to address the problems associated with detecting steel surface defects in contemporary industrial production. The method makes innovative improvements based on the YOLOv8 algorithm and enhances the performance of the novel model mainly through the following aspects. First, the space-to-depth layer followed by a non-strided convolution layer (SPD-Conv) and the efficient multi-scale attention mechanism is introduced into the feature extraction network to enhance the model’s ability to capture fine-grained information and the fusion of multi-scale features. Second, the feature fusion network is optimized by utilizing a weighted bi-directional feature pyramid network and a lightweight network, FasterNet, to improve computational efficiency. Finally, it is shown that ES-BiCF-YOLOv8 reduces the complexity and computational requirements of the model while increasing the detection accuracy utilizing the NEU-DET dataset and deepPCB dataset with substantial experimental validation. The ES-BiCF-YOLOv8 model achieves a 5% improvement of the mean average precision value on the NEU-DET dataset, with the number of parameters and the computational amount only being the baseline 89% and 27%, and also demonstrates good generalization performance on the deepPCB dataset. Furthermore, the experiments demonstrate that ES-BiCF-YOLOv8 can be used for steel surface defect detection in industrial production because it uses less computational resources and can detect in real-time while maintaining high accuracy, in comparison to other popular object detection algorithms. The results of this work not only improve the efficiency and accuracy of steel surface defect detection but also provide ideas for the application of deep learning in the field of industrial detection.

Our primary objective is to address the challenges of unsupervised image-to-image translation, particularly in the domain of applying different visual styles to given content images. However, popular single-image translation techniques face issues such as poor quality, excessive image noise, and discrepancies between generated images and human perception. To overcome these challenges, we propose an innovative approach utilizing a dual-branch attention-guided paradigm to achieve high-quality single-image translation. Our method employs a multiscale pyramid structure and utilizes the generator to perform image transformation after downsampling the input image. In addition, we introduce dual-branch spatial attention modules and hybrid convolution modules to enhance the quality of generated images, mitigate noise, and align more closely with human visual perception. This approach helps enhance the focus on the main subject while minimizing interference from background information, thereby producing excellent image translation results. Furthermore, comprehensive experimental validation and comparisons with benchmark datasets, including the Terra Cotta Warriors dataset, affirm the effectiveness of our method. Specifically, our research significantly improves performance, as evidenced by the decrease in the single-image Fréchet inception distance value to 1.83. This result underscores the superior performance of our method compared with the state-of-the-art approaches.

Autonomous driving represents the future of transportation, and the precise detection of three-dimensional (3D) objects is a fundamental requirement for achieving autonomous driving capabilities. Presently, 3D object detection primarily relies on sensors, such as monocular cameras, stereo cameras, and LiDAR technology. In comparison to stereo cameras and LiDAR, monocular 3D object detection offers the advantages of a wider field of view and reduced cost. However, the existing monocular 3D object detection techniques exhibit limitations in terms of accuracy, particularly when detecting distant objects. To tackle this challenge, we introduce an innovative approach for monocular 3D object detection, specifically tailored for distant objects. The proposed method classifies objects into distant and nearby categories based on the initial depth estimation, employing distinct feature enhancement and refinement modules for each category. Subsequently, it extracts 3D features and, ultimately, derives precise 3D detection bounding boxes. Experimental results using the KITTI dataset demonstrate that this approach substantially enhances the detection accuracy of distant objects while preserving the detection efficacy for nearby objects.

The denoising method based on partial differential equation (PDE) has been proven to be a calculable method and used widely, but it is not good at protecting image detail information and results in staircase effect. To improve the above shortcomings, a PDE image denoising model based on a structure tensor matrix is proposed and explained in this work. Combining two different regular terms and adjusting their proportion, the proposed model uses the eigenvalues of the structure tensor to take the norm parameter value of the second regular term. The experiment results show that the proposed model introduced structure tensor can protect details and suppress the staircase effect effectively, thus obtaining satisfying denoising results.

Aiming at the problem that the outdoor pointer instrument dial is easily obscured by environmental factors, an instrument reading recognition method based on two-dimensional convolution and calculus accumulation is proposed. A weighted multi-feature matching algorithm is proposed to improve the success rate of template matching and accurately obtain the center coordinates of the rotating shaft. The dial image is transformed by polar coordinate transformation and calculus accumulation. Finally, the maximum value is the position of the target pointer, which is automatically read in the two-axis coordinate system. Experiments show that the error compared with the actual values is <0.2%, with good robustness under harsh conditions such as too strong and too dark light intensity and occlusion on the dial.

To overcome the limitations of two-dimensional simultaneous localization and mapping (2D-SLAM) technology, which struggles with closed-loop detection in large spatial environments with unstructured and poor lighting conditions, leading to the accumulation of positioning errors, information loss, and low map construction accuracy, solutions need to be found. We focused on the multimodal fusion algorithm of a depth camera and LiDAR based on particle filter at the front end of SLAM. The precision of map information in complex environments is maintained through the utilization of the multilayer iterative closest point matching method for creating accurate laser odometry and screening the loopback candidate set for closed-loop detection of 2D LiDAR data. The loopback information is calibrated through proximity detection, reducing the global map drift phenomenon and enhancing the operational efficiency of the SLAM algorithm through incremental optimization. The public dataset confirms that the proposed algorithm reduces the average relative pose error by 52% compared with the 2D-SLAM algorithm Cartographer. Moreover, it reduces detection time by an average of 15% while maintaining loopback detection accuracy. The SLAM experiment was validated in a complex real-life environment using the Turtlebot2 robot. The results show that the multimodal fusion SLAM method proposed can accurately reconstruct environmental information and achieve high precision and real-time mapping in complex and expansive scenes.

In the field of multi-style textile defect detection, a common challenge is the difficulty of adapting the inherent detection model to different styles of textile defects. Changes in the color or style of the textile often result in a decrease in the accuracy of defect detection. Relying solely on the model for fine-tuning inspections can lead to catastrophic forgetting, which significantly impacts the performance of the textile defect detector. To address these challenges, a multi-task correlation distillation (MTCD) anomaly detection method based on knowledge distillation and representative sampling is proposed to detect multi-style textile defects. To enable MTCD to detect defects of new-style textiles while maintaining the detection of old-style textiles, two main modules are introduced. The distillation adaptation module (DAM) explores the intra-feature correlation in the feature space of the target detector, allowing the student model to acquire knowledge of new-style textile defect detection while inheriting the teacher model’s detection ability for old-style textile defects. The representative sampling module (RSM) stores representative knowledge of textile defect detection for old-style textiles, facilitating the transfer of knowledge learned from detecting new-style textile defect styles and maintaining the ability to detect defects in old-style textiles. This increases the detection accuracy of the student model for new-style textile defects. The results show that the proposed MTCD method can adapt to the new textile defect detection while maintaining the accuracy of the old textile defect detection and avoiding the problem of catastrophic forgetting. Furthermore, it offers a better balance between stability and plasticity, making it a promising solution for defect detection of multi-style textiles in industrial production environments.

Traditional steganography algorithms use procedures created by human experts to conceal the secret message inside a cover medium. Generative adversarial networks (GANs) have recently been used to automate this process. However, GAN based steganography has some limitations. The capacity of these models is limited. By increasing the steganography capacity, security is decreased, and distortion is increased. The performance of the extractor network also decreases with increasing the steganography capacity. In this work, an approach for developing a generator model for image steganography is proposed. The approach involves building a generator model, called the late embedding generator model, in two stages. The first stage of the generator model uses only the flattened cover image, and second stage uses a secret message and the first stage’s output to generate the stego image. Furthermore, a dual-training strategy is employed to train the generator network: the first stage focuses on learning fundamental image features through a reconstruction loss, and the second stage is trained with three loss terms, including an adversarial loss, to incorporate the secret message. The proposed approach demonstrates that hiding data only in the deeper layers of the generator network boosts capacity without requiring complex architectures, reducing computational storage requirements. The efficacy of the proposed approach is evaluated by varying the depth of these two stages, resulting in four generator models. A comprehensive set of experiments was performed on the CelebA dataset, which contains more than 200,000 samples. The results show that the late embedding model performs better than the state-of-the-art models. Also, it increases the steganography capacity to more than four times compared with the existing GAN-based steganography methods. The extracted payload achieves an accuracy of 99.98%, with the extractor model successfully decoding the secret message.

Digital video splicing has become easy and ubiquitous. Malicious users copy some regions of a video and paste them into another video to create realistic forgeries. It is important to blindly detect such forgery regions in videos. A spatio-temporal co-attention fusion network (SCFNet) is proposed for video splicing localization. Specifically, a three-stream network is used as an encoder to capture manipulation traces across multiple frames. The deep interaction and fusion of spatio-temporal forensic features are achieved by the novel parallel and cross co-attention fusion modules. A lightweight multilayer perceptron decoder is adopted to yield a pixel-level tampering localization map. A new large-scale video splicing dataset is created for training the SCFNet. Extensive tests on benchmark datasets show that the localization and generalization performances of our SCFNet outperform the state-of-the-art. Code and datasets are available at https://github.com/multimediaFor/SCFNet.

Image compression plays a vital role in various applications, such as the storage, transmission, and sharing of digital images. We present a symmetric image compression that leverages improved normalization and attention mechanisms to achieve superior compression performance. We focus on addressing the limitations of conventional normalization techniques in handling diverse image characteristics effectively. An adaptive normalization module is proposed to adjust normalization parameters dynamically based on the content of the input image; it ensures optimal data representation and contributes to improved compression efficiency. Furthermore, a windowed attention mechanism is integrated into the compression network to selectively focus on significant image regions while suppressing noise and redundancies by effectively capturing and preserving important visual features to enhance the overall compression quality and reconstruction fidelity. Comparative analysis with state-of-the-art methods demonstrates the superiority of the proposed symmetric image compression network in terms of compression ratio, reconstruction quality, and subjective visual perception. Moreover, the effects of different normalization and attention configurations on compression performance are thoroughly investigated and analyzed. The results of the experiments validate the effectiveness of the proposed symmetric image compression network with improved normalization and windowed attention, known as the Image Compression Attention Normalization Network. The integration of adaptive normalization and windowed attention mechanisms not only enhances the compression efficiency but also enables adaptability to diverse image characteristics. The proposed compression scheme results are as follows. The inference time for encoding and decoding is 0.152 and 0.163 s, respectively. The Bjøntegaard delta (BD) rate shows a reduction of 74.8936%, and the BD-peak signal-to-noise ratio increases by 6.425 dB. The computational load is measured in floating-point operations and totals 263.614G, with the model utilizing 68.1M parameters. Windowed attention preserves critical data, and adaptive normalization alters image processing, enhancing ablation studies. Despite differences, both modules contribute to emphasizing data retention and adjustments and optimizing image processing for improved results.

Deep learning is widely used in the field of computer vision, but the emergence of adversarial examples threatens its application. How to effectively detect adversarial examples and correct their labels has become a problem to be solved in this application field. Generative adversarial networks (GANs) can effectively learn the features from images. Based on GAN, this work proposes a defense method called “Reconstructing images with GAN” (RIG). The adversarial examples are generated by attack algorithms reconstructed by the trained generator of RIG to eliminate the perturbations of the adversarial examples, which disturb the models for classification, so that the models can restore their labels when classifying the reconstructed images. In addition, to improve the defensive performance of RIG, the attention mechanism (AM) is introduced to enhance the defense effect of RIG, which is called reconstructing images with attention GAN (RIAG). Experiments show that RIG and RIAG can effectively eliminate the perturbations of the adversarial examples. The results also show that RIAG has a better defensive performance than RIG in eliminating the perturbations of adversarial examples, which indicates that the introduction of AM can effectively improve the defense effect of RIG.

In the field of face antispoofing, researchers are increasingly focusing their efforts on multimodal and feature fusion. While multimodal approaches are more effective than single-modal ones, they often come with a huge number of parameters, require significant computational resources, and pose challenges for execution on mobile devices. To address the real-time problem, we propose a fast and lightweight framework based on ShuffleNet V2. Our approach takes patch-level images as input, enhances unit performance by introducing an attention module, and addresses dataset sample imbalance issues through the focal loss function. The framework effectively tackles the real-time constraints of the model. We evaluate the performance of our model on CASIA-FASD, Replay-Attack, and MSU-MFSD datasets. The results demonstrate that our method outperforms the current state-of-the-art methods in both intratest and intertest scenarios. Furthermore, our network has only 0.84 M parameters and 0.81 GFlops, making it suitable for deployment in mobile and real-time settings. Our work can serve as a valuable reference for researchers seeking to develop single-modal face antispoofing methods suitable for mobile and real-time applications.

The implicit neural representation of visual data (such as images, videos, and 3D models) has become a current hotspot in computer vision research. This work proposes a cover selection steganography scheme for neural radiance fields (NeRFs). The message sender first trains an NeRF model selecting any viewpoint in 3D space as the viewpoint key Kv, to generate a unique secret viewpoint image. Subsequently, a message extractor is trained using overfitting to establish a one-to-one mapping between the secret viewpoint image and the secret message. To address the issue of securely transmitting the message extractor in traditional steganography, the message extractor is concealed within a hybrid model performing standard classification tasks. The receiver possesses a shared extractor key Ke, which is used to recover the message extractor from the hybrid model. Then the secret viewpoint image is obtained by NeRF through the viewpoint key Kv, and the secret message is extracted by inputting it into the message extractor. Experimental results demonstrate that the trained message extractor achieves high-speed steganography with a large capacity and attains a 100% message embedding. Additionally, the vast viewpoint key space of NeRF ensures the concealment of the scheme.

As an important source of Earth surface information, remote sensing image has the problems of rough and fuzzy image details and poor perception quality, which affect further analysis and application of geographic information. To address the above problems, we introduce the adversarial feature fusion network with an attention-based mechanism for super-resolution reconstruction of remote sensing images in this paper. First, residual structures are designed in the generator to enhance the deep feature extraction capability of remote sensing images. The residual structure is composed of the depthwise over-parameterized convolution and self-attention mechanism, which work synergistically to extract deep feature information from remote sensing images. Second, coordinate attention feature fusion module is introduced at the feature fusion stage, which can fuse shallow features and deep high-level features. Therefore, it can enhance the attention of the model to different features and better fuse inconsistent semantic features. Finally, we design the pixel-attention upsampling module in the up-sampling stage. It adaptively focuses on the most information-rich regions of a pixel and restores the image details more accurately. We conducted extensive experiments on several remote sensing image datasets, and the results showed that compared with current advanced models, our method can better restore the details in the image and achieve good subjective visual effects, which also verifies the effectiveness and superiority of the algorithm proposed in this paper.

Our study investigates the impact of denoising preprocessing on the accuracy of image segmentation. Specifically, images with Gaussian noise were segmented using the fuzzy c-means method (FCM), local binary fitting (LBF), the adaptive active contour model coupling local and global information (EVOL_LCV), and the U-Net semantic segmentation method. These methods were then quantitatively evaluated. Subsequently, various denoising techniques, such as mean, median, Gaussian, bilateral filtering, and feed-forward denoising convolutional neural network (DnCNN), were applied to the original images, and the segmentation was performed using the methods mentioned above, followed by another round of quantitative evaluations. The two quantitative evaluations revealed that the segmentation results were clearly enhanced after denoising. Specifically, the Dice similarity coefficient of the FCM segmentation improved by 4% to 44%, LBF improved by 16%, and EVOL_LCV presented limited changes. Additionally, the U-Net network trained on denoised images attained a segmentation improvement of over 5%. The accuracy of traditional segmentation and semantic segmentation of Gaussian noise images is improved effectively using DnCNN.

In recent years, deep learning has made significant progress in the field of single-image super-resolution (SISR) reconstruction, which has greatly improved reconstruction quality. However, most of the SISR networks focus too much on increasing the depth of the network in the process of feature extraction and neglect the connections between different levels of features as well as the full use of low-frequency feature information. To address this problem, this work proposes a network based on residual dual-path interactive fusion combined with attention (RDIFCA). Using the dual interactive fusion strategy, the network achieves the effective fusion and multiplexing of high- and low-frequency information while increasing the depth of the network, which significantly enhances the expressive ability of the network. The experimental results show that the proposed RDIFCA network exhibits certain superiority in terms of objective evaluation indexes and visual effects on the Set5, Set14, BSD100, Urban100, and Manga109 test sets.

Crop diseases have an adverse effect on the yield, productivity, and quality of agricultural produce, which threatens the safety and security of the global feature of food supply. Addressing and controlling plant diseases through implementation of timely disease management strategies to reduce their transmission are essential for ensuring minimal crop loss, and addressing the increasing demand for food worldwide as the population continues to increase in a steadfast manner. Crop disease mitigation measures involve preventive monitoring, resulting in early detection and classification of plant diseases for effective agricultural procedure to improve crop yield. Early detection and accurate diagnosis of plant diseases enables farmers to deploy disease management strategies, such interventions are critical for better management contributing to higher crop output by curbing the spread of infection and limiting the extent of damage caused by diseases. We propose and implement a deep and cross residual skip connection separable convolutional neural network (DCRSCSCNN) for identifying and classifying leaf diseases for crops including apple, corn, cucumber, grape, potato, and guava. The significant feature of DCRSCSCNN includes residual skip connection and cross residual skip connection separable convolution block. The usage of residual skip connections assists in fixing the gradient vanishing issue faced by network architecture. The employment of separable convolution decreases the number of parameters, which leads to a model with a reduced size. So far, there has been limited exploration or investigation of leveraging separable convolution within lightweight neural networks. Extensive evaluation of several training and test sets using distinct datasets demonstrate that the proposed DCRSCSCNN outperforms other state-of-the-art approaches. The DCRSCSCNN achieved exceptional classification and identification accuracy rates of 99.89% for apple, 98.72% for corn, 100% for cucumber, 99.78% for grape, 100% for potato, 99.69% for guava1, and 99.08% for guava2 datasets.

Content-based commodity retrieval (CCR) faces two major challenges: (1) commodities in real-world scenarios are often captured randomly by users, resulting in significant variations in image backgrounds, poses, shooting angles, and brightness; and (2) many commodities in the CCR dataset have similar appearances but belong to different brands or distinct products within the same brand. We introduce a CCR neural network called CCR-Net, which incorporates both length loss and salient loss. These two losses can operate independently or collaboratively to enhance retrieval quality. CCR-Net offers several advantages, including the ability to (1) minimize data variations in real-world captured images; and (2) differentiate between images containing highly similar but fundamentally distinct commodities, resulting in improved commodity retrieval capabilities. Comprehensive experiments demonstrate that our CCR-Net achieves state-of-the-art performance on the CUB200-2011, Perfect500k, and Stanford Online Products datasets for commodity retrieval tasks.

Low-light image enhancement, fundamentally an ill-posed problem, seeks to simultaneously provide superior visual effects and preserve the natural appearance. Current methodologies often exhibit limitations in contrast enhancement, noise reduction, and the mitigation of halo artifacts. Negative feedback pulse coupled neural network (NFPCNN) is proposed to provide a well posed solution based on uniform distribution in contrast enhancement. The negative feedback dynamically adjusts the attenuation amplitude of neuron threshold based on recent neuronal ignited state. Neurons in the concentrated brightness area arrange smaller attenuation amplitude to enhance the local contrast, whereas neurons in the sparse area set larger attenuation amplitude. NFPCNN makes up for the negligence of pulse coupled neural network in the brightness distribution of the input image. Consistent with Weber–Fechner law, gamma correction is employed to adjust the output of NFPCNN. Although contrast enhancement can improve detail expressiveness, it might also introduce artifacts or aggravate noise. To mitigate these issues, the bilateral filter is employed to suppress halo artifacts. Brightness is used as coefficient to refine the Relativity-of-Gaussian noise suppression method. Experimental results show that the proposed method can effectively suppress noise while enhancing image contrast.

The attention mechanism has become a crucial technique in deep feature representation for computer vision tasks. Using a similarity matrix, it enhances the current feature point with global context from the feature map of the network. However, the indiscriminate utilization of all information can easily introduce some irrelevant contents, inevitably hampering performance. In response to this challenge, sparsing, a common information filtering strategy, has been applied in many related studies. Regrettably, their filtering processes often lack reliability and adaptability. To address this issue, we first define an adaptive-reciprocal nearest neighbors (A-RNN) relationship. In identifying neighbors, it gains flexibility through learning adaptive thresholds. In addition, by introducing a reciprocity mechanism, the reliability of neighbors is ensured. Then, we use A-RNN to rectify the similarity matrix in the conventional attention module. In the specific implementation, to distinctly consider non-local and local information, we introduce two blocks: the non-local sparse constraint block and the local sparse constraint block. The former utilizes A-RNN to sparsify non-local information, whereas the latter uses adaptive thresholds to sparsify local information. As a result, an adaptive sparse attention (ASA) module is achieved, inheriting the advantages of flexibility and reliability from A-RNN. In the validation for the proposed ASA module, we use it to replace the attention module in NLNet and conduct experiments on semantic segmentation benchmarks including Cityscapes, ADE20K and PASCAL VOC 2012. With the same backbone (ResNet101), our ASA module outperforms the conventional attention module and its some state-of-the-art variants.