2.1

Multilevel memory augmented autoencoder (Multi-MemAE)

The number of layers of the encoder and decoder of the multi-level memory-enhanced auto-encoder (Multi-MemAE) is designed to be four layers respectively, and a basic feature extraction module is placed in each layer of the encoder, followed by a downsampling layer, which is used for decoding. The basic feature processing module used in each layer in the decoder is the same as that of the encoder. The down-sampled features of each layer are fused with the up-sampled feature maps from the corresponding layers of the decoder respectively, and then cascaded through the basic features of each layer in turn. Feature processing module, the basic feature extraction module consists of convolutional layer-BN regularization layer-ReLu activation layer. Downsampling in the Multi-MemAE model is achieved by convolution pooling, and upsampling is achieved by deconvolution. The structure of the Multi-MemAE model is shown in Figure 2.

Figure 2.

Multi-MemAE structure diagram.

The memory enhancement module is constructed as shown in Hu et al.⁹. When training the Multi-MemAE model, based on the idea of weak supervision, the video frame or its optical flow image can be used as input, and then the input image can be reconstructed. The L2 norm is used as the loss function of the model to penalize the distance between the reconstructed image and the input images.

λ_cand λ_d respectively represent the weight coefficients of feature separation and diversity in the memory module.

2.2

Two-stream conditional variational autoencoder (TS-CVAE)

The mathematical expression of the probability distribution of the convolutional neural network model based on the future frame prediction method can be considered as p(I_t+1 | I_1:t), Future frames generated by the model are represented by I_t+1, I_1:t represents a given continuous t frames of video images, I_t+1 is generated by I_1:t. The model takes the RGB image and the optical flow supplementary information as an auxiliary generated image as input information. Its probability distribution can be assumed as p(I_t+1 | I_1:t, H_1:t), H_1:t represents the optical flow image corresponding to the uninterrupted t video frames. The generation of the predicted future frame I_t+1 is jointly determined by the video frame I_t, and the corresponding optical flow image H_t, so it can be regarded as a direct mapping of I_1:t and H_1:t to Convolutional Neural Network model of I_i+1. Considering that the choice of t value in actual model training is usually small, generally between 4 and 6, the content of the video of consecutive t frames and the corresponding optical flow image are very similar, and the video duration is relatively short. According to p(I_t+1 | I_1:t, H_1:t), p(I_i+1 | I_1:t, H_1:t) assumed

to be a conditional variational autoencoder as a generative model for modeling the data, input continuous video frames into the conditional variational autoencoder to extract the latent variable Z of its internal distribution, and then generate the predicted future frame I_t+1 through Z, and use H_1:t as a supplementary condition for the latent variable Z, in order to obtain the optimal distribution p, only when the evidence lower bound ELBO in the variational autoencoder is derived, when the KL divergence values of distribution p(Z|I_1:t) and distribution p(I_t+1|Z) are the smallest, its mathematical expression is:

According to the above equation (3), a two-stream conditional variational autoencoder (TS-CVAE) as shown in Figure 3 is proposed. The model consists of two encoders E_RF and E_Flow, and a decoder D_FFP, the encoder E_Flow encodes the optical flow image H_1:t to obtain the encoded feature E_Flow (H_1:t), and then obtains the prior distribution p(Z|H_1:t), and the encoder E_RF is the video frame I_1:t and the optical flow image H_1:t together. The encoding yields E_RF (I_1:t, H_1:t), and obtain its posterior distribution as q(Z|I_1:t, H_1:t). During training, the TS-CVAE model obtains the latent variable Z from the posterior distribution q(Z|I_1:t, H_1:t), and combines Z with E_Flow (H_1:t) and feeds it into the decoder to generate future frames.

Figure 3.

TS-CVAE structure diagram.

Inspired by existing research, adding skip connections to the method of predicting future frames does not lead to a significant improvement in model performance, so skip connections are not used between encoder E_RF and decoder D_FFP. A variational autoencoder typically assumes the parameters p(I_t+1|Z, H_1:t), q(Z | I_1:t, H_1:t), and p(Z|H_1:t) in equation (3) to be Gaussian distributions. The loss function of conditional variational autoencoder consists of two parts: KL divergence value and prediction frame loss, and its calculation formula is as follows:

In the equation, I_t+1 is the real video frame corresponding to the future frame. In order to make the prediction of future frames clearer, the gradient difference loss function is used to directly penalize the predicted image, and its specific form is as follows:

where α is set to 1 in this experiment. The gradient difference loss function only performs a simple difference between adjacent pixels, in order to reduce the model training time. Combining L_CVAE and L_gdl, the final loss function of the TS-CVAE model is as follows:

Combining the loss functions of the Multi-MemAE part and the TS-CVAE part, the total loss function of the TS- MemCVAE model is as follows:

In the equation, G (•) represents the maximum and minimum normalization, and λ_m and λ_t represent the partial weight coefficients of Multi-MemAE and TS-CVAE, respectively.

2.3

Video frame anomaly score settings

The frame anomaly score of the TS-MemCVAE model during testing consists of two parts: the video frame reconstruction error S_r and the future frame prediction error S_p, and the final frame anomaly score is obtained by fusing the image errors of these two parts^{10, 11}. S_r consists of two parts, the first part is the PSNR of the reconstructed image and the input image, and the second part is composed of three memory modules. S_p is represented by the L2 distance between the predicted image and the input image.

𝜗_r and 𝜗_p in equation (10) represent the weight coefficients of reconstruction error S_r and prediction error S_p, and γ in equation (8) represents the equalization coefficient of reconstruction error. The larger the value of the abnormal score S, the higher the abnormality of the current frame.

3.1

Performance evaluation criteria and experimental details settings

The TS-MemCVAE model is experimented on two benchmark datasets (UCSD-Ped2⁶, CUHK Avenue⁶), and its performance is evaluated using AUC. The model is implemented using Pytorch1.4, the optimizer is Adam, the learning rate is set to 2e-4, and it runs on the experimental platform of NVIDIA Geforce RTX2060 (6G). The size of the images is uniformly adjusted to 224X224, and the pixel range is converted from (0, 255) to (-1, 1). The hyperparameter β, γ, 𝜗_r, 𝜗_p, λ_c, λ_d, λ_m, λ_t are set as: 0.9, 0.01, 0.5, 0.5, 0.15, 0.15, 0.4, 0.6.

3.2

Performance comparison analysis with other models

As can be seen from Table 1, the R-STAE model detects anomalies using a residual spatiotemporal autoencoder composed of 3D convolutions and LSTMs. The DSTN model is to build a GAN structure network, using the optical flow graph as the reconstruction object and the video frame as the auxiliary information, to detect anomalies by the difference between the generated optical flow and the input optical flow. The TS-MemCVAE model takes the video frame as the prediction object and the optical flow images as the supplementary condition. The MGFC-AAE model uses a parallel two-stream variational encoder as the backbone network, and detects anomalies by comparing the difference between the acquired latent variable Z distribution and the prior Gaussian distribution, without using reconstructed images. The GBA-AE model uses a single-encoder dual-decoder network structure to obtain a gradient-based attention feature map with the reconstruction loss of the reconstructed image and the input image, which is fused with the features at the prediction decoder to generate future frames and combined with the ground truth Frame comparison detects anomalies to reconstruct the generalization ability of image-constrained autoencoders. The TS-MemCVAE model uses multiple memory modules to constrain the reconstructed image, and further constrains the predicted image with optical flow information in the prediction task. Compared with R-STAE, DSTN, and MGFC-AAE, the TS-MemCVAE model increases the memory constraint on the generalization of the autoencoder, but the model does not perform well on the CUHK Avenue dataset and cannot extract the intrinsic feature distribution from large datasets well.

Table 1.

AUC comparison of different models on two datasets.

3.3

Experimental results of the model on anomalous datasets

The ROC curves of the TS-MemCVAE model in the UCSD Ped2 dataset and the CUHK Avenue dataset are shown in Figure 4. Figures 5 and 6 show some abnormal behaviors in the abnormal data set (Figures 5a, 5b, and 5c represent the original image, the predicted image, and the difference image, respectively). As can be seen from Figure 6, the model can predict abnormal behaviors such as fast running, and the differential image can relatively clearly show the motion outline of the abnormal behavior. As can be seen from Figure 5, in the UCSD Ped2 dataset, the gap between abnormal behavior and normal behavior is small, and the model contradicts the ROC curve and the predicted results. From the perspective of the composition of the model anomaly score, it can be seen that when the difference between the video frame and the predicted image is small or not at all, the difference between the normal pattern feature in the memory module and the extracted feature is the decisive factor for whether the dominant frame is abnormal or not. The multiple memory modules in the model not only effectively constrain the generalization of the autoencoder, but also play an important role in quantifying the degree of abnormality of video frames.

Figure 4.

ROC curves of the model on both datasets.

Figure 5.

Abnormal behavior detection results of the model on the UCSD Ped2 dataset.

Figure 6.

Abnormal behavior detection results of the model on the CUHK Avenue dataset.

3.4

Ablation experiment

In order to verify the influence of the multi-level memory module in the TS-MemCVAE model and the video frame and its optical flow image in the dual-stream input on its performance, corresponding ablation experiments were performed on each module. The results are shown in Table 2, where RGB represents the input video frame, Flow represents the corresponding optical flow image. It can be seen from Table 2 that the memory module is inserted in the bottleneck and decoding part of the autoencoder. The change of the position of the memory module has little effect on the performance of the model. Multiple memory modules are inserted into different positions of the autoencoder. Generalization plays a considerable constraint role; and the addition of optical flow images enables the model to enhance the acquired motion information features, thereby improving the performance of the model.

Table 2.

Effects of different modules in the model on performance.