2.1. NR-IQA Methods for Synthetic Distortion

Artificial synthetic distortion commonly includes Gaussian noise, motion blur, and JPEG compression distortion; i.e., image quality is degraded by adding various degradation factors artificially. Traditional methods for synthetic distortion are generally divided into those based on natural scene statistics (NSS) and those based on feature learning. These methods first extract hand-designed features from the image and then apply a shallow regression model to regress these features into a predicted quality score. NSS-based methods combine space- or frequency-domain features of the image for quality assessment ⁵. Feature-based learning methods, such as CORNIA ⁶, automatically learn quality-aware features from data. Although such blind image quality evaluation methods based on manual feature extraction can be good at predicting image quality scores for specific distortion types, manual features are often difficult to design, and shallow regression models. This makes it difficult to cope with complex distortion information in the real world, resulting in a lack of prediction accuracy.

With the popularity of deep learning, image quality evaluation methods for deep learning have been introduced successively ⁷. Thanks to the shared parameters of multi-task learning, the risk of model overfitting can be reduced, but it cannot be applied to real distorted images. To alleviate the difficulties in training the network due to the small training dataset in the NR-IQA task, Wu et al. ⁸ proposed a lightweight quality evaluation method to extract the hierarchical features of images, which greatly reduces the parameter complexity of the network while ensuring prediction accuracy. However, it cannot be applied to real distorted images because there is no reference image for real distorted images to train the generator.

Although many of the NR-IQA methods described above have made great advances in the performance of synthetic distortion databases, they cannot be applied to real distorted images. They are not truly general NR-IQA methods and still present challenges when dealing with large-scale data.

2.2. NR-IQA Methods for Authentic Distortion

Approaches for deep learning have made great breakthroughs in the quality evaluation of true distortion. To consider true distortion on top of processing synthetic distorted images, a deep bilinear CNN for quality prediction was proposed by Zhang et al. ⁹, and an end-to-end solution was achieved for quality assessment of synthetic and true distorted images. Su et al. ³ proposed an adaptive hyper network to cope with true distortion, dividing the quality assessment process into three steps: content understanding, perceptual rule, and quality prediction. Zhu et al. ¹⁰ used meta-learning to learn a prior model applied to image quality to improve the accuracy of predictions. Based on this, Zhu et al. ¹¹ further proposed MetaIQA+ to select representative synthetic distortion tasks for the meta-training process through a task selection strategy, effectively reducing the requirement for training data and significantly improving the generalization capability of the model.

In summary, most of the literatures use residual networks with traditional convolution focusing mostly on distorted information about local relationships and failing to capture the global relevance features of images. We build on existing methods to take full account of the perceptual mechanism of HVS and focus on the impact of image semantics on perceptual quality. We also adaptively construct the connection between local regions and the global space in the network model, and initialize the parameters of the model through meta-learning method to improve model generalization capability.

3.1. Meta-Model Based on Attention Mechanism

In IQA scenarios, image regions with objects of interest or relevant to quality assessment tasks usually receive more attention than other regions, so the attention mechanism can effectively enhance the performance of assessment methods and solve the problems of traditional standard convolution. The attention mechanism introduced in this paper can effectively utilize global information and give different attention to focus on pixel-level semantic information to obtain more accurate feature information. We introduce a pixel-level spatial attention mechanism and a channel attention mechanism PSA based on the meta-model, which are added after the first \(3\times3\) convolutional layer of the model network bottleneck.

For robustness and computational efficiency, commonly used networks such as residual networks, the attention module in this paper has not been greatly compressed in both spatial and channel dimension, keeping the size of \([H, W]\) in the spatial dimension and using the size of \(C\)/2 in the channel dimension.

In the channel attention calculation, the feature maps are converted into \(V\) of size \(C/2\times H\times W\) and \(Q\) of size \(1\times H\times W\) by \(1\times 1\) convolution, where the channel of \(Q\)s completely compressed while the channel dimension of \(V\)s still kept at a relatively high level. \(X_{ch}^1\) and \(X_{ch}^2\) are obtained by shape reshaping \(V\) and \(Q\). Since the channel dimension of \(X_{ch}^2\) is compressed, softmax is used to enhance its information. Then matrix multiplication is performed to obtain \(X_{ch}\), denoted as:

When evaluating the image quality, viewers usually pay different attention to different image regions. Similar to channel branching, in the spatial attention mechanism, the feature maps are converted into \(V\) and \(Q\) of size \(C/2\times H\times W\) by \(1\times 1\) convolution. For feature \(Q\), the spatial dimension is compressed using global pooling, which is converted to a size of \(1\times1\), while the spatial dimension of \(V\) features is kept at a relatively large level. After reshaping \(X_{sp}^1\) and \(X_{sp}^2\) are obtained, since the spatial dimension of \(X_{sp}^2\) is compressed, similarly its information is augmented by softmax. \(X_{sp}\) is obtained by matrix multiplication, denoted as:

Then use reshaping to get \(X_{sp}^*\) continue using sigmoid to get the spatial weights, which can be calculated as:

Inspired by early psycho-visual experiments that the critical frequency of contrast sensitivity may be influenced by spatial attention, the spatial and channel weights were multiplied with the original feature maps in their respective dimensions, and the results of the two branches were fused to obtain the final feature maps as follows:

Our attention module fully exploits the modeling capability of the self-attentive mechanism and achieves a more effective long-range modeling with guaranteed computational efficiency. At the same time, the softmax–sigmoid combination makes full use of the high-resolution information stored in the attention module branch.

3.2. Semantic Feature Extraction

For human visual system, the top-down visual perception mechanism has verified that human can comprehend the content of an image before perceiving the quality of the image. However, many current NR-IQA methods ignore the extraction of image content information and lead to a gap with human visual perception. To simulate the working mechanism of HVS, Li et al. ⁴ fused the quality evaluation task with the semantic recognition task by using a pre-trained image classification network to extract the semantic information in the image, where the features of the last convolutional layer are used to fit the quality score.

In this paper, we combine low-level detail information and deep semantic information to take advantage of the complementary feature strengths between different convolutional layers, which facilitates the network to capture distortion information of distorted images. To extract richer content-aware features, we extract multi-level features from the convolutional layers: Layer 1 (\(256\times56\times56\)), Layer 2 (\(512\times28\times28\)), and convolutional Layer 3 (\(1024\times14\times14\)) in ResNet50 instead of using only the deep semantic features of the last convolutional layer. The local feature extraction module is illustrated in Fig. 3.

To reduce the network parameters for fast computation, \(1\times1\) convolution is applied in the local feature extraction branch to reduce the dimensionality. Moreover, the global average pooling operation is performed to reduce the dimension while retaining more picture information for feature fusion. Finally, the refined multi-level features are mapped into vectors by a fully connected layer. With the introduction of multilevel features, the network can now be defined as follows:

3.3. Adaptive Quality Prediction

After combining multi-level features and attention mechanisms, the final quality prediction model can be expressed as:

4.1. Implementation Details

The overall training process of our proposed network consists of the following two steps: (1) during the meta-training phase, the residual networks RestNet50 is trained using the specific distortion NR-IQA tasks constructed from synthetic distortion datasets; (2) during the meta-testing phase, the content-aware network based on the attention mechanism is trained on each dataset. During network training, the network parameters were uniformly trained 16 times using an Adam optimizer with the weight decay rate of 5e\(^{-4}\) and learning rate of 2e\(^{-5}\). Evaluation metrics were used to measure the predicted results using Spearman’s rank order correlation coefficient (SROCC) and Pearson linear correlation coefficient (PLCC).

This paper uses six public datasets: TID2013 ¹², Kadid-10k ¹³, LIVE ¹⁴, CSIQ ¹⁵, LIVEC ¹⁶, and KonIQ-10k ¹⁷ datasets. We use meta-learning for training and testing. Meta-learning involves two levels of validation: one is intra-task validation, i.e., query set, whose loss is used to calculate meta-gradients, which directly participate in the update of outer parameters; the other is inter-task validation, i.e., meta-validation set, to prevent meta-overfitting. When selecting a model for performance testing, we stop early on the validation set. In the experiment, the training set contained 16,749 images, and the validation set contained 3,140 images.

The input images are randomly sampled and horizontally flipped into 25 image blocks of size \(224\times224\) pixels, during training, and the scores of the image blocks sampled in each image are averaged to obtain the final prediction scores during testing. Since the datasets used are all small sample datasets, selecting the appropriate batch-size when adjusting the model parameters can improve the training efficiency and accuracy, so the batch-size of each dataset is adjusted while other parameters are determined: batch-size is 72 for the KonIQ-10k dataset, 96 for the LIVEC dataset, and 72 for the LIVE and CSIQ datasets. Finally, the predicted quality scores and labeled scores are trained by minimizing \(L\) loss function between them, and the calculation process is as follows:

In this paper, the number of parameters is 21.3M and the computational cost is 3.2G.

4.2. Performance Evaluation on the Overall Datasets

In this paper, “accuracy" refers to the ability of an objective evaluation algorithm to predict subjective perceptions. Specifically, it is quantified as follows: PLCC (high) indicates accurate prediction (accuracy), and SROCC (high) indicates correct ranking (monotonicity).

To evaluate the prediction accuracy of our model, eleven representative IQA methods were selected for experimental comparison, including the manual feature extraction-based methods ⁵,⁶,¹⁸, the deep learning-based synthetic distortion IQA methods ⁷,⁸,¹⁹, and the deep learning-based authentic distortion IQA methods ³,⁴,⁹,¹⁰,¹¹. The reported results are listed in Table 1.

As can be seen from the data in Table 1, the best results are highlighted in bold, our proposed method achieves SROCC results of 0.884, 0.917, 0.965, 0.975 and PLCC results of 0.895, 0.930, 0.967, 0.974 on the datasets LIVEC, Koniq-10k, CSIQ, and LIVE, respectively. Meanwhile, our method achieves optimal prediction accuracy for all except the CSIQ dataset, and also achieves suboptimal prediction accuracy on CSIQ after DBCNN. For the average prediction results of the four datasets, our method improves 2.41% for SROCC and 2.39% for PLCC, respectively, over the DBCNN method.

Next, we discuss the experimental results in the following three aspects.

First, the prediction accuracy of our model is better than such methods on most of the datasets compared to the three hand-extracted feature-based methods. The experimental results show that our model can learn more information than the hand-extracted feature-based methods.

Second, our model achieves optimal results for SROCC on all four datasets compared to the synthetic IQA models based on deep learning. In contrast, our method also shows greater potential and excellent prediction accuracy on synthetic datasets, with PLCC results on CSIQ second only to the WaDIQaM method. Our method outperforms the SROCC results of WaDIQaM by about 1.05% on CSIQ and by about 2.20% on LIVE. For the PLCC results, our method outperforms WaDIQaM-NR by about 1.14% on LIVE.

Finally, our model achieves optimal prediction results for both synthetic and authentic distortion compared to the authentic IQA method based on deep learning. Compared with the method SFA, which extracts deep semantic features using convolutional networks, our model achieves significantly better results on KonIQ-10k and LIVEC. Compared with the HyperIQA method for the same content perception, it is verified that the proposed attention mechanism could further refine the features to match the perceptual properties of the human eye. Our method outperforms its SROCC by about 2.91% and PLCC by 1.47% on LIVEC, SROCC by about 1.21% and PLCC by 1.41% on Koniq-10k, SROCC by about 4.55% and PLCC by 2.65% on CSIQ, SROCC by about 1.35% and PLCC by 0.83% on LIVE, and 0.83% PLCC on LIVE.

Overall, the proposed method achieves comprehensive optimal prediction accuracy thanks to three aspects:

1. (1)

   The pre-trained meta-model on the synthetic dataset is able to learn more distortion-related knowledge and adapt well to untrained distortions.

2. (2)

   The proposed local feature extraction module incorporates the attention mechanism into multi-level features.

3. (3)

   The deep semantic features refined by the attention mechanism achieve content-aware distorted image quality prediction.

4.3. Performance Evaluation on Single Distortion Datasets

To verify the prediction accuracy of the proposed method for different distortion types, experiments are conducted in this section on synthetic distortion datasets LIVE and CSIQ. To carry out the experimental procedure, the models of all methods are trained using all the distortion types and tested on the specific distortion types.

Table 2 presents the SROCC results of each method on the LIVE and CSIQ datasets. It can be seen from the left of Table 2, our method achieves the highest prediction accuracy on JP2K, WN, GB, and FF four distortion types, achieving SROCC results of 0.965, 0.988, 0.978, and 0.957, respectively. The prediction results for JPEG distortion type are second only to DBCNN.

It can be clearly noticed from the left of Table 2, our method obtains the best results on JPEG, JP2K, PN, and CC. Specifically, our model outperforms WaDIQaM on JPEG by about 14.89%, JP2K by about 2.75%, PN by about 9.86%, and CC by about 2.93%.

In general, our method outperforms other methods in 8 out of 11 distortion types, mainly because the image content information combined with the attention mechanism is more helpful for learning the perceptual quality, while the meta-learning further improves the generalization performance of the model for various distortion types.

4.4. Ablation Studies

In this section, ablation studies are conducted to verify the effectiveness of each key component of the proposed method. In this paper, the LIVE and LIVEC datasets are selected for the ablation experiments, and the experimental results are presented in Table 3. We use the pre-trained ResNet50 as the baseline model and analyze the impact of each component by SROCC and PLCC metrics. First, Meta, a meta-training component, is added to the baseline model so that the model acquires prior knowledge of distortion trained on synthetic distortion datasets, which to some extent compensates for the pre-trained classification network’s insensitivity to image distortion. Secondly, a meta-testing part is added to the baseline model to extract multi-level features and the image content perception process MTContent, so that the model acquires content perception rules. Then, the attention mechanism PSA is added on top of it to simulate the attention distribution of human eyes when observing images. Finally, the final results are obtained by combining all modules through a meta-learning framework. As can be seen in Table 3, the SROCC and PLCC metrics show a large improvement on the two datasets, which proves the effectiveness of our proposed model.

Taking the images in the LIVEC and KonIQ datasets as examples, the subjective quality scores of the 20 distorted images and the corresponding quality scores predicted by our model are given in this paper. As presented in Fig. 4, the first row is the subjective quality score, the second row is the predicted score with the MTContent module added, and the third row is the predicted score of our model. It can be seen that the multi-level features in the added MTContent module mimic the information generated by the human eye in evaluating an image, and the combination of deep semantic information can eliminate to a certain extent the impact of image content changes and local distortion on quality prediction, and retain more detailed features that are more sensitive to human eye vision, thus achieving reliable prediction results. In addition, after adding the meta-learning process and attention mechanism, the predicted values of our model for most distorted images are basically consistent with the subjective quality scores, which further validates the robustness of the model in this paper to different distortions and the accuracy of predicting image quality.