2.1. Detection of Foreign Objects on Solar Panel Surfaces Based on Intelligent Algorithms

Compared with large solar panels, the area of local foreign objects, such as bird droppings and leaves, is typically smaller. Therefore, we selected the YOLOv5 algorithm to identify the small local foreign objects ⁷. The YOLOv5 algorithm was chosen for the following reasons. First, compared with the YOLOv1–YOLOv4 algorithms, the structural optimization of the YOLOv5 algorithm significantly improves its detection speed and accuracy. As successors of YOLOv5, YOLOv6–YOLOv8 improves the overall accuracy through structural adjustments. However, this also results in a longer training time, which affects the iterations and optimization of the algorithm model. This may become a limiting factor in application environments that require rapid iterations and deployment. Second, in the application scenarios of small-target detection algorithms, the detection accuracy usually only requires being sufficient, and there is no requirement to pursue ultimate accuracy. In this case, the complex structures of YOLOv6–YOLOv8, which can result in higher accuracy, may not necessarily result in better performance but in higher costs. The YOLOv5 algorithm achieves a good balance between detection accuracy and speed, and its classic model architecture has great potential for improvement. Therefore, this study makes improvements based on YOLOv5. In addition, intelligent robots are bound to encounter low-light scenarios (caused by early mornings, evenings, cloudy days, and obstruction by surrounding obstacles) during their continuous inspection of solar panels. This study used the Retinex algorithm to improve the images and the YOLOv5 algorithm for the detection of foreign objects.

The detailed process is shown below.

1. (1)

   The image captured by the camera was converted from red, green, blue (RGB) to hue, saturation, value (HSV) ⁸.

   \begin{equation} \label{eq:1} \left\{\begin{aligned} S(x) & =\begin{cases} 0, & C_{\max}(x)=0,\\ \dfrac{C_{\max}(x)-C_{\min}(x)}{C_{\max}(x)}, & C_{\max}(x)\ne 0,\\ \end{cases}\\ V(x) & =C_{\max}(x),\\ H(x) & =\begin{cases} &\!\!\!\!\! 0°, ~\,\qquad C_{\max}(x)-C_{\min}(x) =0, \\ &\!\!\!\!\! 60°\cdot \left(\displaystyle\frac{G(x)-B(x)}{C_{\max}(x)-C_{\min}(x)}+0\right),\\ & C_{\max}(x)=R(x),\\ &\!\!\!\!\! 60°\cdot \left(\displaystyle\frac{B(x)-R(x)}{C_{\max}(x)-C_{\min}(x)}+2\right),\\ & C_{\max}(x)=G(x), \\ &\!\!\!\!\! 60°\cdot\left(\displaystyle\frac{R(x)-G(x)}{C_{\max}(x)-C_{\min}(x)}+4\right),\\ & C_{\max}(x)=B(x),\\ \end{cases}\\ \end{aligned}\right. \end{equation}

   where \(R(x)\), \(G(x)\), and \(B(x)\) denotes the gray values of the image pixel \(x\) in the R, G, and B color channels, respectively, \(C_{\max}\) and \(C_{\min}\) denote the maximum and minimum values of the pixel in the RGB three-channel component, respectively, and \(S(x)\), \(V(x)\), and \(H(x)\) denote the saturation, luminance, and hue of the pixel, respectively.

2. (2)

   After converting the image to the color space, it was processed using an image enhancement formula ⁹:

   \begin{equation} \label{eq:2} \hspace{-1.3em} \left\{\begin{aligned} & G_F(x)=\frac{\displaystyle\sum_{y\in T} G_d\left(\| y-x\|\right)\cdot G_r\left(\vert V(y)-V(x)\vert\right)\cdot V(y)}{\displaystyle\sum_{y\in T} G_d\left(\| y-x\|\right)\cdot G_r\left(\vert V(y)-V(x)\vert\right)},\\ & G_d\left(\Vert y-x\Vert\right)=\exp\left(-\dfrac{\Vert y-x\Vert^2}{2\sigma_d^2}\right), \\ & G_r\left(\vert V(y)-V(x)\vert\right)=\exp\left(-\dfrac{\vert V(y)-V(x)\vert^2}{2\sigma_r^2}\right), \\ & V_C(x)=\exp\left(\ln \left(V(x)\right)-\ln\left(G_F(x)\right)\right), \\ & S_C(x)=S(x)+t\cdot \left(V_C(x)-V(x)\right)\cdot\varepsilon, \\ & H_C(x)=H(x), \\ \end{aligned}\right. \end{equation}

   where \(G_F(x)\) denotes the result of the brightness of pixel \(x\) after bilateral filtering, \(G_d(\cdot)\) and \(G_r(\cdot)\) denote Gaussian functions, \(T\) represents the neighboring area centered on pixel \(x\), \(y\) denotes the pixel within \(T\), \(S_C(x)\), \(V_C(x)\), and \(H_C(x)\) denote the saturation, brightness, and hue of the pixel after enhancement, respectively ¹⁰.

   

3. (3)

   The image was restored to the RGB space and fused with the original image using a weighted sum:

   \begin{equation} \label{eq:3} \left\{\begin{aligned} \omega(x) &= 1-\dfrac{1}{\exp\left(\displaystyle\frac{\left(R_C(x),G_C(x),B_C(x)\right)}{255}\right)},\\ F(x) &= \omega(x)\cdot \left(R_C(x),G_C(x),B_C(x)\right)\\ &\phantom{=~} + \left(1-\omega(x)\right)\cdot\left(R(x),G(x),B(x)\right), \\ \end{aligned}\right. \end{equation}

   where \(R_C(x)\), \(G_C(x)\), and \(B_C(x)\) denote the values of the enhanced image pixel \(x\) converted back to the RGB color space, \(\omega(x)\) denotes the fusion weight, and \(F(x)\) denotes the pixel \(x\) after the weighted fusion of the original low-light image and enhanced image.

4. (4)

   The fused enhanced image was input into the YOLOv5 model for forward computation ¹¹ to identify and detect foreign objects. The structure of the YOLOv5 model is shown in Fig. 1. During the layer-by-layer forward computation, feature extraction was first performed on the backbone network, and then computations were performed in two CBS [convolution \(+\) batch normalization \(+\) sigmoid-weighted linear unit (SiLU) activation (CBS)] modules and C3 modules three times. The output was the input to the spatial pyramid pooling-fast (SPPF) module, which comprised two CBS modules, three max-pooling layers, and one splicing layer.

5. (5)

   Different scale features extracted by the backbone network were fused to the neck network. First, the features extracted by the SPFF module ¹² were fused with those extracted by the third C3 module of the backbone network after being processed by the CBS module and upsampled in the neck network. The fusion features continued to fuse with the second C3 module of the backbone network after being processed by the C3 module, CBS module, and upsampling in the neck network. The features obtained from the second fusion continued to be fused with the second CBS module of the neck network after processing with the C3 and CBS modules. The features obtained from the third fusion were processed by the C3 and CBS modules again, fused with the features extracted by the first CBS module of the neck network, and processed by the C3 module.

6. (6)

   Target recognition was performed on the fused features of the neck and head networks. The head network used three different scales of candidate boxes to identify the extracted features of the third and fourth CBS modules and the fourth C3 module of the neck network. First, it was determined whether an object within the candidate box was foreign based on its features. If this was true, a reverse calculation was performed on the grid points of the candidate box to obtain the coordinates of the target box in the original image.

2.2. Cleaning of Foreign Objects on Solar Panel Surfaces by Intelligent Robots

An intelligent robot captures images of the surface of a solar panel using cameras that it carries or sets around the solar panel. The process by which an intelligent robot cleans the surface of a solar panel is as follows: Under normal circumstances, the robot moved regularly along a set route and used a brush or electrostatic adsorption device to clean the dust on the surface of solar panels during movement. Simultaneously, the algorithm mentioned earlier was used to identify foreign objects in the images captured by the camera. When a foreign object was identified, the robot prioritized moving towards the object for cleaning. The cleaning process is as follows:

1. (1)

   The intelligent robot moved along a set route on the surface of the solar panel to clean the dust. The movement route allowed the cleaning device of the robot to cover the entire surface of the solar panel while collecting surface images.

2. (2)

   The intelligent robot used the solar-panel surface foreign-object intelligent detection algorithm described earlier to detect foreign objects on the surface.

3. (3)

   If the intelligent robot did not detect foreign objects on the surface of the solar panel, it continued to move along the set cleaning route. If a foreign object was detected, the robot moved in the direction of the foreign object and cleaned it.

4. (4)

   After cleaning the object for a period set based on the demands, the robot left to avoid obstruction and collected images of the solar panel surface to check if it had been cleaned again. If cleaned, the robot returned to its original set route to continue moving and cleaning. Otherwise, the foreign objects were cleaned.

5. (5)

   If the same foreign object was required to be cleaned after reaching the threshold set based on the demands, an alert was sent to the console to notify the staff to handle it manually.

3.1. Experimental Data

Because there is no publicly available dataset, this study used self-collected solar surface images as the dataset for the simulation experiment. Related images were collected at a local photovoltaic base and in the laboratory. Images of photovoltaic panels with foreign objects, such as bird droppings and leaves, in a real environment were captured at the photovoltaic base. Photovoltaic panels were set up in the laboratory, and foreign objects were placed on the surface for the simulations. Images were captured using a WSD-4K-V1.0 camera (Shenzhen Weishida Technology Co., Ltd., China) with eight million pixels and a maximum resolution of 3,840 \(\times\) 2,160 pixels. The camera was placed 2 m away from the photovoltaic panel. Low-light images were captured at 5 a.m. in a real power generation base and captured in a laboratory by controlling the indoor lighting. A total of 4,216 images were obtained, of which 2,782 were obtained from the power generation base and 1,434 were obtained from the laboratory. The two images are shown in Fig. 2. Seventy percent was used as the training set, whereas 30% was used as the test set.

3.2. Experimental Setup

A Pai’an brand automatic solar panel-cleaning robot (Bazhou Lixun Hardware Appliance Factory, China) was used in this study. Its body size was \(654\times 507\times 232\) mm, with front and rear roller brushes, and its body weight was 27.4 kg. In this study, the intelligent algorithm for detecting foreign objects on solar panel surfaces was divided into two parts: low-light image enhancement and surface foreign object recognition. The former method used an improved Retinex algorithm. To verify its effectiveness, it was compared with traditional Retinex and histogram equalization algorithms ¹³.

The relevant parameters of the YOLOv5 model are listed in Table 1. To verify the effectiveness of the YOLOv5 model, it was compared with a faster region-based convolutional neural network (R-CNN) and YOLOv4 algorithms that were equivalently optimized using parameters.

After verifying the algorithm, to test the cleaning ability of the intelligent robot with the assistance of the YOLOv5 algorithm, an intelligent robot that did not use the YOLOv5 algorithm was compared with another robot that used the YOLOv5 algorithm.

When testing the robot, the test site was set at the local photovoltaic power generation base, and the surface of a photovoltaic panel with a specification of \(3\times 4\) m was selected as the test area. To avoid the impact of the simulated surface foreign object obstruction on the photovoltaic panel power generation during the test, the photovoltaic panel within the test range was temporarily disconnected from other photovoltaic equipment.

Before testing the robot, wet tissue paper fragments were attached to the surface of the photovoltaic panel to simulate the obstruction caused by foreign objects. After the test began, the robot that did not use the proposed algorithm moved along the route set in Fig. 3 for cleaning, and the robot using the proposed algorithm also moved along the route set in Fig. 3 but prioritized cleaning the foreign objects detected on the surface. After the robot reached the starting point of the set route, one round of testing was completed, with a total of 30 rounds of testing.

3.3. Evaluation Indicators

This study uses the edge strength, information entropy, and locally orderless errors to measure the enhancement effect of low-light images. The relevant formulas are as follows:

For the evaluation indicators of the improved Retinex algorithm, the precision, recall rate, and \(F\)-value were used to evaluate the accuracy of the foreign object identification, and the intersection over union (IoU) was used to evaluate the accuracy of the foreign object localization. The relevant formulas are as follows:

The cleaning effect of the intelligent robots was measured based on the average time required to complete one round and the residual amount of foreign objects on the surface after cleaning.

3.4. Experimental Results

As listed in Table 2, the improved Retinex algorithm had the best enhancement effect, followed by the traditional Retinex algorithm, and the histogram equalization algorithm had the worst effect. The reason for this is that the histogram equalization algorithm, in principle, increases image brightness by statistically analyzing the grayscale distribution within the image and then adjusting the grayscale levels to make the grayscale distribution of the image more uniform. Although this method is simple and fast, it is prone to oversaturation when adjusting the grayscale image; that is, partial areas are bright and white, resulting in the loss of some image features. The traditional Retinex algorithm enhances the image by eliminating the light component, which has the advantage of maintaining the original color and achieving a large dynamic compression ratio. However, this algorithm performs different enhancement processes on the R, G, and B color components of the pixels and finally performs weighted fusion, resulting in a change in the color ratio. The principle of the improved Retinex algorithm is to first convert the image from the RGB color space into the HSV space, enhance the saturation (S) and luminance (V), and then fuse the H, S, and V components to achieve enhancement. Because the hue (H) is not adjusted, there is no effect on the color.

Table 3 shows that the YOLOv5 algorithm had the best performance, followed by the YOLOv4 algorithm; the faster R-CNN algorithm had the poorest performance. This is because the faster R-CNN algorithm is a two-stage object detection algorithm that first generates an appropriate candidate box based on image features and then identifies the type of target through the image features within the candidate box. This two-stage judgment process results in an accumulation of errors. The YOLOv4 algorithm is a one-stage object detection algorithm that detects object categories and locations through a single neural network forward propagation. Therefore, it is faster, and its SPPF module increases the receptive field of the network, allowing it to obtain more image features and improve detection accuracy. The YOLOv5 algorithm further simplified the model structure based on YOLOv4.

Finally, the cleaning performances of the intelligent robots were tested. The cleaning time and foreign object residue of the robot without the YOLOv5 algorithm and the robot adopting the YOLOv5 algorithm are listed in Table 4. The robot adopting the YOLOv5 algorithm required significantly less time to clean foreign objects on the surface of photovoltaic panels and left less foreign-object residue. This is because the intelligent robot that did not adopt the YOLOv5 algorithm only cleaned along the set route. It could clean a foreign object on the route; however, if the foreign object was slightly sticky, it continued to remain, and the robot did not return to clean it. Upon detecting foreign surface objects, the robot adopting the YOLOv5 algorithm first moved to the foreign object for cleaning and then checked for residue and cleaned again.