2.1. Attempts at Vermin Detection Using Deep Learning

Modern cloud-based detection systems are leveraging deep learning for automatic animal identification ^2,3. Researchers have suggested methods for extracting facial features to identify individual bears ^4,5, along with techniques using deep learning technologies, such as convolutional neural networks (CNNs) ⁶.

Previous studies, including those by and Varte et al. and Korkmaz et al., achieved high detection accuracy using CNN-based models such as the YOLO series ^7,8. However, these methods rely on supervised learning, which requires annotated datasets containing thousands of images of each species. Changing the detection target necessitates retraining, which limits flexibility in real agricultural settings where diverse vermin appear. Furthermore, inference requires high-performance GPU resources, which increases the deployment costs. By contrast, the multimodal LLM (GPT-4o) used in this study can identify a wide variety of animals without further training (zero-shot), owing to its extensive pre-training. This negates the need for expensive dataset construction. Its natural language control allows for the immediate designation of any animal as a detection target, overcoming the “limited class” restrictions mentioned by Korkmaz et al. ⁸.

2.2. Multimodal LLM

A multimodal LLM is an LLM capable of simultaneously processing natural language text alongside information such as images, videos, and audio ^9,10.

These systems can determine image content when image and natural language instructions are provided.

They can recognize image content to a certain extent without further training. Although locally available models can undergo further training, their zero-shot performance without additional training has also been discussed and verified. This study focused on the zero-shot performance of multimodal LLMs.

In this study, we selected GPT-4o as a representative commercial multimodal LLM. Generally, commercial model selection involves a tradeoff between “reasoning accuracy” and “operational efficiency (speed and cost).” For instance, models such as Gemini 1.5 Pro excel in accuracy but incur high costs, whereas lightweight models such as Gemini 1.5 Flash offer efficiency but tend to exhibit low accuracy in complex reasoning tasks ¹¹. In this regard, GPT-4o stands out as a representative model that can effectively resolve this tradeoff. According to “GPT-4o System Card,” compared to its predecessor GPT-4V, GPT-4o not only demonstrates enhanced multimodal reasoning capabilities but also achieves significant improvements in response speed and operational cost-efficiency ¹². Therefore, we selected GPT-4o for its excellent balance between the speed, price, and accuracy required for practical monitoring tasks.

2.3. Differences from Existing Commercial Wildlife Monitoring

Vermin control has traditionally relied on physical methods such as electric fences and capture traps. To improve efficiency, various “smart agriculture” monitoring systems have been commercialized. For example, TRELink (GISupply) integrates sensors with traps to provide hunters with instant alerts upon successful capture ^d. Similarly, the “Smart Donan” project also employs a capture notification system for management of the brown bear population ^b. These systems are effective for post-capture notifications but cannot monitor animal behavior before they enter the trap. Edge artificial intelligence (AI) cameras that have emerged in recent years provide real-time detection but have significant limitations in terms of adaptability. These cameras typically run pre-trained models fixed to specific species. If a new type of vermin is detected, the system must be retrained, which is difficult for facility managers who lack the technical expertise. Furthermore, the expense of edge hardware with GPUs renders their widespread implementation in agricultural regions unaffordable.

2.4. Comparison of Agricultural Robots and Conventional Systems

Although the use of robotics in smart agriculture is advancing, significant challenges remain in terms of cost and practicality. Oliveira et al. comprehensively reviewed agricultural robots and emphasized that high implementation costs and the need for specialized technical knowledge to maintain complex systems are the main factors hindering their adoption by farmers ¹³. To address this economic barrier, Spina et al. proposed developing low-cost robots specialized for image data collection ¹⁴. However, despite cost-reducing efforts, robots equipped with mobility mechanisms remain significantly more expensive than static sensor cameras. Therefore, deploying many robots to cover extensive mountainous areas is impractical. The proposed system is free of moving parts and incorporates economical off-the-shelf sensor cameras. This resolves the “high implementation costs” and “technical complexity” identified as challenges in prior research, enabling easy implementation and operation, even by administrators without specialized knowledge.

4.1. Field Deployment and Data Collection

Cameras were installed and used at three locations in Nanae-cho, Hokkaido, Japan. Fig. 4 shows sample images captured at each of these locations.

The camera at Site A was operated from May 15 to September 30, 2025. At Sites B and C, the camera was operated from July 9 to September 30, 2025. A total of 1,073 images were obtained, 22 of which showed the target vermin.

There were nine deer, 12 bears, and one crow. Sample images of the harmful animals photographed are shown in Fig. 5. The set of images included numerous photographs of birds and those without animals, as well as the 22 images containing harmful animals. Experiments were conducted using these images.

4.2. Operational Metrics and Field Evaluation

To evaluate the feasibility of the proposed system, we assessed operational metrics such as costs, delays, reliability, and user feedback during the implementation period. In terms of cost, the system was highly cost-effective. The total running cost for processing 1,073 images during the operational period was $8.89. The breakdown is as follows: GPT-4o API usage fees were $4.00 (May: $0.60, June: $0.77, July: $0.69, August: $1.27, and September: $0.67), and AWS infrastructure costs were $4.89 (May: $0.74, June: $0.87, July: $1.11, August: $1.22, and September: $0.95). In terms of system performance, the average delay from image capture to email notification was approximately 1 min, providing near-real-time alerts that were sufficient for facility managers to take action. In terms of reliability, the system operated without battery depletion or software crashes throughout this period. However, a communication outage occurred at Location A for Camera 1 from August 31 to September 26 because of an oversight in managing the renewal of the SIM card contract. This shows the importance of communication management in extended field operations. Furthermore, feedback from facility managers was positive. We confirmed that, upon receiving bear notifications, managers immediately conducted on-site inspections. Managers have expressed their desire to use this system for a long time owing to its benefits.

5.1. Models Used in the Experiment

Note that this experiment compared the models operating under different paradigms. GPT-4o and LLaVA perform visual question answering (prompt-based text response generation), whereas YOLO-World and Grounding DINO perform open-vocabulary object detection (bounding box prediction based on keywords). Although their internal mechanisms and output formats differ, the primary goal of the proposed system is to detect specific animals defined by user input and notify facility managers. In this application, converting detection results into binary “presence/absence” data enables direct comparison of practicality. Hence, we evaluated all models using this common metric to determine which paradigm (that is, multimodal LLMs or open-vocabulary detectors) was more effective for the zero-shot design of our vermin monitoring system.

We selected GPT-4o as the leading commercial multimodal LLM (as of August 2024), LLaVA as a widely used open-source vision-language model suitable for local deployment, and YOLO-World and Grounding DINO as state-of-the-art open-vocabulary object detectors. This selection covers both commercial and open-source options as well as different technical paradigms (visual question-answering vs. detection-based approaches).

5.2. Implementation Details

The parameters and environment used in this experiment were as follows.

GPT-4o configuration: We accessed GPT-4o via the OpenAI API using model version gpt-4o-2024-08-06. The parameters were set to \(\textrm{temperature}=0\), \(\textrm{top\_p}=0\), and \(\textrm{seed}=42\), with \(\textrm{max\_tokens}=1200\). The following prompts were used in the experiment.

System prompt: “You are an animal identification AI. Respond with exactly one word from the given options.”

User prompt: “Look at this image and identify what animal is shown. Answer with only one word from these options: bear, deer, crow, fox, none. If none of these animals are visible in the image, answer “none”.”

LLaVA environment: LLaVA 7B was executed locally on the Ollama platform (version 0.14.2). The inference was performed on a PC equipped with an NVIDIA GeForce RTX 4080 GPU (16 GB VRAM). We employed prompts identical to those used for GPT-4o.

Object detector settings: For YOLO-World, we used the YOLO-World-S size model. The target classes were set as “bear,” “deer,” “crow,” and “fox.” We set both the confidence and intersection over union thresholds to 0.5. Similarly, for Grounding DINO, we used the Grounding DINO-T size model with the same target classes. The confidence threshold was set to 0.5, and detections below this value were classified as “none.”

5.3. Experimental Results

The confusion matrices for the inference results of each model are shown in Figs. 6–9. In the figures, the vertical and horizontal axes represent the ground-truth and inference results, respectively.

GPT-4o correctly inferred all 22 detected vermin images. However, it incorrectly inferred bears, deer, crows, and foxes in 17 images in which no animals were present. Examples of these failed inferences are shown in Fig. 10. These images showed vehicle bodies but no animals.

LLaVA often struggled to identify bears, deer, and crows in images. It even mistakenly identified an empty image as containing a “deer.” Fig. 11 shows an example of these inference failures. LLaVA made mistakes in interpreting images of vermin, such as a deer in sunlight and a bear viewed from behind.

YOLO-World misidentified birds as bears in several instances. This method missed the actual vermin in some of the images. Fig. 12 presents examples of these incorrect detections.

Grounding DINO also struggled in this task and failed to identify vermin other than bears. It frequently detected crows where none existed and often misclassified deer as foxes. Examples of these failed inferences are shown in Fig. 13.

We calculated and evaluated the precision, recall, F1-score, and accuracy of the confusion matrix by comparing the model inference results with the ground truth.

• \(\mathrm{TP}\) (true positive): Number of positive classes correctly inferred as positive

• \(\mathrm{FP}\) (false positive): Number of negative classes incorrectly inferred as positive

• \(\mathrm{FN}\) (false negative): Number of positive classes incorrectly inferred as negative

• \(\mathrm{TN}\) (true negative): Number of negative classes correctly inferred as negative

Each metric was calculated using the below formulas. These metrics were then used for evaluation.

Table 1 presents the results for each metric. All models achieved accuracy scores greater than 0.97.

The overall accuracy was high; however, this was partly because of the “none” class being assigned to various images, meaning no object was detected. To obtain a true sense of performance, we must consider the precision for each individual class rather than just the overall accuracy.

The performance per class was analyzed. For bears, Grounding DINO and GPT-4o yielded high F1-scores, and GPT-4o achieved a recall of 1.00. This indicates that there were no detection failures. For deer, GPT-4o achieved a recall of 1.00, and no detection failure was observed. However, the precision was low at 0.64.

LLaVA consistently performed well, with high average scores in every category. However, only GPT-4o could identify “crow.” All models scored zero on “fox” because no ground truth data were available.