Fisheye cameras, with their wide field of view, are highly beneficial for applications such as unmanned systems. Solving the perspective-n-point (PnP) problem that estimates the pose of a camera is particularly important for fisheye cameras. However, most existing PnP methods are primarily developed for pinhole cameras, with limited research focusing on fisheye cameras. This study presents an efficient method for solving the PnP problem for fisheye cameras. The proposed method introduces a mapping from a fisheye camera image to a virtual image plane based on the geometric properties of the fisheye camera. Two-dimensional reference points on the virtual plane are then employed to solve the PnP problem. Error analysis validates the effectiveness of the mapping through numerical computation. The experimental results further demonstrate that the proposed method achieves better accuracy and efficiency than the existing methods.

1. [1] Y. Qian, M. Yang, and J. M. Dolan, “Survey on fish-eye cameras and their applications in intelligent vehicles,” IEEE Trans. on Intelligent Transportation Systems, Vol.23, No.12, pp. 22755-22771, 2022. https://doi.org/10.1109/TITS.2022.3210409
2. [2] K. Terabayashi et al., “Measurement of three-dimensional environment with a fish-eye camera based on structure from motion – error analysis,” J. Robot. Mechatron., Vol.21, No.6, pp. 680-688, 2009. https://doi.org/10.20965/jrm.2009.p0680
3. [3] K. Terabayashi, T. Morita, H. Okamoto, and K. Umeda, “3D measurement using a fish-eye camera based on EPI analysis,” J. Robot. Mechatron., Vol.24, No.4, pp. 677-685, 2012. https://doi.org/10.20965/jrm.2012.p0677
4. [4] Z. Yang, “Large scale visual SLAM with single fisheye camera,” 2014 Int. Conf. on Audio, Language and Image Processing, pp. 138-142, 2014. https://doi.org/10.1109/ICALIP.2014.7009773
5. [5] Y. Wang et al., “CubemapSLAM: A piecewise-pinhole monocular fisheye SLAM system,” Proc. of the 14th Asian Conf. on Computer Vision, Part 6, pp. 34-49, 2019. https://doi.org/10.1007/978-3-030-20876-9_3
6. [6] Y.-W. Choi, J.-W. Choi, S.-G. Im, D. Qian, and S.-G. Lee, “Multi-robot avoidance control based on omni-directional visual SLAM with a fisheye lens camera,” Int. J. of Precision Engineering and Manufacturing, Vol.19, No.10, pp. 1467-1476, 2018. https://doi.org/10.1007/s12541-018-0173-1
7. [7] S. Bhamidipati and G. X. Gao, “SLAM-based integrity monitoring using GPS and fish-eye camera,” Proc. of the 32nd Int. Technical Meeting of the Satellite Division of the Institute of Navigation, pp. 4116-4129, 2019. https://doi.org/10.33012/2019.17117
8. [8] L. Gallagher, G. Sistu, J. Horgan, and J. B. McDonald, “A system for dense monocular mapping with a fisheye camera,” 2023 IEEE/CVF Conf. on Computer Vision and Pattern Recognition, pp. 6479-6487, 2023. https://doi.org/10.1109/CVPRW59228.2023.00689
9. [9] H. Zhu, G. Zhang, Z. Ye, and H. Zhou, “LVIF: A lightweight tightly coupled stereo-inertial SLAM with fisheye camera,” Complex & Intelligent Systems, Vol.10, No.1, pp. 763-780, 2024. https://doi.org/10.1007/s40747-023-01190-5
10. [10] T. Sakuda, H. Chikugo, K. Arai, S. Pathak, and K. Umeda, “Estimation of road surface plane and object height focusing on the division scale in disparity image using fisheye stereo camera,” J. Robot. Mechatron., Vol.35, No.5, pp. 1354-1365, 2023. https://doi.org/10.20965/jrm.2023.p1354
11. [11] P. D. Fiore, “Efficient linear solution of exterior orientation,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.23, No.2, pp. 140-148, 2001. https://doi.org/10.1109/34.908965
12. [12] A. Ansar and K. Daniilidis, “Linear pose estimation from points or lines,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.25, No.5, pp. 578-589, 2003. https://doi.org/10.1109/TPAMI.2003.1195992
13. [13] Y. Zheng, Y. Kuang, S. Sugimoto, K. Aström, and M. Okutomi, “Revisiting the PnP problem: A fast, general and optimal solution,” 2013 IEEE Int. Conf. on Computer Vision, pp. 2344-2351, 2013. https://doi.org/10.1109/ICCV.2013.291
14. [14] L. Kneip, P. Furgale, and R. Siegwart, “Using multi-camera systems in robotics: Efficient solutions to the NPnP problem,” 2013 IEEE Int. Conf. on Robotics and Automation, pp. 3770-3776, 2013. https://doi.org/10.1109/ICRA.2013.6631107
15. [15] L. Kneip, H. Li, and Y. Seo, “UPnP: An optimal O(n) solution to the absolute pose problem with universal applicability,” Proc. of the 13th European Conf. on Computer Vision, Part 1, pp. 127-142, 2014. https://doi.org/10.1007/978-3-319-10590-1_9
16. [16] Y. He, S. Li, and Q. Guo, “Solving perspective-n-point problem with spherical regression,” IOP Conf. Series: Earth and Environmental Science, Vol.234, Article No.012074, 2019. https://doi.org/10.1088/1755-1315/234/1/012074
17. [17] B. Li, Y. Shang, B. Guan, S. Liang, and Q. Yu, “Solving generalized pose problem of central and non-central cameras,” Proc. of the 6th Chinese Conf. on Pattern Recognition and Computer Vision, Part 2, pp. 180-192, 2024. https://doi.org/10.1007/978-981-99-8432-9_15
18. [18] E. Brachmann et al., “DSAC—Differentiable RANSAC for camera localization,” 2017 IEEE Conf. on Computer Vision and Pattern Recognition, pp. 2492-2500, 2017. https://doi.org/10.1109/CVPR.2017.267
19. [19] A. Kendall, M. Grimes, and R. Cipolla, “PoseNet: A convolutional network for real-time 6-DOF camera relocalization,” 2015 IEEE Int. Conf. on Computer Vision, pp. 2938-2946, 2015. https://doi.org/10.1109/ICCV.2015.336
20. [20] V. Lepetit, F. Moreno-Noguer, and P. Fua, “EPnP: An accurate O(n) solution to the PnP problem,” Int. J. of Computer Vision, Vol.81, No.2, pp. 155-166, 2009. https://doi.org/10.1007/s11263-008-0152-6
21. [21] H. Ding, K. Liu, P. Chen, and H. Chen, “A cooperative target 3D tracking method based on EPnP and adaptive Kalman filter,” Proc. of 2019 Chinese Intelligent Automation Conf., pp. 580-591, 2020. https://doi.org/10.1007/978-981-32-9050-1_66
22. [22] Y. Guo, C. Bian, Y. Zhang, and J. Gao, “An EPnP based extended Kalman filtering approach for docking pose estimation of AUVs,” Proc. of 2021 Int. Conf. on Autonomous Unmanned Systems, pp. 2658-2667, 2022. https://doi.org/10.1007/978-981-16-9492-9_261
23. [23] L. Su et al., “Heuristic EPnP-based pose estimation for underground machine tracking,” Symmetry, Vol.14, No.2, Article No.385, 2022. https://doi.org/10.3390/sym14020385
24. [24] X. Xie and D. Zou, “Depth-based efficient PnP: A rapid and accurate method for camera pose estimation,” IEEE Robotics and Automation Letters, Vol.9, No.11, pp. 9287-9294, 2024. https://doi.org/10.1109/LRA.2024.3438037
25. [25] B. Wang, A. Bai, F. Ma, and P. Ji, “A 3D object localization method based on EPNP and dual-view images,” 2024 IEEE 7th Advanced Information Technology, Electronic and Automation Control Conf., pp. 1349-1353, 2024. https://doi.org/10.1109/IAEAC59436.2024.10503831
26. [26] A. Penate-Sanchez, J. Andrade-Cetto, and F. Moreno-Noguer, “Exhaustive linearization for robust camera pose and focal length estimation,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.35, No.10, pp. 2387-2400, 2013. https://doi.org/10.1109/TPAMI.2013.36
27. [27] F. Deng, Y. Wu, Y. Hu, and H. Cui, “Position and pose estimation of spherical panoramic image with improved EPnP algorithm,” Acta Geodaetica et Cartographica Sinica, Vol.45, No.6, pp. 677-684, 2016 (in Chinese).
28. [28] X. Gong, Y. Lv, X. Xu, Y. Wang, and M. Li, “Pose estimation of omnidirectional camera with improved EPnP algorithm,” Sensors, Vol.21, No.12, Article No.4008, 2021. https://doi.org/10.3390/s21124008
29. [29] J. Kannala and S. S. Brandt, “A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.28, No.8, pp. 1335-1340, 2006. https://doi.org/10.1109/TPAMI.2006.153
30. [30] D. Scaramuzza, A. Martinelli, and R. Siegwart, “A flexible technique for accurate omnidirectional camera calibration and structure from motion,” 4th IEEE Int. Conf. on Computer Vision Systems, 2006. https://doi.org/10.1109/ICVS.2006.3
31. [31] C. Mei and P. Rives, “Single view point omnidirectional camera calibration from planar grids,” Proc. of 2007 IEEE Int. Conf. on Robotics and Automation, pp. 3945-3950, 2007. https://doi.org/10.1109/ROBOT.2007.364084
32. [32] J. P. Barreto and H. Araujo, “Issues on the geometry of central catadioptric image formation,” Proc. of the 2001 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, Vol.2, pp. 422-427, 2001. https://doi.org/10.1109/CVPR.2001.990992
33. [33] C. Geyer and K. Daniilidis, “A unifying theory for central panoramic systems and practical implications,” Proc. of the 6th European Conf. on Computer Vision, Part 2, pp. 445-461, 2000. https://doi.org/10.1007/3-540-45053-X_29
34. [34] X. Ying and Z. Hu, “Can we consider central catadioptric cameras and fisheye cameras within a unified imaging model,” Proc. of the 8th European Conf. on Computer Vision, Part 1, pp. 442-455, 2004. https://doi.org/10.1007/978-3-540-24670-1_34
35. [35] C. Brauer-Burchardt and K. Voss, “A new algorithm to correct fish-eye- and strong wide-angle-lens-distortion from single images,” Proc. of 2001 Int. Conf. on Image Processing, Vol.1, pp. 225-228, 2001. https://doi.org/10.1109/ICIP.2001.958994
36. [36] A. Kipnis and A. Shamir, “Cryptanalysis of the HFE public key cryptosystem by relinearization,” Proc. of the 19th Annual Crypto Conf., pp. 19-30, 1999. https://doi.org/10.1007/3-540-48405-1_2