We propose a visual servoing (VS) approach with deep learning to perform precise, robust, and real-time six degrees of freedom (6DOF) control of robotic manipulation to ease the extraction of image features and estimate the nonlinear relationship between the two-dimensional image space and the three-dimensional Cartesian space in traditional VS tasks. Owing to the superior learning capabilities of convolutional neural networks (CNNs), autonomous learning to select and extract image features from images and fitting the nonlinear mapping is achieved. A method for designing and generating a dataset from few or one image, by simulating the motion of an eye-in-hand robotic system is described herein. Therefore, network training requiring a large amount of data and difficult data collection occurring in actual situations can be solved. A dataset is utilized to train our VS convolutional neural network. Subsequently, a two-stream network is designed and the corresponding control approach is presented. This method converges robustly with the experimental results, in that the position error is less than 3 mm and the rotation error is less than 2.5° on average.

1. [1] S. Hutchinson, G. D. Hager, and P. I. Corke, “A tutorial on visual servo control,” IEEE Trans. on Robotics and Automation, Vol.12, No.5, pp. 651-670, 1996.
2. [2] F. Chaumette and S. Hutchinson, “Visual Servo Control, Part I: Basic Approaches,” IEEE Robotics & Automation Magazine, Vol.13, No.4, pp. 82-90, 2006.
3. [3] F. Chaumette and S. Hutchinson, “Visual Servo Control, Part II: Advanced Approaches,” IEEE Robotics & Automation Magazine, Vol.14, No.1, pp. 109-118, 2007.
4. [4] D. Xu, “A tutorial for monicular visual servoing,” Acta Automation Sinica, Vol.44, No.10, pp. 1729-1746, 2018.
5. [5] F. Janabi-Sharifi, L. Deng, and W. J. Wilson, “Comparison of Basic Visual Servoing Methods,” IEEE/ASME Trans. on Mechatronics, Vol.16, No.5, pp. 967-983, 2011.
6. [6] N. R. Gans, S. A. Hutchinson, and P. I. Corke, “Performance Tests for Visual Servo Control Systems, with Application to Partitioned Approaches to Visual Servo Control,” The Int. J. of Robotics Research, Vol.22, Nos.10-11, pp. 955-981, 2003.
7. [7] E. Malis, F. Chaumette, and S. Boudet, “2 1/2 D visual servoing,” IEEE Trans. on Robotics and Automation, Vol.15, No.2, pp. 238-250, 1999.
8. [8] F. Janabi-Sharifi and W. J. Wilson, “Automatic selection of image features for visual servoing,” IEEE Trans. on Robotics and Automation, Vol.13, No.6, pp. 890-903, 1997.
9. [9] M. Paulin and H. G. Petersen, “Automatic feature planning for robust visual servoing,” 2005 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Edmonton, pp. 205-211, 2005.
10. [10] J. Pages, C. Collewet, F. Chaumette, and J. Salvi, “Optimizing plane-to-plane positioning tasks by image-based visual servoing and structured light,” IEEE Trans. on Robotics, Vol.22, No.5, pp. 1000-1010, 2006.
11. [11] Y. Li, Y. F. Li, Q. L. Wang, D. Xu, and M. Tan, “Measurement and Defect Detection of the Weld Bead Based on Online Vision Inspection,” IEEE Trans. on Instrumentation and Measurement, Vol.59, No.7, pp. 1841-1849, 2010.
12. [12] X. Wang, W. Wei, F. Liu, L. Wei, and Z. Liu, “Online Estimation of Image Jacobian Matrix with Time-Delay Compensation,” J. Adv. Comput. Intell. Intell. Inform., Vol.20, No.2, pp. 238-245, 2016.
13. [13] Y. Fang, Z. Xiaoyong, H. Zhiwu, W. Yu, and Y. Wang, “A Switched Extend Kalman-Filter for Visual Servoing Applied in Nonholonomic Robot with the FOV Constraint,” J. Adv. Comput. Intell. Intell. Inform., Vol.19, No.2, pp. 185-190, 2015.
14. [14] A. Saxena, H. Pandya, G. Kumar, A. Gaud, and K. M. Krishna, “Exploring convolutional networks for end-to-end visual servoing,” 2017 IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 3817-3823, 2017.
15. [15] J. Redmon and A. Angelova, “Real-time grasp detection using convolutional neural networks,” 2015 IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 1316-1322, 2015.
16. [16] Q. Bateux, E. Marchand, J. Leitner, F. Chaumette, and P. Corke, “Training Deep Neural Networks for Visual Servoing,” 2018 IEEE Int. Conf. on Robotics and Automation (ICRA), pp. 3307-3314, 2018.
17. [17] Y. LeCun et al., “Backpropagation Applied to Handwritten Zip Code Recognition,” Neural Computation, Vol.1, No.4, pp. 541-551, 1989.
18. [18] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. of the IEEE, Vol.86, No.11, pp. 2278-2324, 1998.
19. [19] I. Lenz, H. Lee, and A. Saxena, “Deep Learning for Detecting Robotic Grasps,” The Int. J. of Robotics Research, Vol.34, Nos.4-5, pp. 705-724.
20. [20] J. L. Schönberger and J.-M. Frahm, “Structure-from-Motion Revisited,” 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 4104-4113, 2016.
21. [21] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Proc. of the 25th Int. Conf. on Neural Information Processing Systems (NIPS’12), Vol.1, pp. 1097-1105, 2012.
22. [22] J. Liu and Y. Li, “An Image Based Visual Servo Approach with Deep Learning for Robotic Manipulation,” The 6th Int. Workshop on Advanced Computational Intelligence and Intelligent Informatics (IWACIII2019), pp. 61-66, 2019.