In this study, we propose a high-speed and low-latency 3D fluorescence imaging method for robotic microscopes. The prototype system consists of a focus-tunable lens called a TAG lens, which operates at several hundred kHz, an image intensifier (I.I.) that enhances faint light such as fluorescence, and a high-speed vision system that can transfer acquired images to the host PC in 500 Hz. The proposed method can acquire images at arbitrary focal lengths at frame rates on the order of 1 kHz by synchronizing the focal-length fluctuation of the TAG lens and the exposure timing of the I.I., whose duration is a few hundred nanoseconds. The low-latency we aim for in this paper is on the order of a few milliseconds. A prototype system was developed to validate the proposed method. High-speed 3D tracking of the Brownian motion of a fluorescent bead of 0.5 μm diameter was demonstrated to verify the feedback performance of the proposed low-latency 3D fluorescence imaging method.

1. [1] D. Ahmed, A. Ozcelik, N. Bojanala, N. Nama, A. Upadhyay, Y. Chen, W. Hanna-Rose, and T. J. Huang, “Rotational manipulation of single cells and organisms using acoustic waves,” Nature Communications, Vol.7, No.1, pp. 1-11, 2016.
2. [2] T. Cacace, M. Paturzo, P. Memmolo, M. Vassalli, P. Ferraro, M. Fraldi, and G. Mensitieri, “Digital holography as 3D tracking tool for assessing acoustophoretic particle manipulation,” Opt. Express, Vol.25, No.15, pp. 17746-17752, 2017.
3. [3] K. Ohara, M. Kojima, S. Takagi, M. Horade, Y. Mae, and T. Arai, “Development of the 3D measurement system in real-time for micro-manipulation,” 2017 IEEE Int. Conf. on Mechatronics and Automation (ICMA), pp. 2022-2027, 2017.
4. [4] H. Oku, M. Ishikawa, Theodorus, and K. Hashimoto, “High-speed autofocusing of a cell using diffraction patterns,” Opt. Express, Vol.14, No.9, pp. 3952-3960, 2006.
5. [5] B. Tamadazte, E. Marchand, S. Dembélé, and N. Le Fort-Piat, “CAD model-based tracking and 3D visual-based control for MEMS microassembly,” The Int. J. of Robotics Research, Vol.29, No.11, pp. 1416-1434, 2010.
6. [6] H. Oku, N. Ogawa, M. Ishikawa, and K. Hashimoto, “Two-dimensional tracking of a motile micro-organism allowing high-resolution observation with various imaging techniques,” Review of Scientific Instruments, Vol.76, No.3, 034301, 2005.
7. [7] A. Mermillod-Blondin, E. McLeod, and C. B. Arnold, “High-speed varifocal imaging with a tunable acoustic gradient index of refraction lens,” Optics Letters, Vol.33, No.18, pp. 2146-2148, 2008.
8. [8] P. Annibale, A. Dvornikov, and E. Gratton, “Electrically tunable lens speeds up 3D orbital tracking,” Biomedical Optics Express, Vol.6, No.6, pp. 2181-2190, 2015.
9. [9] F. C. Cheong, C. C. Wong, Y. Gao, M. H. Nai, Y. Cui, S. Park, L. J. Kenney, and C. T. Lim, “Rapid, high-throughput tracking of bacterial motility in 3D via phase-contrast holographic video microscopy,” Biophysical J., Vol.108, No.5, pp. 1248-1256, 2015.
10. [10] T. Darnige, N. Figueroa-Morales, P. Bohec, A. Lindner, and E. Clément, “Lagrangian 3D tracking of fluorescent microscopic objects in motion,” Review of Scientific Instruments, Vol.88, No.5, 055106, 2017.
11. [11] F. O. Fahrbach, F. F. Voigt, B. Schmid, F. Helmchen, and J. Huisken, “Rapid 3D light-sheet microscopy with a tunable lens,” Optics Express, Vol.21, No.18, pp. 21010-21026, 2013.
12. [12] M. Molaei and J. Sheng, “Imaging bacterial 3D motion using digital in-line holographic microscopy and correlation-based de-noising algorithm,” Optics Express, Vol.22, No.26, pp. 32119-32137, 2014.
13. [13] T. Yamashita, H. Chiba, K. Yamato, and H. Oku, “Development of a coded exposure camera for high-speed 3D measurement using microscope,” Imaging Systems and Applications, ITu3B-2, Optical Society of America, 2018.
14. [14] K. Yamato, H. Chiba, and H. Oku, “High speed three dimensional tracking of swimming cell by synchronous modulation between tece camera and tag lens,” IEEE Robotics and Automation Letters, Vol.5, No.2, pp. 1907-1914, 2020.
15. [15] K. Yamato, H. Chiba, T. Yamashita, and H. Oku, “1-ms Three-Dimensional Feedback Microscope with 69-kHz Synchronous Modulation of Focal Position and Illumination,” IEEE Robotics and Automation Letters, Vol.3, No.3, pp. 1978-1984, 2018.
16. [16] K. Yamato, T. Yamashita, H. Chiba, and H. Oku, “Fast volumetric feedback under microscope by temporally coded exposure camera,” Sensors, Vol.19, No.7, 1606, 2019.
17. [17] J. B. Arous, Y. Tanizawa, I. Rabinowitch, D. Chatenay, and W. R. Schafer, “Automated imaging of neuronal activity in freely behaving Caenorhabditis elegans,” J. of Neuroscience Methods, Vol.187, No.2, pp. 229-234, 2010.
18. [18] Y. Ding and C. Li, “Dual-color multiple-particle tracking at 50-nm localization and over 100-μm range in 3D with temporal focusing two-photon microscopy,” Biomedical Optics Express, Vol.7, No.10, pp. 4187-4197, 2016.
19. [19] P. D. Frymier, R. M. Ford, H. C. Berg, and P. T. Cummings, “Three-dimensional tracking of motile bacteria near a solid planar surface,” Proc. of the National Academy of Sciences, Vol.92, No.13, pp. 6195-6199, 1995.
20. [20] Y. Katayama, O. Burkacky, M. Meyer, C. Bräuchle, E. Gratton, and D. C. Lamb, “Real-Time Nanomicroscopy via Three-Dimensional Single-Particle Tracking,” ChemPhysChem, Vol.10, No.14, pp. 2458-2464, 2009.
21. [21] J. W. Kim and B. H. Lee, “Autofocus tracking system based on digital holographic microscopy and electrically tunable lens,” Current Optics and Photonics, Vol.3, No.1, pp. 27-32, 2019.
22. [22] M. Liebel, J. O. Arroyo, V. S. Beltrán, J. Osmond, A. Jo, H. Lee, R. Quidant, and N. F. v. Hulst, “3D tracking of extracellular vesicles by holographic fluorescence imaging,” Science Advances, Vol.6, No.45, eabc2508, 2020.
23. [23] S. Makise, H. Oku, and M. Ishikawa, “Serial algorithm for high-speed autofocusing of cells using depth from diffraction (dfdi) method,” 2008 IEEE Int. Conf. on Robotics and Automation, pp. 3124-3129, 2008.
24. [24] M. Maru, M. Chen, and K. Hashimoto, “Visual servo microscope for locking on single neuron of a worm,” 2011 IEEE Int. Conf. on Robotics and Biomimetics, pp. 2844-2849, 2011.
25. [25] M. Maru, Y. Igarashi, S. Arai, and K. Hashimoto, “Fluorescent microscope system to track a particular region of C. elegans,” 2010 IEEE/SICE Int. Symposium on System Integration, pp. 347-352, 2010.
26. [26] P. Teunis, F. Bretschneider, and H. Machemer, “Real-time three-dimensional tracking of fast-moving microscopic objects,” J. of Microscopy, Vol.168, No.3, pp. 275-288, 1992.
27. [27] Y. Tsukada and K. Hashimoto, “Feedback regulation of microscopes by image processing,” Development, Growth & Differentiation, Vol.55, No.4, pp. 550-562, 2013.
28. [28] M. Duocastella, B. Sun, and C. B. Arnold, “Simultaneous imaging of multiple focal planes for three-dimensional microscopy using ultra-high-speed adaptive optics,” J. of Biomedical Optics, Vol.17, No.5, 050505, 2012.
29. [29] J. M. Jabbour, B. H. Malik, C. Olsovsky, R. Cuenca, S. Cheng, J. A. Jo, Y.-S. L. Cheng, J. M. Wright, and K. C. Maitland, “Optical axial scanning in confocal microscopy using an electrically tunable lens,” Biomedical Optics Express, Vol.5, No.2, pp. 645-652, 2014.
30. [30] J. Jiang, D. Zhang, S. Walker, C. Gu, Y. Ke, W. H. Yung, and S.-c. Chen, “Fast 3-D temporal focusing microscopy using an electrically tunable lens,” Optics Express, Vol.23, No.19, pp. 24362-24368, 2015.
31. [31] Y. Nakai, M. Ozeki, T. Hiraiwa, R. Tanimoto, A. Funahashi, N. Hiroi, A. Taniguchi, S. Nonaka, V. Boilot, R. Shrestha et al., “High-speed microscopy with an electrically tunable lens to image the dynamics of in vivo molecular complexes,” Review of Scientific Instruments, Vol.86, No.1, 013707, 2015.
32. [32] H. Oku and M. Ishikawa, “High-speed liquid lens for computer vision,” 2010 IEEE Int. Conf. on Robotics and Automation, pp. 2643-2648, 2010.
33. [33] D. Comaniciu and P. Meer, “Mean shift: a robust approach toward feature space analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.24, No.5, pp. 603-619, 2002.
34. [34] Å. Björck, “Numerical methods for least squares problems,” SIAM, 1996.