A Comprehensive Study on Multimodal Object Detection Using ML–DL Fusion for Smart Cities
Abstract
As the smart city infrastructure continues to expand, the demand for reliable real-time object detection systems has increased, especially for complex urban settings. This review examines various approaches used for such systems. To achieve this, multimodal sensor systems, using data from different sensors such as cameras, LiDAR, IoT devices, are important to improve the accuracy of predictions in applications such as intelligent transportation, surveillance and public safety. The paper looks at how these object detection techniques evolved from traditional feature based methods to advanced deep learning systems, such as CNN detectors, or transformer based models. It thus explores hybrid systems, which are generated by fusing machine learning and deep learning methods, along with edge computing. These technologies face challenges including sensor noise, occlusions, and variability in the environment which are critically examined alongside their benefits. Also, this study assesses the applicability, efficiency and accuracy of these existing systems, along with their limitations in handling the complex real-time scenarios. Finally, it discusses the future scope for development of more efficient, robust multimodal perception systems for smart cities.References
[1] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G. and Beijbom, O. (2020) ‘nuScenes: A multimodal dataset for autonomous driving’, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11621–11631. doi: https://doi.org/10.1109/CVPR42600.2020.01164
[2] Ren, S., He, K., Girshick, R. and Sun, J., 2016. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6), pp.1137-1149.
[3] Redmon, J. and Farhadi, A., 2018. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767.
[4] Zhu, X., Su, W., Lu, L., Li, B., Wang, X. and Dai, J., 2020. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159.
[5] Qi, C.R., Liu, W., Wu, C., Su, H. and Guibas, L.J., 2018. Frustum pointnets for 3d object detection from rgb-d data. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 918-927).
[6] Zhou, Y. and Tuzel, O., 2018. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4490-4499).
[7] Ku, J., Mozifian, M., Lee, J., Harakeh, A. and Waslander, S.L., 2018, October. Joint 3d proposal generation and object detection from view aggregation. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 1-8). IEEE.
[8] Molnar, C. (2022) Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, 2nd ed., Leanpub. doi: https://doi.org/10.5281/zenodo.5553937
[9] Geiger, A., Lenz, P. and Urtasun, R., 2012, June. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition (pp. 3354-3361). IEEE.
[10] Satyanarayanan, M. (2017) ‘The emergence of edge computing’, Computer, 50(1), pp. 30–39 doi: https://doi.org/10.1109/MC.2017.9
[11] Viola, P. and Jones, M., 2001, December. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001 (Vol. 1, pp. I-I). Ieee.
[12] Dalal, N. and Triggs, B., 2005, June. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05) (Vol. 1, pp. 886-893). Ieee.
[13] Girshick, R., Donahue, J., Darrell, T. and Malik, J., 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 580-587).
[14] Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y. and Berg, A.C., 2016, September. Ssd: Single shot multibox detector. In European conference on computer vision (pp. 21-37). Cham: Springer International Publishing.
[15] Redmon, J., Divvala, S., Girshick, R. and Farhadi, A., 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 779-788).
[16] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A. and Zagoruyko, S., 2020, August. End-to-end object detection with transformers. In European conference on computer vision (pp. 213-229). Cham: Springer International Publishing.
[17] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S. and Uszkoreit, J., 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
[18] Feng, D., Haase-Schütz, C., Rosenbaum, L., Hertlein, H., Glaeser, C., Timm, F., Wiesbeck, W. and Dietmayer, K., 2020. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions on Intelligent Transportation Systems, 22(3), pp.1341-1360.
[19] Chen, X., Ma, H., Wan, J., Li, B. and Xia, T., 2017. January 21-26). Multi-view 3d object detection network for autonomous driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.
[20] Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J. and Beijbom, O., 2019. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 12697-12705).
[21] Nabati, R. and Qi, H., 2019, September. Rrpn: Radar region proposal network for object detection in autonomous vehicles. In 2019 IEEE international conference on image processing (ICIP) (pp. 3093-3097). IEEE.
[22] Liang, M., Yang, B., Chen, Y., Hu, R. and Urtasun, R., 2019. Multi-task multi-sensor fusion for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 7345-7353).
[23] Cho, H., Seo, Y.W., Kumar, B.V. and Rajkumar, R.R., 2014, May. A multi-sensor fusion system for moving object detection and tracking in urban driving environments. In 2014 IEEE international conference on robotics and automation (ICRA) (pp. 1836-1843). IEEE.
[24] Bishop, C.M. and Nasrabadi, N.M., 2006. Pattern recognition and machine learning (Vol. 4, No. 4, p. 738). New York: Springer.
[25] Dietterich, T.G., 2000, June. Ensemble methods in machine learning. In International workshop on multiple classifier systems (pp. 1-15). Berlin, Heidelberg: Springer Berlin Heidelberg.
[26] Murphy, K.P., 2012. Machine learning: a probabilistic perspective. MIT press.
[27] Shi, W., Cao, J., Zhang, Q., Li, Y. and Xu, L., 2016. Edge computing: Vision and challenges. IEEE internet of things journal, 3(5), pp.637-646.
[28] Li, Y., Yu, A.W., Meng, T., Caine, B., Ngiam, J., Peng, D., Shen, J., Lu, Y., Zhou, D., Le, Q.V. and Yuille, A., 2022. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 17182-17191).
[29] Liu, Z., Tang, H., Amini, A., Yang, X., Mao, H., Rus, D.L. and Han, S., 2023, May. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In 2023 IEEE international conference on robotics and automation (ICRA) (pp. 2774-2781). IEEE.
[30] Xie, L., Xiang, C., Yu, Z., Xu, G., Yang, Z., Cai, D. and He, X., 2020, April. PI-RCNN: An efficient multi-sensor 3D object detector with point-based attentive cont-conv fusion module. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 07, pp. 12460-12467).
[31] Vora, S., Lang, A.H., Helou, B. and Beijbom, O., 2020. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4604-4612).
[32] Yin, T., Zhou, X. and Krahenbuhl, P., 2021. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11784-11793).
[33] Man, Y., Gui, L.Y. and Wang, Y.X., 2023. Bev-guided multi-modality fusion for driving perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21960-21969).
[34] Huang, K., Shi, B., Li, X., Li, X., Huang, S. and Li, Y., 2022. Multi-modal sensor fusion for auto driving perception: A survey. arXiv preprint arXiv:2202.02703.
[35] Deng, J., Shi, S., Li, P., Zhou, W., Zhang, Y. and Li, H., 2021, May. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, No. 2, pp. 1201-1209).
[36] Wang, Y., Chao, W.L., Garg, D., Hariharan, B., Campbell, M. and Weinberger, K.Q., 2019. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8445-8453).
[37] Khaleghi, B., Khamis, A., Karray, F.O. and Razavi, S.N., 2013. Multisensor data fusion: A review of the state-of-the-art. Information fusion, 14(1), pp.28-44.
[38] Zanella, A., Bui, N., Castellani, A., Vangelista, L. and Zorzi, M., 2014. Internet of things for smart cities. IEEE Internet of Things journal, 1(1), pp.22-32.
[39] Satyanarayanan, M., 2017. The emergence of edge computing. computer, 50(1), pp.30-39.
[40] Shokri, R. and Shmatikov, V., 2015, October. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security (pp. 1310-1321).
[41] Doshi-Velez, F. and Kim, B., 2017. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608.
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
