Abstract
The safety of autonomous\connected vehicles primarily relies on their ability to accurately sense the environment. The sensing problem is significantly challenging in weather conditions which include sudden change in lighting, smoke, fog, snow, and rain. Therefore, accurate sensing in adverse weather conditions is a critical and important safety problem that needs to be solved for any successful deployment of autonomous vehicles. There is currently no single sensor currently available in the market that can handle all the possible performance aspects and adverse weather conditions. The objective of this project is to use a combination of Radars and FIR cameras in addition to a LIDAR based system to map the environment and localize the vehicle with respect to the lanes on the road. This project will develop a prototype of an all weather sensing and localization system which will be useful for any autonomous or connected vehicle. The performance of the developed system will be corroborated with several data sets collected at Rellis. Demonstrations of the developed technology will also be done at the yearly Automated Vehicle Symposium, and SAE conferences.
Project Highlights
- Developed a panoramic thermal camera system that provides a video with a 190-degree field of view.
- Developed an object detection system to identify cars and pedestrians from thermal sensor data. Consequently, implemented a sensor fusion algorithm to combine this with radar data for object tracking.
- Validated the system with high-fidelity simulations using Unreal Engine as well as with a high-resolution lidar.
Final Report
EWD & T2 Products
Abhay Singh served as a judge for the Texas Junior Academy of Science (TJAS) in October 2021, which involved high school students from across the state.
Vegamoor participated in the ENDEAVR (Envisioning the Neo-traditional Development by Embracing Autonomous Vehicles Realm) non-profit event organized in Nolanville, TX on February 1, 2020. A prototype of the sensor fusion system was demonstrated, and rides were given to the residents inside the car. The public was also educated about the capabilities of autonomous vehicles. URL: http://endeavr.city/
Student Impact Statement – Abhay Bhadoriya and Vamsi Vegamoor (pdf): The student(s) working on this project provided an impact statement describing what the project allowed them to learn/do/practice and how it benefited their education.
Vegamoor, V. and Bhadoriya, A.S. (2021). “A ROS package for Sensor Fusion with Thermal Cameras”. GitHub repository, https://github.com/VegaVK/flir_adk_multi.
This project was used as part of the research statement by one of the students (Vamsi Vegamoor) for applying to the Dwight D. Eisenhower Transportation fellowship from Federal Highway Administration (US Dept. of Transportation). The fellowship was awarded for the year 2020 and renewed for 2021.
All the data collected in this project have been uploaded in the VTTI Dataverse. The code used for the sensor fusion algorithm has also been made open-source and will be useful as a benchmark for future researchers.
Researchers discovered a gap in research related to vehicle detection in heavy rain/snow. This is a viable extension for this project but will require collecting data outside the state of Texas.
Presentations/Publications
Overview of Autonomous Vehicle Research at Texas A&M, Baidu, Virtual seminar presented on Sep 15, 2021. and Autonomous Vehicle Sensor Fusion Research, Seminar to industry participants from Loomis Inc. presented on Aug 19, 2021, can be found here.
Bhadoriya, A. S., Vegamoor, V. K., & Rathinam, S. (2021). Object Detection and Tracking for Autonomous Vehicles in Adverse Weather Conditions (No. 2021-01-0079). SAE Technical Paper.
Final Dataset
The final datasets for this project are located in the Safe-D Collection on the VTTI Dataverse; DOI: 10.15787/VTT1/B3VKEA.
Research Investigators (PI*)
Sivakumar Rathinam (TAMU)*
Vamsi Vegamoor (TAMU-Student)
Abhay Bhadoriya (TAMU-Student)
Project Information
Start Date: 2019-05-01
End Date: 2021-12-31
Status: Complete
Grant Number: 69A3551747115
Total Funding: $190,409
Source Organization: Safe-D National UTC
Project Number: 04-117
Safe-D Theme Areas
Automated Vehicles
Connected Vehicles
Transportation as a Service
Safe-D Application Areas
Vehicle Technology
Planning for Safety
Operations and Design
More Information
Sponsor Organization
Office of the Assistant Secretary for Research and Technology
University Transportation Centers Program
Department of Transportation
Washington, DC 20590 United States
Performing Organization
Texas A&M University
Texas A&M Transportation Institute
3135 TAMU
College Station, Texas 77843-3135
USA