Vehicle type classification in intelligent transportation systems using deep learning

3.1 Evolution of deep learning in vehicle type classification

The evolution of deep learning in vehicle type classification is an enthralling narrative of technological progress that has redefined the landscape ITS. The advent of deep learning in 2006 marked a pivotal moment in the domain of machine learning, as it opened doors to a transformative approach for understanding and interacting with complex systems [19, 20]. With its innate ability to extract valuable insights from intricate data, deep learning quickly established itself as a linchpin in the development of solutions for big data challenges. It was this potential that set the stage for a captivating journey of evolution [21].

Over the years, deep learning has evolved into a major research area, extending its dominion far beyond the boundaries of machine learning [22, 23]. Its unique capacity to empower machines with the capability to autonomously decipher intricate patterns and features within vast datasets has catapulted it into a multitude of application domains [24, 25]. Notably, deep learning’s triumphant foray into computer vision, natural language processing, and robotics underscored its versatility and transformative power. However, the path to this evolution was not devoid of challenges [26]. Traditional training strategies often found themselves mired in local optima or failed to guarantee convergence, casting a shadow over the adoption of multi-layer neural networks, which held the promise of enhancing feature and representation learning [27].

A pivotal juncture in this trajectory emerged when Hinton and Salakhutdinov [28] introduced a groundbreaking two-stage strategy in 2006. This approach, involving pre-training and fine-tuning, marked a significant milestone by facilitating the first successful application of deep learning. With the proliferation of computing power and the burgeoning size of datasets, deep learning’s popularity soared. The advent of big data and the availability of high-performance computing systems propelled the field further. Remarkable projects, such as the large-scale deep belief network developed by Raina, Madhavan, and Ng, featuring over 100 million parameters and millions of training samples, demonstrated the transformative potential of deep learning. The use of GPU-based frameworks allowed such colossal models to be trained in a fraction of the time that traditional methods would require, heralding a new era in machine learning. Deep learning models, primarily employing a blend of supervised and unsupervised strategies, became the torchbearers for representation learning, enabling the robust classification and recognition tasks seen in applications like computer vision, speech recognition, and text analysis. Thus, the evolution of deep learning in vehicle type classification is a remarkable testament to human ingenuity and technological advancement, paving the way for more sophisticated and efficient ITS solutions [29].

3.2 Research directions in vehicle type classification

Research in vehicle type classification is a dynamic field that constantly seeks to address emerging challenges and explore innovative directions. This section discusses some key research directions in vehicle type classification, shaping the future of intelligent transportation systems. One promising research direction involves the fusion of data from various sensors to enhance vehicle type classification accuracy. Modern vehicles are equipped with a multitude of sensors, such as cameras, LiDAR, radar, and GPS. Combining information from these sensors can provide a holistic understanding of the surrounding environment. Researchers are working on developing sophisticated algorithms that can effectively fuse data from different sensor modalities to improve vehicle classification accuracy. This approach is essential for robust classification in complex scenarios, such as adverse weather conditions or crowded urban environments.

Real-time vehicle type classification is a critical research area with significant practical implications. To support tasks like traffic management, autonomous driving, and law enforcement, systems need to classify vehicles in real-time. Research is focused on developing efficient deep learning models that can run on resource-constrained devices, ensuring rapid processing without compromising accuracy. Furthermore, the integration of edge computing and cloud resources for real-time classification is a prominent research direction to balance processing power and accuracy.

Vehicle type classification models often struggle when faced with data from new environments or regions. Transfer learning and domain adaptation techniques are critical research areas for addressing this challenge. By leveraging knowledge from pre-trained models and adapting them to new domains, researchers aim to improve the generalization of classification models. These approaches can save time and resources while making vehicle classification systems more adaptable to diverse scenarios.

3.3 Recent survey studies in vehicle type classification

Veres and Moussa [17] have presented an overview of the role-modeling strategies employed by deep learning in the context of ITS. They discussed how deep learning can be used to model the behavior of drivers and pedestrians, as well as how it can be used to interpret data from onboard sensors. They also highlighted the potential for deep learning to facilitate more efficient and effective decision-making in ITS. They discuss the technical and problem-specific factors necessary to solve these challenges, examining how practitioners formulate problems to overcome them. This includes examining how the deep learning models utilized the available data, how the data sources were prepared, and how the models were trained and tested. Additionally, they discuss how the models were deployed and the implications of their results. This survey bridges the gap between transportation communities and machine learning by providing insight into upcoming topics and future trends. The survey also explores the challenges and opportunities in using deep learning for transportation applications, such as autonomous vehicles and traffic control. It identifies areas of exploration and potential research directions, such as the need for more high-quality datasets, better methods for model evaluation, and more robust deployment strategies.

Haydari and Yılmaz [30] discussed the latest deep learning applications for traffic control. They mentioned that deep learning can be used to detect objects in traffic, such as cars, pedestrians, and cyclists, as well as to predict traffic patterns. This information can then be used to optimize traffic flow and reduce congestion. The paper focuses on applications for traffic signal control that utilize deep learning, which has been extensively researched in the literature. Deep learning algorithms can learn from data and recognize patterns, making them well-suited for predicting and controlling traffic behavior. Additionally, they can be used to detect obstacles in traffic, which can be used to inform decisions about how to manage the flow of traffic best. By utilizing deep learning algorithms, traffic engineers can develop solutions that improve traffic safety and reduce congestion. A comprehensive discussion is provided on various problem formulations, deep learning variables, and simulation environments for traffic signal control. Several autonomous driving applications have been investigated in the literature using deep learning approaches. The survey provides an overview of existing research by classifying them according to application types, control models, and algorithms studied. They conclude by discussing the issues and challenges concerning deep learning-based transportation applications. These applications range from lanekeeping and traffic sign recognition to autonomous navigation, and they each require different types of deep learning models and algorithms. The survey also highlights the potential of deep learning to address the problems of safety, reliability, and scalability in autonomous driving.

Traffic flow prediction is critical in anticipating key factors affecting road transportation as it produces nonlinear and random data derived from sensors. By predicting traffic flow, officials can plan for congestion, design roads to handle the expected amount of traffic, and even anticipate accidents. The data obtained from sensors can be used to develop models that can accurately predict traffic flow and help improve traffic flow management. According to historical data, traffic flow is defined as the average number of vehicles traveling through a particular area. Forecasting macro-parameters such as volume, density, speed, and flow of traffic accurately can improve the efficiency of a traffic management system. The spatial and temporal traffic features are raw data inputs for predicting traffic flow. In most cases, shallow models cannot reveal spatial and temporal information in big data. In recent years, deep neural networks (DNNs) have demonstrated the capability of capturing and extracting important features for various purposes from large datasets. George and Santra [31] offer an overview of current deep learning approaches and their application to traffic flow prediction by considering a variety of variables, including construction zones, weather conditions, special events, and traffic incidents. As a final note, it highlights open challenges to explore to enhance deep learning algorithms and approaches for forecasting accurate flow rates. The authors also highlight the need for further research into developing robust big data management techniques for collecting, pre-processing, and analyzing large volumes of traffic data necessary for deep learning approaches to work effectively. Furthermore, they emphasize the importance of including data from multiple sources to improve the accuracy of the predictions.

Liu, Zhang and Chen [32] review recent applications of deep learning and neural networks in ITS and explore their potential applications, aiming to present the research community with comprehensive information concerning particular machine learning algorithms employed in traffic prediction and object recognition, two key aspects of ITS. They analyze the advantages and disadvantages of the different approaches, as well as the potential challenges and opportunities posed by deep learning and neural networks in ITS. They also discuss the potential for further application of deep learning to improve traffic prediction accuracy and object recognition performance. In this review, the authors first introduce popular neural networks, such as convolutional neural networks, long-short-term memory networks, gated recurrent units, and hybrid mechanisms. They then focus on using deep learning models to predict traffic and recognize objects, exploring the related data pre-processing, model training, and optimization strategies. Furthermore, they outline the possible future research and development directions in this field. The application and performance of these technologies in ITS are then discussed. Lastly, they discuss some constraints and suggest some promising research directions.

An overview of the reviewed survey studies is provided in Table 1. Consequently, it is necessary to undertake a systematic review and evaluate research concerning vehicle-type classification in ITS that uses deep learning algorithms. This systematic review provides an in-depth and focused analysis to guide future research toward finding robust vehicle-type classification mechanisms based on deep learning.

Despite advancements in vehicle type classification using deep learning, there exists a notable gap in the literature that requires further investigation. The primary literature gap is the lack of a comprehensive review that systematically evaluates the state-of-the-art methods in this field. While there are practical applications of deep learning in vehicle type classification, much of the existing research focuses on experimental aspects rather than the techniques adopted. This gap leaves a void in the scholarly landscape, where a consolidated and in-depth analysis of the methods, along with the latest developments, is lacking. Furthermore, rapid advancements in this field demand a timely review that takes into account the newest findings and technologies.

Additionally, the literature gap extends to the lack of emphasis on the techniques used in vehicle type classification and the recent advances in this domain. Although practical applications have been demonstrated in many studies, the absence of a comprehensive review makes it difficult to determine how these methods can be strengthened to achieve even better results. There is an inherent need for a comprehensive survey that scrutinizes the methods and technologies used in this specific context, bridging the gap between transportation communities and machine learning, and providing insights into upcoming trends, challenges, and research opportunities.

The primary research objective of this study is to fill the identified literature gap by conducting a systematic review and evaluation of research pertaining to vehicle type classification in the context of ITS using deep learning algorithms. The study aims to identify the most commonly used techniques and how they can be enhanced to achieve improved accuracy and efficiency. This objective is essential in the context of rapidly evolving technologies and their integration into transportation systems, as it provides a comprehensive understanding of the state-of-the-art methods.

Another research objective is to offer a timely and focused analysis of the current landscape in vehicle type classification through deep learning. By presenting an overview of the available literature and evaluating the methodologies employed, this research aims to guide future research in the direction of more robust and adaptable vehicle classification mechanisms. In light of the substantial impact of ITS on modern transportation and the increasing demand for autonomous and connected vehicle technologies, the findings from this study will serve as a valuable resource for researchers, policymakers, and practitioners working in this domain.

4.1 Overview of deep learning-based vehicle type classification

Deep learning-based vehicle type classification is a cutting-edge technology that has revolutionized the field of computer vision and transportation management [33]. It is a subset of artificial intelligence that focuses on teaching computers to automatically identify and categorize vehicles based on their visual characteristics. This technology employs deep neural networks, particularly Convolutional Neural Networks (CNNs), to analyze images or video streams and determine the type of vehicle present, such as cars, trucks, motorcycles, or bicycles [34]. One of the primary applications of this technology is in traffic management and surveillance systems. Deep learning algorithms can be deployed in traffic cameras to monitor and analyze the flow of vehicles, assisting in real-time traffic management and accident detection.

Additionally, vehicle type classification is essential for toll collection systems, parking management, and law enforcement purposes, as it enables the automated recognition of vehicles without human intervention. Furthermore, this technology has broader implications for the automotive industry, particularly in the development of autonomous vehicles. Vehicle type classification can aid self-driving cars in understanding the behavior and characteristics of nearby vehicles on the road, thereby enhancing safety and decision-making. Figure 1 shows an artificial neural network with several layers. Each layer is a set of neurons, which are connected to each other [35]. The neurons are activated based on the inputs they receive from the layer above. The output of the layer can then be used to make decisions or predictions. The score times weight in each layer represents some information that should be forwarded to the next layer. Inputs are summed up to produce outputs. The input layer collects the input data, the hidden layers store the corresponding weights, and the output layer generates the output.

OPEN IN VIEWER

Fig. 1

Architecture of deep neural network.

With recent developments in convolutional neural networks, especially Faster Region-Convolutional Neural Networks (Faster R-CNN), image classification and object detection have greatly improved [36]. Faster R-CNNs can use region proposals to identify potential objects in an image and then use the convolutional neural network to classify each region and detect the objects [37, 38]. This makes them more accurate, faster, and efficient than traditional image classification methods. It is superior to traditional machine learning techniques by a wide margin. Wang, Zhang, Wu, Xiao, Qian and Fang [39] propose a new deep learning-based vehicle-type classification scheme. The problem is solved using Faster R-CNN. Several experiments have demonstrated that the algorithm is time-saving, robust, and accurate. The system was tested on an NVDIA Jetson TK1 board with 192 CUDA cores, which is expected to become a leader in computational brains for autonomous vehicles, robotics, and computer vision. The Faster R-CNN algorithm combines a CNN with a region proposal network and can quickly detect objects in an image, making it ideal for the task at hand. Its accuracy and robustness are further proved by the fact that it achieved an accuracy percentage of 90.6 and 90.5% for cars and trucks, respectively. The system was tested on an NVDIA Jetson TK1 board, a powerful and efficient computing platform providing the necessary resources for vision-based applications. According to experimental results, NVDIA Jetson TK1 detects images in around 0.354 seconds and maintains high accuracy rates.

The speed detection and classification of vehicles is a significant research challenge in intelligent transportation systems and road safety. Accurate vehicle detection and classification are essential for many safety-critical applications, such as obstacle avoidance, speed control, and traffic surveillance. Properly classifying and detecting vehicles will also help reduce the risk of accidents, traffic congestion, and other safety hazards. Many systems have been developed to achieve these goals, including technologies for machine learning, image processing, and pattern recognition. Madhan, Neelakandan and Annamalai [40] developed a system for vehicle-type classification and speed detection based on a deep-learning algorithm.

Moreover, they developed an advanced algorithm and its parameters using training datasets to develop a Modular Neural Network (MNN) framework. The MNN framework can identify and classify various types of vehicles and accurately measure their speed. It also can detect and recognize objects in different environments, such as urban and highway traffic. Moreover, it can distinguish between different types of vehicles, including cars, buses, and trucks, at different speeds. It enhances the ability to detect and manage traffic flows in an automated manner by being integrated into the system.

Vehicle counting systems are important in achieving ITS’s objective of ensuring safe and efficient transportation infrastructure. By accurately counting the number of vehicles in a certain area, transportation planners can more accurately assess the current and future needs of the community. This allows them to develop more effective plans for the maintenance and expansion of transportation infrastructure, as well as identify potential areas of congestion and safety concerns. Awang and Azmi [41] propose an improved system for counting vehicles based on vehicle types rather than the current implementation, which only counts vehicles as vehicles and non-vehicles. To attain this goal, they proposed using convolutional neural networks based on the layer-skipping strategy (CNNLS) framework to categorize vehicles into three categories: cars, taxis, and trucks. This vehicle counting system is based on the current traffic census in Malaysia, which records vehicles according to their classes. Due to the almost identical features of taxis and cars, it is difficult to distinguish between these two types of vehicles in this paper. The proposed method could count vehicles using a frontal view captured with a mounted camera on the selected federal road from a self-obtained database. Using the self-obtained database and a mounted camera on the chosen federal road, they tested the method by taking a frontal shot of the vehicle.

Classified vehicles can also be evaluated for observing, tracking, and counting. It remains difficult to exclude irrelevant areas (noise, swinging movements, etc.) and to design an accurate and efficient system. Nasaruddin, Muchtar and Afdhal [42] propose a novel attention-based strategy to clearly separate the interesting region (moving car) from the uninteresting region (the rest of the area). To further enhance the classification efficiency, the deep CNN is fed with the relevant, interesting region to increase its efficiency significantly. The proposed approach is assessed using sequences of numerous challenging outdoors from CDNET 2014 (camera jitter classes, baseline, and bad weather). According to experiments, maintaining a highly accurate rate and classifying moving vehicles cost about 85 frames per second in GPU (and 50 frames per second in CPU). This method achieves a competitive detection accuracy when compared to other state-of-the-art approaches. Additionally, the suggested approach is compared with a recent 3D CNN method, saliency tubes, to verify the results.

Hedeya, Eid and Abdel-Kader [43] developed an ensemble of super-learners to classify vehicle types. It utilizes an essentially single-split densely connected super learner in order to enhance the potential benefits and minimize the drawbacks of the individual base learners DenseNet, ResNet50, and Xception. By learning fusion weights in a data-adaptive manner, the super learner can determine an optimal combination of the base learners. This strategy provides a simple, robust, and enhanced ability to discriminate between classes with similar appearances without the need for logical reasoning or manual features. Several public surveillance datasets are used in the evaluation, including the Beijing Institute of Technology’s dataset and the MIOvision Traffic Camera Dataset (MIO-TCD) for the classification of vehicles. On the MIO-TCD dataset, a three-variant super-learner ensemble was tested with data augmentation and different class weights during training. RXD-CV-CW-NCW and Augmented-RXD rank third amongst the state-of-the-art methods offered in the MIO-TCD classification challenge, which is significant. This study demonstrates that the performance of Augmented-RXD on the MIO-TCD dataset is unaffected by its generalization to the classes shared by the two datasets. Overall, 97.62% accuracy was achieved by the super-learner variants.

Traditional pure convolutional models rely on a limited number of features to extract information from a given environment. This limits the ability of the model to recognize objects in challenging environments, such as bad weather. Furthermore, traditional convolutional models cannot capture global information, which is necessary for accurately recognizing objects. Using the Densely Connected Convolutional Transformer in a Transformer Neural Network (Dense-TNT) framework, Luo, Song, Zhao, Zhang, Zhang, Zhao, Huang and Su [44] propose a novel framework for the classification of vehicle types in complex environments by stacking Densely Connected Convolutional Networks (DenseNet) with Transformer in Transformer (TNT). Three regions of vehicle data and four types of weather conditions were used to evaluate the recognition capability. Experimental results confirm that the proposed vehicle classification model can recognize vehicles under heavy fog conditions with little decay in recognition ability.

Kim and Lim [45] introduced a new scheme for classifying vehicles based on the images obtained from multi-view surveillance cameras. The scheme uses deep learning techniques and image processing algorithms to accurately identify and classify vehicles in real-time. This allows for more efficient and accurate traffic control and the detection of unauthorized vehicles. They proposed four concepts to improve the performance of images from multiple viewpoints with varying resolutions. In particular, they used a combination of transfer learning and data augmentation to increase the robustness of the model.

Additionally, they used a combination of convolutional neural networks (CNN) and support vector machines (SVM) for the detection and classification tasks. Finally, they tested the performance of the model using a variety of datasets to ensure its accuracy. The deep learning method is essential to distinguish multiple views from a single-point image. Data augmentation improves classification capability, while post-processing compensates for imbalanced data by bagging. A novel classification system for vehicle types was developed by combining these schemes. According to the classification challenge dataset, the system achieved 97.8% accuracy in classifying 103,835 images.

A novel method for classifying vehicles utilizing pictures from multi-view traffic surveillance cameras has been developed by Liu, Zhang, Luo and Cai [46]. Most image classification algorithms are designed to maximize the percentage of correct predictions. Images from minority categories are mistakenly categorized as dominant categories in this instance. To overcome this challenge, they suggest a strategy combining deep neural networks with balanced sampling. It is divided into two steps. The first step includes the augmentation of data with balanced sampling to remedy the difficulty of the unbalanced data set. In the second step, convolutional neural network models with various architectures are manufactured using parameters derived from the amplified training data. The proposed method has been demonstrated to have an enhanced mean precision of all categories compared to the baseline algorithms on the MIOvision traffic camera dataset classification challenge data set.

Guo, Li and Jiang [47] propose a semi-supervised vehicle-type classification approach based on ensemble broad learning for ITS. It consists of two major components. A base broad learning system classifiers collection is trained using semi-supervised learning to prevent the burden associated with unlabeled samples and the time-consuming training process. By constructing dynamic ensembles of trained classifiers with different characteristics, the second component determines which vehicle belongs to which type by obtaining the highest type probability and superior generalization performance compared to using a single base classifier alone. This approach allows the system to use the unlabeled samples to supplement the labeled data, improving the classifiers’ accuracy while reducing the time required for training. Furthermore, using dynamic ensembles of trained classifiers helps ensure that the system can make more accurate predictions, even when the data is limited. The proposed method outperformed some mainstream methods in efficiency and effectiveness in numerous experiments on the public datasets MIO-TCD and BIT-Vehicle.

Kolukisa, Yildirim, Ayyildiz and Gungor [48] have developed economical, simple-to-install, and battery-powered 3-D magnetic traffic sensors to classify vehicle types. The sensors are designed to detect magnetic fields radiated by moving vehicles and classify them into different types. The machine learning algorithm is used to process the data gathered by the sensors to identify the vehicle types accurately. Vehicles are classified into three categories on the basis of their structures and sizes: heavy buses, medium passenger cars, and light motorcycles. Using a DNN classifier allows for a robust and accurate classification process, as it can take into account the different structures and sizes of the vehicles. Furthermore, the classifier requires minimal computation power and low energy consumption, which benefits its portability and long battery life. The performance results also demonstrate the efficiency and reliability of the system.

The classification of vehicle types is one of the most important applications in the field of road safety. By classifying vehicles into different types, it is possible to understand better how different types of vehicles interact on the road and how they should be regulated. This helps to ensure that the roads are safe for everyone. Using a Multi-Column Convolutional Neural Network (MCCNN), Bouzi, Bentaieb, Ouamri and Boumedine [49] demonstrated a deep learning-based model for categorizing vehicles into six categories. In place of utilizing a fixed scale convolutional layer, the model uses variant filters to extract features from various scales of images to facilitate classification functioning, particularly in vehicles with like appearances, such as SUVs and sedans. An experimental investigation of the proposed approach is carried out on the BIT dataset, which involves 9850 frontal views of various types and resolutions. Using the whole unbalanced dataset, the MCCNN model achieved a 95.48 percent classification accuracy rate, an improvement over previous research. The accuracy rate of each class was further evaluated using 200 images, resulting in an average of 88.3% for each class.

4.3 Summary and future research directions

Table 2 summarizes the research studies reviewed in this section. The advances previously discussed have led researchers to develop deeper algorithms incorporating more training data. These algorithms have been shown to perform better on various tasks, such as image recognition, natural language processing, and machine translation. Additionally, these algorithms have enabled the development of more sophisticated applications, such as autonomous vehicles and virtual assistants. The introduction of temporal cues to the learning model has also been examined in recent studies, but this has suffered from instability during training. In addition, most of the models proposed were trained and tested under relatively simple conditions. For example, most studies focus on lateral control for a single task. This has limited the development of these algorithms for more complex tasks, such as multi-task control or multi-robot coordination. For example, no decision-making has been integrated for lane changes or turning onto other roads in lane-keeping models. Thus, multiple actions may be performed by the same DNN in the future. In addition, most of these works have been trained and evaluated in simulated environments, which further simplifies the task and would require further testing to confirm their performance in the real world. However, recent developments in this field have shown great promise concerning using deep learning for vehicle-type classification.

The present study provided an innovative perspective to the field of ITS by critically assessing the existing body of research on deep learning-based vehicle type classification. While previous studies had largely focused on experimental aspects, we went beyond by offering a comprehensive analysis of the techniques employed in vehicle classification and the recent advancements within the domain. Our innovative viewpoint was rooted in the need for a systematic and in-depth review that bridged the gap between deep learning communities and transportation researchers. We contributed by highlighting the most commonly used techniques and suggesting how they could be enhanced to achieve superior results, addressing the evolving landscape of ITS. Furthermore, we provided a forward-looking outlook on potential research directions, shedding light on areas that required further exploration, such as the acquisition of high-quality datasets, improved methods for model evaluation, and more robust deployment strategies. This innovative viewpoint was geared towards aiding future research efforts in developing more efficient and accurate vehicle classification systems, a crucial component of modern transportation and safety initiatives. Our contribution thus served as a valuable resource for researchers and practitioners alike, providing a structured guide for harnessing the potential of deep learning in the realm of vehicle type classification and further advancing the field.