Lessons Learned from the Real-World Deployment of a Connected Vehicle Testbed

Abstract

The connected vehicle (CV) system promises unprecedented safety, mobility, environmental, economic, and social benefits, which can be unlocked using the enormous amount of data shared between vehicles and infrastructure (e.g., traffic signals, centers). Real-world CV deployments, including pilot deployments, help solve technical issues and observe potential benefits, both of which support the broader adoption of the CV system. This study focused on the Clemson University Connected Vehicle Testbed (CU-CVT) with the goal of sharing the lessons learned from the CU-CVT deployment. The motivation of this study was to enhance early CV deployments with the objective of depicting the lessons learned from the CU-CVT testbed, which includes unique features to support multiple CV applications running simultaneously. The lessons learned in the CU-CVT testbed are described at three different levels: (i) the development of system architecture and prototyping in a controlled environment, (ii) the deployment of the CU-CVT testbed, and (iii) the validation of the CV application experiments in the CU-CVT. Field experiments with CV applications validated the functionalities needed for running multiple diverse CV applications simultaneously using heterogeneous wireless networking, and meeting real-time and non-real-time application requirements. The unique deployment experiences, related to heterogeneous wireless networks, real-time data aggregation, data dissemination and processing using a broker system, and data archiving with big data management tools, gained from the CU-CVT testbed, could be used to advance CV research and guide public and private agencies for the deployment of CVs in the real world.

Connected vehicle technologies (CVT) offer unprecedented safety, mobility, environmental, economic, and social benefits, which can be unlocked using the enormous amount of data shared between vehicles and infrastructure (e.g., traffic signals, centers) ( 1 ). The real-world connected vehicle (CV) pilot deployments will materialize these promised benefits of the CVT, which are needed to make informed investment and deployment decisions for large-scale deployments by state or local public agencies. The United States Department of Transportation (USDOT) announced a proposed law to install vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications using dedicated short-range communication (DSRC) for all new lightweight vehicles to support different CV applications (i.e., safety, mobility and environmental applications) ( 2 ). From the perspective of this federal agency, state transportation agencies should deploy CVT and use the CV-generated data from the deployment sites to support state-wide planning and policies ( 3 ).

Although the USDOT has recently funded three pilot deployment sites in New York, Wyoming, and Florida to evaluate the efficacy of CV applications ( 4 ), the implementation guidelines for state or local agencies are still in the nascent stages of development. Initiated in Ann Arbor, Michigan in 2010, the CV Safety Pilot Program involved an analysis of the safety benefits of 2800 CVs at 29 infrastructure sites ( 5 ). Although the deployment program was successful in terms of elucidating the safety benefits of CVs, studies related to this program do not provide detailed guidance for the cases where multiple CV applications must be operated simultaneously that would require a heterogeneous wireless networking strategy for selecting specific communication options based on the corresponding application requirements in terms of data delivery delay and coverage range, data analytics capabilities for real-time data streaming, and data archiving strategies using big data management tools and infrastructure. Such deployment cases are required so that state public agencies may capitalize on the benefits from multiple CV applications, and provide academic and research institutions the means to develop innovative CV applications without direct involvement of the on-site CV deployment.

The Clemson University Connected Vehicle Testbed (CU-CVT) is a CV deployment site that includes unique heterogeneous wireless communication, data analytics for real-time data streaming, and data archival capabilities to empower the users with capabilities to operate multiple and diverse CV applications simultaneously. This paper discusses the case study of the deployment of the CU-CVT with the goal of sharing the lessons learned from the CU-CVT deployment, which can be leveraged by other public agencies, and academic or research institutions. Thus, the motivation of this study is to promote early large-scale CV deployments by disseminating findings in the deployment and operation of unique CU-CVT features.

The remainder of this paper is structured as follows. The next section describes existing connected vehicle testbeds and their limitations, and is followed by a section that presents the CU-CVT and the unique features of this testbed. Lessons learned from this testbed are then presented. The final section provides a concluding discussion.

Existing CV Testbeds

The USDOT leads the pilot deployment effort for CV testbeds in the United States through the ITS Strategic Plan ( 6 ). Southeast Michigan was one of the sites for Connected Vehicle Testbed, which is sponsored by the USDOT, and is available to DSRC product developers and CV application developers to test any CV applications. Testbed development began in 2009, and the project was completed in August 2017. Subsequently, the USDOT extended support for the sharing of CV-related data to develop a common technical platform for affiliated partner CV testbeds, such as the California and New York testbeds ( 7 ). Key CV technologies are related to the broadcasting of signal phase and timing (SPaT) data and intersection geometry (i.e., MAP) data, and CV devices (i.e., vehicle awareness devices, aftermarket safety devices, and road side units (RSUs)), and the USDOT preliminary Security Credentials Management System (SCMS) ( 8 ). The two ongoing projects are the “5.9 GHz Dedicated Short-Range Communication Vehicle-based Road and Weather Condition Application: Phase 2” and “Basic Infrastructure Message Development and Standards Support”. The purpose of the first project is to develop and test the “weather condition information” transmission from CVs (i.e., public agency vehicles with connectivity used as CVs) to a central server via RSUs using 5.9 GHz DSRC communication. Public agency maintenance personnel use this stored information for future maintenance purposes.

In 2012, the University of Michigan Transportation Research Institute and the USDOT started a new project to reduce roadway crashes using CV technology (known as Safety Pilot Model Deployment (SPMD) [ 9 ]). The total investment for this three-year SPMD project (2015–2018) was $30 million, which incorporated over 2,800 vehicles and 73 lane-miles of northwest Ann Arbor, Michigan. After successful implementation of SPMD, the Ann Arbor testbed was extended from northeast Ann Arbor to the City of Ann Arbor. The new testbed covers 27 sq. mile city of Ann Arbor including 45 street locations and 12 freeway sites. Every year 1,500 vehicles are to be added in this deployment with the goal of having 5,000 vehicles on the road by 2018. The main goals of this testbed are to ensure that (i) the model deployment will be transferrable to an early operational deployment; (ii) the test facilities will continue to operate and maintain a robust and high-quality test environment; and (iii) the testbed can become an economically sustainable environment from a federally funded program. It will become the world’s largest CV infrastructure deployment, and it is called the Ann Arbor Connected Vehicle Test Environment.

The USDOT awarded a total of $45 million to sites in New York, Wyoming, and Florida in September, 2016 to initiate design, deployment and validate the CV Pilot Deployment Program ( 10 ). The primary focus of the New York City (NYC) pilot deployment in New York is to improve vehicle and pedestrian safety ( 10 , 11 ). This pilot deployment includes 280 RSUs and 10,000 vehicles (cabs and buses), and will be used to evaluate CV safety challenges and benefits in an urban environment. Furthermore, they will evaluate pedestrian safety applications using 100 pedestrian-DSRC units. Multiple locations on Interstate 80 were chosen as the Wyoming CV pilot deployment site in another project ( 12 ). This project deploys DSRC on-board units in 400 CVs (mainly snowplows and heavy vehicles) and provides safety alerts and travel guidance to the commercial CVs. In Florida, the Tampa-Hillsborough Expressway Authority (THEA) pilot deployment plans to implement a variety of V2V and V2I applications ( 13 ). The focus of implementing these applications is to reduce congestion, conflicts between vehicles, and entry of a vehicle from wrong direction at the Selmon Reversible Express Lanes exit. Another purpose of the THEA pilot deployment involves using the CV technologies to improve the safety of pedestrians, increase efficiency of bus operations, and reduce conflicts between vehicles and pedestrians. This deployment uses DSRC as the wireless data transmission medium. Through this CV pilot program, 1,600 cars, ten buses, ten trolleys, 500 pedestrians with smartphone applications, and 40 RSUs along city streets will be deployed in Tampa, Florida.

The first public CV testbed in the United States was developed in 2005 at California on El Camino Real (State Route 82) in Palo Alto between Stanford Ave at the north end and W Charleston Ave ( 14 ). There are 11 consecutive intersections exists along this corridor. More than 50,000 vehicles travel this corridor between San Francisco and San Jose daily. The partners of this testbed are Caltrans, the Metropolitan Transportation Commission and the California PATH program at UC Berkeley. In 2013, CV standards were implemented in the previously developed California CV testbed with the support of USDOT. The focus of this testbed is deploying multimodal intelligent traffic signal systems and environmentally friendly driving applications. The primary objective is to improve overall arterial network performance utilizing traffic signal priority for transit and freight, preemption for emergency vehicles, and pedestrian safety applications. Besides, Virginia has two CV testbeds ( 15 ), the first located in Southwest Virginia in Blacksburg Virginia at the Virginia Smart Road, and the second along Route 460 and Fairfax County in Northern Virginia along I-66 and on the parallel Routes 29 and 50. The Connected Vehicle/Infrastructure University Transportation Center (CVI-UTC) selected these two locations for their CV research based on congestion, high crash rates, and air quality non-attainment. Fifty roadside units were deployed along the roadways of both, which also include different instrumented vehicles (e.g., instrumented automobiles, motorcycles, a motor coach, and a semi-truck).

In existing CV testbeds, CVs are deployed to test CV safety applications using DSRC. Although DSRC is a very low-latency wireless communication medium, it is limited in terms of communication range (∼900 ft (300 m)) ( 16 ). Thus, DSRC alone cannot support the breadth of CV applications (i.e., mobility and environmental applications) outlined by the connected vehicle reference implementation architecture ( 17 ). Other wireless technologies, such as long-term evolution (LTE), WiMAX, and Wi-Fi can support DSRC communications to increase network availability and coverage area, and fulfill data rate requirements at peak period. Thus, it is necessary to use heterogeneous wireless technology in which communication options can be chosen based on the requirements identified by CV applications, and their range, data delivery delay, and throughput. Another limitation of existing testbeds is the missing data analytics platform for real-time data aggregation, processing, and archiving for multiple CV applications running simultaneously. Designing a suitable data delivery system for CVs depends on the real-time data streaming from different devices without duplication and supplying the aggregated data depending on the application demand ( 18 ). Appropriate computational resources are also necessary to support multiple and computationally expensive CV applications.

Unique Features of CU-CVT Testbed

The CU-CVT is developed in a hierarchical architecture. In a CV environment, different types of applications run at different levels (e.g., vehicle level, roadside infrastructure level, system level (i.e., backend server or cloud)) ( 19 ). A layered architecture has proven effective in managing such distributed applications by decoupling the dependency and distributing the computation. The overview of the layered architecture in CU-CVT is shown in Figure 1. The CU-CVT layered architecture has a (i) Mobile Edge, (ii) Fixed Edge, and (iii) System Edge. The Mobile Edges will reside in Layer-1 of the architecture. Each Mobile Edge or each mobile entity (e.g., vehicles, pedestrians) is equipped with a DSRC, 3GPP/LTE Cellular or Wi-Fi-enabled device to collect information related to the Basic Safety Messages (BSMs) such as time stamp, car ID, latitude, longitude and speed. To support different types of application with different levels of computation and memory requirements, DSRC devices are coupled with a computation unit capable of larger memory management, multiprocessor operations, and user-friendly application development. The computation unit also permits the integration of other communication mediums, such as Bluetooth, Wi-Fi, and LTE. The Fixed Edge acts as an entry point to the infrastructure with a very low-latency, single-hop DSRC connection with the Mobile Edge. Residing in Layer-2 of the system, the Fixed Edges communicate with components in Layer-1 and Layer-3. Fixed Edges can be extended to support a video camera and other sensing devices (e.g., weather sensors). Fixed Edges also contain a computational platform for the same reason as the Mobile Edge devices, along with a DSRC-enabled device. Fixed Edge devices are connected to Layer-3 with a high-bandwidth, high-throughput, low-latency and highly reliable communication medium (e.g., fiber optics, wired Ethernet) to act as backhaul support.

Figure 1.

Clemson University Connected Vehicle Testbed (CU-CVT).

The System Edge is situated at the top of the layered architecture, Layer-3. The task of the System Edge is to monitor the overall system to balance and optimize system operations and provide services to different nodes related to connectivity, data storage, application priority, and dissemination of system-wide messages, and to act as a secure portal for external agencies and research interests. One of the main responsibilities of the System Edge is to provide resource optimization and management of communication network resources in heterogeneous wireless networks. Conceptually, the System Edge can be in a cloud or cluster of computers. The System Edge used here supports the various services mentioned above.

CU-CVT system and CV applications are closely intertwined with wireless access in vehicular environments (WAVE) message sets stated in Society of Automotive Engineers (SAE) J2735 ( 20 ) and IEEE 1609.3 standards ( 21 ). CU-CVT supports standard WAVE applications provided by the equipment vendors, which are a set of safety applications including collision avoidance. The CU-CVT testbed provides users and developers the ability to operate a variety of applications and services that are interoperable with standard DSRC WAVE. The applications comply with DSRC standards in terms of transmission channel, frequency, and power ( 22 ). Any other DSRC radios, complying with the 1609 WAVE standards and licensed to operate in the U.S., will be able to work with these applications utilizing standard DSRC channels. The goal is to make the CU-CVT system highly interoperable with any other CV applications that are compatible with WAVE multichannel standards (i.e., IEEE 1609.4) ( 22 ). In addition, the testbed and the hierarchical architecture provide a solid framework for implementing security-enforcing services. The CU-CVT testbed and CV applications are designed to use the vendor-specific security keys that are embedded into each licensed DSRC unit. The applications also provide methods to enable and disable such privacy enforcements. With the hierarchical structure of the testbed, implementation of SCMS ( 23 ) or other forms of security and privacy methodologies can be easily implemented. In addition to the security and privacy platform available on DSRC networks, CU-CVT testbed has a virtual private network (VPN) on top of Clemson University campus network. The private network being used for the testbed provides an impenetrable backbone for data communication between Fixed Edges and a System Edge. Collectively, the VPN and the use of secret private keys exchanged before initiating any communication satisfy the privacy requirements for the testbed and the application data.

Lessons Learned: CU-CVT Deployment

This section presents the lessons learned at different levels of the CU-CVT deployment: (i) system design and prototyping; (ii) system deployment; and (iii) system validation.

System Design and Prototyping

In the process of deploying the real-world CV testbed, the first step involves defining the system architecture, selecting an appropriate communication medium for use among the CV components and software components, and evaluating and justifying these selections by developing a prototype system. In this section, the lessons learned from the system design architecture and prototyping in a controlled environment are detailed.

System Design Architecture

The focus of this sub-section is to define the rationale for selecting a layered design architecture, heterogeneous wireless communication medium, data analytics using real-time data streaming, and data archiving with big data management tools in the process of concept development of the CU-CVT.

Layered Architecture

The key motivations for choosing the layered architecture of the testbed are to reduce data delivery delay and data loss rate, and increase throughput between diffferent edges of the CU-CVT testbed ( 19 ). DSRC provides a low-latency, high-throughput communication link from a Mobile Edge to a Fixed Edge and many applications, such as queue prediction, traffic data collection, and video monitoring, which may need computation at the Fixed Edge rather than sending the data to a System Edge. Thus, the overall effectiveness of the application is improved in terms of analyzing the data, which is a primary motivation for designing the layered architecture. Layered architecture can also support the integration of roadway sensors (e.g., video cameras, weather sensors, roadway traffic sensors) with the CVT deployment while reducing high bandwidth requirements. There are several advantages of this layered design architecture including proximity, intelligence, and scalability. Proximity of the edge provides the flexibility to communicate efficiently with the next immediate edge layer, and information distribution will be quicker than with a centralized system. The next immediate layer is defined as the closest edge layer in terms of the physical and logical sense. With the growing number of CVs, massive amounts of raw data will be collected and processed into usable formats for different CV applications. Thus, intelligence functions must be distributed efficiently between different layers to reduce the computational time for computationally expensive applications. This layered architecture can be easily scalable by changing the number of edges in different layers to meet the demand of computational resources.

Heterogeneous Wireless Communication (Het-Net)

To ensure the reliability of data transfer for ITS applications, a combination of different wireless communication technologies such as LTE, Wi-Fi, and WiMAX has been used ( 24 , 25 ). However, establishing the proper communication networks is a challenge in a layered architecture with dynamic vehicular networks. Traditional network management is unsuitable for highly mobile vehicular networks because of its dependency on network access, and packet management and routing requirements. Het-Net allows selection between different wireless communication options based on their availability, reliability, and data delivery requirement of the CV applications ( 26 , 27 ). Based on the CV application requirements, Het-Net optimizes networking resources, which can provide a better solution to improve communication and content delivery issues with CVT.

Data Analytics using Real-Time Data Streaming

One of the primary challenges in layered architecture involves managing the data delivery system between different components. To mitigate this challenge, CU-CVT has adopted a publish–subscribe-based data delivery system among the connected components ( 18 ). A broker-based system, which is a publish–subscribe-based data delivery system, is a popular data delivery system in different areas that include social media, e-commerce, and manufacturing ( 18 ). In general, in this system, producers (e.g., sensors, RSUs) produce data, and consumers (e.g., CV applications, emergency management center, and traffic management center) consume data. Such frameworks include Kafka, RabbitMQ, ActiveMQ, WebsphereMQ, and Message Queuing Telemetry Transport (MQTT) ( 18 ).

Data Archiving using Big Data Management Tools

Along with the data delivery system, a data storage system is also necessary. A database is a piece of software and storage area where data can be stored that can be used for data analysis and reporting. In a CV environment, the different types of structured and unstructured data generated from different sources (e.g., on-board units, RSUs, sensors) must be aggregated and stored at different edge levels of the layered architecture. Structured data can be stored in relational databases (structured query languages (SQL)), and non-structured data are suitable for storage in NoSQL databases ( 28 ). Oracle, MySQL, and SQL Servers are the most popular relational database systems used in industry, whereas MongoDB, CouchDB, and BigTable are the most popular NoSQL databases ( 29 , 30 ). Both have some advantages and disadvantages in terms of performance, usability, and scalability. The evaluation and selection of the most suitable data management tool for the CV environment is described in the sub-section “Data Archiving using Big Data Management Tools” of the System Deployment section.

System Prototype Development in a Controlled Environment

Before the real-world deployment of the CV system, it is important to develop a prototype of the application and system architecture in a controlled environment, especially in a controlled roadway. Given that a focus of the CV applications is that of safety-critical issues, deploying any application without such testing may create a hazardous situation. The first phase of application or research development involves testing the process in a controlled lab environment where the operation of the application, logic, and errors are debugged and perfected. Therefore, a particular corridor at Clemson University was selected (i.e., the Student Organic Farm, as shown in Figure 2) as the real-world CV application test site. After the successful development and testing in a controlled environment, the application was deployed in the real world, which eventually involved CU-CVT.

Figure 2.

Organic farm prototype deployment in a controlled environment.

System Deployment

In this section, the lessons learned from the CU-CVT deployment are described in detail.

Location Selection

Identifying the optimal locations of physical devices is a critical aspect of field deployment. This system was deployed on the main campus of Clemson University (Clemson, SC) as illustrated in Figure 3, with three Fixed Edges positioned along Perimeter Road. The location of the RSU depends on the surrounding environment, roadway geometry, opportunities for roadside mounting, and the purpose of the testbed. The specific range of functionality of the RSUs (owing to DSRC range limitation) requires placing them at intervals to ensure only a minimal range overlap. The blue line in Figure 3 shows the range from the field test.

Figure 3.

(a) Packet loss at C36 location, (b) Packet loss at C42 location.

The effect of RSU location on packet loss at two locations (C36 and C42) on the Perimeter Road deployment is shown in Figure 3. The RSU device used is the same for both RSU locations. One RSU shows higher coverage (Figure 3a), whereas the other location (Figure 3b) shows less coverage because of the horizontal and vertical curve of the roadway. This is because of external factors such as interference and signal blockages. Thus, the location should be selected depending on the road way geometry and topography. The RSUs should be placed at locations with minimum interference from trees, houses, and other signal blockages. For the CU-CVT deployment, a corridor with multiple signals was chosen, and signalized intersections were used as the RSU locations. An intersection that allows the use of RSU for CV safety, and mobility applications related to intersections (e.g., queue warning, stop-sign gap assist) is considered to be a suitable location. Consecutive intersections have a certain spacing, allowing installation of one RSU at each intersection and covering the whole corridor. It also serves the purpose of covering multiple directions, which may be required for some CV applications (e.g., stop-sign gap assist). Finally, the close proximity of the RSU to the signal controller provides easy access to the controller, which may also be valuable in V2I applications.

System Compatibility and Integration

One of the key issues for real-world deployment is system compatibility. These deployment experiences showed that different vendors have different ways of implementing applications, services, communication, and data streaming application programming interface. The main problem experienced is that application development environment varies in different vendor devices. The testbed should be independent of vendor choice. As there is no specific standard for device configuration and it is the early phase of equipment production, one needs to write one’s own code to make it compatible with the platform. Another approach involves developing a generalized CVT distributed application services package, such as ThinGs in a Fog (TGIF) (19), to provide services, such as geo-location, messaging, HetNet and WAVE services, for application developers.

Heterogeneous Wireless Communication

Het-Net allows the selection of the best available communication options based on their availability and reliability, as well as data delivery requirements of the corresponding CV applications. Figure 4 illustrates an example deployment scenario with three communication options, Wi-Fi, LTE, and DSRC, in which a wireless network will be selected based on the requirements of the CV application to be supported, coverage area and availability of the network, and handoff delay for switching from one wireless communication option to another wireless communication option. As shown in Figure 4, CVs (Mobile Edges) use DSRC network to send data to the Fixed Edges while they are within the DSRC coverage area of a Fixed Edge (RSU). From the Fixed Edge, data collected from the CVs will be transferred to the System Edge using backhaul network (either through wired, Wi-Fi, or any long-range wireless communication network). On the other hand, CVs use LTE wireless network to transfer data directly to the System Edge while they are outside the DSRC coverage area of a Fixed Edge (RSU). CVs can also use Wi-Fi wireless network to transfer data directly to the System Edge if Wi-Fi network is available. This study has deployed two Het-Net instances (instance 1: switching of networks between LTE and Wi-Fi; instance 2: switching of networks between LTE and DSRC) based on the availability of the communication options and subject to the CV application requirements. An application layer handoff, which involves switching or “handoff” from one network to another, is an elemental approach for managing heterogeneous networks. The application layer handoff works with no change to hardware and software of the Mobile Edge (on-board unit) and Fixed Edge (RSU). A hard-handoff is performed when one wireless network is switching to another wireless network to implement Het-Net services. The hard-handoff can be defined as when a wireless communication network connection is broken before connecting with another network.

Figure 4.

Heterogeneous wireless communication scenarios.

Data Analytics with Real-Time Data Streaming

The messaging service is equipped with a broker or the ability to publish/subscribe to a broker. For the CU-CVT deployment, the MQTT broker was used ( 30 ). MQTT is a lightweight open source publish/subscribe messaging transport protocol, and it is easy to implement. MQTT is superior to the hypertext transfer protocol (HTTP) messaging service, because it provides better throughput, has lower power consumption, lower network bandwidth utilization, and lower latency. The publish/subscribe protocol makes the communication between different edges efficient as it enables a large number of mobile clients (e.g., Mobile Edge) reliably communicate with Fixed Edges or a server (e.g., System Edge), and vice versa in an edge centric connected vehicle testbed like the CU-CVT. Publish/subscribe decouples a client (message sender) from another client (message receiver) by making use of a unique set of strings for each type of message called Message Title. The MQTT broker filters all the received messages and distributes those messages accordingly. MQTT uses subject-based filtering of messages. CV applications are dynamic in nature, and most applications require efficient real-time data streaming and data distribution between different edges of the CU-CVT. Thus, data exchange in V2V and V2I communication is a major challenge. Moreover, CV applications operate in a constrained environment, where there are often limitations in bandwidth, computation, and storage. Given the requirements of a lightweight broker-based communication method, MQTT is a good choice for the field deployment of CV applications.

Data Archiving with Big Data Management Tools

Data storage is an essential part of CV applications. Many CV applications have to deal with big data, where there is a large volume of streaming data and a large variety of data (i.e., real-time location, vehicle speed, position, acceleration). The choice of database for archiving data should be uniform across all layers of field deployment. For the CU-CVT deployment, MongoDB was used as the uniform data storage platform for CV applications. As CV applications will need to deal with huge amounts of data, a non-relational database is a better choice than a relational one. NoSQL databases offer advantages over SQL databases in several aspects, such as scalability and data diversity (i.e., different types of data). Among many NoSQL databases, MongoDB was chosen for the CU-CVT deployment because it has many advantages over other NoSQL databases. MongoDB is a popular open source NoSQL database tool for big data management. Being open source is a major advantage. MongoDB offers the query ability of a relational database while maintaining the features of a NoSQL database. Moreover, if the data are location based, MongoDB is handy because it has some built-in spatial functions. MongoDB stores data using the very popular JavaScript Object Notation (JSON) format, which is compatible with all programming platforms. Different CV applications require different data storage requirements and different data structures. Therefore, a fast, scalable, and flexible data storage platform is needed that can store any type of data, of any size, at any time while the application is running. Moreover, different applications may operate on different platforms. A universal data storage platform is needed that will perform effectively across all CV applications, and MongoDB is the solution.

Inter-Agency Cooperation

The challenge of developing a testbed for field implementation requires collaboration with multiple stakeholders. For the CU-CVT deployment, significant collaboration was required between the researchers and the Clemson University Information Technology (IT) department. The research team often works closely with the IT department of the university, as the campus Wi-Fi network (Eduroam) and optical fiber is used for backhaul support. To install a RSU at the testbed, a light pole close to the traffic signal was chosen. There was regular communication with the Clemson University campus physical planning department for the bucket truck to install the RSU at the Perimeter Road (where the CV-CVT is located) light pole. The light pole was selected so that there are optical fiber services from the pole to the backend resources. Without diligent support from the Clemson University campus physical planning and IT departments, the proper deployment of the testbed would have been impossible. The use of a state-owned road requires permission from the state Department of Transportation (state DOT) to set up the roadside RSUs and secure any access to the state DOT infrastructure, specifically the DOT’s communication network and traffic control devices and equipment.

System Validation in the CU-CVT

The lessons learned from system validation include validation experiences on DSRC system performance ascertainment, heterogeneous wireless network, data analytics with real-time data streaming, and system integration.

DSRC System Performance Ascertainment

The CV testbed is deployed on a geographically spread out network, including dynamic scenarios ranging from high-speed less congested scenarios to slow-speed highly congested scenarios. For these scenarios, it becomes very important to understand how the system performs to provide reliable communication between different Mobile Edges and Fixed Edges. The coverage range obtained for the three Fixed Edges in the CU-CVT is shown in Figure 5, with the received signal strength indicator value measured by the Fixed Edges. This coverage of the DSRC range provides an observation on network availability and allows for strategic planning for heterogeneous network management and switching between different RSUs.

Figure 5.

DSRC network coverage of the CU-CVT testbed.

Heterogeneous Wireless Communication

In this study, two Het-Net instances were evaluated (instance 1: switching of networks between LTE and Wi-Fi; instance 2: switching of networks between LTE and DSRC) in terms of handoff delay. The average delay recorded between successive packet arrivals is used to estimate the handoff delay during switching from one network to another network. UDP data transfer protocol is used to send data packets from a CV to the roadside infrastructure (i.e., Fixed Edge) or server (i.e., System Edge). In the first instance, Wi-Fi and LTE wireless Het-Net service is implemented for transferring data from vehicles to a System Edge (i.e., Server). For receiving data packets through Wi-Fi, the 802.11 link must be active and a CV must be within the coverage area of the Wi-Fi access point. The hard-handoff delays during switching from Wi-Fi to LTE and switching from LTE to Wi-Fi are 3.61 and 3.10 seconds, respectively.

In the second instance, LTE and DSRC wireless Het-Net service is implemented for transferring data from CVs to the roadside infrastructure or server. A vehicle equipped with DSRC radio (i.e., Mobile Edge or CV) can send data to Fixed Edge (i.e., RSU) when the CV is inside the DSRC coverage area of the Fixed Edge. Before entering the coverage area of the Fixed Edge, the LTE wireless network option of the CV is always active. As the CV is equipped with DSRC communication capabilities, it receives the beacon-like message when it enters the coverage area of the Fixed Edge with DSRC, and an application inside DSRC radio of a CV initiates the handoff process. The application executes a hard-handoff while switching from LTE to DSRC. The opposite handoff process (switching from DSRC to LTE) is executed when the CV leaves the DSRC coverage area of the Fixed Edge and receives the beacon-like messages from the LTE wireless network. The handoff delays during switching from DSRC to LTE and switching from LTE to DSRC are 0.23 and 0.03 seconds, respectively, which are much smaller than the corresponding handoff delays between Wi-Fi and LTE.

Data Analytics with Real-Time Data Streaming

Raw data (e.g., time stamp, car ID, latitude, longitude and speed), known as BSMs, are collected from each CV. The campus Fixed Edge used Wi-Fi to communicate with the backhaul server (System Edge). Vehicles equipped with CV on-board equipment (i.e., Mobile Edge) could thus collect real-time data from other CVs as those vehicles traveling within their DSRC range. All Mobile Edge device-equipped vehicles send data to the RSU using DSRC if these vehicles are within the DSRC coverage area of a Fixed Edge. Optical fiber is used to connect with the System Edge with the Fixed Edge when real-time communication is required. A database system is hosted using MongoDB to store data at the Fixed Edge and at the System Edge. After collecting raw data from the DSRC on-board unit, the DSRC-RSU will publish raw data to the MQTT broker system, and later the raw data processing unit subscribes the MQTT broker to consume the required data. The MQTT broker, the MongoDB database, and other computational processes reside in an edge-computing device. The processed data are stored for a short period at the MongoDB database to run real-time CV applications in the Fixed Edge. On the other hand, in the System Edge, a MongoDB database is used to store data for a longer period so that other CV applications can use the stored data.

System Integration and Validation: A Case Study with Collision Avoidance and Queue Detection Application

To verify the accurate integration of the hardware and software components with reliable communication systems to ensure the requisite safety and mobility CV benefits, the overall system integration was validated by testing the collision avoidance and queue detection applications.

Collision Avoidance Application

From the field tests with the CU-CVT, the research team recorded the time required to deliver emergency safety messages from a vehicle that stopped suddenly (CV-1 or Mobile Edge 1) to the immediate following vehicle (CV-2 or Mobile Edge 2) using DSRC. Three different ranges of vehicle speeds (i.e., 25 -30 mph, 35 -40 mph, and 45 -50 mph) were used for the field tests. Table 1 provides the average safety distance between CV-1 and CV-2 in three test runs with three different ranges of vehicle speeds (i.e., 25–30 mph, 35–40 mph, and 45–50 mph). Also calculated was the minimum safety distance requirement for CV-2 (Table 1, Column 3) when CV-2 applied the brake after receiving the first warning message, with a deceleration rate of 11.2 ft/s² ( 31 ). The motivation of this field test is to determine if measured DSRC message delivery latency can fulfill the requirement of a collision avoidance application after receiving the first warning message. As shown in Table 1, the minimum safety distance required is less than the average safety distance available between CV-1 and CV-2 at all three ranges of vehicle speeds considering maximum speed of the range. In addition, the observed average message delivery latency is significantly less than the latency requirements for CV safety applications ( 32 ). The receiving time of the safety message in the vehicles downstream depends on several factors, such as vehicle movement patterns, vehicle speeds, and any interference or noise in the environment. Thus, the distance to the upstream vehicle at the time of receiving the safety message varied for different tests. Simultaneously, the exchange of safety messages between CV-1 and CV-2 served as a prompt for the following three events:

Using LTE, the Het-Net system in CU-CVT delivered the collision-warning message from CV-1 and CV-2 to another vehicle (CV-3) which was beyond the DSRC range;

CV-1 sent BSMs to the nearest Fixed Edge via DSRC as CV-1 was within its DSRC range;

The Fixed Edge sent the BSMs to the System Edge layer using the Wi-Fi communication option available in CU-CVT.

Table 1.

Collision Avoidance Results for CV-2

Vehicle speed (mph)	Safety distance		Message delivery latency
Vehicle speed (mph)	Average safety distance of three test runs between CV-1 and CV-2 (when 1st warning message packet received)(ft)	Minimum safety distance requirement(considering 11.2 ft/s² deceleration rate)(ft)	DSRCaverage message delivery latency(ms)	Latency requirements for CV safety applications(ms) ( 32 )
25–30	156	38	2.4	200
35–40	201	118	1.8
45–50	316	240	1.7

In previous work ( 26 ), the research team proved that the latency for LTE wireless communication is above the required latency for CV safety applications but within the latency requirement to receive possible upstream incident warning. This discretion in latency proves the suitability of Het-Net for CV applications enabling multiple wireless networking options based on the latency requirements of the CV applications. A database system was hosted using MongoDB to store data at the Fixed Edge and at the System Edge. After collecting BSMs from the DSRC-enabled CV-1, the DSRC-enabled RSU at the Fixed Edge will publish BSMs to the MQTT broker system, and MongoDB archives BSMs from the MQTT broker. BSMs are also archived in the MongoDB database at the System Edge for future use.

Traffic Queue Detection Application

A queue ahead of a traveling vehicle is one of the major causes of rear-end collisions that can also disrupt traffic throughput by introducing shockwaves into the upstream traffic flow ( 33 ). Daily recurring congestion, work zones, incidents, and/or weather conditions are also causatives of traffic queues ( 33 ). Rear-end collisions and the impact of resulting traffic flow shockwaves can be significantly reduced through the rapid detection of a traffic queue, the prompt formulation of an appropriate response message for approaching vehicles, and the dissemination of queue information to the approaching vehicles in real time. A threshold-based queue detection algorithm was developed by USDOT using V2V and V2I communication utilizing DSRC ( 33 ). Typically, the threshold-based algorithm uses average speed of the vehicle (i.e., less than 5 mph) as well as the average separation distance (i.e., less than 20 ft) within DSRC communication range. Each CV in the DSRC range can broadcast a series of BSMs every 1/10th of a second. The research team implemented the threshold-based queue detection algorithm and evaluated queue detection application accuracy in the CU-CVT testbed. The queue detection application was used to effectively showcase the usage of different features of the testbed for developing and deploying CV applications. Both the average speed and average separation distance were used to detect a traffic queue in the testbed.

An application running in the Mobile Edge broadcasts vehicular information (i.e., time stamp, car ID, latitude, longitude and speed) to a Fixed Edge within the communication range of the DSRC. Using a python script at Fixed Edges, the separation distance between two consecutive CVs was calculated using the latitude and longitude of each vehicle. The speed and separation distance are also averaged of all three CVs for each second. Using the average speed and average separation distance, the queue was detected by running the developed threshold-based algorithm at Fixed Edges when three CVs stopped at the traffic signal in front of the Jervy Gym location. The accuracy of the threshold-based queue detection method is 81% for three test runs, and the threshold-based algorithm was run 167 times (every 1 second) for traffic queue detection. To calculate the accuracy of the threshold-based algorithm, the threshold-based queue prediction algorithm was compare with field-collected video camera data, which represents the ground truth data. As the accuracy of the algorithm depends on average separation distance between vehicles, the reason for lower accuracy is a higher separation distance between the CVs higher than the threshold because of the presence of non-connected vehicles between CVs. The backhaul network consisting of a Wi-Fi network is used to connect the Fixed Edges with the System Edge when real-time communication is required. Big data software, MongoDB, was used to store data at both the Fixed Edge and at the System Edge. This effort was used to demonstrate the suitability of the testbed for deploying both safety and mobility CV applications.

Conclusions

CU-CVT has several unique features, such as layered architecture, heterogeneous wireless communication medium, data analytics using real-time data streaming, and data archiving with big data management tools. The layered architecture is validated through a system design process. As the layered architecture is easily scalable depending on the deployment area and penetration level of CVs, the CU-CVT design is applicable for future large-scale deployment. In addition, the developed heterogeneous wireless technique can be easily implemented in many other CV testbeds. The selection between different wireless communication options (e.g., Wi-Fi, LTE) is based on the feasibility, accessibility, and data delivery requirements of the application. The use of this heterogeneous network technology also makes the expansion of traffic data collection for traffic management possible in terms of the data collection coverage area in a CV environment. The testbed and supporting software application programming interfaces are designed to make the application developed for the testbed independent of both different network technologies and networking devices. The testbed is also easily reconfigurable for forthcoming networking technologies such as 5G. The research team’s goal has been to develop applications and systems that can be easily integrated to any platform and systems, and that support any new networking technologies. The motivation has been to develop a connectivity strategy in which the applications can be developed independently of networking and other CV device attributes. These features make the CU-CVT testbed and application development platform reach a longer life cycle in terms of future CVT research, transferability of the work, and supporting current and future CVT technology development.

Lessons learned were identified at three different levels: (i) the development of design architecture and prototyping in a controlled environment, (ii) the deployment of the CU-CVT testbed, and (iii) the validation of the deployment with a CV application experiment. Although all wireless communication technologies, such as DSRC, LTE, and Wi-Fi are easily accessible by CVs, DSRC is usually preferred for CV safety applications because of its very low communication delay. The performance of heterogeneous wireless technologies, Wi-Fi, DSRC, and LTE was evaluated for the collision avoidance application and traffic data collection and archiving purposes. The field tests found that the heterogeneous wireless network can support multiple applications simultaneously, including safety and mobility applications. For data analytics in real-time data streaming, data aggregation and processing should be done based on on-demand data requests. Brokering systems can support data distribution for real-time data analytics by reducing data redundancy. In addition, real-time data archiving is required at the System Edge level for mobility and safety applications using big data management tools. The experimental results for the identified unique features of the CU-CVT testbed may advance future CV research and serve as reference to agencies for the deployment of CV infrastructure in the real world. The CU-CVT was found to be an invaluable resource for evaluating CV technologies and applications.

Footnotes

Acknowledgements

This material is based on the work supported by the Center for Connected Multimodal Mobility (C²M²) (USDOT Tier 1 University Transportation Center) headquartered at Clemson University, Clemson, South Carolina, USA and the National Science Foundation (NSF) under U.S. Ignite Grant #1531127. The authors also wish to acknowledge the editorial assistance of Mr. Godfrey Kimball in the preparation of this draft.

Author Contributions

The authors confirm contribution to the paper as follows: Study conception and design: Mashrur Chowdhury; Data collection: Mizanur Rahman; Anjan Rayamajhi; Sakib Khan; Mhafuzul Islam and Zadid Khan; Analysis and interpretation of results: Mashrur Chowdhury; Mizanur Rahman; Anjan Rayamajhi; Sakib Khan; Mhafuzul Islam, Zadid Khan and James Martin; Draft manuscript preparation: Mashrur Chowdhury; Mizanur Rahman; Anjan Rayamajhi; Sakib Khan; Mhafuzul Islam, Zadid Khan and James Martin. All authors reviewed the results and approved the final version of the manuscript.

The Standing Committee on Vehicle-Highway Automation (AHB30) peer-reviewed this paper (18-06561).

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the Center for Connected Multimodal Mobility (C²M²) and NSF, and the U.S. Government assumes no liability for the contents or use thereof.

References

Connected Vehicle Benefits . https://www.its.dot.gov/factsheets/pdf/ConnectedVehicleBenefits.pdf. Accessed July 28, 2017.

Department of Transportation, Federal Motor Vehicle Safety Standards; V2V Communications, Notice of Proposed Rulemaking , January 2017.

Federal Motor Vehicle Safety Standards; V2V Communications . https://www.federalregister.gov/documents/2017/01/12/2016-31059/federal-motor-vehicle-safety-standards-v2v-communications. Accessed July 28, 2017.

Federal Highway Administration. Vehicle to Infrastructure Deployment Guidance and Products, 2015. https://www.its.dot.gov/meetings/pdf/V2I_DeploymentGuidanceDraftv9.pdf, Accessed July 28, 2017.

Safety Pilot Model Deployment. Lessons Learned and Recommendations for Future Connected Vehicle Activities. https://ntl.bts.gov/lib/59000/59300/59361/FHWA-JPO-16-363.pdf. Accessed July 28, 2017.

Barbaresso

Cordahi

Garcia

Hill

Jendzejec

Wright

USDOT’s Intelligent Transportation Systems (ITS) ITS Strategic Plan 2015-2019. No. FHWA-JPO-14-145, Washington, DC: US Department of Transportation, Intelligent Transportation Systems Joint Program Office, 2014.

U.S. Government Accountability Office. Intelligent Transportation Systems: Vehicle-to-Infrastructure Technologies Expected to Offer Benefits but Deployment Challenges Exist. GAO-15-775. http://www.gao.gov/products/GAO-15-775. Accessed July 28, 2017.

Affiliated Test Bed Memorandum of Agreement , version 20140130. https://www.its.dot.gov/presentations/CV_Safety_sept2012/pdf/Day%202%20-%20Affiliated%20Test%20Bed.pdf. Accessed July 28, 2017.

The Ann Arbor Connected Vehicle Test Environment. http://www.aacvte.org/. Accessed July 28, 2017.

10.

New York City DOT Pilot. https://www.its.dot.gov/pilots/pilots_nycdot.htm. Accessed July 28, 2017.

11.

Additional information regarding New York City deployment initial Concept of Operation. http://www.its.dot.gov/pilots/pdf/NYC_ConOpsWebinar.pdf. Accessed July 28, 2017.

12.

Wyoming DOT Pilot. https://www.its.dot.gov/pilots/pilots_wydot.htm. Accessed July 28, 2017.

13.

Tampa Pilot. https://www.its.dot.gov/pilots/pilots_thea.htm. Accessed July 28, 2017.

14.

California CV TestBed , California. http://caconnectedvehicletestbed.org/index.php/index.php. Accessed November 9, 2017.

15.

Virginia Connected Vehicle Testbed. http://cvi-utc.org/test-bed-development/. Accessed July 28, 2017.

16.

Mak T.

Sengupta

Vehicle-to-vehicle Safety Messaging in DSRC. Proc., 1st ACM International Workshop on Vehicular ad hoc Networks, 2004, pp. 19–28, ACM.

17.

Connected Vehicle Reference Implementation Architecture. http://www.iteris.com/cvria/html/applications/applications.html. Accessed July 28, 2017.

18.

Chowdhury

Rahman

Dey

Apon

Luckow

Ngo

L. B.

A Distributed Message Delivery Infrastructure for Connected Vehicle Technology Applications. IEEE Transactions on Intelligent Transportation Systems, 2017.

19.

Rayamajhi

Rahman

Kaur

Liu

Chowdhury

McClendon

Wang

K. C.

Gosain

Martin

ThinGs In a Fog (TGIF): System Illustration with Connected Vehicles. Proc., 2017 IEEE 85th Vehicular Technology Conference (VTC Spring), Sydney, Australia, 2017.

20.

IEEE Standard for Wireless Access in Vehicular Environments (WAVE) – Networking Services . IEEE Std 1609.3-2016 (Revision of IEEE Std 1609.3-2010), 2016, pp. 1–160. DOI: 10.1109/IEEESTD.2016.7458115

21.

IEEE Standard for Wireless Access in Vehicular Environments. Multichannel Operation. https://standards.ieee.org/findstds/standard/1609.4-2016.html. Accessed November 9, 2017.

22.

IEEE Approved Draft Standard for Wireless Access in Vehicular Environments–Security Services for Applications and Management Messages - Amendment 1. IEEE P1609.2a/D8, 2017, pp. 1–122.

23.

Security Credential Management System Proof–of–Concept Implementation . https://www.its.dot.gov/pilots/pdf/SCMS_POC_EE_Requirements.pdf. Accessed November 9, 2017.

24.

Zhou

Chowdhury

Wang

K.C.

Fries

R.N.

A Framework for Performance Evaluation of Communication Alternatives for Intelligent Transportation Systems. Journal of Intelligent Transportation System, Vol. 13, No. 3, 2009, pp. 111–126.

25.

Zhou

Dey

K. C.

Chowdhury

Wang

K. C.

Process for Evaluating the Data Transfer Performance of Wireless Traffic Sensors for Real-time Intelligent Transportation Systems Applications. IET Intelligent Transport Systems, Vol. 11, No. 1, 2016, pp. 18–27.

26.

Dey

K. C.

Rayamajhi

Chowdhury

Bhavsar

Martin

Vehicle-to-vehicle (V2V) and Vehicle-to-infrastructure (V2I) Communication in a Heterogeneous Wireless Network–Performance Evaluation. Transportation Research Part C: Emerging Technologies, Vol. 68, 2016, pp. 168–184.

27.

Dey

K. C.

Yan

Wang

Shen

Chowdhury

Qiu

Soundararaj

A Review of Communication, Driver Characteristics, and Controls Aspects of Cooperative Adaptive Cruise Control (CACC). IEEE Transactions on Intelligent Transportation Systems, Vol. 17, No. 2, 2016, pp. 491–509.

28.

NoSQL Database. http://nosql-database.org/. Accessed July 28, 2017.

29.

Database Engine. https://db-engines.com/en/system/CouchDB%3BGoogle+Cloud+Bigtable%3BMongoDB. Accessed July 28, 2017.

30.

MQTT. http://mqtt.org/. Accessed July 28, 2017.

31.

Association of State Highway and Transportation Officials. A Policy on Geometric Design of Highways and Streets, 6th ed. American Association of State Highway and Transportation Officials, Washington, D.C., 2011.

32.

Segupta

Jiang

Chrysler

Design and Analysis of Highway Safety Communication Protocol in 5.9 GHz Dedicated Short Range Communication Spectrum. Proc., the 57th IEEE Semiannual Vehicular Technology Conference, Vol. 4, 2003, pp. 2451–2455.

33.

Report on Dynamic Speed Harmonization and Queue prediction Algorithm Design. Final Report – FHWA-JPO-14-168. February 28, 2014. https://ntl.bts.gov/lib/54000/54800/54895/FHWA-JPO-14-168.pdf. Accessed July 28, 2017.