Similar to how the cloud revolutionized data storage, Event Streaming Platforms (ESPs) are radically changing data processing and the ability to extract data insights. In this article, we explore how ESPs are driving change in the transportation industry.
Increasingly, transportation studies and insights involve various data collection requests—data points like volume, turn movement counts, travel times, or signal phasing data all help create a picture of a given transportation area. Due to resource limitations, this data has historically been collected for a specific time of day (i.e., morning peak, afternoon peak, and off-peak traffic periods). This limited dataset correspondingly translated into a limited understanding of the transportation landscape.
Traffic data collection methods and resolution of the collected data have evolved over the years, moving from human surveyors for manual collection to automated data collection using magnetic sensors, radars, and cameras. However, until the last decade, the collected data was not available in real-time, so it was often used to analyze past conditions without the possibility of active traffic management.
In the last several years, a vast amount of data was collected that enables real-time monitoring, active performance measures, and dynamic actions. Still, data insights are often siloed in separate facilities and offer little consideration for the entire transportation system. There is often no data sharing between departments, meaning individual departments use valuable budget resources to chase the same data insights, leading to increased data collection costs and reduced efficiency.
…And that’s where Event Streaming Platforms come in
Increasing communication and data sharing between departments and offices, as well as introducing automation in data analysis, is possible by building an ESP. An ESP’s main goal is to provide data exchange and advanced data analytics. An ESP can remove barriers between departments and jurisdictions as well as introduce data from third parties to provide an advanced traffic management system.
With an ESP, the focus moves from snapshots of individual facilities to a big-picture understanding of the entire transportation system. ESPs enable integrated management based on multiagency, multijurisdictional, and multimodal approaches and result in better-informed travelers, improved situation awareness, optimized use of existing infrastructure, enhanced responses and control, and dynamic management to achieve common objectives.
How Event Streaming Platforms help make roadways safer
ESPs have several use cases, including the ability to collect and analyze various data (e.g., speed; travel times; connected vehicle [CV] messages; road, traffic, and weather conditions; incidents; work zone conditions; etc.) to provide real-time information that can be shared with motorists through different Intelligent Transportation Systems (ITS) channels.
Consolidating CV data into existing transportation systems enables or enhances many ITS applications. The abundance of real-time data instantly produces and delivers traveler information that improves transportation safety by warning drivers of unexpected conditions, reducing primary incidents and secondary crashes. Traveler information is easily broadcasted through the 511 app, variable message signs, or CV onboard units (OBU). Additionally, introducing CV data from basic safety messages (BSM), intersection geometry (MAP), and Signal Phasing and Timing (SPaT) messages increases the performance of existing Advanced Transportation Measurement Systems (ATMS) and Automated Traffic Signal Performance Measures (ATSPM) systems.
ESP also improves asset management, including pavement, traffic signs, and marking conditions, and can provide real-time status and remote reboot options for CV devices. ESPs enable access to the tolling system, speeds, volume, and all historical data, including work orders.
How Event Streaming Platforms work
The ESP acts as a central nervous system where incoming data gets aggregated, filtered, and processed. The ESP needs to handle extremely large amounts of incoming data. As an example, one CV produces over 800,000 BSMs daily, which means a single CV produces about 1.7 gigabytes (GB) of data per day. Over a year, 100 CVs will create 63.072 terabytes (TB) of data. Due to an increase in CV data flow across various systems that cannot be accommodated by on-premises solutions, ESP relies on edge computing on cloud platforms and various technologies such as Publishers/Subscribers (Pub/Sub) service, microservice layer, or BigQuery.
To handle the large amount of incoming data, ESPs often use a technology known as Kafka. The Kafka platform is based on a Pub/Sub model that promptly allows processing and filtering data abundance. Kafka also allows for many different data pipelines to exist concurrently. For example, Kafka Topics for real-time weather and CV data may exist at the same time. The Kafka pipeline also allows a Real-Time Data Hub (RTDH) to process relevant data topics in near real-time to help make actionable insights for transportation. This architecture also allows for the data to be processed in a data lake for long-term storage and analysis.
Designing and developing an efficient ESP that will benefit a department of transportation (DOT) in operations ranging from designing intersections and roadways to providing real-time roadway alerts to the public requires an experienced software and data architect who can design an extensible and reliable ESP. From choosing the right cloud platform to designing the overall system, the process and complexity of establishing an ESP should not be underestimated. However, the benefits of building and establishing a trusted ESP can forever change the way DOTs are managed and drastically increase the quality and consistency of transportation operations.
Top considerations for Event Streaming Platform success
ESP is designed to be secure, reliable, and highly scalable. Top considerations for ESP success include:
- A high level of data management
- Understanding the quality of the data
- Maintaining data ownership and access control
- Third-party data ingestion
- Fostering data sharing
- Skillful workforce with experience in emerging technologies
Contact us
Trihydro’s transportation technology team has experience building various data exchange platforms within transportation systems. Connect with us if you plan to implement CV, test CV devices, or merge CV data into an existing system.
