Vigil #5 - The Finishing Touches
Bringing it all together with a Grafana dashboard and parallel data streams.
Vigil is a real-time traffic accident detection system that I've been building and documenting here. This is the final post in the series, where I'll cover the last leg of the journey: building a user interface with Grafana and scaling up the system to handle multiple camera feeds in parallel.
A Quick Recap
In the previous posts, I laid the groundwork for Vigil. I started by figuring out how to extract frames from online traffic cameras, then built a basic end-to-end pipeline to process them, and even took a detour to understand the nitty-gritty of video processing. The goal has always been to create a system that can monitor a large number of traffic cameras and detect accidents in real time.
Now, with a functional pipeline that can extract frames from a camera and run them through a machine learning model, it's time to address two crucial aspects: visualization and scalability.
Giving Vigil a Face: The Grafana Dashboard
So far, the output of Vigil has been a stream of text in the terminal. While that's fine for debugging, it's not a practical way to monitor the status of dozens or even hundreds of cameras. I needed a user interface, a "face" for Vigil.
Enter Grafana. For those who are not familiar with it, Grafana is an open-source platform for monitoring and observability. It's incredibly versatile and allows you to create beautiful, interactive dashboards from various data sources. In our case, the data source is InfluxDB, a time-series database where Vigil stores information about each camera feed.
The setup is defined in the docker-compose.yaml file, which orchestrates the different services that make up Vigil. I've included Grafana and InfluxDB in the mix, so a single docker-compose up command is all it takes to get everything running.
The result is a simple but effective dashboard that shows a list of all the camera feeds Vigil is monitoring. For each camera, I can see:
Camera Name and Road
Status
Incidents
Camera Link
Last Update:
Here is a sneak peek of what the dashboard looks like:
This dashboard transforms Vigil from a command-line tool into a proper monitoring system. It provides a bird's-eye view of the entire network of cameras and makes it easy to spot any incidents at a glance.
Scaling Up: Parallelising the Streams
The second major challenge was scalability. The initial version of Vigil processed camera feeds one by one, in a sequential loop. This was fine for a handful of cameras, but it wouldn't work for the hundreds of feeds I planned to monitor. The processing time for each feed would add up, and the system would quickly fall behind, defeating the "real-time" aspect of the project.
The solution was to parallelise the data streams. Instead of processing one feed at a time, I modified the system to handle multiple feeds concurrently. I used Python's ThreadPoolExecutor to create a pool of worker threads, where each thread is responsible for a chunk of camera feeds.
The start.py script now orchestrates this process. It fetches the full list of cameras, divides it into smaller chunks, and assigns each chunk to a thread in the pool. Each thread runs a FeedExtractor instance, which extracts frames from its assigned cameras and sends them to the FeedClassifier for processing.
This parallel architecture is a game-changer for Vigil's performance. It allows the system to scale horizontally – to a point – by simply adding more threads to the pool. The result is a much more efficient and responsive system that can keep up with a large number of real-time video streams.
The Final Picture
With the Grafana dashboard and the parallelized data streams in place, Vigil has reached a state of completion. It's a system that can:
Fetch a list of traffic cameras from an online source.
Extract video frames from these cameras in parallel.
Run the frames through a machine-learning model to detect accidents.
Store the results in a time-series database.
Display the status of all cameras on a real-time dashboard.
This project has been a fantastic learning experience. It started as a simple idea and evolved into a complex system with multiple moving parts. From video processing and machine learning to data streaming and visualisation, Vigil has touched upon many different aspects of modern software engineering.
Of course, there's always room for improvement. The accident detection model could be fine-tuned, the system could be made more robust, and the dashboard could be enhanced with more features. But for now, I'm happy with what I've accomplished. Vigil is a working proof-of-concept that demonstrates the power of combining computer vision, machine learning, and modern software tools to solve real-world problems.
Thank you for following along on this journey. I hope you've found it interesting and perhaps even inspiring. As always you can check all the code at Github.
Until the next project, happy coding!