Close Calls (Insight Data Engineering)

Close Calls is a data engineering project combining an Apache Spark batching pipeline and a custom visualization using Google Maps and Aurelia to display near-misses of worldwide air traffic. The data source was roughly 80GB/day of JSON files from adsb-exchange.com. In total, I had around 500 GB of data and was able to process it all in faster than real time.