BREAKING DATA SILOS FOR EFFECTIVE DECISION MAKING

K. Fordham & N. Brown (Auckland Council), S. Greenwood & N. Tomasi (Mott MacDonald New Zealand Limited)

ABSTRACT

Reality is complicated and messy. Key decisions that might allocate resources, communicate to the public, or direct rapid responses to events, all strongly depend on the full context of the situation. While the amount of data collected every day is growing at an ever-increasing rate, the interpretation of these datasets for decision making often present challenges. Complex problems can rarely be resolved by the analysis of a single data source. The integration of multiple datasets into a unified system is crucial to inhibit data silos, enable big-data analysis, and deliver effective outcomes. The Auckland Council and Mott MacDonald have spent several years employing such unified systems, which provide real-time data analysis from a wide range of sources to deliver targeted insights.

The Auckland Council Rainfall Analysis System is an example of these integrated systems. This cloud solution analyses data such as rain radar, rain gauges, water flow/level sensors, public requests for service, cameras, weather warning and GIS layers. It supports the Auckland Council’s flood response by consolidating datasets and specialist services (such as rain radar processing), making this information easy to access across the organisation and their partners. Crucially, the analysis of these datasets is visualised in an appropriate manner for operators and decision-makers to quickly and reliably identify areas impacted by storm events and to get a comprehensive understanding of the situation on-site to prioritise resources efficiently, without the need to compare information from different proprietary software packages.

Other examples of integrated systems include Safeswim, the water quality digital twin for public beaches. This solution allows the public to make informed decisions around the health risks of swimming and reflects a wide range of integrated datasets and models the Auckland Council, Mott MacDonald and other partners such as Watercare, Puhoi Stour, DHI and Surf Life Saving New Zealand have collaborated on. To provide real-time accurate forecasts of water quality, consideration of each dataset in isolation is insufficient, consolidating data to provide a unique source of truth is critical to ensure the reliability of the information shared with the public.

These schemes operate upon a strong platform specially built to enable data integration and analysis, owing its success to a series of key development plans and design architecture choices. Early and continued investment has allowed the platform to be appropriately planned in collaboration with organisations with a shared vision, keep improving, and staying on the right track to deliver the desired outcomes. The platform is scalable, to meet the challenges of working with large datasets such as the rain radar, which provides >20,000 virtual rain gauge sites across the Auckland region with 60 second temporal resolution. A robust API also unlocks the ability to build automated workflows for any third party who can access and add to these datasets for further analysis.

To reach the intended outcomes, clear communication is necessary between all teams involved, from software developers working on the data ingestion through to end-users interpreting data and interacting with the platform. Each step in the pipeline from acquiring and storing the data, processing them in conjunction with other data sources, and visualising it to communicate information about the physical world need to complement each other.

The results are award winning innovative solutions recognised throughout the globe as best in class. Breaking data silos and enabling collaboration in the industry supports the Auckland Council in delivering better outcomes for the community of Auckland.