SharkSmart choose Snowflake to protect the data of the world’s largest shark monitoring program.
In October 2015, prompted by the rising incidence of shark interactions, especially along the NSW North Coast, the NSW Government made a resolute commitment to the $16 million NSW Shark Management Strategy. This strategy was meticulously designed to trial innovative technologies aimed at bolstering protection for beachgoers while concurrently minimising harm to sharks and the broader marine ecosystem.
Among the groundbreaking technologies introduced were the SMART Drumlines and Shark Listening Stations, instrumental in the tagging and tracking of White, Tiger, and Bull Sharks along the New South Wales coastline.
The initial trial of these technologies proved remarkably successful and marked the inception of an enduring journey. Over the years, this initiative has grown substantially and now encompasses an impressive network of 37 listening stations and 305 SMART Drumlines strategically positioned along the picturesque coast of New South Wales. This expansive network stands as a testament to the program's evolution, establishing it as the most extensive and sophisticated shark tracking program globally.
“The quest was to find a platform that could streamline data sharing, eliminating the need for cumbersome CSV exchanges. Such a solution would not only reduce administrative burden but also save valuable time for the dedicated NSW research team.”
STORY HIGHLIGHTS
Enhanced Data Resilience and Protection: The SharkSmart program's adoption of Snowflake dramatically improved data resilience. The transition from a vulnerable Excel file to Snowflake's robust platform eliminated the looming risk of data corruption. With Snowflake's architecture, the program gained a high level of data protection, shielding its valuable dataset from potential threats. This transformation provided an essential layer of business resilience, ensuring the program's critical data remained intact and secure.
Streamlined Administration and Resource Efficiency: Snowflake emerged as a game-changer by streamlining data administration for the SharkSmart team. In a unique context where there were no dedicated Database Administration (DBA) resources, Snowflake's managed platform offered a solution that significantly reduced administrative complexity.
Versatile Data Management: Snowflake's adaptability was pivotal in managing the program's multifaceted data ecosystem. Handling data from various sources in both batch and near real-time, Snowflake provided a unified platform for data ingestion.
CHALLENGE
Since its inception in 2015, the SharkSmart program has been instrumental in collecting data from tagged sharks, contributing significantly to the understanding of shark movements. By 2023, this invaluable dataset had grown to over 260,000 rows of notifications. Initially, the team had been managing this wealth of data in Excel. However, with this magnitude, Excel was struggling, and file corruption became an imminent threat.
As the number of sharks tagged continued to rise, it translated into an exponential increase in listening station notifications. The SharkSmart team recognised the need for a contemporary data warehousing solution that could address these challenges in a cost-effective manner while minimising the technical administration.
Notably, the NSW Shark Tagging program is just one piece of the global marine tagging landscape. This international connection meant that marine animals tagged by other organisations were also being detected by the NSW listening stations. Consequently, the NSW detection data is routinely shared with these global research organisations through CSV exchanges. The quest was to find a platform that could streamline data sharing, eliminating the need for cumbersome CSV exchanges. Such a solution would not only reduce administrative burden but also save valuable time for the dedicated NSW research team.
SOLUTION
The pivotal decision to choose Snowflake as the backbone of SharkSmart's data management was driven by several compelling factors. As the program thrived and the volume of data surged, a robust and scalable data warehousing solution became an imperative requirement.
Scalability and Performance: Snowflake emerged as the natural choice due to its reputation as a highly scalable and exceptionally performant data warehouse. The ability to seamlessly accommodate the ever-expanding dataset, which now encompassed over 260,000 rows of notifications, was a paramount consideration. Snowflake's architecture allowed for the program's rapid growth without compromising data retrieval speeds, ensuring that the SharkSmart team could access critical insights in a timely manner.
Minimised Administration: An essential consideration in the selection process was the fact that the SharkSmart team operated without dedicated DBA resources. This distinctive aspect posed a unique challenge; the chosen system needed to be user-friendly, minimising administrative overhead and technical complexities. Snowflake's cloud-based architecture and intuitive management interfaces ensured that the team could effectively manage their data without the need for extensive technical expertise. This user-friendly approach significantly reduced the administrative burden and enabled the team to focus on their core mission—tracking and safeguarding sharks while ensuring the safety of beachgoers.
Versatile Data Sourcing: SharkSmart's data ecosystem sources information from multiple systems in both batch and near real-time. This multifaceted data flow presented a unique challenge. However, Snowflake's versatility shines here as it adeptly handles both batch and real-time data, offering a unified platform for all data ingestion needs. This seamless integration ensures that the SharkSmart team can work with the most up-to-date data in their mission to monitor and protect marine life.
Enhancing Data Quality at Sea: One of Snowflake's critical data sources is generated by staff during the dynamic process of shark tagging at sea. Amidst the awe-inspiring backdrop of the open ocean, researchers engage with sharks ranging from 2 meters to upwards of 4 meters in length. In this unique and challenging environment, data entry errors inevitably arise. The SharkSmart researchers lack SQL expertise, making it essential to provide them with a straightforward interface for rectifying data quality issues. Recognising the need for a user-friendly solution, the SharkSmart team turned to a Snowflake-embedded Streamlit app. This innovative app not only streamlines the correction of data quality but also minimises the administrative complexities typically associated with custom applications.
In essence, Snowflake's scalability, performance, user-friendly interface, and innovative Streamlit App stand as a testament to the SharkSmart program's mission to harness technology, streamline data management, and champion the cause of marine conservation. This comprehensive solution empowers the team to manage their expanding dataset with ease while continuing to drive their critical mission forward.
RESULTS
The implementation of Snowflake within the SharkSmart program delivered a series of remarkable results, underscoring the program's enhanced operational resilience and efficiency:
Enhanced Business Resilience: One of the primary challenges addressed by Snowflake was the impending risk of data corruption within the large Excel file previously used to manage the program's burgeoning dataset. By transitioning to Snowflake, the SharkSmart program fortified its business resilience. Snowflake's robust architecture and high level of data resilience ensured that the program's critical data was protected from potential corruption risks. This transformation provided peace of mind, eliminating the specter of data loss due to file corruption.
Streamlined Administration: Unlike legacy databases that demand extensive technical expertise and constant DBA involvement, Snowflake's managed nature ensures that routine administrative tasks are handled seamlessly within the platform. With no dedicated DBA resources at their disposal, Snowflake provided a robust solution without the need for additional headcount.
Versatile Data Handling: The SharkSmart program's data ecosystem presented a complex challenge with information flowing from multiple systems in both batch and near real-time scenarios. Snowflake's versatility was a game-changer in this regard. It seamlessly accommodated both batch and real-time data, providing a unified platform for data ingestion.
Effortless Data Sharing: One of the program's unique needs was the ability to share data. Snowflake's data sharing capabilities eliminate the need for cumbersome CSV exchanges, enabling seamless sharing of data with other research organisations around the world.
In summary, Snowflake's integration empowered the SharkSmart program to overcome multiple challenges while enhancing their business resilience. It replaced a vulnerable Excel file, reduced administrative complexities, managed diverse data sources, and facilitated effortless data sharing. This comprehensive transformation has not only fortified their data management but also propelled the program's mission in the realm of marine conservation with newfound efficiency and effectiveness.
For more information get in touch with us via email at info@pivotanalytics.com.au or calling 1300 475 510.
