Fivetran introduces Metadata API – enables end-to-end data analysis and visibility

Fivetran has introduced Fivetran’s Metadata API to enable the tracking of data “in-flight” from source to destination as it moves through Fivetran-managed pipelines. With this additional visibility, customers can integrate with governance and observability tools to give data teams more control over who has access to what data. Enabling automated data governance, the Fivetran Metadata API also provides data stewards, security teams and data engineering teams the needed visibility to answer where the data came from, who accessed it, and what changes have occurred in the pipeline.

“Every enterprise knows it must be data-driven, but traditional data governance has been a barrier with manual processes and reactive enforcement of policies. That's not a scalable approach, especially as data infrastructure grows to thousands of pipelines,” said Fraser Harris, VP of Product at Fivetran. “With Metadata API, our customers get out-of-the-box data governance automations and data quality workflows so they can proactively identify and take action on governance issues before they become a problem. Our automated in-flight approach enables data access at scale without increasing risk to the business.”

Complex legislation such as Europe’s GDPR, the California Consumer Privacy Act (CCPA), and other state laws impose fines on companies that do not comply with complex rules to safeguard data privacy. The Fivetran Metadata API helps enterprises address compliance requirements, easily integrating into their existing privacy and security strategies while bolstering the value of investments they’ve made in data catalogues and data quality solutions.

With the Fivetran Metadata API:

Data analysts are provided with a deep understanding of where data is coming from and are able to run impact analyses on it

Data stewards know end users have access to data that has been handled securely and is compliant with governance requirements

Security and legal teams can complete security audits and ensure the data moved is in compliance with organisational policies

Data architects and engineers will soon be able to understand upstream schema changes and ensure downstream processes are updated.

Creating an automated data governance experience – Fivetran partners with leading data catalogues

Fivetran is excited to launch better metadata management with four leading data catalogue vendors: Atlan, data.world, Alation and Collibra. Here are the combined benefits:

Information about all data can be consolidated into a single data catalogue, for a complete view and seamless user experience

End-to-end data lineage graphs are available for data, despite data passing through multiple systems and tools

By centralising governance in a single tool, data stewards can better ensure policies and processes are being applied to data as necessary 

The ability to source trace data at a column level back to its origin helps confirm data quality and builds trust that the data is accurate and safe to use.

“Fivetran’s Metadata API solves a major gap in extracting data from operational systems into modern analytical systems by delivering the much-needed context. The availability of the Metadata API will accelerate development of reliable and secure data-intensive applications by exposing lineage, impact analysis, and security and privacy aspects,” said Sanjeev Mohan, Principal at SanjMo, a data and analytics expert and former Research Vice President at Gartner.

“Our customers are excited to move faster with more visibility, powered by metadata from Fivetran,” said Prukalpa Sankar, Co-Founder at Atlan. “No need to manually stitch together sources anymore. With the new Metadata API, customers can now bring metadata from Fivetran into Atlan for a truly automated data governance experience. This end-to-end lineage covers everything from upstream sources to the warehouse and BI layer, helping customers stay informed across complex data stacks with powerful root cause and impact analysis.”

“data.world's interface with Fivetran enables data producers and data consumers to understand and trust the data synchronisation and mapping along all databases, data sources and applications that matter to enterprises,” said Jon Loyens, CPO & Co-Founder at data.world. “Together with Fivetran, we provide granular data visibility,  including metadata of source-destination column pairs, simplifying data discovery, data governance and actionable insights.”