-
-
OpenMetadata
Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
If you don't know what Wardley maps are, you should stop reading this article and go and study them here. You'll get more out of learning about maps than reading this article.
There are a lot of new vendors working on solutions to tackle Data Discoverability, Quality, Observability, Reliability or Lineage. Regardless of what question these services are proposing to tackle, I see all them all working to improve trust in an organisations data. For as long as there have been data systems and insights being produced, data teams are always asked to defend their numbers and ensure they are correct. These questions are valid and necessary as important decisions are often made based on those numbers. However, they do take time to answer and issues are often caused by data changing rather than any software bug. As data volumes grew and and all parts of the data stack expanded, it has become more difficult for teams to keep a track of all the data in their platform. Companies like Monte Carlo, Datafold, Bigeye and Metaplane all have products to help data teams keep on top of state of the data in their data warehouses. These tools all operate by tracking where data is sourced from, how it gets into the data warehouse, transformation and then profiling it at rest. Combined with open frameworks like Open Lineage and Open Metadata, these tools have the potential to improve organisations confidence in their data. Up to now, the vendors for these tools are targeting the data teams as their customer. Data teams understand the problem and are happy to outsource it if possible. I think the real potential for these tools will come when data customers are the ones using them. If data customers can bypass the data teams and check how much trust they can put in the insights being generated, that would be a game-changer. Self-service data trust if you will. Taking this a step further, if our industry ever gets to the point where the modern data stack starts to drive customer facing applications, tools that automate and verify data trust will become essential.