What is Data Observability?

Рік тому

Learn more about Databand → ibm.biz/IBM-databand
Check out IBM Analytics → ibm.biz/ibm-analytics
Data observability can help data teams and engineers better understand the health of data in their system and automatically identify, troubleshoot and resolve issues in near real-time.
In this video, Ryan Yackel talks about the difficulty in managing the health and quality of your data sets, what data observability is, and how it can help you fix issues with your data before it impacts your bottom-line.
Get started for free on IBM Cloud → ibm.biz/ibm-free-cloud-tier
Subscribe to see more videos like this in the future → ibm.biz/subscribe-now
#databand #observability #DataScience #MachineLearning #Deeplearning #ArtificialIntelligence #BigData

КОМЕНТАРІ: 21

@Flankymanga Рік тому

Hello guys. Thank you for this blackboard session. I have a suggestion: Theory is nice and it is the first step to get into the specific IT topic. But as they say: seeing is believing... could you please make a video with practical use case, how data observability practices are being utilized to solve data pipeline issues? In other words: How data observability looks and works in real life on real tools. That would be awesome to see. And if you could merge this topic with Data Governance that would be totally awesome! I am loving these blackboard explanations! Thank you very much!

@AISmallBusinessHub 10 місяців тому

Thanks for breaking this down clearly. It makes it easy my product designer brain to understand the concepts

@dineshshekhawat2021 Рік тому

Very insightful, learned a very useful concept on a high level. Keep bringing such videos!

@alexpishvanov736 Рік тому

I really like this idea to shift thinking paradigm from been software engineer to data engineer:)

@nikitashrestha684 Рік тому

Hello, thanks for this explanation. I would like to see more videos on NLP domain. To be more specific using pre-trained tranformers for text analytics ( Transfer Learning )

@robfletcher6274 Рік тому

Great explanation, nice one

@mustufabaig1483 Рік тому

What I understand from this video is that data observability is just a grouping of "data quality", "data lineage", "ETL Logging" etc. into one umbrella. These are common concepts within data warehousing/engineering. Some teams solve them with frameworks/vendor tools, some through custom development and some use both approaches. Data observability is just a new name for it. good presentation though.

@ladonwilliams1333 Рік тому

Absolutely great overview. Appreciate the train analogy. I got a good understanding of the premise of Data Observability, plus the bonus of a concise understanding of data engineering and pipelines. Need this gentleman explaining more concepts!

@MsVikas99 2 місяці тому

Really great overview. Thank you

@billlabranche Рік тому

Great illustration to the concept of Data Observability - thanks for making it simple enough to understand. PS... I'm very impressed with your "backwards writing skills"!!!!

@rajarshisadhya8174 Рік тому

Thanks much for the really cool explanation. Wondering whether there is any tool available that provides the capability to really ‘Observe’ any changes/incident in the data pipeline?

@thedatawhisperer Рік тому

Great job!

@ARATHI2000 9 місяців тому

Understood in theory..Thanks! Several years ago, one issue we used to run into was some of our Enterprise customers were missing sending data on defined schedule into our SaaS env. Our ETL data ingestion job (sort of data pipeline tool) won't help since it won't be kicked off. So we wrote something outside of the ETL job to ensure we can catch such data misses by the customer...could/should this be part of data observability?

@paddyarunachalam7057 6 місяців тому

Thank you. Great session. Could I ask what technology you use to put this together?

@SoniaRaval 9 місяців тому

Great overview and love the train analogy

@jorge-hernandez-ramirez Рік тому

Good!!

@silicon6086 Рік тому

Is Data Observability same as DataOps ?

@ladonwilliams1333 Рік тому

I think that Data Observability is more of a technical implementation to identify/remediate issues in the data pipeline that is managed by Data Engineers and others potentially involved in the process due to cross-functional responsibilities. Whereas DataOps is a methodology to foster a culture of effective communication and collaboration between data stewards, data users, and executives/business process owners. So, Data Observability helps to ensures quality data so that DataOps can glean business value from data presented. In essence, Data Observability integrally folds into DataOps.

@vio4jesus Рік тому

SoooOoooooo we've been moving data for years with ETL tools. Tools that generally don't require a super coder. Some of these tools have some level of monitoring built in, and some have data quality modules. BUT NOW..... let's go back to hand-writing code to move data. Hhhhhhmmm seems like a bad idea.

@davidmurphy563 Рік тому

I think it would have been a better presentation if he had stuck to concrete examples more; like with the betting company. The train was cute, and I did get it, but he explained something abstract by making it even more abstract. Heading in the other direction and injecting some hard reality would have been more instructive. At the end of the day, this is not an academic exercise, it's there to solve real world problems.