Snowflake Vs Databricks - 🏃♂️ A Race To Build THE Cloud Data Platform 🏃♂️
Переглядів 132,505
День томуDatabricks vs Snowflake
Two data storage solutions that started in very different worlds converging on the data platform.
Both want to be your one stop shop.
Your data warehouse and data lake
Your data lakehouse...
But really they want to be your data operating system.
If you're trying to pick between Databricks And Snowflake, or perhaps another data storage solution, then we should talk! Set up a free consultation today
calendly.com/ben-rogojan/cons...
Looking to start you're own data engineering/analytics consulting company, then you should check out my new course here
courses.technicalfreelancerac... - and use the coupon code "deconsult" to get 50% off
Also if you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer In 2022
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
Or check out my blog
www.theseattledataguy.com/
Articles from snowflake and databricks
databricks.com/blog/2021/11/1...
www.snowflake.com/blog/indust...
If you would like to learn more about data engineering, then check out Googles GCP certificate
* bit.ly/3BSBk62
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.
@SeattleDataGuy 8 місяців тому
If you need data infra or data strategy help, then feel free to set-up a free consultation! - calendly.com/seattledataguy/30-minute-meeting-requirements-yt
@kevintaylor463 8 місяців тому
Snowflake is for BI and more traditional analytics. Excels in data warehousing, storage, analytics. Data Warehouse engineers and Data Analysts. Databricks is for big data processing (machine learning, AI workloads). Data Engineers & Data Scientists.
@19FizZ91 Місяць тому
Ty!
@markwan7141 Рік тому
Love this. I'm a Solutions Architect and the information you give out is priceless and accurate. Well done!
@riffsoffov9291 Рік тому
Thanks for the video. Also thanks for the relevant title, rather than clickbait or one tailored for youtube's algos. I found you because Codestrap mentioned you (youtube's algos are useless for finding people who know what they're talking about).
@nicky_rads Рік тому
Nice video detailing two very popular data products, thanks ! Personally a big fan of notebook dev, but as you explain both products are solid and can serve customers well.
@eklok5000 Рік тому
Awesome video and comparison! Was missing some details on their features in the sense of functionality e.g. a Delta lake might have time travel while snowflake has some pretty Advanced sql functionality that one might use for gdpr hashing. On these „implementation“ details i Would love to see some content. Also bq vs snowflake would be very interesting i guess
@robwozniak9912 10 місяців тому
Well done! You very succinctly summarized the essence of both platforms.
@mg00 Рік тому
Micropartitions is just another name for... partitions aka shards, something that all databases use to scale out (because that's the only way to do it). There have been dozens of databases doing this for data warehouse/OLAP uses for decades. Snowflake (like BigQuery) was more powerful because of cloud-native scaling rather than provisioning real hardware, not just separating storage and compute. The other thing was usability from being in the cloud and running on object stores with features like zero-copy clones, sharing datasets across companies, etc.
@isaactucker-rasbury710 Рік тому
Thank you for this video. Looking at options for my team and this is really useful!
@agilejro Рік тому
I feel Streamlit has helped increase the depth and specialty application possibilities for Snowflake while keeping the streamline usability, do you feel this is true?
@andrew3068 Рік тому
our boy has been hitting the gym. loved the philosophical background on them at the beginning.
@SeattleDataGuy Рік тому
Just a little
@arjunrattan8190 Рік тому
Very informative. Curious, which Netflix show's clip you put in the scaling section?
@pavankumar-ni3my 6 місяців тому
I primarily do ETL activities for a product based org and would like to know when to use Databricks and Snowflake or rather how DB/Snowflake can help ETL engineers who migrate data from one product to another. I couldn't understand this video properly because I was expecting a one line answer on when one should use Databricks/Snowflake
@alexanderpotts8425 Рік тому
coming from an SSIS background I figured I'd end up landing in snowflake when I got to working with cloud tech, but I've really, really taken to spark and databricks. even if writing pipeline code in notebooks scares me sometimes
@Aussified Рік тому
SSIS is more aligned with what Azure Data Factory does and ADF is tightly integrated with Azure Databricks.
@thomsondcruz5456 Рік тому
Snowflake provides credit usage per second for each T-shirt size no VM costs to run the service. Databricks has its DBU cost per second for it’s service plus there is an underlying cloud VM cost to run databricks. This could be a plus or minus because databricks allows you to select the type of VM you want - Compute optimised vs Memory Optimised
@houmandehesh740 10 місяців тому
this is amazing thank you
@brothermalcolm Рік тому
I must be watching too many of your videos lately - searching for databricks landed me almost straight here
@SeattleDataGuy Рік тому
That's pretty impressive considering I just uploaded this 👀
@Rex_793 Рік тому
@@SeattleDataGuy you were top on my recommendations too; the youtube algo knows. 🏃
@SeattleDataGuy Рік тому
Thanks for your support!
@rajandalawai5640 8 місяців тому
I suggest that topics such as this that are heavy on content, it is better to go a little slower
@HistoricCookie992 Рік тому
Hello SDG, Would you suggest IBM Data Engineering followed by IBM Data warehousing and GCP at the last on coursera or DataCamp Data Engineering track ? Time isn't a problem. Yes, I'm switching career. I'm already learning Python and Sql already.
@user-zv9um9pb6w Рік тому
You missed core difference's. Databricks has a strong flexible offering that includes, ml, geospatial, ect. Serverless, ephemeral clusters ect.. snowflake does hosted data warehouse well... Thats what makes it good. You don't want to mess with managing complexity use snowflake ( with databricks) if not use databricks
@laananihamza1192 Рік тому
Hi there thank you so much for this great video, can you please tell me how databricks/snowflake could help me as a power bi/tableau developper ? and why we should consider them instead of a more simpler etl tool like power query for example ? Thanks again
@permiek Рік тому
Spark (and therefore Databricks) is really a game changer
@audreydelou Рік тому
So much game changer that databricks had to develop a new engine in memory because spark is not powerful enough
@victoriwuoha3081 Рік тому
Nicely explained.
@SeattleDataGuy Рік тому
Glad it was helpful!
@janardhand6200 Рік тому
Big like from india.. nice presentation
@ig2947 Рік тому
Thank you for great video. So if I have Databricks with Deltalake, do I need a seperaste data warehouse ? I mean will I be missing anything, by not having a DW ?
@Skandawin78 3 місяці тому
Excellent video, thanks
@SeattleDataGuy 3 місяці тому
thank you!
@KearlHat 6 місяців тому
Hi SeattleDataGuy - Which product has the best support? From the vendor, partner & community perspective? Thanks!
@cyclonus01 Рік тому
Thanks for the overview. Our organization went through a very contentious evaluation internally between these two platforms. Ultimately ended up going with SnowFlake due to internal politics. Would also love to see a video comparing all four: bigquery redshift snowflake databricks
@gj4king1 Рік тому
Which would you have gone with?
@jamaswin88 Рік тому
What would have gone with without the politics
@cyclonus01 Рік тому
@@gj4king1 probably DataBricks.
@cyclonus01 Рік тому
@@jamaswin88 probably DataBricks.
@SeattleDataGuy Рік тому
That'd be an intense video!
@HannayGwyn 11 місяців тому
"Using tools for what they're for", - yes, 100% ❤
@ashishk81 Рік тому
What databrick do differently which gcp, azure or AWS can't do ?
@digithat6496 Рік тому
Been loving snowpark sno far. It's like a wrapper of spark but by snowflake.
@SeattleDataGuy Рік тому
Any cool use cases that you have implemented? I haven't got to really play as deep in snowpark as I would like.
@digithat6496 Рік тому
@@SeattleDataGuy Yeah I love the new UDTF functions on snowfpark where you can like call python written udtf in SnowSQL with your py files or other packages in it. It's more like macros on dbt. Was able to join multiple tbales on different snowalke servers using snowpark and push it to a prod warehouse. I'm thinking to implement complex udfs like I did with pandas hopefully that works.
@michakrzeminski2857 Рік тому
its not spark underneath
@seasql Рік тому
need both databricks and snowflake for any enterprise application.. databricks ingestion the data from various sources and processing the data and stores in any database such as snowflake and postgres... snowflake is mainly used to OLAP application.. we cannot stores the data in delta table in databricka.. it's taking more time to return the data from delta table due to the more processing steps involved in spark.. job,stages,task,partition and cores but these kind of processing steps is not involved in snowflake and snowflake return data at earliest..
@kamusial Рік тому
have you tried Databricks SQL warehouse?
@EderFerreiraDias Рік тому
Great video.
@audreydelou Рік тому
not up to date
@bharanidharanm2653 3 місяці тому
I got into a LinkedIn thread dispute where a snowflake executive replied calling out snowflake is not proprietary and whoever calls it is a canard as per his viewpoint
@user-sb4hm3jc6m 5 місяців тому
The issue with Snowflake in my opinion: is that it is too abstract and too over simplified, i mean honestly it's good to work with, and you can do great things really fast. But then, after the rush of migration, when you want to scale and optimize the cost and performance of your workloads because it starts to be used for real. Then you realize that the only way to have better performance it to change warehouse size. Of course you can try to tune your queries you'd say, but the query plan is not very chatty, it's very minimalistic to say the least. You cannot tune the engine to fit your need, as you can do with Spark with partition related tuning of playing with different joining strategies ... You have absolutely no control on things like codecs, or compression, or dataformats ... And when you ask Snowflake support to know more about what is going on underneath, then you understand that, it is not going anywhere. My point is, that you'll have a very limited number of cost-free options to optimise your jobs and queries. And it can be ok for some companies because at the end of the day, you pay what you get, but it can also be very frustrating for others.
@SebastianWallkotter Місяць тому
I must disagree with the diminishing returns in Snowflake. From XS - 4XL I have thus far experienced a consistent inverse linear relationship, i.e., increase the cluster by one t-shirt size -> compute time reduces by 50%. Occasionally performance of the bigger cluster was even better, because more RAM means less disk spill for certain jobs, but I never noticed the inverse. This worked for small jobs (couple thousand rows moved between tables) all the way to our currently largest runs (joining around 1B rows against around 200B+ rows).
@xd4rkneo 2 місяці тому
One thing that I have not understood so far is whether Databricks can be perceived as a data virtualization tool (since they are not promoting it to be such). Yet to my understanding, Databricks is a perfect example for data virtualization. Am I wrong here?
@thomasnourse7506 Рік тому
Snowflake also has clustered warehouses... not more work... and also, the bigger the warehouse the faster performance therefore the same cost, just data retrieved faster... you need to catch up.
@tahirashafmithapuri2686 Рік тому
Good analysis but I want to add as we have been heavy snowflake users. Features like replication, Time-Travel and Data Shares do add a lot of value to our ecosystem.
@SeattleDataGuy Рік тому
Data sharing is honestly great! I have seen several teams save so much development time because of that feature!
@audreydelou Рік тому
Snowpark makes the difference enabling users to run python, java or scala code inside of Snowflake 😃
@deemahdee Рік тому
Can you make any comparisons or parallels with Palantir?? Not sure how Palantir competes with these companies
@SeattleDataGuy Рік тому
Been working to get access to palantir
@thomasadams6860 Рік тому
Have experience will a 3. Palantir is databricks but with a ui and a lot more walled garden. Snowflake is a good warehouse but starts to fail at massive scales. You need iceberg or delta with trino spark to do many trillions of rows.
@SeattleDataGuy Рік тому
@Thomas Adams I would agree with the palantir to databricks similarity. Its the fact that they treat everything like files 😆
@deemahdee Рік тому
@@thomasadams6860 what do you mean walled garden?
@thomasadams6860 Рік тому
@@deemahdee Palantir wants you to stay strictly in their ecosystem for everything and make it very difficult to use different things easily.
@juliustuckayo8973 Рік тому
Waking up to another nugget of gold here
@SeattleDataGuy Рік тому
Glad you enjoyed it!
@geehaf Рік тому
Like what you do. :)
@SeattleDataGuy Рік тому
Thank you!
@JohnKonye Рік тому
Do much content around Data Security?
@SeattleDataGuy Рік тому
Not currently. But I am aiming to dig into a few options in this space.
@JoeG2324 Рік тому
I'm thinking about having my team get started with databricks, but concerned about the cost. I think we get 3000 a month to spend, but after we start getting billed. Does anyone know how quickly that bill starts to add up? We're not doing any kind of streaming of data. we'd be mostly using it to run jobs once a day. Some of the queries can take anywhere from 5 to 10 minutes to run and then they output to various tables. I just don't want my department getting hit with a 30,000 dollar bill or something at month end
@SeattleDataGuy Рік тому
Have you talked to a databricks account executive? When you talk to them, they should give you some perspective on controlling costs. Also there are things you can do such as autotermination(similar to snowflake) which will shutdown your clusters when its not running. There are also lots of other things you can do to avoid costs such as putting limits. Personally I know a lot more about Snowflake cost management(which based on what I have looked up is far easier). But overall they both want you to spend as much as possible but they don't want you to leave. So when you talk to an AE, make sure they give some documentation on that(otherwise you have a terrible AE). docs.databricks.com/clusters/clusters-manage.html
@JoeG2324 Рік тому
@@SeattleDataGuy hey thanks. yeah, I need to setup a meeting with one of the account manager. hopefully, we can implement it without incurring too many costs. it looks like a great product.
@SeattleDataGuy Рік тому
@@JoeG2324 Are you going the machine learning route or the data warehouse route? I am curious why the databricks choice vs other options? What was the tipping point or feature that really made it make sense?
@JoeG2324 Рік тому
@@SeattleDataGuy thanks,we're going the datawarehouse route, my team doesn't handle any machine learning. The reason we're going with databricks is because my company already has a relationship with them as many other groups in the company use it. They have much larger budgets than my team so cost is def a concern
@jimthorstad Рік тому
@@JoeG2324 Hi Joe, once you schedule your jobs the Databricks clusters start and stop automatically and they also auto scale so if you’re data volume varies from day today you won’t have to oversize or risk under sizing your clusters. I’m sure you’re going to have fun setting up your first use case. If you have any trouble finding the right contacts at Databricks feel free to PM me and I’ll get you connected.
@badoiuecristian Рік тому
Databricks is clearly the winner here. The Snowflake offers simplicity but at the cost of customisation. If u have an app and want to optimise the cost of usage by selecting specific hardware for processing big data you can't do it in Snowflake as you can't touch the back end. I am certified in both and by now there isn't even a realistic comparison. Snowflake plays catch-up to spark. Not even mentioning the python native api that u can use with Databricks to do anything vs sql like syntax u need to use with snowflake.
@ryanromero2049 Рік тому
What do you think about Palantir?
@krishnayogi Рік тому
Indeed databricks is giving run for money to the big cloud guys at this moment
@alfredsfutterkiste7534 Рік тому
You make however a great argument in favor of Snowflake in general. Simplicity and hassle-free data warehousing are hugely attractive to companies struggling to attract and retain data engineering talent. Just pull in tables with Stitch, Fivetran or Airbyte and let a group of SQL monkeys dbt the data into shape. Pay the higher cloud bill because these SQL monkeys hired obv. can’t optimize queries but it doesn’t matter for the most part. Cheaper than hiring experienced data engineers who can turn all the knobs of your dwh to optimize it.
@neech235 Рік тому
@@alfredsfutterkiste7534 you can do exactly that in Databricks too though- it is actually more flexible to talent diversity as it supports all of the most popular languages used by analysts, including SQL. It has the simplicity that Snowflake offers but also far more depth and customisation
@badoiuecristian Рік тому
@@alfredsfutterkiste7534 well I disagree with the statement that hassle-free is better if you get limited fine tuning opportunity. Yes if you work with 2FB of data you don't care but as soon as you start working with TB of data which have to feed into an app that a user is frequently using then you feel the limitations. In my humble opinion you can have simple aka default settings but still leave the option for fine tuning.
@AnkitYadav-nh5by Рік тому
Hey SDG, have you looked at Cloudera’s Cloudera Data Platform (CDP). What are your thoughts on that ?
@shreshti82 Рік тому
CDP is based on Spark and other open source solutions. We reviewed it but given up, as there is a lot of management work. It's better to head to Snowflake / Databricks based on your current situation of data, if it's in cloud / on-prem.
@qwerty99060 Рік тому
For data engineer beginner, aws is better or azure?
@fa11en1ce Рік тому
They are the same, but there are more AWS jobs
@tomthetitan101 Рік тому
AWS has some fantastic free learning resources on EdX - I'd start with the cloud practitioner course if you know nothing.
@spencerhan6983 Рік тому
Microsoft runs the enterprise, aws runs the internet. Now it's your choice for your career vision.
@culpritdesign Рік тому
Azure data factory is very easy to learn and use
@SS-ud6nt Рік тому
Aws
@GoldJSilverio Рік тому
i felt this was a bit unstructured ....
@scarothers20 Рік тому
Both platforms are top notch. But at the end of the day Databricks takes the cake.. And If done correctly, MUCH more cost effective than Snowflake.
@audreydelou Рік тому
Snowflake has near zero administration, with no downtime for scaling or maintenance and provisioning takes so long with DBX ...
@scarothers20 Рік тому
@@audreydelou until your stuck with Snowflakes proprietary data formats. Without being biased, ill stick with open source route. Snowflake cost is everyone's main complaint..
@culpritdesign Рік тому
Azure synapse is a great contender
@vaibhavs120 Місяць тому
But it's expensive as compared to snowflake
@echoestakingls1218 Рік тому
PALANTIR?
@SeattleDataGuy Рік тому
Just filmed a video, need to edit and get approved
@SelvarajPrabhuMoorthy 8 днів тому
Snow flakes simply Structured data for analytics
@thomasnourse7506 Рік тому
Snowflake is on AWS, Azure, and GCP now.... just FYI
@user-bs8ku6cg9f Рік тому
where is palantir??
@SeattleDataGuy Рік тому
I finished my first video with them, waiting for some reviews from them
@user-bs8ku6cg9f Рік тому
@@SeattleDataGuy okey thanks buddy
@vancediaz7656 Рік тому
✋ p̾r̾o̾m̾o̾s̾m̾
@SeattleDataGuy Рік тому
Not sure what this means
@unitycatalog Рік тому
Databricks is light years ahead of snowflake
@adamesd3699 9 місяців тому
Different people have different perspectives on this, and it all depends on use case anyway.
@dave11F Рік тому
Unfortunate that snowflake does not support the R language.
@TheCheukhin Рік тому
Databricks completely defeated Snowflake.
@kylelarson5074 Рік тому
Both inferior solutions, I dont see either of these companies offering natural language processing applications for no code operational data analysis as other more innovative ML and A.I. native companies have.
@SeattleDataGuy Рік тому
Please do share some of the solutions you prefer!
@kylelarson5074 Рік тому
@@SeattleDataGuy I am not as generous as you are 😉
@SeattleDataGuy Рік тому
@@kylelarson5074 Well at least you're honest 😆
@vijayrajan5792 Рік тому
I would like to connect & talk with you on LinkedIn.
Seattle Data Guy
Переглядів 54 тис.
Студия Квартал 95 Online
Переглядів 400 тис.
УКРАИНА СЕГОДНЯ - ГРОШІ
Переглядів 1,1 млн
Суспільне Харків
Переглядів 449 тис.
Seattle Data Guy
Переглядів 21 тис.
SQLBits
Переглядів 53 тис.
Darshil Parmar
Переглядів 138 тис.
Seattle Data Guy
Переглядів 5 тис.
Зeлeный чaй
Переглядів 1,7 млн
Screen
Переглядів 694 тис.