This is the Bible for DLT! Worth to loop and study! Well done🙌

Ha, the CEO knowing where the raw data is stored.. almost died laughing there.

Excellent Demo! Would you please provide the PDF file of this demo as well as the code for us to practise? looking forward to hearing from you.

Super good stuff. I wonder if the the function call inside the loop @1:13:22 should have been `create_report(r)` instead of `create_table(r)`?

interesting! I now have an idea of what delta live tables can do

43:10 this is awesome man.

I am not sure I understood the repeating references to the "errors in our docs"? Can you please clarify? What would be a reasone to publish docs with the errors, please? Is there quality control over these docs?

Would it be possible to create unmanaged tables with a location in datalake using DLT pipelines ?

anyone have a link to the slides?

where can i have the PPT? and demo code?

@michaelarmbrust2076 While using apply_changes, how do we handle duplicates in the sequence by column in a stateless way? Does dropDuplicates deduplicate data for the micro-batch like a forEachBatch would? or would it attempt to deduplicate the whole stream unless a watermark is given?

question here, why i run the same will get error "16:08:48 Running with dbt=1.6.2 16:08:49 Registered adapter: databricks=1.6.4 16:08:49 Unable to do partial parsing because saved manifest not found. Starting full parse. 16:08:51 Found 2 models, 0 sources, 0 exposures, 0 metrics, 471 macros, 0 groups, 0 semantic models 16:08:51 16:14:02 Concurrency: 8 threads (target='databricks_cluster') 16:14:02 16:14:02 1 of 2 START sql streaming_table model default.device .......................... [RUN] 16:14:03 1 of 2 OK created sql streaming_table model default.device ..................... [OK in 0.53s] 16:14:03 2 of 2 START sql materialized_view model default.device_activity ............... [RUN] 16:14:04 2 of 2 ERROR creating sql materialized_view model default.device_activity ...... [ERROR in 0.82s] 16:14:04 16:14:04 Finished running 1 streaming_table model, 1 materialized_view model in 0 hours 5 minutes and 12.60 seconds (312.60s). 16:14:04 16:14:04 Completed with 1 error and 0 warnings: 16:14:04 16:14:04 Runtime Error in model device_activity (models/example/device_activity.sql) [TABLE_OR_VIEW_NOT_FOUND] The table or view `main`.`default`.`device` cannot be found. Verify the spelling and correctness of the schema and catalog. If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog." from my understanding the table only can created by DLT pipeline, DBT cannot create the table. but you succesd in create the streaming table and MV. May i know why?

37:10 no azure storage accounts?

Quick Question : If a record is deleted from Source table hard delete how apply_changes cdc will handle ?

Why can't we run the code in the cell for debugging? I have found DLTs have so much limitation and hard to debug.

Please share the repo for cicd atleast

Straming=is always costly... If you trigger it manually or on schedule it is not streaming...

John Carmack, is that you?

This is the way how you can make Data engineer job easy and pay less to them.

Hard to follow unless you know a lot already.

just another stereotype reinvention of a wheel