I do! Congratulations!! That's so awesome to hear!

Love this!

Congrats man, how it's going ❤

amazing

Congrats!

Yes! ETL is something I've been trying to learn and I got a new job where I will be able to learn more but having a source to learn the foundations would be great!

I would like to see this as well! Thank you!

I will love to have you do this for us at a granular level. Many thanks

I write a lot of my own ETLs, with T-SQL/powershell/cmd, etc., most very large amounts of delimited text based data into a Star schema type of data structure. I always build it to tolerate changing file structures, using a “Stage then Load” type of method, bulk load and stage to all Varchar defined columns then Dynamically build and merge (upsert) from stage to prod/fact table by stage/fact column name match via the system catalog or some type of metadata definition tables for column mapping/join and key columns to use for merging (mostly time series) data. It’s important to use features defined sample sizes (i.e. Batchsize) so as not to cause excessive T log or resource usage on the back end, especially if a shared Relational data server. Many of these vendor products cannot handle files that change, whether data type, column / header name change, etc. without the load failing, or requiring a new definition/format setup each time. I’ve actually come across one or two products that attempt a single transaction when loading very large data files to tables, that caused excessive T-log growth, and brought the DB server to a halt. I write mine so it will always attempt the load to “known” columns, then report any that were new or missing. I also define processes to “transform” based on staging to fact column data type match, one example being many files from excel sheets have this “serial” date value that needs to be cast to datetime from a 5 digit numerical value. I need to go the next step though and how to incorporate AI with it. We get tons of loan payment data to be analyzed, probably 40-50 GB a month, with 30 or so years currently stored in SQL tables, one table 4-5 TB in size, both page compressed and partitioned, used for both very selective queries and queries that process 2-4 year data samples. Fun stuff. I would love to learn new methods that successfully replace pre aggregating, that work fast and are very flexible. I’ve Worked in the past with products like Hyperion, Cognos, SSIS, etc. as well as some vendor DW products like Paladyne, but they are always an 80% solution, with the remaining 20% being 80% of the work.

For sure - I'm implementing an Azure Data Lake at my company right now and that's exactly what we are trying to avoid lol

@@AlexTheAnalyst adls gen2 using hierarchical namespaces? good technology :)

A data swamp is a great addition to the list, although it's of course the thing you want to avoid.

Better lakehouse than leakhouse :D

Lakehouse is a term that marries the Data Lake and the Data warehouse. It does this by adding an abstraction layer to help organize and normalize data in the data lake with a combination of a hive meta store and a Delta format.

Yeah that's usually what a data lake is for - some type of centralized system

I'm so honored to hear that! Keep up the good work!

I’m new to a higher level data position than I’m used to. I would love this kind of content as well.

A data warehouse is queried using SQL, if the data warehouse is built using a relational database technology, such as MySQL, Oracle RDBMS, SQL Server. You usually don't query data lakes. They are mainly used for AI/Data Science purposes, and since they also contain non-structured data, you would not use SQL (Structured Query Language) which is designed for relational databases.

Is this THE Moses Varghese?? Thank you man! :D

ur welcome

Glad it was helpful!

Glad it was helpful!

You're welcome!

Glad to hear it!

Like a Data Lake-house then? :)