List: Data Engineering | Curated by Kris Lee

Jan 10, 2024
33 stories
Data Engineering
Ayush Thakur
Python Decorators That Can Reduce Your Code By HalfHi there, welcome to my blog! Today I’m going to share with you some amazing Python decorators that can reduce your code by half. Sounds…
Nov 14, 2023
35
Nov 14, 2023
35
Oindrila Chakraborty
How to “Optimize” the “Delta Tables” in DatabricksHow “Delta Tables” Become “Slow” to “Query”?
Sep 15, 2023
1
Sep 15, 2023
1
Matt Weingarten
Successful Engineering With Databricks
Sep 26, 2023
2
Sep 26, 2023
2
Youssef Mrini
How to pass the Associate Databricks Data Engineering Certification ?Number of questions : 45
Jul 4, 2022
Jul 4, 2022
In
OCTAVE — John Keells Group
by
OCTAVE - John Keells Group
How to run Unit Tests on Azure Databricks Notebooks using Azure DevOps CICD PipelinesThis is the second article in a series about Azure DevOps and MLOps. Missed Part 1? Read it here:
Aug 19, 2021
2
Aug 19, 2021
2
In
DataDailyRead
by
Venkatesh Pappakrishnan, Ph.D.
Python Vs Scala for Spark in production systemsWhich is the right one for your project?
May 16, 2022
May 16, 2022
In
Dev Genius
by
Amit Singh Rathore
Data Engineering Design Patternssome ideas p
Aug 19, 2022
3
Aug 19, 2022
3
In
TDS Archive
by
Robert O'Brien
How To Upload Your Pandas DataFrame To Your Database 10x FasterSpeed up your data science pipelines with this simple trick.
Dec 14, 2020
4
Dec 14, 2020
4
In
TDS Archive
by
Giorgos Myrianthous
How to Run Airflow Locally With DockerA step by step guide for running Airflow with Docker on your local machine
May 13, 2022
4
May 13, 2022
4
In
Better Programming
by
Jamie Burns
How to Optimize Queries in SQL ServerWhen you’ve got a query that’s running a little slow, here’s an approach you can take to make it fast
May 12, 2022
2
May 12, 2022
2
In
Geek Culture
by
Shawn Shi
Decouple Long-running Tasks from HTTP Request Processing — Using Azure Service BusDiscuss how to use Azure Service Bus as a message broker instead of an in-memory message broker and how topics and queues are…
Mar 30, 2022
Mar 30, 2022
 This story is no longer available
In
TDS Archive
by
Iman Samizadeh, Ph.D.
A brief introduction to two data processing architectures — Lambda and Kappa for Big DataBig Data, Internet of things (IoT), Machine learning models and various other modern systems are becoming an inevitable reality today…
Mar 15, 2018
6
Mar 15, 2018
6
In
The Startup
by
Shawn Shi
Data Engineering — How to Build an ETL Pipeline Using SSIS in Visual Studio 2019Discussing how to build an ETL pipeline for database migration using SQL Server Integration Services (SSIS) in Visual Studio 2019.
Oct 8, 2020
2
Oct 8, 2020
2
In
CodeX
by
Saikat Dutta
How to Become a Data EngineerA complete roadmap on how you can learn and become a Data Engineer in 2022
Apr 16, 2022
16
Apr 16, 2022
16
Tamara Janina Fingerlin
DataTalksClub: Data Engineering Zoomcamp — Week 1(sharing my notes and learning process during the free Data Engineering Zoomcamp from the DataTalksClub)
Jan 19, 2022
Jan 19, 2022
Matt Weingarten
Data Engineering Best Practices
Apr 14, 2022
5
Apr 14, 2022
5
In
TDS Archive
by
Edwin Tan
Apply Functions to Pandas DataFrame Using map(), apply(), applymap() and pipe()What is Pandas map, apply, applymap and pipe?
Apr 4, 2022
Apr 4, 2022
In
Towards Dev
by
Hanson Chiu
Quick guide to understand various database types and common data architecture patternsI’ve been working on data-related works and projects for more than 6 years. With evolving data technologies (also the buzzwords especially…
Apr 3, 2022
Apr 3, 2022
In
Dev Genius
by
Haq Nawaz
Python ETL Pipeline: Incremental data load Source Change DetectionWe will continue with the ETL incremental data load approaches. The incremental data load in ETL (Extract, Transform and Load) is the ideal…
Apr 4, 2022
2
Apr 4, 2022
2