
Job Market Intelligence - end-to-end project
Brainstorming I've been thinking about reviving my recent Databricks Bootcamp Project....
Tag archive

Brainstorming I've been thinking about reviving my recent Databricks Bootcamp Project....

Keep Fabric as the execution plane and Airflow as the orchestration plane. Thin tasks, terminal-state success, and what breaks when one tool does both.

I had nine Olist e-commerce CSVs — 1.55 million rows — and no pipeline. I wanted a production-shaped...

Airflow 3.4.0 arrived while I was preparing a talk for Apache Airflow & Apache Magpie Meetup....
One pipeline, three stacks, four failures injected on purpose. What each stack needs to stay correct, how long a run takes, and what it costs to install.
Introduction Most data platforms eventually reach the same problem. Jobs that once ran...
A review of Airflow DAG author privileges as a code execution boundary.
For a long time our answer to flaky pipelines was "just add retries". Three retries, then five. It...

Amazon Managed Workflows for Apache Airflow (MWAA) sends Airflow metrics to Amazon CloudWatch without...
Airflow vs Dagster vs Prefect: compare tasks, assets and flows, scheduling, lineage, testing, deployment, managed options and migration, with one ETL in each.

This article was first published on the Bruin blog. A data orchestrator is the system that runs a...
Notes from the second course: source systems, batch vs streaming ingestion, ETL vs ELT, and orchestrating pipelines with Airflow on AWS.