Free interactive career roadmap

Data Engineer Roadmap

A complete six-month intermediate data engineering roadmap covering advanced SQL, Python pipelines, dimensional modelling, dbt, orchestration, Spark, lakehouse formats, streaming, data quality, cloud architecture and production operations. Build one analytics platform from raw ingestion to trusted batch and real-time data products using mostly official documentation.

6 months duration24 guided weeks10–12 hours per week

Your progress0 / 24 weeks

Saved on this device

01Month 1 — SQL, Python and Data Foundations

02Month 2 — Warehousing and Analytics Engineering

03Month 3 — Orchestration and Reliable Batch Pipelines

04Month 4 — Spark and Lakehouse Processing

05Month 5 — Streaming, Quality and Cloud Operations

06Month 6 — Production Capstone and Interviews

SQL, Python and Data Foundations

Build strong query, scripting and data-contract habits before introducing distributed tools.

W01

Advanced SQL for AnalyticsUse SQL to express complex transformations and inspect performance.

10–12 hours per week0/5

Learn: Study joins, CTEs, windows, subqueries, aggregates and query plans.Practise: Solve cohort, funnel, ranking and rolling-metric exercises.Build: Create analyst-ready order and customer queries over PostgreSQL.Verify: Check grain, duplicates and totals against source records.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A versioned SQL workbook with ten business queries and validation checks.

W02

Python for Data PipelinesWrite maintainable ingestion code rather than notebook-only scripts.

10–12 hours per week0/5

Learn: Study files, iterators, typing, logging, exceptions and packaging.Practise: Parse CSV and JSON with malformed, missing and duplicate records.Build: Create a CLI ingestion package with configuration and structured logs.Verify: Run twice and prove the result is idempotent.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A tested Python ingestion package with configuration, logs and idempotent loading.

W03

Data Modelling and ContractsDefine table grain, keys and quality expectations before transformation.

10–12 hours per week0/5

Learn: Study normalised sources, dimensional models, facts, dimensions and slowly changing attributes.Practise: Model orders, customers, products and events at explicit grains.Build: Write schema and column contracts for raw and curated tables.Verify: Trace one metric from source field to final definition.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A source-to-target model, data dictionary and contract set.

W04

Local Batch PipelineCombine ingestion, validation and modelling into one repeatable run.

10–12 hours per week0/5

Learn: Study extract-load-transform boundaries and raw versus curated layers.Practise: Create failure fixtures for missing files, bad types and duplicate keys.Build: Load raw commerce files, clean them and populate analytics tables.Verify: Rebuild from zero and reconcile counts and revenue totals.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

Version 1 of the commerce analytics platform with raw, clean and mart tables.

MONTH 01 MILESTONEA reproducible local ingestion pipeline and analytics schema with validated raw and clean layers.

Warehousing and Analytics Engineering

Create tested, documented transformations with reusable modelling conventions.

W05

Warehouse Architecture and File FormatsChoose storage and layers based on workload rather than tool popularity.

10–12 hours per week0/5

Learn: Study warehouses, lakes, lakehouses, row versus column storage and Parquet.Practise: Compare CSV and Parquet size and scan behaviour on the same data.Build: Design raw, stage and analytics layers for the platform.Verify: Explain where mutable database tables and immutable files belong.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A storage-layer decision record and Parquet comparison notebook.

W06

dbt Models and LineageTurn transformation SQL into a dependency-aware software project.

10–12 hours per week0/5

Learn: Study sources, refs, staging, marts, materialisations and lineage.Practise: Create staging models with naming and type cleanup.Build: Build intermediate and dimensional marts in dbt.Verify: Generate the lineage graph and verify every mart traces to declared sources.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A layered dbt project with clear source-to-mart lineage.

W07

Data Tests and DocumentationMake trust rules executable and visible to consumers.

10–12 hours per week0/5

Learn: Study generic tests, custom tests, source freshness and documentation.Practise: Add uniqueness, null, relationship, accepted-value and business-rule tests.Build: Document models, columns, owners and metric definitions.Verify: Introduce a bad record and confirm the relevant test fails clearly.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A tested and generated data catalog for the commerce marts.

W08

Incremental Models and Change HandlingProcess changed data efficiently without losing correctness.

10–12 hours per week0/5

Learn: Study incremental materialisations, late data, snapshots and slowly changing dimensions.Practise: Simulate updated customers and late-arriving orders.Build: Add incremental fact processing and customer-history tracking.Verify: Compare a full refresh with incremental results and reconcile differences.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

Version 2 with incremental loads, historical dimensions and reconciliation evidence.

MONTH 02 MILESTONEA dbt project with staging, intermediate and mart layers, tests, documentation and incremental logic.

Orchestration and Reliable Batch Pipelines

Schedule dependencies, retries and backfills while keeping pipeline runs observable and safe.

W09

Airflow Core Concepts and DAGsModel workflows as explicit tasks, dependencies and data intervals.

10–12 hours per week0/5

Learn: Study DAGs, tasks, scheduling, runs, operators and TaskFlow.Practise: Create small DAGs and inspect logical dates versus wall-clock time.Build: Orchestrate ingestion, dbt transformation and validation tasks.Verify: Trigger manually and verify dependency order and task artifacts.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A working DAG that runs the complete daily batch pipeline.

W10

Retries, Idempotency and BackfillsMake reruns safe and historical recovery predictable.

10–12 hours per week0/5

Learn: Study retries, timeouts, catchup, data intervals, backfills and atomic writes.Practise: Fail a task after partial work and design a safe rerun.Build: Add partition-aware loads and a controlled backfill procedure.Verify: Backfill three days and prove no duplicates or missing partitions.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

An idempotent, partition-aware pipeline with a verified backfill runbook.

W11

Pipeline Testing and CICatch broken DAGs, transformations and contracts before deployment.

10–12 hours per week0/5

Learn: Study DAG parsing tests, unit tests, integration checks and dbt build selection.Practise: Create fixtures for valid, empty and malformed source partitions.Build: Run Python tests, DAG checks and dbt tests in CI.Verify: Break a dependency and confirm CI reports the failure precisely.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A CI workflow that validates code, DAGs, transformations and data contracts.

W12

Containerised Orchestration and AlertsRun the stack consistently and make failures actionable.

10–12 hours per week0/5

Learn: Study Docker Compose, Airflow components, secrets and notifications.Practise: Start scheduler, webserver, metadata database and pipeline dependencies locally.Build: Add failure callbacks and a concise operational dashboard or report.Verify: Diagnose one planted failure from task logs and metadata.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

Version 3 with containerised Airflow, alerts and a failure-triage runbook.

MONTH 03 MILESTONEA containerised Airflow pipeline with tested DAGs, backfills, alerts and documented operations.

Spark and Lakehouse Processing

Scale transformations beyond a single process and understand modern table formats.

W13

Spark Execution Model and DataFramesUnderstand lazy evaluation, partitions, shuffles and the DataFrame API.

10–12 hours per week0/5

Learn: Study driver/executors, transformations, actions, plans and partitioning.Practise: Translate pandas and SQL transformations into PySpark DataFrames.Build: Build a Spark job for order-event enrichment and aggregation.Verify: Inspect the physical plan and identify shuffle boundaries.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A tested PySpark batch job with plan notes and partition metrics.

W14

Spark SQL PerformanceReduce unnecessary scans and shuffles using evidence.

10–12 hours per week0/5

Learn: Study partitioning, joins, caching, adaptive execution and file sizing.Practise: Compare broadcast and shuffle joins on controlled data.Build: Tune one representative transformation and record before/after metrics.Verify: Explain why each tuning change helps and when it could hurt.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A performance experiment with reproducible plans and measurements.

W15

Parquet Layout and Partition DesignOrganise files for efficient reads without creating a small-file problem.

10–12 hours per week0/5

Learn: Study row groups, column chunks, statistics, compression and partition pruning.Practise: Write datasets with two layouts and compare scan behaviour.Build: Choose partition columns and compaction rules for order events.Verify: Verify predicate and column pruning with query plans.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A partitioned Parquet dataset with layout benchmark and compaction policy.

W16

Apache Iceberg and Lakehouse TablesAdd schema evolution, snapshots and reliable table operations to object storage.

10–12 hours per week0/5

Learn: Study catalogs, metadata, snapshots, partition evolution and time travel.Practise: Create, update and query an Iceberg table through Spark.Build: Migrate the curated orders dataset to Iceberg and test schema evolution.Verify: Time-travel to a prior snapshot and document recovery steps.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

Version 4 with Spark jobs, Parquet raw layers and an Iceberg curated table.

MONTH 04 MILESTONEA Spark pipeline writing partitioned Parquet and Iceberg tables with measured performance choices.

Streaming, Quality and Cloud Operations

Process events continuously while defining correctness, freshness and operational limits.

W17

Kafka FundamentalsUnderstand durable event logs, partitions, offsets and consumer groups.

10–12 hours per week0/5

Learn: Study brokers, topics, keys, ordering, replication and delivery semantics.Practise: Produce and consume keyed order events locally.Build: Define event schemas and partition strategy for the platform.Verify: Restart consumers and explain observed offset and duplicate behaviour.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A Kafka event contract plus reproducible producer and consumer demo.

W18

Spark Structured StreamingTreat streams as unbounded tables while reasoning about time and state.

10–12 hours per week0/5

Learn: Study sources/sinks, triggers, output modes, checkpoints, windows and watermarks.Practise: Aggregate order events by event time with late records.Build: Build a Kafka-to-curated streaming job with checkpointing.Verify: Restart after failure and verify state and output behaviour.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A fault-recoverable streaming aggregation with event-time windows.

W19

Data Quality, Freshness and ObservabilityMeasure whether data is correct, complete, timely and traceable.

10–12 hours per week0/5

Learn: Study freshness, volume, schema, distribution and lineage signals.Practise: Create normal and deliberately broken pipeline runs.Build: Publish quality and freshness checks with owners and severity levels.Verify: Diagnose a stale mart from source through orchestration to output.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A data-health report with freshness SLOs, lineage and incident evidence.

W20

Cloud Data ArchitectureMap the local platform to managed cloud components without losing portability or cost awareness.

10–12 hours per week0/5

Learn: Study object storage, IAM, encryption, catalogs, compute and warehouse options.Practise: Create a cloud component map and monthly cost assumptions.Build: Design raw, stage and analytics layers with least-privilege access.Verify: Run a tabletop failure and recovery review for one critical pipeline.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

Version 5 architecture with security boundaries, cost notes and recovery plan.

MONTH 05 MILESTONEA local real-time pipeline with Kafka, Structured Streaming, quality checks and a cloud-ready architecture.

Production Capstone and Interviews

Harden the platform and present evidence of reliable data delivery and sound design.

W21

Governance, Privacy and AccessProtect sensitive data through classification, minimisation and controlled access.

10–12 hours per week0/5

Learn: Study PII classification, encryption, retention, masking and auditability.Practise: Label sensitive fields and define producer/consumer access needs.Build: Add masking or tokenisation for a sample sensitive field.Verify: Verify an unauthorised role cannot access raw sensitive data.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A data-classification inventory, access matrix and verified masking control.

W22

Data System DesignDesign ingestion, storage, processing and serving from concrete requirements.

10–12 hours per week0/5

Learn: Study batch versus streaming, SLAs, backfills, schemas and failure recovery.Practise: Design clickstream, CDC and financial-reporting pipelines.Build: Write one complete architecture for the capstone's expected scale.Verify: Defend latency, consistency, storage and cost trade-offs.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A data-system design document with scale, contracts, SLOs and recovery paths.

W23

SQL, Python and Pipeline InterviewsPractise the analysis and debugging tasks common in data-engineering interviews.

10–12 hours per week0/5

Learn: Review SQL windows, Python transforms, data modelling and pipeline incidents.Practise: Complete timed SQL, coding, modelling and debugging exercises.Build: Prepare explanations for late data, duplicates, skew and failed backfills.Verify: Run one mock incident from alert to corrected data.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

Four mocks: SQL, Python, modelling and production pipeline diagnosis.

W24

Final Platform and Case StudyShow a coherent path from raw data to trusted consumer outcomes.

10–12 hours per week0/5

Learn: Review ingestion, transformation, orchestration, scale, streaming, quality and governance.Practise: Ask a reviewer to run a backfill and trace one metric end to end.Build: Polish repository, local stack, diagrams, data catalog and incident notes.Verify: Fix the top review findings and map evidence to target job descriptions.Document the decisions, evidence and remaining limitations for this week.

DELIVERABLE

A runnable commerce data platform, architecture, catalog, quality dashboard and five-minute walkthrough.

MONTH 06 MILESTONEA portfolio-grade batch-and-streaming platform, case study and practical interview evidence.