DP-203: Microsoft Azure Data Engineer Online Training Course

Partition Strategy

• Implement partition strategies for files, analytical workloads, streaming workloads, and Azure Synapse Analytics.
• Identify when partitioning is needed in Azure Data Lake Storage Gen2.

Create and Execute Queries

• Create and execute queries by using a compute solution that leverages SQL serverless and Spark cluster

Implement and Recommend Database Templates

• Implement and recommend Azure Synapse Analytics database templates.

Manage Metadata and Data Lineage

• Push new or updated data lineage to Microsoft Purview.
• Browse and search metadata in Microsoft Purview Data Catalog.

Data Transformation

• Design and implement incremental loads.
• Transform data using Apache Spark, Transact-SQL (T-SQL), Azure Synapse Pipelines, Azure Data Factory, and Azure Stream Analytics.
• Cleanse, handle duplicates, missing data, late-arriving data, split, shred JSON, encode and decode data.
• Configure error handling for transformations.
• Normalize, denormalize values, and perform exploratory data analysis.

Batch Processing

• Develop batch processing solutions using Azure Data Lake Storage, Azure Databricks, Azure Synapse Analytics, and Azure Data Factory.
• Use PolyBase to load data to a SQL pool.
• Implement Azure Synapse Link and query replicated data.
• Create data pipelines, scale resources, configure batch size, create tests, and integrate Jupyter or Python notebooks.
• Upsert data, revert to previous states, configure exception handling and batch retention, and read from/write to a delta lake.

Stream Processing

• Create a stream processing solution using Stream Analytics and Azure Event Hubs.
• Process data using Spark structured streaming, create windowed aggregates, handle schema drift, process time series data, and manage data across partitions.
• Configure checkpoints, watermarking, scale resources, optimize pipelines, handle interruptions, and replay archived stream data.

Batch and Pipeline Management

• Trigger batches, handle failed batch loads, and validate batch loads.
• Manage and schedule data pipelines in Azure Data Factory or Azure Synapse Pipelines.
• Implement version control for pipeline artifacts and manage Spark jobs in a pipeline.

Data Security

• Implement data masking, encryption (at rest and in motion), and row-level/column-level security.
• Implement Azure role-based access control (RBAC), POSIX-like ACLs for Data Lake Storage Gen2, and data retention policies.
• Implement secure endpoints, resource tokens in Azure Databricks, and manage sensitive information.

Monitoring

• Implement logging with Azure Monitor, configure monitoring services, and monitor stream processing.
• Measure performance of data movement, data pipeline performance, and query performance.
• Schedule and monitor pipeline tests, interpret Azure Monitor metrics and logs, and implement a pipeline alert strategy.

Optimization and Troubleshooting

• Compact small files, handle skew and data spill, optimize resource management, tune queries using indexers and cache.
• Troubleshoot failed Spark jobs and pipeline runs, including activities executed in external services.