Maintaining and automating data workloads
Effective data engineering includes ensuring the integrity and readiness of your data workloads. This is achieved with advanced automation techniques using tools including Cloud Composer, logging and monitoring using GCP native tools, and recovery from failure.
Topics Include:- Optimizing Resources including minimizing costs according to business requirements, proper resource management, and job management techniques.
- Designing automation and repeatability using tools such as Cloud Composer.
- Organizing workloads based upon business requirements. This includes knowing the tradeoffs and best uses for slot pricing, caching, and when it is best to use interactive or batch jobs.
- Monitoring and troubleshooting processes by ensuring data process observability, monitoring usage, troubleshooting error messages, and managing workloads.
- Maintaining awareness of failures and mitigating impact be designing for fault tolerance, running jobs in multiple regions or ones, accounting for bad data, and ensuring data availability.
GCP Professional Data Engineer Certification Preparation Guide (Nov 2023)
→ Maintaining and automating data workloads
Module Topics
Optimizing resourcesDesigning automation and repeatability
Organizing workloads based on business requirements
Monitoring and troubleshooting processes
Maintaining awareness of failures and mitigating impact
Optimizing resources
Analyze and optimize compute requirements for the business, accounting for both business-critical data processes, development, and data processing.
Topics Include:- Minimizing costs per required business need for data
- Ensuring that enough resources are available for business-critical data processes
- Deciding between persistent or job-based data clusters (e.g., Dataproc)
Designing automation and repeatability
Leverage GCP's native Airflow implemenation, Cloud Composer, to develop task orchestration.
Topics Include:- Creating directed acyclic graphs (DAGs) for Cloud Composer
- Scheduling jobs in a repeatable way
Organizing workloads based on business requirements
Leverage flexible pricing options to achieve cost-optimization while using core components or when querying BigQuery data.
Topics Include:- Flex, on-demand, and flat rate slot pricing (index on flexibility or fixed capacity)
- Interactive or batch query jobs
Monitoring and troubleshooting processes
No matter how well your process or pipelines are developed errors in both applications and data occur. Leverage GCP's logging layer to quickly and efficiently spot and fix errors. Use BigQuery's admin portal to manage workloads and reservations.
Topics Include:- Monitoring planned usage
- Troubleshooting error messages, billing issues, and quotas
- Manage workloads, such as jobs, queries, and compute capacity (reservations)
Maintaining awareness of failures and mitigating impact
Leverage GCP's native stack to ensure high availability and distributed architecture. Use all the tools needed to manage risk according to the business requirements.
Topics Include:- Designing system for fault tolerance and managing restarts
- Running jobs in multiple regions or zones
- Preparing for data corruption and missing data
- Data replication and failover (e.g., Cloud SQL, Redis clusters)