Designing data processing systems Foxtrot Communications

Designing data processing systems

At the core of every data engineering project is the overall data processing system architecture. Learn the core components and best practices for data processing in GCP.

Topics include:

High level architectural concerns such as security, core components or subsystems, and data sovereignty.
Data reliability, such as preparing data, monitoring core components, disaster recovery, and data integrity concerns.
Data flexibility and portability, mapping business requirements, data cataloging, and data governance
Designing data migrations on a high level. This includes gathering stakeholder requirements, GCP migration components, migration validation strategy, and best practices for dataset architecture.

GCP Professional Data Engineer Certification Preparation Guide (Nov 2023)
→ Designing data processing systems

Designing for security and compliance

Ensuring a secure and legally compliant solution will provide a high-quality product and will inspire confidence in your stakeholders and users.

Topics Include:

Identity and Access Management (e.g., Cloud IAM and organization policies)
Data security
Data security (encryption and key management)
Privacy (e.g., personally identifiable information, and Cloud Data Loss Prevention API)
Regional considerations (data sovereignty) for data access and storage
Legal and regulatory compliance

<img src="https://storage.googleapis.com/ftc-stp-artifacts/eZe7RZYwT90rsJ5JN84v/Designing%20for%20security%20and%20compliance.jpg" class="img_25">
<p>
Ensuring a secure and legally compliant solution will provide a high-quality product and will inspire confidence in your stakeholders and users.
</p>
Topics Include:
<ul>
<li> Identity and Access Management (e.g., Cloud IAM and organization policies)</li>
<li>Data security</li>
<li>Data security (encryption and key management)</li>
<li>Privacy (e.g., personally identifiable information, and Cloud Data Loss Prevention API)</li>
<li>Regional considerations (data sovereignty) for data access and storage</li>
<li>Legal and regulatory compliance</li>
</ul>

Designing for reliability and fidelity

A high degree of reliability and consistency will inspire confidence in your users and stakeholders. Ensure your product is ready to handle any errors in data processing or failures in your architecture.

Topics Include:

Preparing and cleaning data (e.g., Dataprep, Dataflow, and Cloud Data Fusion)
Monitoring and orchestration of data pipelines
Disaster recovery and fault tolerance
Making decisions related to ACID (atomicity, consistency, isolation, and durability) compliance and availability
Data validation

<p>
A high degree of reliability and consistency will inspire confidence in your users and stakeholders.  Ensure your product is ready to handle any errors in data processing or failures in your architecture. 
</p>
Topics Include:
<ul>
<li>Preparing and cleaning data (e.g., Dataprep, Dataflow, and Cloud Data Fusion)</li>
<li>Monitoring and orchestration of data pipelines</li>
<li>Disaster recovery and fault tolerance</li>
<li>Making decisions related to ACID (atomicity, consistency, isolation, and durability) compliance and availability</li>
<li>Data validation</li>
</ul>

Designing for flexibility and portability

Account for an ever evolving product and data schema in your solution. Embrace multi-cloud and hybrid cloud architectures and tie everything together with an efficient data catalog.

Topics Include:

Mapping current and future business requirements to the architecture
Designing for data and application portability (e.g., multi-cloud and data residency requirements)
Data staging, cataloging, and discovery (data governance)

Designing data migrations

A key requirement of any professional data engineer is data migration from any source into GCP. Take advantage of native GCP services to ensure a consistent and efficient migration.

Topics Include:

Analyzing current stakeholder needs, users, processes, and technologies and creating a plan to get to desired state
Planning migration to Google Cloud (e.g., BigQuery Data Transfer Service, Database Migration Service, Transfer Appliance, Google Cloud networking, Datastream)
Designing the migration validation strategy
Designing the project, dataset, and table architecture to ensure proper data governance