When will I have access to the lectures and assignments?

To access the course materials, assignments and to earn a Certificate, you will need to purchase the Certificate experience when you enroll in a course. You can try a Free Trial instead, or apply for Financial Aid. The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.

What will I get if I subscribe to this Certificate?

When you enroll in the course, you get access to all of the courses in the Certificate, and you earn a certificate when you complete the work. Your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile.

Source Systems, Data Ingestion, and Pipelines

4 days left! Save on skills that make you shine with 40% off 3 months of Coursera Plus. Save now

Source Systems, Data Ingestion, and Pipelines

This course is part of DeepLearning.AI Data Engineering Professional Certificate

Instructors: Joe Reis

Top Instructor

13,195 already enrolled

4 modules

Gain insight into a topic and learn the fundamentals.

136 reviews

Intermediate level

Recommended experience

Flexible schedule

3 weeks at 10 hours a week

Learn at your own pace

92%

Most learners liked this course

4 modules

Gain insight into a topic and learn the fundamentals.

136 reviews

Intermediate level

Recommended experience

Flexible schedule

3 weeks at 10 hours a week

Learn at your own pace

92%

Most learners liked this course

What you'll learn

Gather stakeholder needs and translate them into system requirements.
Implement a batch and a streaming ingestion process on AWS to ingest data from various source systems.
Integrate aspects of security, data management, DataOps and orchestration into the data systems you build.

Skills you'll gain

Tools you'll learn

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

4 assignments

Taught in English

91% of learners achieved a positive career outcome

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

Build your Cloud Computing expertise

This course is part of the DeepLearning.AI Data Engineering Professional Certificate

When you enroll in this course, you'll also be enrolled in this Professional Certificate.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate from DeepLearning.AI

There are 4 modules in this course

In this course, you will explore various types of source systems, learn how they generate and update data, and troubleshoot common issues you might encounter when trying to connect to these systems in the real world. You’ll dive into the details of common ingestion patterns and implement batch and streaming pipelines. You’ll automate and orchestrate your data pipelines using infrastructure as code and pipelines as code tools. You’ll also explore AWS and open source tools for monitoring your data systems and data quality.

Module details

In lesson 1, you will explore source systems data engineers commonly interact with. Then in lesson 2, you will learn how to connect to various source systems and troubleshoot common connectivity issues.

What's included

21 videos9 readings1 assignment1 programming assignment3 ungraded labs

21 videosTotal 131 minutes

Welcome to Course 26 minutes
Course 2 Overview2 minutes
Different Types of Source Systems7 minutes
Relational Databases7 minutes
SQL Queries7 minutes
NoSQL Databases7 minutes
Database ACID Compliance6 minutes
Lab Walkthrough - Interacting with Amazon DynamoDB NoSQL Database7 minutes
Object Storage5 minutes
Logs5 minutes
Streaming Systems7 minutes
Lesson Overview3 minutes
Connecting to Source Systems4 minutes
Basics of IAM and Permissions7 minutes
Basics of Networking in the Cloud4 minutes
AWS Networking Overview - VPCs & Subnets11 minutes
AWS Networking - Internet Gateway & NAT Gateway7 minutes
AWS Networking - Route Tables6 minutes
AWS Networking - Network ACLs & Security Groups7 minutes
Lab Walkthrough - Troubleshooting Database Connectivity on AWS13 minutes
Week 1 Summary1 minute

9 readingsTotal 45 minutes

Program Syllabus5 minutes
[IMPORTANT] Guidelines before you start the labs in this course10 minutes
[Optional] FAQ VS Code Lab Environment5 minutes
Join the DeepLearning.AI Forum to ask questions, get support, or share amazing ideas!2 minutes
[Optional] Connecting to an Amazon RDS MySQL Database10 minutes
Basics of AWS IAM10 minutes
[Optional] AWS Networking Overview- VPC1 minute
Week 1 Resources1 minute
Lecture Notes W11 minute

1 assignmentTotal 30 minutes

Week 1 Quiz30 minutes

1 programming assignmentTotal 120 minutes

Assignment 1: Troubleshooting Database Connectivity on AWS120 minutes

3 ungraded labsTotal 360 minutes

Practice Lab 1: Interacting With a Relational Database Using SQL120 minutes
Practice Lab 2: Interacting With Amazon DynamoDB NoSQL Database120 minutes
Practice Lab 3: Interacting With Amazon S3 Object Storage120 minutes

This week you will dive deep into the batch and streaming ingestion patterns. You will identify use cases and considerations for each, and then build a batch and a streaming ingestion pipeline. When looking at batch ingestion, you will compare and contrast the ETL and ELT paradigms. You will also explore various AWS services for batch and streaming ingestion.

What's included

11 videos6 readings1 assignment1 programming assignment1 ungraded lab

11 videosTotal 60 minutes

Week 2 Overview3 minutes
Data Ingestion on a Continuum6 minutes
Conversation with a Marketing Analyst4 minutes
ETL vs. ELT8 minutes
REST API4 minutes
Lab Walkthrough - Batch Processing to Get Data From an API10 minutes
Conversation with a Software Engineer5 minutes
Streaming Ingestion Details6 minutes
Kinesis Data Streams Details6 minutes
Lab Walkthrough - Streaming Ingestion6 minutes
Week 2 Summary2 minutes

6 readingsTotal 32 minutes

Batch and Streaming Tools10 minutes
Summary of the Differences: ETL vs. ELT5 minutes
What is Change Data Capture (CDC)?10 minutes
Summary: General Considerations for Choosing Ingestion Tools5 minutes
Week 2 Resources1 minute
Lecture Notes W21 minute

1 assignmentTotal 30 minutes

Week 2 Quiz30 minutes

1 programming assignmentTotal 120 minutes

Assignment 2: Batch Data Processing from an API120 minutes

1 ungraded labTotal 120 minutes

Practice Lab: Streaming Ingestion120 minutes

In the first lesson, you will explore DataOps automation practices, including applying CI/CD to both data and code, and using infrastructure as code tools like Terraform to automate the provisioning and management of your resources. Then in lesson 2, you will explore DataOps observability and monitoring practices, including using tools like Great Expectation to monitor data quality, and using Amazon CloudWatch to monitor your infrastructure.

What's included

17 videos5 readings1 assignment1 programming assignment2 ungraded labs

17 videosTotal 123 minutes

Week 3 Overview3 minutes
[Optional] Conversation with Chris Bergh14 minutes
DataOps Automation4 minutes
Infrastructure as Code7 minutes
Terraform - Creating an EC2 Instance10 minutes
Terraform - Defining Variables and Outputs6 minutes
Terraform - Defining Data Sources and Modules7 minutes
Lab Walkthrough - Implementing DataOps with Terraform8 minutes
Data Observability5 minutes
[Optional] Conversation with Barr Moses16 minutes
Monitoring Data Quality4 minutes
[Optional] Conversation with Abe Gong7 minutes
Great Expectations - Core Components4 minutes
Great Expectations - Workflow Example7 minutes
[Optional] Conversation with Chad Sanderson14 minutes
Amazon CloudWatch5 minutes
Week 3 Summary1 minute

5 readingsTotal 32 minutes

[Optional] Additional Terraform Configuration Example - Exercise15 minutes
[Optional] Note regarding Practice Lab 15 minutes
[Optional] Conversation Takeaways & Additional Notes10 minutes
Week 3 Resources1 minute
Lecture Notes W31 minute

1 assignmentTotal 30 minutes

Week 3 Quiz30 minutes

1 programming assignmentTotal 120 minutes

Assignment 3: Testing Data Quality with Great Expectations120 minutes

2 ungraded labsTotal 240 minutes

Practice Lab 1: Implementing DataOps with Terraform120 minutes
Practice Lab 2: Implementing Monitoring with Amazon CloudWatch120 minutes

This week, you will learn all about orchestrating your data pipeline tasks. You'll identify the various orchestration tools, but will focus on Airflow -- one of the most popular and widely used tools in the field today. You'll explore the core components of Airflow, the Airflow UI, and how to create and manage DAGs using various Airflow features.