Introduction to Data Engineering

In this module, you will learn about the different entities that come together to form a modern data ecosystem and the role Data Engineers, Data Scientists, Data Analysts, Business Analysts, and Business Intelligence Analysts play in this ecosystem. You will learn what data engineering is and the key tasks in a data engineering lifecycle. You will also gain an understanding of the responsibilities of a data engineer, the skillsets they need in order to be successful, and what a typical day in the life of a data engineer looks like. At the end of the module, you will be guided to create a Lite account on IBM Cloud.

In this module, you will learn about the data engineering ecosystem, the different types of data structures, file formats, sources of data, and the languages data professionals use in their day-to-day tasks. You will gain an understanding of several different types of data repositories such as relational and non-relational databases, data warehouses, data marts, and data lakes. You will learn about ETL and ELT processes, data pipelines, and data integration platforms. You will also gain an understanding of what big data is, and the tools used for processing and storing big data. During the course of this module, you will be guided to provision an instance of IBM Db2 using the Cloud Lite account you created in the previous module.

In this module, we will walk you through the data engineering lifecycle. You will learn about the architecture of a data platform, factors for selecting and designing data stores, and the different facets of security as it applies to data platforms and data lifecycle management. You will also learn about the process, steps, and tools used for gathering, importing, wrangling, and querying data. You will gain an understanding of performance monitoring and the steps you can take to troubleshoot performance issues. We will also talk about governance regulations, why we need them, and how technology enables compliance to regulations. During the course of this module, you will be guided to load data from a CSV file into the IBM Db2 instance you created in the previous module. You will also be guided to explore your dataset using some basic SQL queries that will be provided to you.