AWS Glue

Service for organizing analytical data and converting it into usable forms.

Back to glossary Go to related practice questions

In one sentence

Service for organizing analytical data and converting it into usable forms.

Overview

AWS Glue is a serverless data integration service that performs data extract, transform, load, and catalog management.

Common use cases

ETL processing
Data catalog management
Preprocessing for S3 data lakes

CLF-C02 exam points

An ETL service
Manages metadata with the Glue Data Catalog
Combines with Athena and Redshift

SAA-C03 exam points

Used in batch analytics platforms
Useful for transforming data on S3
Consider cost based on job execution time

Practical Use

AWS Glue is a data integration service that runs ETL (Extract / Transform / Load) processing and manages data catalogs in a serverless manner for data lakes and data warehouses. Its core components include Glue Data Catalog (a central repository of metadata), Glue Crawler (scans data sources to automatically infer schemas and register them in the catalog), Glue ETL jobs (run transformations on Apache Spark or in a Python shell environment), Glue Studio (designs jobs in a GUI), and Glue DataBrew (a no-code data preparation tool). Because AWS provides the Apache Spark-based job execution infrastructure as a fully managed offering, cluster management is unnecessary. A typical configuration is the "serverless analytics foundation" that aggregates data such as logs, CSV, and Parquet in S3 as a data lake, infers schemas with Glue Crawler, registers them in Glue Data Catalog, and queries them with SQL from Amazon Athena. Because Athena references Glue Data Catalog as its metastore, tables cataloged with Glue can be queried directly from Athena. Amazon Redshift Spectrum and Amazon EMR can also reference the same Data Catalog, with the major advantage of being able to share metadata across multiple analytics tools. A typical pattern in ETL jobs is to read CSV or JSON from S3, apply column type conversions, missing value imputation, JOINs, and aggregations, and then convert the data into columnar formats such as Parquet or ORC and write it back to S3. In the exam context, CLF-C02 asks at the level of "it is an ETL service", "Glue Data Catalog can manage metadata", and "it can be combined with Athena or Redshift". SAA-C03 asks in depth about the configuration of S3 data lake + Glue + Athena, role division with Lake Formation, and cost optimization of Glue jobs (DPU count and job execution time).

Official Documentation

Cost notes

Billed by job execution time and crawler runs
Not needed for normal operation of small sites

Security notes

Limit the S3 access scope of the Glue execution role
Do not include personal data in the data being analyzed

Related services

Amazon Simple Storage Service

A representative AWS storage service for storing files securely and at low cost.

Amazon Athena

A service to query logs and CSVs stored in S3 with SQL.

Amazon Redshift

An AWS-managed data warehouse for large-scale data analysis.

Related architecture diagrams

Related architecture diagrams have not been added for this term yet.