IJS Ilham Jaya Saputra. All Projects

Data Engineering

Fashion Studio ETL Pipeline

An extract, transform, and load project for fashion product data. I separated collection, cleaning, and storage into modules so the same curated dataset can be delivered to different destinations.

Ilham Jaya Saputra
Ilham Jaya Saputra

Statistics, Universitas Airlangga

PythonPandasBeautifulSoupGoogle Sheets APISQLAlchemyPostgreSQL

The Task

Product listings contain fields that need cleaning before analysis, including prices, ratings, color counts, sizes, and gender labels. This project brings those fields into a consistent tabular dataset and supports three output destinations.

Pipeline Design

The entry point calls separate extraction, transformation, and loading modules. It checks required configuration before starting and stops if the transformed dataset is empty. Separating the stages makes the data flow easier to inspect and gives each module a focused responsibility.

Cleaning and Validation

The transformation module removes invalid product titles, extracts numeric prices and ratings, converts color counts, and strips prefixes from size and gender values. It then removes rows with missing values and duplicates and enforces column data types.

For this exercise, the price conversion uses a fixed exchange rate of IDR 16,000 per USD defined in the source code. It is a project parameter rather than a live exchange-rate feed.

Storage Outputs

The loading module can write the cleaned table to CSV, update a Google Sheets worksheet, and write to a PostgreSQL products table through SQLAlchemy. Each destination checks for empty input and reports errors to the caller.

The three output destinations make the cleaned dataset available for spreadsheet exploration, flat-file analysis, or database queries.

Project Sources