Data Engineering
Fashion Studio ETL Pipeline
An extract, transform, and load project for fashion product data. I separated collection, cleaning, and storage into modules so the same curated dataset can be delivered to different destinations.
Statistics, Universitas Airlangga
The Task
Product listings contain fields that need cleaning before analysis, including prices, ratings, color counts, sizes, and gender labels. This project brings those fields into a consistent tabular dataset and supports three output destinations.
Pipeline Design
The entry point calls separate extraction, transformation, and loading modules. It checks required configuration before starting and stops if the transformed dataset is empty. Separating the stages makes the data flow easier to inspect and gives each module a focused responsibility.
Cleaning and Validation
The transformation module removes invalid product titles, extracts numeric prices and ratings, converts color counts, and strips prefixes from size and gender values. It then removes rows with missing values and duplicates and enforces column data types.
For this exercise, the price conversion uses a fixed exchange rate of IDR 16,000 per USD defined in the source code. It is a project parameter rather than a live exchange-rate feed.
Storage Outputs
The loading module can write the cleaned table to CSV, update a Google Sheets worksheet, and write to a PostgreSQL products table through SQLAlchemy. Each destination checks for empty input and reports errors to the caller.
The three output destinations make the cleaned dataset available for spreadsheet exploration, flat-file analysis, or database queries.