Data pipeline model repository

One framework to develop, deploy and operate data workflows with Python and SQL.

Introduction to Data Engineering

high performance better alternative to Airbyte, Singer, Meltano

💜🌈📊 A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Metabase, Dbt, Polars, Docker. Data from kaggle and youtube-api 🌺

Banking Data Warehouse Pipeline

Let your pipe lines flow thru the Python code in xonsh.

Pipeline de dados automatizado para extração e armazenamento de previsões meteorológicas para o setor de turismo.

Data Engineering Pipeline practice with Amazon Sales Data

Leveraging AWS Cloud Services, an ETL pipeline transforms YouTube video statistics data. Data is downloaded from Kaggle, uploaded to an S3 bucket, and cataloged using AWS Glue for querying with Athena. AWS Lambda and Glue converts to Parquet format and stores it in a cleansed S3 bucket. AWS QuickSight then visualizes the materialised data.

A simple ETL for temperature data from the Openweathermap API, storing it into an Azure SQL Database

Ayush @ Data Engineering Portfolio

The Security Reference Architecture (SRA) implements typical security features as Terraform Templates that are deployed by most high-security organizations, and enforces controls for the largest risks that customers ask about most often.

In this project, I have created an end to end solution for analyzing the bing latest news data. I have used the microsoft fabric for all the tools.

Pipeline to automate the collection of board game and expansion data from BoardGameGeek's XML API2. Data is stored in Google Cloud Storage and BigQuery. Data is modelled using DBT in a star schema. (Terraform, GCP, Mage, Python, dbt)

Zillow Data Pipeline: Extracts data from Zillow, transfers it through AWS services, and performs analytics. Utilizes Python scripts, AWS Lambda, S3, Amazon RedShift, and QuickSight. Explore docs/images for architecture visuals.

Data Engineering 🛠️ is like the backbone of data processing 📊, managing data pipelines 🚀, warehouses 🏢, and lakes 🌊. It's the bridge 🌉 between raw data and actionable insights, powering businesses 🚀 with efficient data management and analytics 📈.