# coffeeverse: technical walkthrough

# ☕ Coffeeverse: Azure Cloud ETL Pipeline Walkthrough  

Azure-native ETL pipelines are a cornerstone of modern data engineering. The [**Coffeeverse**](https://github.com/anix-lynch/coffeeverse) repo demonstrates how to build a production-grade ETL pipeline using Azure’s serverless stack—Blob Storage, Azure Functions, Cosmos DB, and dbt—all orchestrated by Azure Data Factory. This walkthrough is for developers familiar with Azure and ETL concepts.  

## What problem does this solve?  

Building scalable, cost-effective ETL pipelines is a common challenge in data engineering. Coffeeverse addresses this by showcasing:  
1. **Event-driven architecture**: Using Azure Functions for serverless transformations.  
2. **Multi-layer data processing**: Bronze (raw), Silver (cleaned), Gold (analytics-ready) layers.  
3. **Zero-cost production**: Entirely runs on Azure Free Tier.  
4. **End-to-end orchestration**: Azure Data Factory and dbt for seamless ETL workflows.  

This pipeline is ideal for developers looking to learn Azure-native ETL or deploy a production-grade analytics dashboard.  

## Architecture  

```mermaid  
graph TD  
    A[Blob Storage] --> B[Azure Functions]  
    B --> C[Cosmos DB]  
    C --> D[dbt: Staging]  
    D --> E[dbt: Marts]  
    E --> F[Streamlit Dashboard]  
```  

- **Blob Storage**: Ingests raw data (Bronze layer).  
- **Azure Functions**: Transforms raw data into Silver layer and writes to Cosmos DB.  
- **Cosmos DB**: Stores cleaned data for further transformations.  
- **dbt**: Creates staging tables (Silver) and marts (Gold) for analytics.  
- **Streamlit**: Visualizes analytics-ready data.  

## Setup in 3 steps  

1. **Clone the repo and set up environment**:  
   ```bash  
   git clone https://github.com/anix-lynch/coffeeverse.git  
   cd coffeeverse  
   make setup  
   ```  

2. **Run locally**:  
   ```bash  
   make run-app  
   ```  

3. **Deploy to Azure**:  
   ```bash  
   make deploy-infra  
   make deploy-functions  
   ```  

See [`docs/RUNBOOK.md`](docs/RUNBOOK.md) for detailed instructions.  

## Key code walkthrough  

Here’s the core transformation logic in Azure Functions (`pipelines/transform.py`):  

```python  
import json  
from azure.cosmos import CosmosClient  

def transform_and_load(raw_data):  
    # Parse raw data  
    data = json.loads(raw_data)  

    # Clean & enrich data  
    cleaned_data = {  
        "id": data["id"],  
        "name": data["name"].strip(),  
        "quantity": int(data["quantity"]),  
        "timestamp": data["timestamp"]  
    }  

    # Load to Cosmos DB  
    client = CosmosClient.from_connection_string(os.environ["COSMOS_DB_CONN_STR"])  
    database = client.get_database_client("coffeeverse")  
    container = database.get_container_client("silver")  
    container.upsert_item(cleaned_data)  
```  

This function:  
1. Parses raw JSON data.  
2. Cleans and enriches the data (e.g., trimming strings, converting types).  
3. Upserts the cleaned data into Cosmos DB.  

## Before vs After  

**Before**: Raw data in Blob Storage (Bronze layer)  
```json  
{ "id": "123", "name": " Espresso ", "quantity": "10", "timestamp": "2023-10-01T12:00:00Z" }  
```  

**After**: Cleaned data in Cosmos DB (Silver layer)  
```json  
{ "id": "123", "name": "Espresso", "quantity": 10, "timestamp": "2023-10-01T12:00:00Z" }  
```  

**After**: Analytics-ready data in Streamlit (Gold layer)  
```
📊 Coffee Metrics  
Espresso: 10 units (2023-10-01)  
```  

## How to extend this / contribute  

1. **Add new data sources**: Modify the Bronze layer ingestion logic in Azure Functions.  
2. **Enhance transformations**: Add dbt models for advanced analytics (e.g., forecasting).  
3. **Improve visuals**: Customize the Streamlit dashboard for specific use cases.  
4. **Optimize costs**: Explore Azure’s pricing tiers for larger-scale deployments.  

Contributions are welcome! Open issues or submit PRs with improvements.  

⭐ **Star the repo** if you find it useful:  
[GitHub - Coffeeverse](https://github.com/anix-lynch/coffeeverse)  

Happy coding! ☕👩‍💻