Skip to content

Projects

Data engineering, architecture and AI projects: from lakehouses and MLOps to agents, MCP and AI deployed at the edge.

LLM Extraction with Evals: Know When to Trust the Output

CI/CD Status

Goal: Extracts invoice fields with Claude or Gemini, keeps the exact text behind every value, scores each field and sends doubtful documents to a person.

Tools: Python, Claude (structured outputs), Gemini, Pydantic, pytest, GitHub Actions

Impact: Shows why field accuracy alone misleads: a regex baseline gets 86% of fields right but only 25% of documents.

Architecture

LLM extraction with evals: Documents (Invoices, Gold labels) → Extract (Claude, Gemini, Regex baseline) → Score (Normalize, Field scoring, Evidence check) → Route (Auto-accept, Human review)Documents
Invoices60 generated, 4 layouts
Gold labelsknown answers
Extract
Claudestructured outputs
Geminiresponse schema
Regex baselinethe bar to beat
Score
Normalize1.234,50 = 1234.50
Field scoringaccuracy per field
Evidence checkquote must be in the doc
Route
Auto-acceptevery field backed
Human reviewanything doubtful

Claude Code Data Template: Guardrails for AI on Data Projects

CI/CD Status

Goal: A dbt + DuckDB starter where Claude Code follows data rules, and hooks block DROP, production targets and secrets before they run.

Tools: Claude Code (hooks, subagents, skills), dbt, DuckDB, Python, pytest, GitHub Actions

Impact: 34 hook tests and 12 dbt data tests, including a reconciliation that fails if a join starts dropping orders.

Architecture

Claude Code data guardrails: Rules (CLAUDE.md, Permissions) → Before a tool runs (guard_bash, guard_files) → After an edit (check_model) → Helpers (Subagents, Skills, dbt + DuckDB)Rules
CLAUDE.mdlayers, grain, done
Permissionsdbt allowed, .env hidden
Before a tool runs
guard_bashDROP, prod, force push
guard_filessecrets, .env
After an edit
check_modeltests, no SELECT *
Helpers
Subagentsreviewer, investigator
Skillsnew-model, reconcile
dbt + DuckDBdemo shop, 12 tests

table-diff: Prove a Data Migration Is Right

CI/CD Status

Goal: Compares a legacy table with its migrated copy by hash, from day to row to cell, and filters out the harmless noise every migration adds.

Tools: Python, DuckDB, SQL, pytest, GitHub Actions

Impact: Compares 1,000,000 rows in 0.09s on a laptop and finds exactly the 3 planted bugs; exit codes make it a CI gate.

Architecture

table-diff: migration validation: Inputs (Legacy table, Migrated table, DuckDB) → Normalize (Noise rules) → Three passes (Partition pass, Row pass, Cell pass) → Output (Report, Exit code)Inputs
Legacy tableParquet, CSV, DuckDB
Migrated tablesame key, new platform
DuckDBevery pass is SQL
Normalize
Noise rulesfloats, spaces, NULLs
Three passes
Partition passcount + hash sum per day
Row passmatch by key
Cell passwhich columns changed
Output
ReportMarkdown or JSON
Exit code0 match · 1 diff · 2 error

Data Incident Casebook

CI/CD Status

Goal: Three reproducible data incidents worked like investigations: symptom, hypotheses, evidence, root cause, fix and a guard that stays.

Tools: Python, DuckDB, SQL, pytest, GitHub Actions

Impact: Match rate restored from 57% to 98%, a 15% revenue overstatement closed, and 5 reusable guard checks.

Architecture

Data incident casebook: Incident (Generated data, Pipeline under test) → Investigation (Hypotheses, Evidence queries, Root cause) → Fix (Fix, Guard checks) → Output (Case files, CI)Incident
Generated datano client data
Pipeline under teststaging → model → report
Investigation
Hypothesescheapest first
Evidence queriesone per hypothesis
Root causeconfirmed by data
Fix
Fixsmallest safe change
Guard checksstay in the pipeline
Output
Case filesgenerated, tested in CI
CIPython 3.11–3.13

AI-Powered Data Lake: Claude + MCP Server

Goal: Medallion-architecture data lake that ingests CRM, ERP and API data, uses Claude to advise on data quality, generate SQL and business insights, and exposes the lake to agents through an MCP server

Tools: Claude (Sonnet + Opus), MCP, AWS S3, Glue, Athena, PySpark, dbt, CloudFormation, pytest

Impact: End-to-end pipeline from raw data to business-ready KPIs, where AI agents query the lakehouse directly via MCP

Architecture

AI-Powered Data Lake: Sources (CRM, ERP, APIs & events) → Bronze → Silver (S3 Bronze, Glue + PySpark, Quality checks) → Gold (dbt on Athena, KPIs & reports, CloudFormation) → AI layer (Claude Sonnet, Claude Opus, MCP server)Sources
CRMHubSpot, Salesforce
ERPSAP exports
APIs & eventsWhatsApp, web
Bronze → Silver
S3 Bronzeraw, as received
Glue + PySparkclean, dedupe, UTC
Quality checksschema + rules
Gold
dbt on Athenadaily metrics
KPIs & reportsrevenue, growth
CloudFormationS3, Glue, Athena
AI layer
Claude Sonnetprofiling, NL → SQL
Claude Opusbusiness insights
MCP serveragents query lake

Offline AI Tutor with Gemma 4 for Rural Peru

Goal: Educational tutor running Gemma 4 locally on a community edge server, so primary students without internet can use AI from basic phones over local WiFi

Tools: Gemma 4, Ollama, Google ADK, FastAPI, Angular, RAG (TF-IDF, scikit-learn), Docker Compose

Impact: Built for GDG Lima's Build with Gemma competition (Local AI & Edge track): AI deployed where the user is, with no connectivity required

Architecture

Offline AI Tutor (Gemma 4): Students (Basic phones, Local WiFi) → Edge server (Web UI, FastAPI) → Agent (Google ADK, Gemma 4, Gemini API) → Knowledge (PyMuPDF, SQLite, TF-IDF search)Students
Basic phonesbrowser, no data
Local WiFischool intranet
Edge server
Web UIno CDN, works offline
FastAPIREST API
Agent
Google ADKagent + tools
Gemma 4local via Ollama
Gemini APIoptional cloud mode
Knowledge
PyMuPDFMINEDU materials
SQLitechunks, chats
TF-IDF searchRAG, scikit-learn

Enterprise RAG & LLM Platform

CI/CD Status

Goal: Production-grade RAG system with vector database, LLM integration, and automated evaluation using Ragas

Tools: ChromaDB, OpenAI/Groq, FastAPI, Ragas, Prometheus, Docker, CI/CD

Impact: Complete knowledge management pipeline with metrics tracking, performance monitoring, and model versioning

Architecture

📄 Ingestion PipelineDocument Loading(PDF, Notion, Web)ChunkingRecursive SplitterEmbeddingtext-embedding-3 (1536d)🧠 Vector StoreChromaDB💬 RAG API (FastAPI)Query Processing(Hybrid Search RRF k=60)LLM Generation(GPT-4 / Llama 3)Evaluation (Ragas)Faithfulness, relevancy

US Accidents Data Lakehouse (Spark)

CI/CD Status

Goal: Spark-based lakehouse with Medallion Architecture (Bronze/Silver/Gold), streaming pipelines, ML training, and Airflow orchestration

Tools: PySpark, Delta Lake, MLflow, Airflow, Databricks, Terraform, Great Expectations

Impact: Scalable data platform processing millions of records with integrated ML pipelines, Delta Lake optimizations, and infrastructure provisioning

Architecture

Batch 6M rowsSpark cleanGeo aggScheduleAutomated data ingestion from Kaggle US accidents dataset
Bronze Ingestion
Bronze layer with full history and ACID transactions
Raw Delta Table
Distributed data cleaning with Apache Spark
Spark Cleaning
Silver layer with conformed schemas and optimized storage
Conformed Delta
Geographic hotspot analysis for accident clustering
Hotspot Aggregates
Weather correlation with accident patterns
Weather Impact
Temporal trend analysis for accident patterns
Hourly/Daily Trends
Machine learning pipeline with experiment tracking
ML Pipeline
End-to-end pipeline orchestration with Apache Airflow
Airflow DAG

Market Risk Analytics Platform

Goal: Calculate financial risk metrics (VaR, CVaR, Greeks) using Databricks with Airflow and Control-M orchestration

Tools: Databricks, Delta Lake, Control-M, Airflow, Great Expectations, PySpark

Impact: Enterprise-grade risk analysis system with data quality gates, Control-M integration, and CI/CD pipeline

Architecture

📡 IngestionWeb ScrapersPython/BS4Market NewsCloud RunScheduled Jobs1min frequency⚙️ Storage & Risk EngineBigQueryData Warehouse📊 Risk Metrics• VaR 95%/99%• CVaR• Greeks (Δ, Γ, V)📈 Serving & AnalyticsDuckDBWASM EngineIn-browser SQLFrontendReact/RechartsInteractive Charts

Retail Analytics Platform (dbt + Databricks)

Goal: Modern analytics engineering platform with dimensional modeling (Kimball), incremental models, and >80% test coverage

Tools: dbt, Databricks, Delta Lake, dbt_expectations, Z-ORDER, Liquid Clustering

Impact: Fully-tested analytics pipeline with customer segmentation (RFM), Delta Lake optimizations, and comprehensive documentation

Architecture

POS DataPostgreSQL⚙️ Data Processing (Databricks + dbt)🥉 BRONZERaw Ingest🥈 SILVER (dbt)Transform🥇 GOLD MARTSRFM Analysis • Customer CLV • Sales AggregatesBI DashboardMetabase

Telco Churn MLOps Platform

CI/CD Status

Goal: End-to-end MLOps pipeline with MLflow tracking, FastAPI serving, drift detection (Evidently), and automated CI/CD

Tools: MLflow, FastAPI, Evidently AI, Prometheus, Docker, Great Expectations, GitHub Actions

Impact: Complete ML lifecycle system with model registry, drift monitoring, API serving, and automated deployment pipeline

Architecture

FeaturesModel promoteDeployFeature engineering and data preprocessing for churn prediction
Data Prep
XGBoost gradient boosting for churn classification
Training
Model evaluation with AUC-ROC and Precision-Recall metrics
Evaluation
Experiment tracking and model artifact storage with MLflow
MLflow Tracking
MLflow Model Registry for staging and production models
Model Registry
REST API for real-time churn prediction
FastAPI Service
Data drift and model performance monitoring with Evidently AI
Drift Monitoring
Time-series metrics collection with Prometheus
Metrics
Operational dashboards and alerting with Grafana
Dashboards

Road Accident Prevention in Peru (Thesis)

Goal: Normalization, analysis and prediction of road accidents in Peru using 2008–2023 historical data

Tools: PySpark, Parquet, scikit-learn (Random Forest), Flask, Jupyter

Impact: ETL pipeline from Excel to Parquet, temporal and regional feature engineering, and a web app that predicts accidents by region, date and weather

Architecture

Road Accident Prediction, Peru: Raw data (Excel files, Regions, dates) → ETL (Python ETL, Parquet / CSV) → Analysis (PySpark EDA, Features) → Prediction (Random Forest, Flask web app)Raw data
Excel files2008–2023 records
Regions, datesmixed formats
ETL
Python ETLclean, standardize
Parquet / CSVoptimized storage
Analysis
PySpark EDAbig data analysis
Featureslags, trends, night
Prediction
Random Forestscikit-learn
Flask web appregion, date, weather

JobFlow: GenAI Job Search Agent

Goal: Agent that finds Data Engineer roles daily, scores them against your CV and uses an LLM to write a tailored application pack per job

Tools: Python, JobSpy, SQLite, Groq / OpenAI, RenderCV, Jinja2, GitHub Actions

Impact: Automated daily run that deduplicates postings, filters out unrelated roles and outputs a tailored PDF CV, cover letter and recruiter message per job

Architecture

JobFlow GenAI Agent: Collect (GitHub Actions, Job boards, JobSpy) → Store (SQLite, Status) → Score (Match score, Top matches) → LLM apply pack (Tailored CV, Cover letter, Recruiter note)Collect
GitHub Actionsruns every day
Job boardsLinkedIn, Indeed +2
JobSpyparallel scraping
Store
SQLiteupsert, no repeats
Statusnew vs applied
Score
Match scoreCV + keywords
Top matchesmin score filter
LLM apply pack
Tailored CVGroq / OpenAI
Cover letterJinja2 template
Recruiter noteready to send