LLMs
Show where LLMs affected a real decision in the project. Include the implementation, constraint, test or measurement that makes the skill verifiable.
CAREER PROJECT · Advanced · 2–3 weekends
RAG Evaluation Workbench
Build retrieval-augmented generation with repeatable quality evaluation instead of stopping at chat.
LLMsRetrievalEvaluationData pipelinesObservability
WHAT THIS PROJECT PROVES
Make the engineering visible.
This brief is designed as portfolio evidence for a AI Engineer path. The goal is not to claim a hiring outcome; it is to make your implementation, judgment and tradeoffs easier to inspect.
Retrieval
Show where Retrieval affected a real decision in the project. Include the implementation, constraint, test or measurement that makes the skill verifiable.
Evaluation
Show where Evaluation affected a real decision in the project. Include the implementation, constraint, test or measurement that makes the skill verifiable.
Data pipelines
Show where Data pipelines affected a real decision in the project. Include the implementation, constraint, test or measurement that makes the skill verifiable.
Observability
Show where Observability affected a real decision in the project. Include the implementation, constraint, test or measurement that makes the skill verifiable.
BUILD PLAN
Finish the core before expanding the scope.
Use this as a project sequence, not a rigid tutorial. Your stack can change; the evidence should still show what you built, why you built it that way and how you know it works.
Define the scope and success criteria
Turn the brief into a small finishable system. Decide what the core RAG Evaluation Workbench must do, what you will deliberately leave out, and how you will show that it works.
Build the core workflow
Build retrieval-augmented generation with repeatable quality evaluation instead of stopping at chat. Keep the first pass focused on the main user or system flow before adding stretch features.
Add engineering proof
Create evaluation dataset and retrieval pipeline. Document the decisions, failure states, tests, performance, security or operational tradeoffs that matter for this project.
Ship and package the evidence
Publish the strongest artifacts, including quality dashboard, failure analysis, cost/latency notes. Make it easy for a reviewer to understand the problem, architecture, tradeoffs and what you personally implemented.
PORTFOLIO PROOF CHECKLIST
Leave artifacts a reviewer can inspect.
A strong project page should make it possible to understand the system without guessing what you personally built.
01
Evaluation dataset
02
Retrieval pipeline
03
Quality dashboard
04
Failure analysis
05
Cost/latency notes
06
A concise README explaining the problem, architecture and tradeoffs
07
A working demo, API example or reproducible run path
08
Evidence of testing, monitoring, performance, accessibility or reliability where relevant
RESUME / PORTFOLIO FRAMING
Describe what you actually did.
Implemented a RAG pipeline with repeatable evaluation, failure analysis and latency/cost instrumentation.
Use this as a starting structure, then replace it with your real implementation details, metrics and constraints. Do not claim outcomes you did not measure.
INTERVIEW TALKING POINTS
Be ready to defend the decisions.
These prompts turn the project from a repository link into a technical conversation about judgment, tradeoffs and evidence.
Why did you choose your approach to LLMs, and what alternative did you reject?
Answer with details from your implementation. Prefer concrete constraints, measurements and alternatives over generic best practices.
Where could RAG Evaluation Workbench fail under real usage, and how would you detect or recover from that failure?
Answer with details from your implementation. Prefer concrete constraints, measurements and alternatives over generic best practices.
How did Retrieval affect the rest of the design?
Answer with details from your implementation. Prefer concrete constraints, measurements and alternatives over generic best practices.
What did you measure or test to prove Observability instead of only claiming it?
Answer with details from your implementation. Prefer concrete constraints, measurements and alternatives over generic best practices.
If you had one more week, what would you improve first and why?
Answer with details from your implementation. Prefer concrete constraints, measurements and alternatives over generic best practices.
STRETCH GOALS
Go deeper only after the core is solid.
Reranking
Add it only if it strengthens the engineering story instead of delaying a finished, deployable core.
Hybrid search
Add it only if it strengthens the engineering story instead of delaying a finished, deployable core.
Tracing
Add it only if it strengthens the engineering story instead of delaying a finished, deployable core.
RELATED PORTFOLIO PROJECTS
Build another signal in the same path.
Choose another project only when it proves a different part of the role instead of repeating the same evidence.
Intermediate2 weekends
Support Ticket Triage Assistant
Classify and route support tickets with confidence thresholds, human review and evaluation.
ClassificationLLMsEvaluationHuman-in-loopAPIs
Open project brief →Advanced2 weekends
Document Extraction Pipeline
Extract structured fields from messy documents and evaluate correctness against labeled examples.
OCR/visionStructured outputEvaluationPipelinesValidation
Open project brief →Intermediate1–2 weekends
Semantic Search Engine
Index a real corpus, implement semantic retrieval and compare it with keyword baselines.
EmbeddingsVector searchEvaluationAPIsRanking
Open project brief →