Vector databases, evaluation, observability and serving tools
The supporting tools every real AI application needs: vector databases to search your documents (Chroma, pgvector, Pinecone, Qdrant, Weaviate), evaluation tools to test quality (promptfoo, Ragas, DeepEval), observability to see what's happening (Langfuse, LangSmith), and ways to run models yourself (Ollama, vLLM) or through cloud platforms.
🎯 By the end of this lesson you will be able to:
- Choose a vector database for a prototype and for production
- Use evaluation and observability tools to test and monitor AI quality
- Know when to run models locally or self-hosted, and the tools for each
In this lesson
Vector databases: search by meaning
RAG (Lesson 9) needs somewhere to store the embeddings of your document chunks and find the closest ones to a question. That is a vector database. Some are simple libraries for prototypes, some are features of databases you already use, and some are managed cloud services.
| Tool | What it is | Best for |
|---|---|---|
| Chroma | Open-source, runs inside your app or as a server | Prototypes, learning, small projects |
| pgvector | An extension that adds vector search to PostgreSQL | Teams already using Postgres; keeps data in one place |
| Pinecone | Fully managed cloud vector database | Production without running infrastructure |
| Qdrant, Weaviate, Milvus | Open-source vector databases, self-hosted or cloud | Large collections, filtering, control over hosting |
| Elasticsearch, OpenSearch | Search engines with keyword and vector search | Hybrid search on existing search infrastructure |
Chroma in five lines
import chromadb
client = chromadb.Client()
policies = client.create_collection("policies")
policies.add(ids=["p1", "p2"], documents=["New staff get 25 days of annual leave.", "Refunds take 5 working days."])
print(policies.query(query_texts=["How much holiday do I get?"], n_results=1)["documents"])
# [['New staff get 25 days of annual leave.']] ← found by meaning, not by the word "holiday"Chroma creates the embeddings for you with a small default model, which makes it ideal for learning.
pgvector: vector search in the database you already have
CREATE EXTENSION vector; CREATE TABLE chunks (id bigserial PRIMARY KEY, content text, embedding vector(1536)); -- Find the 5 chunks closest in meaning to a question's embedding SELECT content FROM chunks ORDER BY embedding <=> '[...question embedding...]' LIMIT 5;
The number 1536 must match the size of the embeddings your embedding model produces.
Evaluation tools: test quality like code
Evaluation tools run your test cases (Lessons 19–23) automatically, so every prompt or model change is checked before release.
- promptfoo: define prompts, models and test cases in a simple config file; run in your terminal or CI; compare models side by side; also includes red-team testing.
- Ragas: ready-made metrics for RAG, such as faithfulness to sources and how relevant the retrieved chunks are.
- DeepEval: write AI tests in Python like normal unit tests, with many built-in metrics.
A promptfoo test file comparing two models
prompts:
- "Summarise this complaint in 3 bullets: {{email}}"
providers:
- anthropic:messages:claude-...
- openai:gpt-...
tests:
- vars:
email: "Order 4471 arrived broken. I want a replacement, not a refund."
assert:
- type: contains
value: "replacement"
- type: llm-rubric
value: "At most 3 bullet points and no made-up details"
# Run with: npx promptfoo evalBoth models answer every test case, and you see a table of passes and failures, which is evidence for choosing a model.
Observability: see every step
Observability tools record each request as a trace: the prompt, retrieved chunks, tool calls, model answer, time taken, tokens and cost. When a user reports a bad answer, you can see exactly why.
| Tool | Notes |
|---|---|
| Langfuse | Open source; tracing, prompt management, evaluations and cost tracking; self-host or cloud |
| LangSmith | From the LangChain team; tracing and evaluation, works with or without LangChain |
| Arize Phoenix | Open-source tracing and evaluation, strong for RAG analysis |
| OpenTelemetry | The open standard for traces; its generative AI conventions let you send AI traces to many monitoring tools |
Running models: cloud, local or self-hosted
- Provider APIs: the simplest option; the provider runs everything.
- Cloud AI platforms: Amazon Bedrock, Microsoft Foundry (formerly Azure AI Foundry) and Google Vertex AI offer many models inside your existing cloud account, security and billing.
- Ollama: runs open models on your own laptop or server with one command, which is great for learning, privacy and offline use.
- llama.cpp: the engine behind many local tools; runs models efficiently even without a big GPU.
- vLLM: high-speed serving of open models on GPU servers, with an OpenAI-compatible API, for production self-hosting.
Run an open model on your own computer with Ollama
# 1. Install Ollama from ollama.com, then: ollama pull llama3.2 # download a small open model (a few GB) ollama run llama3.2 "Explain RAG in one sentence" # For production GPU serving, vLLM starts an OpenAI-compatible server: vllm serve <hugging-face-model-id> # then call http://localhost:8000/v1 with any OpenAI SDK
Nothing leaves your computer. Small local models are less capable than the largest hosted models, so test them on your own tasks.
What this means for your role
Developer
Prototype with Chroma and a local model in Ollama; move to pgvector or a managed vector database and add tracing before production.
Tester / QA
Put promptfoo or DeepEval tests in CI so every prompt or model change is checked against your evaluation set.
Business Analyst / PO
Ask early whether data may leave the company. That decides between provider APIs, cloud platforms in your account, or self-hosting.
Tech Lead / Architect
Choose the vector store, evaluation tool and tracing tool once for the platform (Lesson 24), and make tracing mandatory for production AI.
Project / Delivery Manager
Self-hosting saves per-token costs but adds GPU, operations and people costs; compare total cost of ownership (Lesson 25).
Scrum Master / Agile
Make evaluation results and trace reviews part of the team's regular rhythm, such as a quick look at failures in each review.
Try it: your own private AI in 10 minutes
- Install Ollama and run a small open model with a question about your field.
- Install Chroma (pip install chromadb) and add three sentences from a document you know.
- Ask a question in different words and check whether Chroma finds the right sentence by meaning.
Key takeaways
- Vector databases range from Chroma for prototypes, to pgvector inside Postgres, to managed services like Pinecone and open-source Qdrant, Weaviate and Milvus.
- Evaluation tools such as promptfoo, Ragas and DeepEval turn quality checks into automated tests.
- Observability tools such as Langfuse and LangSmith trace every step, token and cost.
- Run models through provider APIs, cloud platforms (Bedrock, Microsoft Foundry, Vertex AI), locally with Ollama, or self-hosted with vLLM.
Key terms
- Vector database
- A database that stores embeddings and finds the items closest in meaning to a query.
- Trace
- A record of every step in one AI request: prompts, retrieval, tool calls, answer, time and cost.
- Self-hosting
- Running models on your own servers or cloud account instead of a provider's API.
- OpenAI-compatible API
- An API that accepts the same request format as OpenAI's, so the same code works with many servers.
Check your understanding
Answer all 5 questions. Score 3 or more to complete the lesson.
1. Your team already runs PostgreSQL and wants vector search with the least new infrastructure. What fits?
2. What does an observability tool such as Langfuse record?
3. Which tool lets you run open models on your own laptop with one command?
4. What is promptfoo mainly used for?
5. Why does an 'OpenAI-compatible' API matter?