Nishaglobal Education logo

Vector databases, evaluation, observability and serving tools

The supporting tools every real AI application needs: vector databases to search your documents (Chroma, pgvector, Pinecone, Qdrant, Weaviate), evaluation tools to test quality (promptfoo, Ragas, DeepEval), observability to see what's happening (Langfuse, LangSmith), and ways to run models yourself (Ollama, vLLM) or through cloud platforms.

Lesson 43 of 4420 min readQuiz: 5 questions

🎯 By the end of this lesson you will be able to:

  • Choose a vector database for a prototype and for production
  • Use evaluation and observability tools to test and monitor AI quality
  • Know when to run models locally or self-hosted, and the tools for each
In this lesson
  1. Vector databases: search by meaning
  2. Evaluation tools: test quality like code
  3. Observability: see every step
  4. Running models: cloud, local or self-hosted
  5. What this means for your role
  6. Key takeaways
  7. Quiz

Vector databases: search by meaning

DocumentsPDFs, pagesEmbedding modeltext → vectorsVector databaseChroma · pgvectorLLMAPI · Ollama · vLLMAnswer + sourcesEvaluationpromptfoo · Ragas · DeepEvalObservabilityLangfuse · LangSmith · PhoenixEvaluation and tracing watch every step
A production RAG stack: documents become vectors in a vector database, the LLM answers with sources, and evaluation and tracing watch every step.

RAG (Lesson 9) needs somewhere to store the embeddings of your document chunks and find the closest ones to a question. That is a vector database. Some are simple libraries for prototypes, some are features of databases you already use, and some are managed cloud services.

ToolWhat it isBest for
ChromaOpen-source, runs inside your app or as a serverPrototypes, learning, small projects
pgvectorAn extension that adds vector search to PostgreSQLTeams already using Postgres; keeps data in one place
PineconeFully managed cloud vector databaseProduction without running infrastructure
Qdrant, Weaviate, MilvusOpen-source vector databases, self-hosted or cloudLarge collections, filtering, control over hosting
Elasticsearch, OpenSearchSearch engines with keyword and vector searchHybrid search on existing search infrastructure

Chroma in five lines

import chromadb

client = chromadb.Client()
policies = client.create_collection("policies")
policies.add(ids=["p1", "p2"], documents=["New staff get 25 days of annual leave.", "Refunds take 5 working days."])
print(policies.query(query_texts=["How much holiday do I get?"], n_results=1)["documents"])
# [['New staff get 25 days of annual leave.']]  ← found by meaning, not by the word "holiday"

Chroma creates the embeddings for you with a small default model, which makes it ideal for learning.

pgvector: vector search in the database you already have

CREATE EXTENSION vector;
CREATE TABLE chunks (id bigserial PRIMARY KEY, content text, embedding vector(1536));

-- Find the 5 chunks closest in meaning to a question's embedding
SELECT content FROM chunks ORDER BY embedding <=> '[...question embedding...]' LIMIT 5;

The number 1536 must match the size of the embeddings your embedding model produces.

Evaluation tools: test quality like code

Evaluation tools run your test cases (Lessons 19–23) automatically, so every prompt or model change is checked before release.

  • promptfoo: define prompts, models and test cases in a simple config file; run in your terminal or CI; compare models side by side; also includes red-team testing.
  • Ragas: ready-made metrics for RAG, such as faithfulness to sources and how relevant the retrieved chunks are.
  • DeepEval: write AI tests in Python like normal unit tests, with many built-in metrics.

A promptfoo test file comparing two models

prompts:
  - "Summarise this complaint in 3 bullets: {{email}}"
providers:
  - anthropic:messages:claude-...
  - openai:gpt-...
tests:
  - vars:
      email: "Order 4471 arrived broken. I want a replacement, not a refund."
    assert:
      - type: contains
        value: "replacement"
      - type: llm-rubric
        value: "At most 3 bullet points and no made-up details"

# Run with:  npx promptfoo eval

Both models answer every test case, and you see a table of passes and failures, which is evidence for choosing a model.

Observability: see every step

Observability tools record each request as a trace: the prompt, retrieved chunks, tool calls, model answer, time taken, tokens and cost. When a user reports a bad answer, you can see exactly why.

ToolNotes
LangfuseOpen source; tracing, prompt management, evaluations and cost tracking; self-host or cloud
LangSmithFrom the LangChain team; tracing and evaluation, works with or without LangChain
Arize PhoenixOpen-source tracing and evaluation, strong for RAG analysis
OpenTelemetryThe open standard for traces; its generative AI conventions let you send AI traces to many monitoring tools

Running models: cloud, local or self-hosted

  • Provider APIs: the simplest option; the provider runs everything.
  • Cloud AI platforms: Amazon Bedrock, Microsoft Foundry (formerly Azure AI Foundry) and Google Vertex AI offer many models inside your existing cloud account, security and billing.
  • Ollama: runs open models on your own laptop or server with one command, which is great for learning, privacy and offline use.
  • llama.cpp: the engine behind many local tools; runs models efficiently even without a big GPU.
  • vLLM: high-speed serving of open models on GPU servers, with an OpenAI-compatible API, for production self-hosting.

Run an open model on your own computer with Ollama

# 1. Install Ollama from ollama.com, then:
ollama pull llama3.2                       # download a small open model (a few GB)
ollama run llama3.2 "Explain RAG in one sentence"

# For production GPU serving, vLLM starts an OpenAI-compatible server:
vllm serve <hugging-face-model-id>          # then call http://localhost:8000/v1 with any OpenAI SDK

Nothing leaves your computer. Small local models are less capable than the largest hosted models, so test them on your own tasks.

What this means for your role

Developer

Prototype with Chroma and a local model in Ollama; move to pgvector or a managed vector database and add tracing before production.

Tester / QA

Put promptfoo or DeepEval tests in CI so every prompt or model change is checked against your evaluation set.

Business Analyst / PO

Ask early whether data may leave the company. That decides between provider APIs, cloud platforms in your account, or self-hosting.

Tech Lead / Architect

Choose the vector store, evaluation tool and tracing tool once for the platform (Lesson 24), and make tracing mandatory for production AI.

Project / Delivery Manager

Self-hosting saves per-token costs but adds GPU, operations and people costs; compare total cost of ownership (Lesson 25).

Scrum Master / Agile

Make evaluation results and trace reviews part of the team's regular rhythm, such as a quick look at failures in each review.

Try it: your own private AI in 10 minutes

  1. Install Ollama and run a small open model with a question about your field.
  2. Install Chroma (pip install chromadb) and add three sentences from a document you know.
  3. Ask a question in different words and check whether Chroma finds the right sentence by meaning.

Key takeaways

  • Vector databases range from Chroma for prototypes, to pgvector inside Postgres, to managed services like Pinecone and open-source Qdrant, Weaviate and Milvus.
  • Evaluation tools such as promptfoo, Ragas and DeepEval turn quality checks into automated tests.
  • Observability tools such as Langfuse and LangSmith trace every step, token and cost.
  • Run models through provider APIs, cloud platforms (Bedrock, Microsoft Foundry, Vertex AI), locally with Ollama, or self-hosted with vLLM.

Key terms

Vector database
A database that stores embeddings and finds the items closest in meaning to a query.
Trace
A record of every step in one AI request: prompts, retrieval, tool calls, answer, time and cost.
Self-hosting
Running models on your own servers or cloud account instead of a provider's API.
OpenAI-compatible API
An API that accepts the same request format as OpenAI's, so the same code works with many servers.

Check your understanding

Answer all 5 questions. Score 3 or more to complete the lesson.

  1. 1. Your team already runs PostgreSQL and wants vector search with the least new infrastructure. What fits?

  2. 2. What does an observability tool such as Langfuse record?

  3. 3. Which tool lets you run open models on your own laptop with one command?

  4. 4. What is promptfoo mainly used for?

  5. 5. Why does an 'OpenAI-compatible' API matter?

Answer all 5 questions to check.

Practise more