Case study · Diploma thesis 2026 · HTL Leonding

Development of an MCP Server with Semantic Search for LeoWiki

The school wiki's keyword search fails on natural-language questions. We built the missing layer: a vector database with German‑optimized embeddings behind an MCP server, so AI assistants like Claude, ChatGPT and Mistral Le Chat can answer questions from the wiki — permission-aware, self-hosted.

Team · Jan Ritt & Imre Obermüller Supervisor · Rainer Stropek

📄 Read the full thesis (PDF, 170 pages)

Result: 17× better retrieval

Measured on 78 verified question-answer pairs across 12 wiki namespaces (10,841 indexed chunks). Bars show MRR on a 0–1 scale:

DokuWiki keyword search

Full-question queryAND logic · 49 / 78 return nothing

0.051

Hand-optimized keywordsbest-case manual · 29 / 78 return nothing

0.365

Semantic jump ≈ +0.39 MRR →

Semantic RAG retrieval

Octen-Embedding-4Bbest local model · top-10 hit 94.9%

0.883

17×MRR vs full-question keyword search (0.883 vs 0.051)

2.4×vs hand-optimized keywords (0.883 vs 0.365)

+89.8 pptop-10 hit rate (5.1% → 94.9%)

Architecture

Two phases: an offline preparation pipeline (fetch → evaluate → chunk → embed → deploy) and a runtime query flow on the Raspberry Pi 5.

Enrichment

Content evaluation with GPT-4o-mini (text) and Qwen-2.5-VL (images & PDF graphics) before content-aware chunking.

MCP server

FastMCP 3.0 (Python) with a two-tool RBAC search design, five middleware layers, MCP resources & prompts — over stdio and HTTP Streamable.

Security

OAuth 2.1 via Scalekit, JWT role decoding and role-based access control on every search — teachers, students and admins each see only their content.

Embedding evaluation — an engineering trade-off

Five models benchmarked on the full corpus. The local quality leader (Octen-4B) could not run on the target hardware — so production pragmatically uses the API model that ties it within 1.1 points of MRR:

Octen-Embedding-4Blocal · 4.0B · MTEB #2 (DE) · nDCG 0.899
quality leader

0.883

94.9%hit

text-embedding-3-largeOpenAI API · 3,072 dim · nDCG 0.897
production

0.872

97.4%hit

Tier 1 ≥ 0.87 · production-grade

PIXIE-Rune-v1.0local · 568M · MTEB #3 (DE) · nDCG 0.823

0.802

88.5%hit

snowflake-arctic-embed-l-v2.0local · 568M · MTEB #7 (DE) · nDCG 0.822

0.788

92.3%hit

bge-m3-unsupervisedRAGFlow default · 568M · nDCG 0.800

0.756

93.6%hit

Production tier (MRR ≥ 0.87) MRR · 0–1 % badge = top-10 hit rate

Methodology: MRR, nDCG@10, MAP, precision@k, hit rate + LLM-as-Judge and RAGAS · chunk-size experiments (256/512/1024) · hybrid vs. dense retrieval · keyword baseline · Wilcoxon signed-rank tests with bootstrap confidence intervals.

MTEB DE reference AI provider landscape

Deployment — production on 5 watts

Four Docker Compose services on a Raspberry Pi 5 (8 GB, ARM64): Caddy for TLS & reverse proxy, the MCP server, Qdrant for vector search, and a watchdog for JSONL auto-ingestion. Mean end-to-end search latency: ~226 ms (stdio) / ~321 ms (HTTP).