What we did
Smart Legal Pro is, at its core, a stack of highly specialised AI functions: document analysis, contract comparison, letter drafting, contract revision, lawyer pre-chat, follow-up questions, reply analysis. Historically, each of these functions had its own prompt, its own parser, its own way of handling errors. That worked — but it was technical debt that made every optimisation expensive.
Over the past few weeks we rebuilt the foundation from the ground up. No new buttons in the UI, no loud re-launch. But under the hood, nothing is the same as before.
1. One pipeline for all AI calls
Every edge function now calls the model through the same hardened layer:
- PII masking before the model sees anything (names, addresses, IBANs, case reference numbers) — reversible when writing the result back.
- Schema validation of the response against a registered JSON schema entry.
- Schema repair loop: if validation fails, the model receives the concrete errors back and corrects them in a targeted way — instead of blindly regenerating from scratch.
- Refund path: if even that fails, the credit deduction is automatically reversed. Nobody pays for a broken answer anymore.
2. Playbooks instead of hardcoded rules
Jurisdiction-specific logic ("in a German traffic-fine notice, a missing calibration certificate is not a red flag") now lives exclusively in the playbooks table — per case type, per language, per country, with an XX fallback. The composer pulls the matching playbook and layers it over the generic system prompt as an overlay.
In concrete terms: we can treat the law differently for Germany, Austria and Switzerland without touching the code. Current status: 149 active playbooks for the DACH region, all curated from a single source.
3. Full telemetry for every run
Every single AI call now lands in prompt_runs with:
- Function, model, tier, language, country
- Input and output tokens, cached tokens (for prompt-caching savings), cost in credits
- Latency (P50/P95), success/failure, schema repair attempts
- Anonymised run ID for later few-shot curation
The Platform Admin has its own view for this (/admin/prompt-health): filterable by function, language, country, tier — with stats on success rate, latency, cost and repair rate. This is the foundation on which we'll optimise every single prompt variant data-driven over the coming weeks, instead of by gut feeling.
4. Schema registry and few-shot curation
Two new admin tools make the optimisation practical:
- Prompt Schemas (
/admin/prompt-schemas): a central registry of all response schemas, versioned, activatable live. Nobody needs to deploy anything for a schema update anymore. - Few-Shot Curation (
/admin/prompt-few-shot): successful runs land as candidates in a table, get reviewed, approved, and become available as examples from the next call onward. Quality improves with every approved sample.
5. Bugs that fell out along the way
While migrating, a few pieces of legacy debt turned up that we fixed on the spot:
- The letter revision dialog showed an error toast despite successfully creating the letter — a mismatch between the edge response and the frontend parser. Fixed.
- `generate-letter-from-template` (the primary letter path from the wizard) was still on the old pipeline. Now on the same footing as everything else.
- `analyze-reply` and `lawyer-prechat-stream` weren't writing telemetry at all. Now fully visible.
What you'll notice
Directly: fewer "oops" toasts, no phantom errors, clean refunds in the rare cases where the model really does get something wrong.
In the medium term: noticeably better answers, because for the first time we can see, data-driven, where which prompt variant performs well and where it doesn't. Optimisation goes from guesswork to an engineering discipline.
What's coming next
- Gather an initial data baseline from live DE runs (4–6 weeks).
- On that basis: targeted prompt refreshes per function, build up a few-shot library.
- Only then: further languages, further countries. Germany has to be genuinely excellent first.
We're not building fireworks. We're building a foundation.
