feat(guardrail): similarità pre-scrittura su POST /v1/memories

Guardrail deterministico FUORI dall'LLM (stessa architettura di egeos-copilot): - Strato 1: text_hash SHA-256 normalizzato -> BLOCK 409 (EXACT_DUPLICATE) - Strato 2: similarità semantica top-3 BGE-M3 cosine -> BLOCK/WARN/ALLOW (soglie configurabili: GUARDRAIL_BLOCK_THRESHOLD 0.85, WARN 0.70) - Supersede esplicito bypassa il guardrail (correzione intenzionale) - text_hash e flag guardrail nel payload; audit create_blocked - Indice payload su text_hash - Test: 6 nuovi (duplicato esatto, similarità alta/moderata, nessun candidato, supersede bypass, disabilitato, text_hash) — 28/28 passano
2026-08-18 12:58:28 +02:00
parent 91ec4c82d7
commit eccb2cb870
4 changed files with 262 additions and 4 deletions
@@ -34,7 +34,7 @@ Requisiti: Docker + Compose v2, Ollama con modello `bge-m3` sul host

 | Endpoint | Descrizione |
 |---|---|
-| `POST /v1/memories` | Crea record (text, kind, agent_id, scope, **project_id obbligatorio**, source, expires_at, supersedes_id, supersede_reason) |
+| `POST /v1/memories` | Crea record (text, kind, agent_id, scope, **project_id obbligatorio**, source, expires_at, supersedes_id, supersede_reason). Applica il guardrail di similarità pre-scrittura |
 | `POST /v1/memories:search` | Ricerca semantica (query, kind, project_id, scope, top_k, include_superseded, min_score) |
 | `GET /v1/memories/{id}` | Recupera per UUID |
 | `DELETE /v1/memories/{id}` | Elimina per UUID |
@@ -43,6 +43,25 @@ Requisiti: Docker + Compose v2, Ollama con modello `bge-m3` sul host

 Auth: header `X-API-Key` (chiave condivisa, accesso completo). Rate limit 120 req/min per chiave. Audit log in JSON lines (docker logs).

+## Guardrail di similarità (v1)
+
+Enforcement deterministico FUORI dall'LLM, prima di ogni scrittura su `POST /v1/memories`:
+
+1. **Strato 1 — hash esatto**: SHA-256 del testo normalizzato (`text_hash` nel payload). Se esiste un record attivo con lo stesso hash → `409 BLOCK (EXACT_DUPLICATE)`.
+2. **Strato 2 — similarità semantica top-3**: embedding BGE-M3 cosine sui record attivi (esclusi i superseded).
+   - top-1 ≥ `GUARDRAIL_BLOCK_THRESHOLD` (default 0.85) → `409 BLOCK (KNOWN_SOLUTION)`
+   - top-1 ≥ `GUARDRAIL_WARN_THRESHOLD` (default 0.70) → `WARN`: salva con flag `guardrail` nel payload
+   - altrimenti → `ALLOW`
+
+Il **supersede esplicito** (`supersedes_id`) è una correzione intenzionale: bypassa il guardrail.
+
+Configurazione (env): `GUARDRAIL_ENABLED` (default true), `GUARDRAIL_BLOCK_THRESHOLD`, `GUARDRAIL_WARN_THRESHOLD`. Soglie di partenza da calibrare sul corpus reale.
+
+Risposta BLOCK (409):
+```json
+{"detail": {"error": "duplicate_memory", "reason": "KNOWN_SOLUTION", "matches": [{"memory_id": "...", "score": 0.92}], "message": "..."}}
+```
+
 ## Sicurezza

 - Qdrant bindato su 127.0.0.1; gateway solo su interfaccia VPN
@@ -52,6 +52,15 @@ API_KEYS: set[str] = {k.strip() for k in os.environ.get("API_KEYS", "").split(",
 RATE_LIMIT_PER_MIN = int(os.environ.get("RATE_LIMIT_PER_MIN", "120"))
 MAX_TEXT_LEN = int(os.environ.get("MAX_TEXT_LEN", "8000"))

+# Guardrail di similarità pre-scrittura (v1, 2026-08-17)
+# Approccio a strati: hash esatto SHA-256 -> BLOCK; similarità semantica top-3
+# (BGE-M3 cosine) -> BLOCK / WARN / ALLOW. Enforcement FUORI dall'LLM.
+GUARDRAIL_ENABLED = os.environ.get("GUARDRAIL_ENABLED", "true").lower() == "true"
+# Soglie cosine BGE-M3 (valori di partenza da calibrare sul corpus)
+GUARDRAIL_BLOCK_THRESHOLD = float(os.environ.get("GUARDRAIL_BLOCK_THRESHOLD", "0.85"))
+GUARDRAIL_WARN_THRESHOLD = float(os.environ.get("GUARDRAIL_WARN_THRESHOLD", "0.70"))
+GUARDRAIL_VERSION = "similarity-v1"
+
 # Metriche: push a VictoriaMetrics (stesso pattern dell'energy engine domotics)
 VM_PUSH_URL = os.environ.get("VM_PUSH_URL", "http://host.docker.internal:8428/api/v1/import/prometheus")
 VM_PUSH_INTERVAL = int(os.environ.get("VM_PUSH_INTERVAL", "30"))
@@ -94,7 +103,7 @@ async def lifespan(_app: FastAPI):
                SPARSE_VECTOR_NAME: qm.SparseVectorParams(modifier=qm.Modifier.IDF),
            },
        )
-        for field in ("agent_id", "project_id", "scope", "kind", "supersedes_id", "superseded_by"):
+        for field in ("agent_id", "project_id", "scope", "kind", "supersedes_id", "superseded_by", "text_hash"):
            qdrant.create_payload_index(
                collection_name=COLLECTION,
                field_name=field,
@@ -278,6 +287,83 @@ def _now_iso() -> str:
    return datetime.now(timezone.utc).isoformat()


+# ---------------------------------------------------------------------------
+# Guardrail di similarità pre-scrittura
+# ---------------------------------------------------------------------------
+
+def _normalize_text(text: str) -> str:
+    """Normalizzazione canonica: lowercase, accenti rimossi, spazi normalizzati."""
+    import unicodedata
+
+    s = unicodedata.normalize("NFD", text.lower())
+    s = "".join(c for c in s if not unicodedata.combining(c))
+    return " ".join(s.split())
+
+
+def _text_hash(text: str) -> str:
+    """SHA-256 del testo normalizzato (strato 1: duplicati esatti)."""
+    return hashlib.sha256(_normalize_text(text).encode("utf-8")).hexdigest()
+
+
+def _find_similar(text: str, vector: list[float], top_k: int = 3) -> list[dict]:
+    """Top-k record attivi più simili (esclude i superseded)."""
+    qfilter = qm.Filter(must=[qm.IsEmptyCondition(is_empty=qm.PayloadField(key="superseded_by"))])
+    hits = qdrant.query_points(
+        collection_name=COLLECTION,
+        query=vector,
+        query_filter=qfilter,
+        limit=top_k,
+        with_payload=True,
+    ).points
+    return [
+        {
+            "memory_id": h.id,
+            "score": round(float(h.score), 4),
+            "text": (h.payload or {}).get("text", ""),
+            "kind": (h.payload or {}).get("kind", ""),
+            "project_id": (h.payload or {}).get("project_id", ""),
+        }
+        for h in hits
+    ]
+
+
+def _decide_guardrail(text: str, vector: list[float]) -> dict:
+    """Applica il guardrail a 2 strati. Ritorna {decision, reason, matches}."""
+    # Strato 1 — hash esatto (duplicato identico)
+    text_hash = _text_hash(text)
+    qfilter = qm.Filter(
+        must=[
+            qm.FieldCondition(key="text_hash", match=qm.MatchValue(value=text_hash)),
+            qm.IsEmptyCondition(is_empty=qm.PayloadField(key="superseded_by")),
+        ]
+    )
+    exact = qdrant.query_points(
+        collection_name=COLLECTION,
+        query=vector,
+        query_filter=qfilter,
+        limit=1,
+        with_payload=True,
+    ).points
+    if exact:
+        return {
+            "decision": "BLOCK",
+            "reason": "EXACT_DUPLICATE",
+            "matches": [{"memory_id": exact[0].id, "score": 1.0}],
+        }
+
+    # Strato 2 — similarità semantica top-3
+    matches = _find_similar(text, vector, top_k=3)
+    if not matches:
+        return {"decision": "ALLOW", "reason": "NO_CANDIDATE", "matches": []}
+
+    top1 = matches[0]["score"]
+    if top1 >= GUARDRAIL_BLOCK_THRESHOLD:
+        return {"decision": "BLOCK", "reason": "KNOWN_SOLUTION", "matches": matches}
+    if top1 >= GUARDRAIL_WARN_THRESHOLD:
+        return {"decision": "WARN", "reason": "MODERATE_SIMILARITY", "matches": matches}
+    return {"decision": "ALLOW", "reason": "NEW_SOLUTION", "matches": matches}
+
+
 def _parse_ts(value: Optional[str]) -> Optional[float]:
    """Converte ISO 8601 in timestamp Unix (per i range query Qdrant)."""
    if not value:
@@ -402,6 +488,30 @@ async def add_memory(
    vector = await embed(body.text)
    sparse = _sparse_encode(body.text)

+    # Guardrail di similarità pre-scrittura (enforcement FUORI dall'LLM).
+    # Il supersede esplicito è una correzione intenzionale: bypassa il guardrail.
+    guardrail: Optional[dict] = None
+    if GUARDRAIL_ENABLED and not body.supersedes_id:
+        guardrail = _decide_guardrail(body.text, vector)
+        if guardrail["decision"] == "BLOCK":
+            _audit(
+                key,
+                "create_blocked",
+                kind=body.kind,
+                agent_id=body.agent_id or "shared",
+                reason=guardrail["reason"],
+                matches=[m["memory_id"] for m in guardrail["matches"]],
+            )
+            raise HTTPException(
+                status_code=409,
+                detail={
+                    "error": "duplicate_memory",
+                    "reason": guardrail["reason"],
+                    "matches": guardrail["matches"],
+                    "message": "Memoria già presente o quasi identica: usa supersedes_id per correggere la versione attiva, oppure riformula il contenuto.",
+                },
+            )
+
    payload: dict[str, Any] = {
        "text": body.text,
        "kind": body.kind,
@@ -415,7 +525,15 @@ async def add_memory(
        "supersedes_id": superseded_id,
        "supersede_reason": body.supersede_reason,
        "embedding_model": EMBED_MODEL,
+        "text_hash": _text_hash(body.text),
    }
+    if guardrail:
+        payload["guardrail"] = {
+            "version": GUARDRAIL_VERSION,
+            "decision": guardrail["decision"],
+            "reason": guardrail["reason"],
+            "matches": guardrail["matches"],
+        }
    point_vector: dict[str, Any] = {"": vector}
    if sparse is not None:
        point_vector[SPARSE_VECTOR_NAME] = sparse
@@ -437,7 +555,14 @@ async def add_memory(
        )
        _audit(key, "supersede", old_id=superseded_id, new_id=memory_id, kind=body.kind, agent_id=payload["agent_id"])
    else:
-        _audit(key, "create", memory_id=memory_id, kind=body.kind, agent_id=payload["agent_id"])
+        _audit(
+            key,
+            "create",
+            memory_id=memory_id,
+            kind=body.kind,
+            agent_id=payload["agent_id"],
+            guardrail=payload.get("guardrail", {}).get("decision", "ALLOW"),
+        )
    response = {"memory_id": memory_id, "created_at": payload["created_at"], "supersedes_id": superseded_id}
    if idem_key:
        _idempotency[idem_key] = {"hash": _payload_hash(body), "response": response, "ts": time.time()}
@@ -12,6 +12,8 @@ from pathlib import Path
 # Disabilita il push metriche nei test
 os.environ["METRICS_ENABLED"] = "false"
 os.environ["API_KEYS"] = "test-key"
+# Guardrail disabilitato di default nei test esistenti (abilitato nei test del guardrail)
+os.environ["GUARDRAIL_ENABLED"] = "false"

 sys.path.insert(0, str(Path(__file__).resolve().parent.parent))

@@ -33,6 +35,7 @@ class FakeQdrant:
        self.points: dict[str, FakePoint] = {}
        self.collection_exists = False
        self.upsert_calls = 0
+        self.query_score = 0.9  # score di default per query_points (configurabile nei test)

    def get_collections(self):
        class _C:
@@ -94,7 +97,7 @@ class FakeQdrant:
                            ok = False
                if not ok:
                    continue
-            results.append(type("H", (), {"id": p.id, "score": 0.9, "payload": pl})())
+            results.append(type("H", (), {"id": p.id, "score": self.query_score, "payload": pl})())
        return type("R", (), {"points": results[:limit]})()


@@ -0,0 +1,111 @@
+"""Test del guardrail di similarità pre-scrittura (Memory Gateway).
+
+Casi: duplicato esatto (hash) -> BLOCK 409; similarità alta -> BLOCK 409;
+similarità moderata -> WARN (salva con flag); nessun candidato -> ALLOW;
+supersede esplicito bypassa il guardrail.
+"""
+
+import pytest
+from conftest import auth_headers, make_record
+
+
+@pytest.fixture(autouse=True)
+def enable_guardrail(monkeypatch):
+    """Abilita il guardrail per i test di questa suite (il conftest lo disabilita di default)."""
+    import main as gateway
+
+    monkeypatch.setattr(gateway, "GUARDRAIL_ENABLED", True)
+
+
+def test_duplicato_esatto_bloccato_409(client):
+    """Stesso testo normalizzato -> hash uguale -> BLOCK (409)."""
+    body = make_record(text="Il cliente non accede al portale COSMO-SkyMed")
+    r1 = client.post("/v1/memories", json=body, headers=auth_headers())
+    assert r1.status_code == 200
+
+    # Stesso testo con maiuscole/spazi diversi -> stesso hash normalizzato
+    body2 = make_record(text="  IL CLIENTE NON ACCEDE   al portale COSMO-SkyMed ")
+    r2 = client.post("/v1/memories", json=body2, headers=auth_headers())
+    assert r2.status_code == 409
+    detail = r2.json()["detail"]
+    assert detail["error"] == "duplicate_memory"
+    assert detail["reason"] == "EXACT_DUPLICATE"
+
+
+def test_similarita_alta_bloccata_409(client):
+    """Score top-1 >= soglia BLOCK (0.85) -> 409 KNOWN_SOLUTION."""
+    client.fake_qdrant.query_score = 0.9
+    r1 = client.post("/v1/memories", json=make_record(text="primo record"), headers=auth_headers())
+    assert r1.status_code == 200
+
+    r2 = client.post("/v1/memories", json=make_record(text="secondo record simile"), headers=auth_headers())
+    assert r2.status_code == 409
+    assert r2.json()["detail"]["reason"] == "KNOWN_SOLUTION"
+
+
+def test_similarita_moderata_warn_salva(client):
+    """Score top-1 tra 0.70 e 0.85 -> WARN: salva con flag guardrail."""
+    client.fake_qdrant.query_score = 0.75
+    r1 = client.post("/v1/memories", json=make_record(text="primo record"), headers=auth_headers())
+    assert r1.status_code == 200
+
+    r2 = client.post("/v1/memories", json=make_record(text="secondo record simile"), headers=auth_headers())
+    assert r2.status_code == 200
+    memory_id = r2.json()["memory_id"]
+    # Il record salvato deve avere il flag guardrail WARN
+    saved = client.fake_qdrant.points[memory_id].payload
+    assert saved["guardrail"]["decision"] == "WARN"
+    assert saved["guardrail"]["reason"] == "MODERATE_SIMILARITY"
+
+
+def test_nessun_candidato_allow(client):
+    """Score top-1 sotto soglia WARN -> ALLOW, nessun flag guardrail."""
+    client.fake_qdrant.query_score = 0.5
+    r1 = client.post("/v1/memories", json=make_record(text="primo record"), headers=auth_headers())
+    assert r1.status_code == 200
+
+    r2 = client.post("/v1/memories", json=make_record(text="secondo record"), headers=auth_headers())
+    assert r2.status_code == 200
+    memory_id = r2.json()["memory_id"]
+    saved = client.fake_qdrant.points[memory_id].payload
+    assert saved["guardrail"]["decision"] == "ALLOW"
+    assert saved["guardrail"]["reason"] == "NEW_SOLUTION"
+
+
+def test_supersede_bypassa_guardrail(client):
+    """Il supersede esplicito è una correzione intenzionale: bypassa il guardrail."""
+    client.fake_qdrant.query_score = 0.9
+    r1 = client.post("/v1/memories", json=make_record(text="record originale"), headers=auth_headers())
+    assert r1.status_code == 200
+    old_id = r1.json()["memory_id"]
+
+    # Supersede con testo molto simile -> deve passare (correzione intenzionale)
+    r2 = client.post(
+        "/v1/memories",
+        json=make_record(text="record originale corretto", supersedes_id=old_id, supersede_reason="correzione"),
+        headers=auth_headers(),
+    )
+    assert r2.status_code == 200
+    assert r2.json()["supersedes_id"] == old_id
+
+
+def test_guardrail_disabilitato_salva_sempre(client, monkeypatch):
+    """Con GUARDRAIL_ENABLED=false non si blocca nulla."""
+    import main as gateway
+
+    monkeypatch.setattr(gateway, "GUARDRAIL_ENABLED", False)
+    client.fake_qdrant.query_score = 0.9
+    r1 = client.post("/v1/memories", json=make_record(text="primo"), headers=auth_headers())
+    assert r1.status_code == 200
+    r2 = client.post("/v1/memories", json=make_record(text="primo"), headers=auth_headers())
+    assert r2.status_code == 200
+
+
+def test_text_hash_salvato_nel_payload(client):
+    """Ogni record salvato deve avere text_hash (per lo strato 1 del guardrail)."""
+    r = client.post("/v1/memories", json=make_record(text="record con hash"), headers=auth_headers())
+    assert r.status_code == 200
+    memory_id = r.json()["memory_id"]
+    saved = client.fake_qdrant.points[memory_id].payload
+    assert "text_hash" in saved
+    assert len(saved["text_hash"]) == 64  # SHA-256 hex