Overview
Modern reasoning-enhanced LLM agents hallucinate tool names that don't exist in their registry — a failure mode documented across GPT, Claude, Llama, and Qwen models. The 2026 literature measures Llama-3.1-8B at 62–99% phantom-tool rate depending on task complexity. No commercial product detected this before Sentinel.
Sentinel wraps any autonomous agent with a three-layer detection cascade. Layer 1 is an in-memory registry hash that passes real tools through in under 1 millisecond at zero cost. Layer 2 uses Gemini text-embedding-001 with three fusion heuristics — Levenshtein name distance (F1), argument key Jaccard overlap (F2), and top-1 vs top-2 similarity gap (F3) — to catch near-miss phantoms. Confident matches above 0.85 trigger AUTO_CORRECT, injecting the real tool name back into the agent's next turn. Ambiguous cases (0.60–0.85) escalate to Layer 3: a Gemini 2.5-flash semantic verifier producing structured JSON judgments.
The system is deployed and measurable. Live at https://sentinel.66-245-207-218.nip.io on Vultr Milan behind Caddy and Let's Encrypt TLS. Registry: 20 Claude Code and MCP tools. Latency benchmark: p50 = 0.179ms on 2,000 calls — 56 times under the 10ms budget. Phantom detection F1 = 1.000 on a 55-example corpus. A real-time dashboard streams every intercept decision via SSE.
Three distribution surfaces ship in v0.1.0: a Claude Code PreToolUse hook (one line in settings.json), a public REST endpoint at /detect for any agent platform, and the live dashboard. Built on four published 2026 papers (arXiv:2510.22977, 2601.05214, 2602.08082, 2605.09252). Apache 2.0. 141 tests passing.
Note: the demo URL is substituted with the GitHub repository link.
Tech stack
No comments yet
Be the first to comment on this project.
sentinel
Full-Stack Self-Healing DevOps & Security Copilot The Concept: An AI operations engineer that doesn't just suggest code fixes—it monitors cloud infra, detects uptime/security drops, automatically provisions fixes, and generates audit reports for enterprise compliance.
Agency Compiler
Agentic creative project compiler. Most creatives, agencies and designers dont know how to use ai, none the less agents. We created a platform that allows a creative agency or studio owner to instantly spin up agents based on the project ask, resources and scale.
ProofRun
ProofRun helps developers answer a critical question before shipping: will this update break behavior users rely on? An AI agent inspects two Git revisions, designs targeted experiments, and proposes repairs. An independent Docker-based verifier runs those experiments against both versions, preserves the original tests, and checks whether the repair actually works. Developers receive an update, skip, or postpone recommendation with recorded evidence and a reviewable patch. In a live synthetic…