tokenmizer

---

The Problem

Every AI session has a context limit. When you hit it:

• The model forgets every decision, rationale, and context built over hours
• You waste 10–30 minutes re-explaining the project every new session
• Large files (CSV, PDF, Excel) eat your entire token budget instantly

How TokenMizer Solves It

TokenMizer is a local proxy between your app and any LLM. Every request goes through a pipeline that builds a live knowledge graph, compresses inputs, caches responses, and auto-checkpoints before context runs out.

Your App  →  TokenMizer (:8000)  →  Claude / GPT / Gemini / any LLM
                    │
          ┌─────────┴──────────────┐
          │   6-Layer Pipeline     │
          │   L0  File Intel       │  CSV/PDF/Excel → schema + sample
          │   L1  Compression      │  15–40% input reduction
          │   L2  Output Trim      │  5–15% output reduction
          │   L3  Semantic Cache   │  100% on repeated queries
          │   L4  Graph Memory     │  session continuity
          │   L5  Prompt Cache     │  90% on repeated system prompts
          └────────────────────────┘

---

Architecture

Decision Memory — 4-State Model

Status	Meaning	In Resume
🟢 ACTIVE	Current — in effect	✅ Always
🟡 SUPERSEDED	Replaced by newer decision	⚠️ 7 days
🔴 INVALIDATED	Explicitly wrong/cancelled	⚠️ Always (warning)
⬜ ARCHIVED	Superseded >7 days ago — aged out	❌ Never

History is never deleted. "Why did we switch from React to Next.js?" — always answerable: ask GET /api/graph/{session}/why?q=react (or the why_decision MCP tool) and get the full old → new trail with trigger, reason, and evidence per hop.

From Storage to Reasoning

The graph doesn't just store facts — it answers questions over them:

Capability	Endpoint / Tool	What it answers
Ontology	GET /api/ontology	The formal vocabulary: node/edge types with semantics, and the status state machine (which lifecycle transitions are legal)
Causal chains	GET /api/graph/{id}/why?q=... · MCP why_decision	"Why is X the current choice?" — walks the supersession chain with trigger/reason/evidence per hop
Reasoning view	GET /api/graph/{id}/reasoning	Active decisions per topic, recent changes, decision timeline, and a consistency audit
Consistency audit	(part of /reasoning)	Contradictions the tracker missed, superseded decisions with lost history, dangling references

All reasoning is deterministic and local — no LLM calls, no extra cost.

---

Quick Start