Back to work CONTEXT-AWARE REAL-TIME PERCEPTION TOOLING

Sensei

A low-latency desktop teaching agent that watches which UI elements you click in Logic Pro through macOS's Accessibility API and offers single-sentence spoken guidance in real time, grounded by strict prompt engineering rather than a fine-tuned model.

Architecture

perception.txt

click in Logic Pro
        │
        ▼
Swift daemon: CGEvent tap
  AXUIElementCopyElementAtPosition
  + 5-level ancestor walk, child enumeration
        │
        ├── AX labels resolve ─────────────┐
        │                                  ▼
        │                     tutor context (Tier 1 static map
        │                      + Tier 2 dynamic focus, <1K tokens)
        │                                  │
        └── AX labels empty / ambiguous    │
                        │                  │
                        ▼                  │
              vision fallback (on demand)  │
              screenshot crop → LLM        │
                        │                  │
                        └────────┬─────────┘
                                 ▼
                    Claude Haiku, rolling 5-click window
                                 │
                                 ▼
                    single-sentence guidance → macOS `say`

Perception & ingestion

The perception layer is Accessibility-tree-primary, vision-secondary. A Swift click-logger daemon captures every click system-wide via a CGEvent tap, then resolves it with `AXUIElementCopyElementAtPosition` and a hierarchy walk: a bare, unlabeled element like a fader knob resolves through its ancestor chain to a full semantic path (Mixer to Stereo Out to volume fader). This alone is sufficient for the large majority of teaching interactions, at effectively zero latency and no API cost per click. Vision, a screenshot crop sent to an LLM, is invoked only on demand: when the AX tree returns empty labels with no useful ancestor context, or the student asks a visual question directly. Vision never sits on the critical path.

Context engineering

Tutor context is built in two tiers. Tier 1 is a full, pruned Accessibility-tree crawl taken once at session start, compressed to a token budget, and loaded as a persistent map of everything that exists in the app. Tier 2 is a re-crawl of the subtree closest to each click, giving the tutor the current state of the active UI area. The tutor is instructed to reference only elements present in one of the two tiers.

State & persistence (not yet built)

The current implementation is a vertical slice: the click logger is piped directly into a Python process that calls Claude Haiku with a rolling window of the last five click events plus a hardcoded lesson step, and speaks the response with macOS's built-in `say` command. There is no event bus and no persistence layer yet, by design.

Architecture & Trade-offs

What was built, and what was deliberately chosen against.

Accessibility tree primary, vision secondary

A validation pass against Logic Pro found that hierarchy walking resolves the Accessibility tree's sparsity problem for menus, dialogs, the library browser, and the mixer, covering roughly ninety percent of teaching interactions at sub-millisecond latency and no cost. Vision is reserved for the remaining cases where AX labels stay opaque all the way up, such as the bare workspace canvas, keeping the expensive, slower path off the common one.

A two-tier context strategy to eliminate hallucination

An early version of the tutor hallucinated UI elements because it reasoned from training knowledge instead of the actual screen. Giving it a complete static map of the application (Tier 1) plus a live, per-click focus window (Tier 2), and instructing it to never reference an element outside that map, removed the hallucination entirely and made step completion detectable: a dialog present in Tier 2 that later disappears means the step is done.

A vertical slice before infrastructure

Rather than building an event bus and a storage layer first, the riskiest unknowns (does model inference feel responsive in the loop, does the guidance sound like a tutor, does voice delivery feel natural) were tested end to end in one build cycle: click logger wired straight to an LLM call with a hardcoded lesson step and system TTS. Storage and a bus can be inserted later without touching the perception or teaching layers.

Stack

Swift (Accessibility & Core Graphics APIs), Python, Anthropic SDK (Claude Haiku), macOS `say`

Status

Proof of concept. The perception layer and the vertical slice are validated end to end against Logic Pro; event bus, persistence, and lesson-state tracking are deliberately not yet built.

This repository is private. Email bharath@bharathk.dev for access.

Back to work