# Chat

> OpenEng Chat is a local, private coding and integration agent: a thin web UI over a sealed on-device engine, so models, prompts and data stay on your machine by default. The web app is live; the engine isn't published yet.

OpenEng Chat is a local, private coding and integration agent. The interface at `chat.openeng.app` is a thin web client; everything that matters — model inference, repository indexing, file edits, build and test runs, and every conversation — happens inside a sealed engine running on your own machine at `127.0.0.1:50056`. By default, no prompt, model file, or line of your code leaves the device. The exceptions are ones you set up: a model you register with the `external` backend sends its prompts to the server you point it at ([Provider vs. backend](#provider-vs-backend)), and an MCP server you add receives the tool calls the agent sends it ([MCP servers](#mcp-servers)). This page is the complete reference: architecture, models, groups, cross-app automation, the agent loop, MCP tools, and the security model.

> [!WARNING] The Chat engine is not published yet
> The web app at [chat.openeng.app](https://chat.openeng.app) is live, but no `openeng-chat` engine binary has been published for any platform yet. The installer at `https://openeng.app/engine/chat/install.sh` exists, but it stops with a message that no Chat engine binary is published for your platform, so the app cannot be paired until the engine ships. Chat is free, like every engineering app. This page describes the app and engine as built, so it is ready when the engine is released.

## Overview & architecture

Chat is split into two halves joined by a single hardened boundary:

- **The thin UI** — A React app served from `chat.openeng.app` (installable as a PWA). It renders conversations, the model and group pickers, the live execution graph, and diff/tool cards. It holds no model, runs no inference, and stores no code — it is a view onto the engine.
- **The sealed engine** — A native binary, `openeng-chat`, bound to loopback `127.0.0.1:50056`. It owns the model registry, the repo index, the agent loop, downloads, MCP hosting, and all sibling-app bridging. All heavy work is on-device.

### The sealed boundary

The browser talks to the engine over gRPC-Web with a per-frame cryptographic seal. On connect, the UI performs a single handshake — `CreateSession`, the *only* unsealed RPC — exchanging X25519 public keys. From that point, when the UI offered a key, every request and response frame is individually encrypted with a fresh ephemeral key: **X25519 → HKDF-SHA256 → ChaCha20-Poly1305**. The key exchange isn't authenticated, and a request frame that fails to unseal is passed on as it arrived. Because gRPC-Web permits only unary and server-streaming calls, each live stream (an agent turn, an index run, a download) is a `stream FooEvent` paired with a unary control RPC.

The wire schema has no field for an OpenEng secret or product prompt. Your own credentials and prompts are fields that reach only your engine over the sealed loopback: the system prompt and model settings you give a group or a model, and an MCP server's headers (for example an `Authorization` token). Model files stay inside the engine under `~/.openeng/chat/` and never cross the wire. Your own code, paths, and commands are legitimate payload and flow freely.

> [!NOTE]
> The handshake also returns a `boot_id` (the engine's boot epoch). The UI uses it to decide whether an in-flight turn from before a page refresh is still valid to re-attach to — a turn is only resumed if it belongs to the same engine boot.

### Starting the engine

The engine is launched from a terminal with a short-lived key minted for your signed-in browser session:

```bash
openeng-chat serve --oauth=<one-time key>
```

The connect screen mints this one-time key from your authenticated browser and shows the full command ready to copy. Key facts about the boot flow:

- **OAuth is the only boot path.** There is no `OPENENG_KEY` or offline mode — the engine authenticates the pairing against your account.
- **The key is short-lived** (about five minutes) and cached in the browser. A cached, unexpired key is reused across refreshes with no network call; a fresh key is minted only on a cache miss/expiry or an explicit *Regenerate*. Minting is rate-limited: the OpenEng API mints at most five unused keys an hour per account, shared by all OpenEng apps (a key an engine has used no longer counts), and each app also counts the keys it mints in your browser.
- **Connection is automatic.** The connect page (`/pair` gate) polls with a heartbeat; the instant the engine is reachable it completes the handshake, negotiates the seal, and forwards you to wherever you were headed.

> [!TIP]
> The engine is useful before you download anything. With no local model installed, a coding turn degrades to a deterministic, retrieval-only answer (top-ranked relevant symbols plus the repo map), so you can verify the whole pipeline without a multi-gigabyte download.

## Using the app

This section walks through the app at `chat.openeng.app` as built. The sections after it are the reference for what the engine does underneath.

### The workspace

- **Left menu:** **Chats**, with **Settings** at the bottom. **Chats** opens general chat.
- **Header:** a button that hides or shows the side panel, the **chat group switcher** (the Default group first, then your groups, and a **Settings** row at the end), a status dot for the engine, a search button for the command palette, and your account menu.
- **URLs:** general chat is `/chat` (one conversation: `/chat/<chat-id>`); a group is `/projects/<group-id>/chat` (one conversation: `/projects/<group-id>/chat/<chat-id>`), and its analysis is `/projects/<group-id>/analysis`; Settings is `/settings` (add `?category=models` or `?category=mcp` for a category). `/` and any other path open `/chat`. When you open the app, it returns you to where you left off (the engine stores your last page, so this works across browsers and restarts); the first time, it opens your latest chat.
- **Command palette:** <kbd>⌘</kbd>+<kbd>K</kbd> (<kbd>Ctrl</kbd>+<kbd>K</kbd> on Windows and Linux) opens any group, **Settings**, or **Models**.

### General chat

**Chats** opens the Default group (see [The Default group](#the-default-group)): a plain conversation with a local model, with no repository and no coding tools. The side panel lists your conversations, with **New chat** at the top and a delete button on each. Pick a model in the composer (or leave it on `auto`), type your message, and press <kbd>Enter</kbd> (<kbd>Shift</kbd>+<kbd>Enter</kbd> for a new line). The attach button (**Add image, audio, or file**) and the record button add [attachments](#attachments). While a reply is being written, **Stop** cancels it. If no usable model is downloaded yet, the composer says "no model — add one in Models".

### Chat groups

Create a group in **Settings → Groups** with **Add group**: a **Group name**, a **Default system prompt** (Markdown, with **Write** and **Preview**), and **App connections** to clusters, repositories, or terminal tabs in your other running OpenEng apps. Each group in the list has an edit button and a remove button. Switch groups from the header's group switcher or the command palette. Inside a group, the side panel lists that group's conversations, and every message you send there carries the group's system prompt. See [Chat groups](#groups) and [Connecting your other OpenEng apps](#app-connectivity).

### Coding groups and Analysis

A group bound to a folder on your machine is a coding agent (see [Folder-bound coding groups](#folder-bound-coding-groups) for how a group gets a folder). Its composer asks for a coding task, and an **auto-apply edits** checkbox lets the agent's edits land without asking you first. The transcript shows each step of a turn: the plan, the code it retrieved, each tool call and its result (expand one for details), each edit as a diff (marked `applied` once it lands), and a pass or fail mark for each build or test run.

A group's **Analysis** tab summarizes the index the engine builds of the group's folder: counts of **Files**, **Symbols**, **Edges**, and **Languages**, the files per language, **Hotspots** (the most-referenced symbols, ranked by PageRank), and **Entry points** (files that nothing else references). The first visit indexes the folder and shows its progress; **Reindex** rebuilds the index. Analysis needs a group with a folder.

### Models and MCP servers

**Settings → Models** lists every model in the registry with its size, its kind, and whether it is downloaded or active. **Download** starts a download (or rejoins one in progress) with a live progress bar, **Add model** registers a new model, and the trash button removes one; see [Models](#models). **Settings → MCP servers** registers, tests, and removes MCP tool servers; see [MCP servers](#mcp-servers).

### When the engine is offline

If the engine stops while you are working, the app shows the pairing screen and brings you back to the same page as soon as the engine is running again. If you refresh during a reply, the app reattaches to the reply in progress, as long as the engine has not restarted. The account menu (your avatar) shows your email, the engine and app versions, a **Release notes** link, and **Sign out**.

## Models

Every model Chat can run lives in a single, fully file-backed registry at `~/.openeng/chat/state/models.json`. What you see in Settings → Models is exactly what is in that file — nothing is hidden or hardcoded. On first run the default Qwen2.5-Coder catalog is *seeded into* the file as real, editable, removable entries.

| Seeded model | id | Approx. size |
| --- | --- | --- |
| Qwen2.5-Coder 0.5B (Q4\_K\_M) | `qwen2.5-coder-0.5b` | \~468 MB |
| Qwen2.5-Coder 1.5B (Q4\_K\_M) | `qwen2.5-coder-1.5b` *(default)* | \~1.0 GB |
| Qwen2.5-Coder 3B (Q4\_K\_M) | `qwen2.5-coder-3b` | \~2.0 GB |

### Provider vs. backend — the critical distinction

Two independent axes describe a model. Confusing them is the single most common mistake, so the registry keeps them strictly separate:

- **Provider = where it is downloaded from** — The download *source* only: `huggingface` (a repo + file), `url` (a direct link), `ollama` (the Ollama registry), `lmstudio`, or `builtin`. The provider says nothing about how the model runs.
- **Backend = how it runs** — The *runtime*: `llama.cpp`, `mistral.rs`, `whisper`, `onnx`, or `external`. The backend alone decides whether inference is on-device or proxied to a server.

The consequence: **provider = Ollama with backend = llama.cpp pulls a GGUF from the Ollama registry and then runs it locally** through llama.cpp. Only `backend = external` proxies inference to a running server (Ollama, LM Studio, or any OpenAI-compatible endpoint) — and in that case there is nothing to download, the provider merely names which server holds the model. Every non-external backend needs its file on disk, regardless of which provider supplied it.

> [!WARNING] Warning
> "Ready to use" is decided by the backend, not the provider. An `external` model is always ready (no local file). A `llama.cpp` or `mistral.rs` model is only ready once its file exists on disk — even if you pulled it from Ollama.

### Adding a model

Settings → Models → *Add model* registers any model into the dynamic registry. The required fields depend on the provider:

| Provider | What you supply | Runs |
| --- | --- | --- |
| HuggingFace | Repo id + a file within it (the `.gguf`) | Locally (default `llama.cpp`) |
| Direct URL | A direct download URL to the weights | Locally |
| Ollama | The model name/tag to pull, e.g. `qwen2.5-coder:3b` | Locally (pulled GGUF) *or* external |
| LM Studio | The model name/tag as it appears in the server | External (default) |

Provider defaults: Ollama and LM Studio default to `backend = external`; everything else defaults to `llama.cpp`. The on-disk file is namespaced by the model's assigned id so two repos that share a GGUF filename never collide (built-ins keep their bare filename for backward compatibility).

### Configuring: kind, modalities, and params

Beyond the source, each entry carries runtime metadata:

- **Kind** — `chat`, `editor`, `vision`, `image`, `audio`, `transcribe`, `tts`, or `embed`. The kind (with the backend) routes a turn to the right execution path.
- **Modalities** — `text`, `vision`, `audio`, `image`, `video`; defaults to `["text"]`.
- **Params** — an opaque per-model JSON blob for runtime settings (context length, GPU layers, quantization hints).

The kind + backend combination selects the on-device path. Notable cases handled entirely locally: `image` + `mistral.rs` generates pictures via a diffusion model; `vision` + `mistral.rs` runs a multimodal model over attached images; `audio` + `mistral.rs` handles recorded audio; `transcribe` + `whisper` is speech-to-text via whisper.cpp. The `external` backend instead routes chat, image, and TTS to an OpenAI-compatible server.

> [!NOTE]
> The engine ships in feature-gated variants because ggml-based backends cannot all co-link in one binary. Which local backends a given build exposes depends on how it was compiled; an unavailable backend degrades gracefully rather than crashing a turn.

### Downloads and the active model

Downloads are **engine-owned, refresh-resilient, and auto-start on add**. When you register a downloadable model the engine begins pulling it immediately; you don't have to click anything. `ListModels` reports live `downloading`, `downloadedBytes`, and `totalBytes` for each entry, so the progress bar in the model list is simply the engine's own state re-read. Because the engine keeps downloading regardless of the browser, closing the tab or refreshing never interrupts a pull — reopening rejoins it in progress. The manual *Download* button is idempotent: it re-kicks and rejoins an in-flight download rather than starting a second one.

The **active model** is the one used when the composer's picker is set to *auto*. Resolution order: the pinned model if it is downloaded, else the default (`qwen2.5-coder-1.5b`), else the first downloaded editor. The composer picker lists every downloaded, runnable (non-embed) model; choosing one pins it for that turn.

Removing a model offers two levels: *remove from list only* (drops the entry but keeps the file on disk, so re-adding reuses it) and *remove & delete file*. Built-in seed entries cannot be dropped from the list, but their downloaded file can be deleted to reclaim space — the entry stays so it can be re-downloaded later.

## Chat groups

A **group** is the central organizing concept in Chat. It is *not* a folder — it is a name plus a default system prompt (and, optionally, connections to other apps). Every chat lives inside a group, and the group's system prompt rides along with every message you send in it.

### The Default group

The Default group is always present and needs no setup. It is a plain, projectless workspace: pick a model in the composer and start chatting. It has no repository, no retrieval, and no coding tools — just a conversation with the selected local model.

### Creating a group

Settings → Groups → *Add group* opens a modal (not a folder chooser) with two fields:

- **Group name** — a single line, e.g. *Research*, *Writing*, *Backend service*.
- **Default system prompt** — a multi-line Markdown editor with a *Write ⇆ Preview* toggle. Write edits the raw Markdown; Preview renders it exactly as the model-facing prompt would read.

The prompt is **prepended to every message in the group**. It leads the composed system message so its instructions frame everything the engine's base protocol adds. Two groups may share a name — the name is a label, not an identity; each gets a distinct id.

### The system prompt on every turn — and auto-summary

Because the group prompt is user-supplied and unbounded, it is clipped to a fixed budget (\~600 estimated tokens) when composed with the base protocol, so a huge prompt can never crowd out the engine's mandatory tool-call instructions — the base always survives intact as the tail. Separately, a **large group system prompt is condensed into a summary at save time**, and that condensed form is what rides each turn. The result: an over-long system prompt still shapes every message without ever blowing the context window.

### Editing and removing

Editing a group re-opens the same modal and replaces its name, prompt, and connections in full. Removing a group forgets it from the registry; for a folder-bound group this never touches the working tree on disk. Groups persist to `~/.openeng/chat/state/projects.json` so they survive restarts and are shared across browser tabs on the same machine.

### Folder-bound coding groups

A group whose path is bound to a working tree becomes a full **coding agent**. Where a folderless group is a plain conversation, a folder-bound group indexes the repository on first use (a tree-sitter symbol graph and a PageRank-ranked repo map), retrieves relevant context for each message, and gains the code tools — `read_file`, `list_dir`, `grep`, `edit_file`, `write_file`, `run_cmd`, and `verify`. The agent reads the repo, proposes edits as diffs, runs your build/tests, reads the failures, and iterates.

> [!WARNING] Warning
> The coding agent is an engine capability keyed on a group having a bound working-tree path. The current *Add group* modal creates folderless (name + prompt) groups; binding a folder to activate the code agent is an engine-level operation and is not yet a first-class button in the group modal. A group's id reflects this: folder-bound groups are identified by a hash of their canonical path, folderless groups by a generated `g-…` id.

## Connecting your other OpenEng apps

A group can attach **connections** to resources in sibling OpenEng apps, turning Chat into a cross-app operator. Once attached, the agent can act on exactly those resources *autonomously* — it decides which tools to call, reads the results, and iterates.

### How the bridge works

The Chat engine reaches sibling engines over **internal sealed gRPC** — the same handshake-and-seal used by the browser, but engine-to-engine on loopback. It connects to each app's local engine, authenticating with that app's `0600` handshake token, and speaks its native protocol:

| App | Loopback port | What a connection points at |
| --- | --- | --- |
| Kubernetes | `50054` | A cluster / context |
| Git | `50053` | A repository |
| Terminal | `50052` | A tab (its shell) |

Availability is a fast probe — the group-config picker checks whether each app engine's endpoint and token files exist. If an app isn't running it shows as unavailable and is simply skipped; a down sibling is never fatal.

### Attaching connections

In the group modal, the *App connections* picker lists every detected sibling app. Expanding a running app enumerates its resources (clusters, repos, tabs); tick the ones this group may act on. The selections are stored on the group as `{ app, connection_id, label }` triples.

### The agent tools

Attaching connections exposes typed tools to the model during a turn. Only the tools whose app is connected on the current group appear in the tool catalog:

- **Kubernetes** — `k8s_namespaces`, `k8s_pods`, `k8s_pod_logs` (tail/previous/container), `k8s_describe`, `k8s_resources`.
- **Git** — `git_status`, `git_log` (limit/path), `git_search` (query/mode).
- **Terminal** — `terminal_run` — **executes a command in the attached tab's shell** — and `terminal_history`.

### Authorization & scope

The security boundary is strict: **a tool call may only touch a connection configured on the current group.** Before any cross-app call, the requested `connection`/`repo` argument is resolved against the group's connection list:

- If exactly one connection of that app is configured, the argument may be omitted.
- If several are configured, the model must name one by id or label; ambiguity is refused.
- An id match takes precedence over a label match, so one connection's label can never shadow another's id (a confused-deputy guard).
- A connection with an empty id is never usable — it would silently resolve to the app engine's *current* context, outside the group's scope, so it is filtered out entirely.

### Example: autonomous multi-app debugging

Attach a Kubernetes cluster and the relevant Git repo to a group, then ask *"why is my payments pod failing?"*. The agent, on its own, will: list namespaces, list pods, read the failing pod's logs, describe the resource, cross-reference recent commits in the connected repo, and then explain the root cause — each step surfacing in the execution graph as it happens.

## Using a chat

### Sending a message

Type in the composer and press <kbd>Enter</kbd> to send (<kbd>Shift</kbd>+<kbd>Enter</kbd> for a newline). A folderless or Default group runs a conversational turn steered by the group prompt; a folder-bound group runs the autonomous coding agent. The composer's model picker and, for coding groups, the *auto-apply edits* toggle sit beneath the input.

### Attachments

Drop or attach files and the model reads them. Handling depends on type:

- **Documents** (scripts, logs, `.txt`/`.md`/`.csv`/`.json`/`.yaml`/source, etc.) are decoded to text and folded into the prompt, so even a purely local llama.cpp model actually sees the file's contents. Binary/non-UTF-8 blobs are skipped rather than injected as garbage.
- **Images** ride to the multimodal (vision) path.
- **Audio** can be recorded in-browser (captured as 16 kHz mono WAV so the engine's whisper/symphonia path can decode it) or attached, then routed to the transcription/audio path.
- **Video** is sampled into frames in the browser and attached as images for a vision model — no server-side video decoder required.

### The live execution graph

While a turn runs, a "thinking" panel above the composer shows a compact execution graph. The header names the current action with a spinner — *"Deciding what to do next"*, *"Kubernetes · reading pod logs"*, *"Reading the attached file"*, *"Writing the answer"* — and the trail below lists the steps already completed (a check) or failed (an ✕). It is a real-time window into the agent's reasoning, not a fixed progress bar.

### The autonomous agent loop

An agentic turn is a loop: the model plans, emits a single tool call, the engine executes it, appends the result to a running scratchpad, and feeds it back — repeating until the model produces a final answer or calls `done`. The loop has a generous runaway backstop of **24 iterations**; the group's system prompt, not that number, is what shapes how the agent iterates. If it exhausts the limit without finishing, it says so honestly rather than reporting success.

In coding groups, tools are gated by risk. Read and search tools (`read_file`, `list_dir`, `grep`) run immediately. Write and run tools (`edit_file`, `write_file`, `run_cmd`) require approval unless *auto-apply edits* is on — the proposed change is shown as a unified diff and applied only on your approval. `verify` runs your build/tests (auto-detected when the command is left empty) and streams the outcome back into the loop so the agent can react to failures.

### Smart-Summary rolling memory

A long conversation never runs out of context. After each turn the engine folds the new (user, assistant) exchange into a single rolling summary, re-compressed to a fixed budget so it stays bounded no matter how many turns accumulate. When any text to summarize is itself larger than the window, it is chunked, each chunk summarized (map), the parts joined and recursively reduced until it fits. Every path is fail-soft: if the model errors, a deterministic head/tail clip guarantees a bounded result.

This is backed by a hard **"never fails on tokens"** guarantee: if an inference call overflows the context window, it is retried once with a minimal, un-overflowable prompt (system + the current task only), silently — a recovered overflow never leaks an error line to you.

> [!TIP]
> History is lossless. Tool cards, diffs, and verify results are persisted alongside each message and restored when you reopen a chat. And if you refresh mid-turn, the UI re-attaches to the still-running turn (same engine boot) and resumes its live stream.

### Cancellation

The *Stop* button cancels the in-flight turn. Cancellation truncates generation cleanly — any partial reply is persisted, but a canceled turn is reported as canceled, never as a success, and is not folded into the rolling summary.

## MCP servers

Chat is an MCP (Model Context Protocol) host. Register external tool servers and the agent can call their tools alongside the built-in and cross-app tools. Settings → MCP servers manages them; servers persist to `~/.openeng/chat/state/mcp-servers.json`.

- **stdio transport** — The engine spawns a subprocess. Provide the **command**, its **args**, and any **env**. Example: `npx -y @modelcontextprotocol/server-filesystem /path`.
- **http transport** — The engine posts JSON-RPC to a **URL** with optional **headers** (e.g. an `Authorization` token). Example: `http://127.0.0.1:8000/mcp`.

*Test* connects, initializes, and lists a server's tools, recording the tool count. During a chat you select which servers are active per turn (the tool pills in the composer of a conversational group); the engine opens sessions to the selected servers, flattens their tools into the catalog, and routes each call to the server that advertised it (first wins on a name collision). Server failures are surfaced in the turn rather than aborting it, and MCP tools compose with a group's app connections in the same tool loop.

## Privacy & security

Private by default: prompts leave the machine only through an `external` backend or an MCP server you add.

- **Sealed loopback** — The engine binds only `127.0.0.1`, authenticates every call with its local handshake token, and, when the app offered a key at the handshake, encrypts every later frame with ChaCha20-Poly1305. Cross-app engine-to-engine calls use the same seal plus each app's `0600` local token. The exact-origin lockdown the Linux Terminal, Kubernetes and Data engines have is still pending here: in its current source the Chat engine answers any web origin and hands its token to any caller.
- **Local model files & prompts** — Weights stay inside the engine under `~/.openeng/chat/`, and inference is on-device (or, for `external` backends, on a server *you* point at). The system prompts and model settings you enter travel from the app to your engine over the sealed loopback, and go further only to an `external` backend you set up, which receives the group's system prompt with the conversation.
- **What the wire carries** — The protocol has no field for an OpenEng secret or product prompt. Your code, paths and commands, the prompts and model settings you set, and your MCP servers' headers are your own payload, and they reach only your engine. No telemetry.

Attachments, repository contents, terminal commands, and cluster queries are all your own data acting on your own machine. The trust boundary is the loopback socket; everything of value stays inside it unless you point a model's `external` backend or an MCP server outside your machine.