Chat

OpenEng Chat is a local, private coding and integration agent. The interface at chat.openeng.app is a thin web client; everything that matters — model inference, repository indexing, file edits, build and test runs, and every conversation — happens inside a sealed engine running on your own machine at 127.0.0.1:50056. By default, no prompt, model file, or line of your code leaves the device. The exceptions are ones you set up: a model you register with the external backend sends its prompts to the server you point it at (Provider vs. backend), and an MCP server you add receives the tool calls the agent sends it (MCP servers). This page is the complete reference: architecture, models, groups, cross-app automation, the agent loop, MCP tools, and the security model.

Overview & architecture

Chat is split into two halves joined by a single hardened boundary:

The sealed boundary

The browser talks to the engine over gRPC-Web with a per-frame cryptographic seal. On connect, the UI performs a single handshake — CreateSession, the only unsealed RPC — exchanging X25519 public keys. From that point, when the UI offered a key, every request and response frame is individually encrypted with a fresh ephemeral key: X25519 → HKDF-SHA256 → ChaCha20-Poly1305. The key exchange isn't authenticated, and a request frame that fails to unseal is passed on as it arrived. Because gRPC-Web permits only unary and server-streaming calls, each live stream (an agent turn, an index run, a download) is a stream FooEvent paired with a unary control RPC.

The wire schema has no field for an OpenEng secret or product prompt. Your own credentials and prompts are fields that reach only your engine over the sealed loopback: the system prompt and model settings you give a group or a model, and an MCP server's headers (for example an Authorization token). Model files stay inside the engine under ~/.openeng/chat/ and never cross the wire. Your own code, paths, and commands are legitimate payload and flow freely.

Starting the engine

The engine is launched from a terminal with a short-lived key minted for your signed-in browser session:

openeng-chat serve --oauth=<one-time key>

The connect screen mints this one-time key from your authenticated browser and shows the full command ready to copy. Key facts about the boot flow:

• OAuth is the only boot path. There is no OPENENG_KEY or offline mode — the engine authenticates the pairing against your account.
• The key is short-lived (about five minutes) and cached in the browser. A cached, unexpired key is reused across refreshes with no network call; a fresh key is minted only on a cache miss/expiry or an explicit Regenerate. Minting is rate-limited: the OpenEng API mints at most five unused keys an hour per account, shared by all OpenEng apps (a key an engine has used no longer counts), and each app also counts the keys it mints in your browser.
• Connection is automatic. The connect page (/pair gate) polls with a heartbeat; the instant the engine is reachable it completes the handshake, negotiates the seal, and forwards you to wherever you were headed.

Using the app

This section walks through the app at chat.openeng.app as built. The sections after it are the reference for what the engine does underneath.

The workspace

• Left menu: Chats, with Settings at the bottom. Chats opens general chat.
• Header: a button that hides or shows the side panel, the chat group switcher (the Default group first, then your groups, and a Settings row at the end), a status dot for the engine, a search button for the command palette, and your account menu.
• URLs: general chat is /chat (one conversation: /chat/<chat-id>); a group is /projects/<group-id>/chat (one conversation: /projects/<group-id>/chat/<chat-id>), and its analysis is /projects/<group-id>/analysis; Settings is /settings (add ?category=models or ?category=mcp for a category). / and any other path open /chat. When you open the app, it returns you to where you left off (the engine stores your last page, so this works across browsers and restarts); the first time, it opens your latest chat.
• Command palette: ⌘+K (Ctrl+K on Windows and Linux) opens any group, Settings, or Models.

General chat

Chats opens the Default group (see The Default group): a plain conversation with a local model, with no repository and no coding tools. The side panel lists your conversations, with New chat at the top and a delete button on each. Pick a model in the composer (or leave it on auto), type your message, and press Enter (Shift+Enter for a new line). The attach button (Add image, audio, or file) and the record button add attachments. While a reply is being written, Stop cancels it. If no usable model is downloaded yet, the composer says "no model — add one in Models".

Chat groups

Create a group in Settings → Groups with Add group: a Group name, a Default system prompt (Markdown, with Write and Preview), and App connections to clusters, repositories, or terminal tabs in your other running OpenEng apps. Each group in the list has an edit button and a remove button. Switch groups from the header's group switcher or the command palette. Inside a group, the side panel lists that group's conversations, and every message you send there carries the group's system prompt. See Chat groups and Connecting your other OpenEng apps.

Coding groups and Analysis

A group bound to a folder on your machine is a coding agent (see Folder-bound coding groups for how a group gets a folder). Its composer asks for a coding task, and an auto-apply edits checkbox lets the agent's edits land without asking you first. The transcript shows each step of a turn: the plan, the code it retrieved, each tool call and its result (expand one for details), each edit as a diff (marked applied once it lands), and a pass or fail mark for each build or test run.

A group's Analysis tab summarizes the index the engine builds of the group's folder: counts of Files, Symbols, Edges, and Languages, the files per language, Hotspots (the most-referenced symbols, ranked by PageRank), and Entry points (files that nothing else references). The first visit indexes the folder and shows its progress; Reindex rebuilds the index. Analysis needs a group with a folder.

Models and MCP servers

Settings → Models lists every model in the registry with its size, its kind, and whether it is downloaded or active. Download starts a download (or rejoins one in progress) with a live progress bar, Add model registers a new model, and the trash button removes one; see Models. Settings → MCP servers registers, tests, and removes MCP tool servers; see MCP servers.

When the engine is offline

If the engine stops while you are working, the app shows the pairing screen and brings you back to the same page as soon as the engine is running again. If you refresh during a reply, the app reattaches to the reply in progress, as long as the engine has not restarted. The account menu (your avatar) shows your email, the engine and app versions, a Release notes link, and Sign out.

Models

Every model Chat can run lives in a single, fully file-backed registry at ~/.openeng/chat/state/models.json. What you see in Settings → Models is exactly what is in that file — nothing is hidden or hardcoded. On first run the default Qwen2.5-Coder catalog is seeded into the file as real, editable, removable entries.

Provider vs. backend — the critical distinction

Two independent axes describe a model. Confusing them is the single most common mistake, so the registry keeps them strictly separate:

The consequence: provider = Ollama with backend = llama.cpp pulls a GGUF from the Ollama registry and then runs it locally through llama.cpp. Only backend = external proxies inference to a running server (Ollama, LM Studio, or any OpenAI-compatible endpoint) — and in that case there is nothing to download, the provider merely names which server holds the model. Every non-external backend needs its file on disk, regardless of which provider supplied it.

Adding a model

Settings → Models → Add model registers any model into the dynamic registry. The required fields depend on the provider:

Provider defaults: Ollama and LM Studio default to backend = external; everything else defaults to llama.cpp. The on-disk file is namespaced by the model's assigned id so two repos that share a GGUF filename never collide (built-ins keep their bare filename for backward compatibility).

Configuring: kind, modalities, and params

Beyond the source, each entry carries runtime metadata:

• Kind — chat, editor, vision, image, audio, transcribe, tts, or embed. The kind (with the backend) routes a turn to the right execution path.
• Modalities — text, vision, audio, image, video; defaults to ["text"].
• Params — an opaque per-model JSON blob for runtime settings (context length, GPU layers, quantization hints).

The kind + backend combination selects the on-device path. Notable cases handled entirely locally: image + mistral.rs generates pictures via a diffusion model; vision + mistral.rs runs a multimodal model over attached images; audio + mistral.rs handles recorded audio; transcribe + whisper is speech-to-text via whisper.cpp. The external backend instead routes chat, image, and TTS to an OpenAI-compatible server.

Downloads and the active model

Downloads are engine-owned, refresh-resilient, and auto-start on add. When you register a downloadable model the engine begins pulling it immediately; you don't have to click anything. ListModels reports live downloading, downloadedBytes, and totalBytes for each entry, so the progress bar in the model list is simply the engine's own state re-read. Because the engine keeps downloading regardless of the browser, closing the tab or refreshing never interrupts a pull — reopening rejoins it in progress. The manual Download button is idempotent: it re-kicks and rejoins an in-flight download rather than starting a second one.

The active model is the one used when the composer's picker is set to auto. Resolution order: the pinned model if it is downloaded, else the default (qwen2.5-coder-1.5b), else the first downloaded editor. The composer picker lists every downloaded, runnable (non-embed) model; choosing one pins it for that turn.

Removing a model offers two levels: remove from list only (drops the entry but keeps the file on disk, so re-adding reuses it) and remove & delete file. Built-in seed entries cannot be dropped from the list, but their downloaded file can be deleted to reclaim space — the entry stays so it can be re-downloaded later.

Chat groups

A group is the central organizing concept in Chat. It is not a folder — it is a name plus a default system prompt (and, optionally, connections to other apps). Every chat lives inside a group, and the group's system prompt rides along with every message you send in it.

The Default group

The Default group is always present and needs no setup. It is a plain, projectless workspace: pick a model in the composer and start chatting. It has no repository, no retrieval, and no coding tools — just a conversation with the selected local model.

Creating a group

Settings → Groups → Add group opens a modal (not a folder chooser) with two fields:

• Group name — a single line, e.g. Research, Writing, Backend service.
• Default system prompt — a multi-line Markdown editor with a Write ⇆ Preview toggle. Write edits the raw Markdown; Preview renders it exactly as the model-facing prompt would read.

The prompt is prepended to every message in the group. It leads the composed system message so its instructions frame everything the engine's base protocol adds. Two groups may share a name — the name is a label, not an identity; each gets a distinct id.

The system prompt on every turn — and auto-summary

Because the group prompt is user-supplied and unbounded, it is clipped to a fixed budget (~600 estimated tokens) when composed with the base protocol, so a huge prompt can never crowd out the engine's mandatory tool-call instructions — the base always survives intact as the tail. Separately, a large group system prompt is condensed into a summary at save time, and that condensed form is what rides each turn. The result: an over-long system prompt still shapes every message without ever blowing the context window.

Editing and removing

Editing a group re-opens the same modal and replaces its name, prompt, and connections in full. Removing a group forgets it from the registry; for a folder-bound group this never touches the working tree on disk. Groups persist to ~/.openeng/chat/state/projects.json so they survive restarts and are shared across browser tabs on the same machine.

Folder-bound coding groups

A group whose path is bound to a working tree becomes a full coding agent. Where a folderless group is a plain conversation, a folder-bound group indexes the repository on first use (a tree-sitter symbol graph and a PageRank-ranked repo map), retrieves relevant context for each message, and gains the code tools — read_file, list_dir, grep, edit_file, write_file, run_cmd, and verify. The agent reads the repo, proposes edits as diffs, runs your build/tests, reads the failures, and iterates.

Connecting your other OpenEng apps

A group can attach connections to resources in sibling OpenEng apps, turning Chat into a cross-app operator. Once attached, the agent can act on exactly those resources autonomously — it decides which tools to call, reads the results, and iterates.

How the bridge works

The Chat engine reaches sibling engines over internal sealed gRPC — the same handshake-and-seal used by the browser, but engine-to-engine on loopback. It connects to each app's local engine, authenticating with that app's 0600 handshake token, and speaks its native protocol:

Availability is a fast probe — the group-config picker checks whether each app engine's endpoint and token files exist. If an app isn't running it shows as unavailable and is simply skipped; a down sibling is never fatal.

Attaching connections

In the group modal, the App connections picker lists every detected sibling app. Expanding a running app enumerates its resources (clusters, repos, tabs); tick the ones this group may act on. The selections are stored on the group as { app, connection_id, label } triples.

The agent tools

Attaching connections exposes typed tools to the model during a turn. Only the tools whose app is connected on the current group appear in the tool catalog:

Authorization & scope

The security boundary is strict: a tool call may only touch a connection configured on the current group. Before any cross-app call, the requested connection/repo argument is resolved against the group's connection list:

• If exactly one connection of that app is configured, the argument may be omitted.
• If several are configured, the model must name one by id or label; ambiguity is refused.
• An id match takes precedence over a label match, so one connection's label can never shadow another's id (a confused-deputy guard).
• A connection with an empty id is never usable — it would silently resolve to the app engine's current context, outside the group's scope, so it is filtered out entirely.

Example: autonomous multi-app debugging

Attach a Kubernetes cluster and the relevant Git repo to a group, then ask "why is my payments pod failing?". The agent, on its own, will: list namespaces, list pods, read the failing pod's logs, describe the resource, cross-reference recent commits in the connected repo, and then explain the root cause — each step surfacing in the execution graph as it happens.

Using a chat

Sending a message

Type in the composer and press Enter to send (Shift+Enter for a newline). A folderless or Default group runs a conversational turn steered by the group prompt; a folder-bound group runs the autonomous coding agent. The composer's model picker and, for coding groups, the auto-apply edits toggle sit beneath the input.

Attachments

Drop or attach files and the model reads them. Handling depends on type:

• Documents (scripts, logs, .txt/.md/.csv/.json/.yaml/source, etc.) are decoded to text and folded into the prompt, so even a purely local llama.cpp model actually sees the file's contents. Binary/non-UTF-8 blobs are skipped rather than injected as garbage.
• Images ride to the multimodal (vision) path.
• Audio can be recorded in-browser (captured as 16 kHz mono WAV so the engine's whisper/symphonia path can decode it) or attached, then routed to the transcription/audio path.
• Video is sampled into frames in the browser and attached as images for a vision model — no server-side video decoder required.

The live execution graph

While a turn runs, a "thinking" panel above the composer shows a compact execution graph. The header names the current action with a spinner — "Deciding what to do next", "Kubernetes · reading pod logs", "Reading the attached file", "Writing the answer" — and the trail below lists the steps already completed (a check) or failed (an ✕). It is a real-time window into the agent's reasoning, not a fixed progress bar.

The autonomous agent loop

An agentic turn is a loop: the model plans, emits a single tool call, the engine executes it, appends the result to a running scratchpad, and feeds it back — repeating until the model produces a final answer or calls done. The loop has a generous runaway backstop of 24 iterations; the group's system prompt, not that number, is what shapes how the agent iterates. If it exhausts the limit without finishing, it says so honestly rather than reporting success.

In coding groups, tools are gated by risk. Read and search tools (read_file, list_dir, grep) run immediately. Write and run tools (edit_file, write_file, run_cmd) require approval unless auto-apply edits is on — the proposed change is shown as a unified diff and applied only on your approval. verify runs your build/tests (auto-detected when the command is left empty) and streams the outcome back into the loop so the agent can react to failures.

Smart-Summary rolling memory

A long conversation never runs out of context. After each turn the engine folds the new (user, assistant) exchange into a single rolling summary, re-compressed to a fixed budget so it stays bounded no matter how many turns accumulate. When any text to summarize is itself larger than the window, it is chunked, each chunk summarized (map), the parts joined and recursively reduced until it fits. Every path is fail-soft: if the model errors, a deterministic head/tail clip guarantees a bounded result.

This is backed by a hard "never fails on tokens" guarantee: if an inference call overflows the context window, it is retried once with a minimal, un-overflowable prompt (system + the current task only), silently — a recovered overflow never leaks an error line to you.

Cancellation

The Stop button cancels the in-flight turn. Cancellation truncates generation cleanly — any partial reply is persisted, but a canceled turn is reported as canceled, never as a success, and is not folded into the rolling summary.

MCP servers

Chat is an MCP (Model Context Protocol) host. Register external tool servers and the agent can call their tools alongside the built-in and cross-app tools. Settings → MCP servers manages them; servers persist to ~/.openeng/chat/state/mcp-servers.json.

Test connects, initializes, and lists a server's tools, recording the tool count. During a chat you select which servers are active per turn (the tool pills in the composer of a conversational group); the engine opens sessions to the selected servers, flattens their tools into the catalog, and routes each call to the server that advertised it (first wins on a name collision). Server failures are surfaced in the turn rather than aborting it, and MCP tools compose with a group's app connections in the same tool loop.

Privacy & security

Private by default: prompts leave the machine only through an external backend or an MCP server you add.

Attachments, repository contents, terminal commands, and cluster queries are all your own data acting on your own machine. The trust boundary is the loopback socket; everything of value stays inside it unless you point a model's external backend or an MCP server outside your machine.