Chat

OpenEng Chat is a local, private coding and integration agent. The interface at chat.openeng.app is a thin web client; everything that matters — model inference, repository indexing, file edits, build and test runs, and every conversation — happens inside a sealed engine running on your own machine at 127.0.0.1:50056. No prompt, model file, or line of your code ever leaves the device. This page is the complete reference: architecture, models, groups, cross-app automation, the agent loop, MCP tools, and the security model.

Overview & architecture

Chat is split into two halves joined by a single hardened boundary:

The sealed boundary

The browser talks to the engine over gRPC-Web with a per-frame cryptographic seal. On connect, the UI performs a single handshake — CreateSession, the only unsealed RPC — exchanging X25519 public keys. From that point every request and response frame is individually encrypted: X25519 → HKDF-SHA256 → ChaCha20-Poly1305. Because gRPC-Web permits only unary and server-streaming calls, each live stream (an agent turn, an index run, a download) is a stream FooEvent paired with a unary control RPC.

The protocol carries a deliberate egress invariant: the wire schema has no field through which a secret, a credential, or a product prompt could escape. Model files, system prompts, and model configuration are resolved entirely inside the engine from ~/.openeng/chat/ and are never serialized onto the wire. Your own code, paths, and commands are legitimate payload and flow freely; nothing else can.

Starting the engine

The engine is launched from a terminal with a short-lived key minted for your signed-in browser session:

openeng-chat serve --oauth=<key>

The connect screen mints this one-time key from your authenticated browser and shows the full command ready to copy. Key facts about the boot flow:

• OAuth is the only boot path. There is no OPENENG_KEY or offline mode — the engine authenticates the pairing against your account.
• The key is short-lived (about five minutes) and cached in the browser. A cached, unexpired key is reused across refreshes with no network call; a fresh key is minted only on a cache miss/expiry or an explicit Regenerate. Minting is rate-limited.
• Connection is automatic. The connect page (/pair gate) polls with a heartbeat; the instant the engine is reachable it completes the handshake, negotiates the seal, and forwards you to wherever you were headed.

Models

Every model Chat can run lives in a single, fully file-backed registry at ~/.openeng/chat/state/models.json. What you see in Settings → Models is exactly what is in that file — nothing is hidden or hardcoded. On first run the default Qwen2.5-Coder catalog is seeded into the file as real, editable, removable entries.

Provider vs. backend — the critical distinction

Two independent axes describe a model. Confusing them is the single most common mistake, so the registry keeps them strictly separate:

The consequence: provider = Ollama with backend = llama.cpp pulls a GGUF from the Ollama registry and then runs it locally through llama.cpp. Only backend = external proxies inference to a running server (Ollama, LM Studio, or any OpenAI-compatible endpoint) — and in that case there is nothing to download, the provider merely names which server holds the model. Every non-external backend needs its file on disk, regardless of which provider supplied it.

Adding a model

Settings → Models → Add model registers any model into the dynamic registry. The required fields depend on the provider:

Provider defaults: Ollama and LM Studio default to backend = external; everything else defaults to llama.cpp. The on-disk file is namespaced by the model's assigned id so two repos that share a GGUF filename never collide (built-ins keep their bare filename for backward compatibility).

Configuring: kind, modalities, and params

Beyond the source, each entry carries runtime metadata:

• Kind — chat, editor, vision, image, audio, transcribe, tts, or embed. The kind (with the backend) routes a turn to the right execution path.
• Modalities — text, vision, audio, image, video; defaults to ["text"].
• Params — an opaque per-model JSON blob for runtime settings (context length, GPU layers, quantization hints).

The kind + backend combination selects the on-device path. Notable cases handled entirely locally: image + mistral.rs generates pictures via a diffusion model; vision + mistral.rs runs a multimodal model over attached images; audio + mistral.rs handles recorded audio; transcribe + whisper is speech-to-text via whisper.cpp. The external backend instead routes chat, image, and TTS to an OpenAI-compatible server.

Downloads and the active model

Downloads are engine-owned, refresh-resilient, and auto-start on add. When you register a downloadable model the engine begins pulling it immediately; you don't have to click anything. ListModels reports live downloading, downloadedBytes, and totalBytes for each entry, so the progress bar in the model list is simply the engine's own state re-read. Because the engine keeps downloading regardless of the browser, closing the tab or refreshing never interrupts a pull — reopening rejoins it in progress. The manual Download button is idempotent: it re-kicks and rejoins an in-flight download rather than starting a second one.

The active model is the one used when the composer's picker is set to auto. Resolution order: the pinned model if it is downloaded, else the default (qwen2.5-coder-1.5b), else the first downloaded editor. The composer picker lists every downloaded, runnable (non-embed) model; choosing one pins it for that turn.

Removing a model offers two levels: remove from list only (drops the entry but keeps the file on disk, so re-adding reuses it) and remove & delete file. Built-in seed entries cannot be dropped from the list, but their downloaded file can be deleted to reclaim space — the entry stays so it can be re-downloaded later.

Chat groups

A group is the central organizing concept in Chat. It is not a folder — it is a name plus a default system prompt (and, optionally, connections to other apps). Every chat lives inside a group, and the group's system prompt rides along with every message you send in it.

The Default group

The Default group is always present and needs no setup. It is a plain, projectless workspace: pick a model in the composer and start chatting. It has no repository, no retrieval, and no coding tools — just a conversation with the selected local model.

Creating a group

Settings → Groups → Add group opens a modal (not a folder chooser) with two fields:

• Group name — a single line, e.g. Research, Writing, Backend service.
• Default system prompt — a multi-line Markdown editor with a Write ⇆ Preview toggle. Write edits the raw Markdown; Preview renders it exactly as the model-facing prompt would read.

The prompt is prepended to every message in the group. It leads the composed system message so its instructions frame everything the engine's base protocol adds. Two groups may share a name — the name is a label, not an identity; each gets a distinct id.

The system prompt on every turn — and auto-summary

Because the group prompt is user-supplied and unbounded, it is clipped to a fixed budget (~600 estimated tokens) when composed with the base protocol, so a huge prompt can never crowd out the engine's mandatory tool-call instructions — the base always survives intact as the tail. Separately, a large group system prompt is condensed into a summary at save time, and that condensed form is what rides each turn. The result: an over-long system prompt still shapes every message without ever blowing the context window.

Editing and removing

Editing a group re-opens the same modal and replaces its name, prompt, and connections in full. Removing a group forgets it from the registry; for a folder-bound group this never touches the working tree on disk. Groups persist to ~/.openeng/chat/state/projects.json so they survive restarts and are shared across browser tabs on the same machine.

Folder-bound coding groups

A group whose path is bound to a working tree becomes a full coding agent. Where a folderless group is a plain conversation, a folder-bound group indexes the repository on first use (a tree-sitter symbol graph and a PageRank-ranked repo map), retrieves relevant context for each message, and gains the code tools — read_file, list_dir, grep, edit_file, write_file, run_cmd, and verify. The agent reads the repo, proposes edits as diffs, runs your build/tests, reads the failures, and iterates.

Connecting your other OpenEng apps

A group can attach connections to resources in sibling OpenEng apps, turning Chat into a cross-app operator. Once attached, the agent can act on exactly those resources autonomously — it decides which tools to call, reads the results, and iterates.

How the bridge works

The Chat engine reaches sibling engines over internal sealed gRPC — the same handshake-and-seal used by the browser, but engine-to-engine on loopback. It connects to each app's local engine, authenticating with that app's 0600 handshake token, and speaks its native protocol:

Availability is a fast probe — the group-config picker checks whether each app engine's endpoint and token files exist. If an app isn't running it shows as unavailable and is simply skipped; a down sibling is never fatal.

Attaching connections

In the group modal, the App connections picker lists every detected sibling app. Expanding a running app enumerates its resources (clusters, repos, tabs, boards); tick the ones this group may act on. The selections are stored on the group as { app, connection_id, label } triples.

The agent tools

Attaching connections exposes typed tools to the model during a turn. Only the tools whose app is connected on the current group appear in the tool catalog:

Authorization & scope

The security boundary is strict: a tool call may only touch a connection configured on the current group. Before any cross-app call, the requested connection/repo argument is resolved against the group's connection list:

• If exactly one connection of that app is configured, the argument may be omitted.
• If several are configured, the model must name one by id or label; ambiguity is refused.
• An id match takes precedence over a label match, so one connection's label can never shadow another's id (a confused-deputy guard).
• A connection with an empty id is never usable — it would silently resolve to the app engine's current context, outside the group's scope, so it is filtered out entirely.

Example: autonomous multi-app debugging

Attach a Kubernetes cluster and the relevant Git repo to a group, then ask "why is my payments pod failing?". The agent, on its own, will: list namespaces, list pods, read the failing pod's logs, describe the resource, cross-reference recent commits in the connected repo, and then explain the root cause — each step surfacing in the execution graph as it happens.

Using a chat

Sending a message

Type in the composer and press Enter to send (Shift+Enter for a newline). A folderless or Default group runs a conversational turn steered by the group prompt; a folder-bound group runs the autonomous coding agent. The composer's model picker and, for coding groups, the auto-apply edits toggle sit beneath the input.

Attachments

Drop or attach files and the model reads them. Handling depends on type:

• Documents (scripts, logs, .txt/.md/.csv/.json/.yaml/source, etc.) are decoded to text and folded into the prompt, so even a purely local llama.cpp model actually sees the file's contents. Binary/non-UTF-8 blobs are skipped rather than injected as garbage.
• Images ride to the multimodal (vision) path.
• Audio can be recorded in-browser (captured as 16 kHz mono WAV so the engine's whisper/symphonia path can decode it) or attached, then routed to the transcription/audio path.
• Video is sampled into frames in the browser and attached as images for a vision model — no server-side video decoder required.

The live execution graph

While a turn runs, a "thinking" panel above the composer shows a compact execution graph. The header names the current action with a spinner — "Deciding what to do next", "Kubernetes · reading pod logs", "Reading the attached file", "Writing the answer" — and the trail below lists the steps already completed (a check) or failed (an ✕). It is a real-time window into the agent's reasoning, not a fixed progress bar.

The autonomous agent loop

An agentic turn is a loop: the model plans, emits a single tool call, the engine executes it, appends the result to a running scratchpad, and feeds it back — repeating until the model produces a final answer or calls done. The loop has a generous runaway backstop of 24 iterations; the group's system prompt, not that number, is what shapes how the agent iterates. If it exhausts the limit without finishing, it says so honestly rather than reporting success.

In coding groups, tools are gated by risk. Read and search tools (read_file, list_dir, grep) run immediately. Write and run tools (edit_file, write_file, run_cmd) require approval unless auto-apply edits is on — the proposed change is shown as a unified diff and applied only on your approval. verify runs your build/tests (auto-detected when the command is left empty) and streams the outcome back into the loop so the agent can react to failures.

Smart-Summary rolling memory

A long conversation never runs out of context. After each turn the engine folds the new (user, assistant) exchange into a single rolling summary, re-compressed to a fixed budget so it stays bounded no matter how many turns accumulate. When any text to summarize is itself larger than the window, it is chunked, each chunk summarized (map), the parts joined and recursively reduced until it fits. Every path is fail-soft: if the model errors, a deterministic head/tail clip guarantees a bounded result.

This is backed by a hard "never fails on tokens" guarantee: if an inference call overflows the context window, it is retried once with a minimal, un-overflowable prompt (system + the current task only), silently — a recovered overflow never leaks an error line to you.

Cancellation

The Stop button cancels the in-flight turn. Cancellation truncates generation cleanly — any partial reply is persisted, but a cancelled turn is reported as cancelled, never as a success, and is not folded into the rolling summary.

MCP servers

Chat is an MCP (Model Context Protocol) host. Register external tool servers and the agent can call their tools alongside the built-in and cross-app tools. Settings → MCP servers manages them; servers persist to ~/.openeng/chat/state/mcp-servers.json.

Test connects, initializes, and lists a server's tools, recording the tool count. During a chat you select which servers are active per turn (the tool pills in the composer of a conversational group); the engine opens sessions to the selected servers, flattens their tools into the catalog, and routes each call to the server that advertised it (first wins on a name collision). Server failures are surfaced in the turn rather than aborting it, and MCP tools compose with a group's app connections in the same tool loop.

Privacy & security

Privacy is structural, not a policy promise. The design makes the private path the only path.

Attachments, repository contents, terminal commands, and cluster queries are all your own data acting on your own machine. The trust boundary is the loopback socket; everything of value stays inside it.