Data

OpenEng Data lets you query any database — SQL, document, key-value, graph, object store, or streaming log — from one browser workspace at data.openeng.app, while every byte of database access runs locally on your own machine. A small signed engine binary (openeng-data) listens only on 127.0.0.1:50057; the web app is a thin client that renders panels and issues operations over an encrypted loopback boundary. Your connection strings and cloud keys are entered once, stored 0600 under ~/.openeng/data/, and never leave the device except to open the database you pointed them at.

Overview

Data is the sixth app in the OpenEng suite and reuses the same architecture as the Terminal and Kubernetes engines: a thin cloud-hosted UI driving a local, sealed gRPC engine. The split is deliberate — the browser is where you read rows, documents, graphs and files; the engine is where credentials live and where every driver connection is opened.

The engine categorises every source into one of six UI categories, which decides the panels the browser renders and the query dialect the playground expects:

The sealed local engine

Everything the browser does flows through a single encrypted boundary. The engine serves gRPC-Web on 127.0.0.1:50057 behind a persisted, publicly-trusted TLS certificate for engine.openeng.app (resolved to loopback), so the browser connects with no certificate-trust step. On top of TLS, every RPC frame except the opening handshake is individually sealed: an X25519 key agreement is run per session, a symmetric key is derived with HKDF-SHA256, and each frame is encrypted with ChaCha20-Poly1305.

You start the engine once, pairing it with a one-time key copied from the Connect screen:

./build.sh                          # cargo build --release → ~/.local/bin/openeng-data
openeng-data serve --oauth=<KEY>    # the one-time key from the Connect screen
openeng-data --version

The engine is zero-config: the loopback port is fixed at 50057, there are no environment-variable knobs, and self-update is always on. At boot it enforces a mandatory, non-skippable update (it fetches, verifies and re-execs into the newest stable build before serving). While running, a newer stable is staged in the background; the session reports it (update_available / staged_version) and the UI raises a restart banner that calls RestartEngine to re-exec in place — the live session is carried internally so you never re-pair.

Two kinds of connection

A connection is a data source you can list, switch between, and query — the same way the Kubernetes app lists clusters. There are two kinds, both persisted 0600 under ~/.openeng/data/ and both selected from the header switcher:

Adding a connection

The add-connection form (in the full-frame Settings pane) has a segmented toggle between the two modes. In Direct mode you pick an engine and fill structured fields — host, port, database, username, password, plus per-engine toggles — and the UI assembles the DSN for you; the raw connection string is never shown for network engines. In Cloud connector mode you paste the connector URL, the API-key value (sent as the X-API-Key header), and optionally an Authorization: Bearer token and a plaintext write key.

The direct engines the UI offers, and the DSN it builds for each:

Because the browser cannot read local paths, the SQLite and Filesystem pickers call the engine's BrowsePath RPC to walk the local machine and let you pick a file or directory. A direct engine that the engine does not compile a driver for is rejected at add time with a message pointing you to a cloud connector instead.

Switching the active connection (SetCurrent), testing one (TestConnection pings the source and returns a server-version string), and removing one are all one-click. Direct handles are opened lazily on first use and cached; a re-add invalidates the cached handle.

Native drivers

Direct connections are served by typed async Rust drivers compiled into the engine — no shell-outs, no external CLIs. Each driver implements three operations: a cheap ping (server version / status), list_children (one lazy level of the schema tree), and run_query (a query/command producing a unified result).

Three more direct drivers ship in the engine for completeness:

• DynamoDB (document category) — browse tables and run PartiQL (SELECT * FROM "table", and INSERT/UPDATE/DELETE for writes).
• Apache Kafka / Redpanda / MSK (streaming category) — the tree is topics → partitions; a "query" is a topic (orders), a partition (orders/2), or an explicit start (orders/2/earliest) that the engine resolves to the latest N messages.
• MongoDB (document category) — a hand-rolled OP_MSG/SCRAM wire client. It is gated behind the optional mongo Cargo feature and is off by default, because the upstream mongodb crate does not build on the pinned toolchain.

The engine normalises many aliases to a driver key: postgresql/pg/cockroach/crdb/redshift → postgres; mariadb → mysql; valkey → redis; mongodb+srv → mongodb; redpanda/msk/warpstream → kafka; fs/file/local → filesystem.

The adaptive schema tree

The left panel is an adaptive object hierarchy that reshapes itself per engine. It is loaded lazily, one level at a time, through the ListChildren RPC: the browser sends the path trail from the root (empty = the connection root) and a per-driver interpreter returns that node's children. Each returned node carries a kind (database, schema, table, view, collection, key, label, reltype, bucket, object, dir, file, …), an icon hint, a detail string (row count, key type, file size), a has_children flag, and a runnable flag.

A filter box narrows the current level (passed as the filter argument, so filtering happens in the engine/connector, not just in the browser). For large listings — a Redis keyspace, an S3 bucket — the tree uses the streaming twin ListChildrenStream, which appends nodes as the source enumerates them so the panel paints progressively; a superseding filter aborts the in-flight stream.

Starter queries and query targeting

A runnable leaf (a table, collection, key, label, reltype, or file) can be double-clicked to drop a starter query into the playground — SELECT * FROM "schema"."table" LIMIT 100 for a SQL table, db.users.find({}) for a Mongo collection, GET key for a Redis key, MATCH (n:Label) RETURN n LIMIT 50 for a Neo4j label. Identifiers are quoted per engine (backticks for MySQL, double quotes elsewhere) so hyphens, mixed case and reserved words work.

For SQL, graph and document sources the tree's root segment is a database, so expanding a database node also makes it the active query target — a typed query then runs against the database you are browsing rather than the connection's default (which for Neo4j is often an empty neo4j graph).

The query playground and results

The right panel is a database-specific playground: a query/command editor with a Run action (⌘/Ctrl+Enter), an engine badge, an optional target-database field (for SQL, document and graph sources), a read-only indicator, and the results area. The editor's contents are shared through the store, so the tree, the ⌘K command palette, and the History view all write into the same editor. Every run is recorded to a local run history.

Queries run through RunQueryStream: the engine sends a QueryMeta header (columns, result kind, write-flag), then batches of rows / documents / graph data as they arrive, then a terminal summary — so results paint progressively. Batches amortise the per-frame seal (never one row per chunk). A default row cap of 1000 applies when the client sets none (hard-capped at 50,000) so a large result can neither blow the sealed frame nor the browser; a truncated result is flagged.

The unified result is rendered in the right shape by its result_kind:

The status bar reports execution time, rows affected (for writes), and truncation. Each statement is classified read-vs-write so the result is stamped is_write and, on a read-only connection, a write is refused before it runs.

Cloud connectors

A cloud connector lets you point the Data app at a database or object store that lives in a cloud you cannot reach directly. Instead of opening the database, the engine calls a small HTTPS service — a connector — that you deploy into your own cloud (an AWS Lambda, a GCP Cloud Function, or an Azure Function). The connector reaches the target inside your VPC and holds its credentials there; the engine only ever stores the connector's URL and your API key, and only ever sees the connector's trimmed JSON responses.

What a data connector does

A data connector speaks the OpenEng tools protocol over three HTTPS endpoints. The engine authenticates every request with an x-api-key header and an Authorization: Bearer header (the API key is reused as the bearer when you do not set a separate token):

GET  {base}/manifest        → { "engine": "postgres",
                                "readOnly": true,
                                "readOnlyCapable": true }

POST {base}/call            { "name": "<tool>", "arguments": { … } }
     x-api-key: <key>       → { "result": <value> }   |   { "error": "…" }

POST {base}/call-stream     NDJSON frames: {"t":"meta|chunk|end|error", …}

The /manifest tells the engine which database the connector fronts (so the UI picks the right panels) and whether it is currently read-only. Tree browsing and query execution map onto /call tool invocations; the engine prefers the NDJSON /call-stream endpoint so large listings and result sets stream, and transparently falls back to buffered /call when a connector predates streaming (a missing /call-stream route returns 404 with no error body). A 401/403 from the gateway is tagged with a sentinel so the UI can raise a re-authenticate dialog instead of surfacing a generic error.

Breadth and connectivity

The connector catalogue is capability-abstracted: connectors are named for what they do, not for one vendor's product. There are 210 data-app connectors — most capabilities packaged for AWS, GCP and Azure — spanning far more than raw databases:

Each connector fronts the right native protocol behind the scenes. As one example, the Neo4j connector ships a pure-standard-library native Bolt client (the binary Bolt protocol on port 7687, with PackStream and Node/Relationship/Path decoding) alongside its HTTP client — so a bolt://-only deployment such as Neo4j Aura or a hardened cluster works, mapping its results into the same shape the HTTP path returns.

Read-only mode and write keys

A cloud connector is read-only by default. What grants write access is a plaintext write key, sent as an x-write-key header; the connector grants writes only when it matches the connector's configured WRITE_KEY hash. You generate a write key at connectors.openeng.app and enter it on the connect form; leave it blank and the connection stays read-only. The manifest's readOnly / readOnlyCapable flags drive the read-only badge and banner in the UI.

Configuration, scopes, and security

Data is intentionally almost knob-free. The engine is fixed to loopback 50057, self-update is mandatory and always on, and there are no environment variables to tune it; all state lives under ~/.openeng.

• Credentials are local and 0600. Direct DSNs live under ~/.openeng/data/connections/ and connector URL + API key (+ optional bearer + write key) under ~/.openeng/data/connectors/, written with owner-only permissions on Unix.
• Credentials never re-cross the wire outward. Stored secrets are echoed back only by GetConnection, and only to pre-fill the local edit / re-auth dialog on the same sealed loopback that entered them.
• No secret on the wire schema. A build-time schema test enforces that no RPC field carries a product secret; the DSN and connector API key are the only marked exceptions, both being your own credentials for your own sources.
• Read-only scope. Read-only is a connector concept: a connector connection refuses writes unless a valid write key is presented, and the engine additionally blocks any statement it cannot prove is a read. A direct connection runs whatever the underlying database user is permitted to do — scope it with a read-only database role if you want a hard guarantee.
• The one cloud call the engine makes for itself is the boot-time OAuth one-time-key exchange that authenticates you; database traffic is otherwise entirely direct (to your DB) or to your own connector.

Alongside connections, the engine persists two generic UI-owned stores that never carry credentials: a client-state key/value store (opaque JSON shared across every browser on the machine — the run history, editor preferences) and a stale-while-revalidate cache (last read result plus an engine-driven refresh timestamp) so panels render instantly from cache while the engine revalidates in the background.