Data

OpenEng Data lets you query SQL, document, key-value, graph, object-store and streaming databases from one browser workspace at data.openeng.app: ten database engines directly, and more through a connector in your own cloud. Every direct database connection runs locally on your own machine; a connector connection runs its queries in your own cloud, where its cloud credentials stay. A small engine binary (openeng-data, checked against its published SHA-256) listens only on 127.0.0.1:50057; the web app is a thin client that renders panels and issues operations over an encrypted loopback boundary. Your connection strings and cloud keys are entered once, stored 0600 under ~/.openeng/data/, and never leave the device except to open the database you pointed them at.

Overview

Data is part of the OpenEng suite and reuses the same architecture as the Terminal and Kubernetes engines: a thin cloud-hosted UI driving a local, sealed gRPC engine. The split is deliberate — the browser is where you read rows, documents, graphs and files; the engine is where credentials live and where every driver connection is opened.

The engine categorizes every source into one of six UI categories, which decides the panels the browser renders and the query dialect the playground expects:

Install and pair the engine

Data is free. It is one of the five engineering apps on the OpenEng platform and shares your OpenEng sign-in, but it runs on its own engine.

1. Install the engine. On Linux (x86_64) or macOS (Apple silicon or Intel; for now the macOS build is the previous release, see Platforms), run:

   bashCopy

   curl -fsSL https://openeng.app/engine/data/install.sh | bash
   

   The installer downloads the binary for your platform, checks it against its published SHA-256 checksum when it can (it prints checksum verified; see Check a download yourself), and installs openeng-data into ~/.local/bin (set OPENENG_BIN_DIR to use another directory), adding that directory to your PATH if needed. There is no native Windows build; on Windows, use WSL.

2. Sign in. Open data.openeng.app and sign in with your Google account.

3. Start the engine. The pairing screen (/pair, titled "Connect to your data engine") shows a start command with a one-time key, ready to copy:

   bashCopy

   openeng-data serve --oauth=<one-time key>
   

   Run it on the same machine. The app connects as soon as the engine is up (or when you click Retry connection) and opens the page you were heading to.

Using the app

This section walks through the console at data.openeng.app. The sections after it are the reference for what the engine does underneath.

The workspace

• Left menu: Explorer, Query, and History, with Settings at the bottom. Each section has its own URL (/explorer, /query, /history, /settings); / and any other path open /explorer. A link you open before signing in or pairing brings you back to the same page afterward.
• Header: a button that hides or shows the side panel, the connection switcher, a status dot for the engine, a search button for the command palette, and your account menu. The connection switcher lists every connection with its engine's icon, a cloud badge for connector connections, a lock for read-only ones, and a check on the active one. Switching connections clears the query editor. The switcher can also refresh the list and delete a connection (after you confirm); deleting the active connection selects the next one.
• Command palette: ⌘+K (Ctrl+K on Windows and Linux) jumps to a section, switches the connection, or, when a connection is active, drops a starter query for any of its top-level objects into the editor.

Adding and managing connections

Settings → Connections lists every connection with its engine, category, and host. Connect switches to a connection and opens Explorer; the active connection is marked Connected; the trash button removes one. New connection opens the add form (Hide the add form closes it), which has two modes:

• Direct database (the default): pick one of ten engines (PostgreSQL, CockroachDB, MySQL / MariaDB, MongoDB, Redis, Neo4j, DynamoDB, Apache Kafka, SQLite, or Filesystem) and fill in its fields; the app builds the connection string for you (see Adding a connection). For SQLite and Filesystem, Browse opens a picker that walks your machine through the engine: "Choose a database file" highlights .db, .sqlite, .sqlite3, and .db3 files, and "Choose a folder" commits the open folder with Select this folder.
• Cloud connector: a name, the connector URL, and its API key (sent as the X-API-Key header). Tick Use an authorization bearer token to send a separate Authorization: Bearer token, or Configure a write key (grants write access) to add a write key. The engine reads the connector's manifest to learn which database it fronts and whether it is read-only. Object stores such as Amazon S3 are available only this way. See Cloud connectors.

Explorer

Explorer shows the active connection's objects in its Objects side panel, next to the query editor. What you see depends on the database: databases, schemas, and tables; collections; keys; labels and relationship types; buckets, folders, and files; or topics and partitions (see The adaptive schema tree). Each level loads when you expand it. The filter box narrows the current level, the refresh button reloads it, and the top-level objects are also searchable in the command palette. Click or double-click a table, collection, key, label, file, or topic (or use its run button) to drop a starter query into the editor; see Starter queries. Expanding a database also makes it the database your queries run against.

Running queries

Query is the same editor and results area as Explorer, without the tree. The editor's hint shows the connection's own query language. Click Run or press ⌘+Enter (Ctrl+Enter on Windows and Linux). SQL, document, and graph connections also show a field for the database to run against. On a read-only connection, a lock appears with the banner "Read-only connection — write / update / delete statements are blocked by the connector.", and write statements are refused.

Results appear in the shape the database returned: a table, JSON documents, a graph, message cards for Kafka (partition, offset, timestamp, key, and value), a single value, or text. When a query also returns a graph or an execution plan, Table, Graph, and Plan tabs (or Result and Plan) let you switch views. The status bar shows success or the error, a badge for writes, a warning when results were truncated, and how long the query took. See The query playground and results.

History

History (/history) lists the queries you have run since you opened the app in this browser tab, newest first (up to 100). Click an entry to switch back to its connection and load its query, and its target database, into the editor. Clear empties the list. History is kept in the tab's memory only, so reloading the page starts it empty.

Settings and the engine

Settings has two categories: Connections (see above) and Engine, which shows whether the engine is connected, its fixed address (127.0.0.1:50057), its version, and the account you are signed in with.

If the active cloud connector rejects its stored credentials (HTTP 401 or 403), the app opens a Re-authenticate connector dialog pre-filled with the stored URL, API key, and bearer token. Correct them and click Save & retry. If you cancel, the connection is marked inaccessible and a bar under the header offers Re-authenticate to reopen the dialog. Direct connections never show this dialog.

The account menu (your avatar) shows your email, the engine and app versions, a Release notes link, and Sign out.

Notes for each database

The sealed local engine

Everything the browser does flows through a single encrypted boundary. The engine serves gRPC-Web on 127.0.0.1:50057 behind a persisted, publicly-trusted TLS certificate for engine.openeng.app (resolved to loopback), so the browser connects with no certificate-trust step. On top of TLS, when the client offered a key at CreateSession (the opening handshake, the only unsealed RPC), every later frame is individually sealed with a fresh ephemeral X25519 key, HKDF-SHA256 and ChaCha20-Poly1305. The key exchange isn't authenticated, and a request frame that fails to unseal is passed on as it arrived.

You start the engine once, pairing it with a one-time key copied from the Connect screen (install it first; see Install and pair the engine):

openeng-data serve --oauth=<one-time key>    # the one-time key from the Connect screen
openeng-data --version

The engine is zero-config: the loopback port is fixed at 50057, the only environment variable it reads is OPENENG_HOME (which moves the whole ~/.openeng root), and self-update is always on. At boot it enforces a mandatory, non-skippable update (it fetches, verifies and re-execs into the newest stable build before serving). While running, a newer stable is staged in the background; the session reports it (update_available / staged_version) and the UI raises a restart banner that calls RestartEngine to re-exec in place — the live session is carried internally so you never re-pair. The one exception, from the release notes: restarting this way when updating from Data 0.0.3 asks for a fresh one-time key once; updating at the next start needs nothing.

Two kinds of connection

A connection is a data source you can list, switch between, and query — the same way the Kubernetes app lists clusters. There are two kinds, both persisted 0600 under ~/.openeng/data/ and both selected from the header switcher:

Adding a connection

The add-connection form (in the full-frame Settings pane) has a segmented toggle between the two modes. In Direct mode you pick an engine and fill structured fields — host, port, database, username, password, plus per-engine toggles — and the UI assembles the DSN for you; the raw connection string is never shown for network engines. In Cloud connector mode you paste the connector URL, the API-key value (sent as the X-API-Key header), and optionally an Authorization: Bearer token and a plaintext write key.

The direct engines the UI offers, and the DSN it builds for each:

Because the browser cannot read local paths, the SQLite and Filesystem pickers call the engine's BrowsePath RPC to walk the local machine and let you pick a file or directory. A direct engine that the engine does not compile a driver for is rejected at add time with a message pointing you to a cloud connector instead.

Switching the active connection (SetCurrent), testing one (TestConnection pings the source and returns a server-version string), and removing one are all one-click. Direct handles are opened lazily on first use and cached; a re-add invalidates the cached handle.

Native drivers

Direct connections are served by typed async Rust drivers compiled into the engine — no shell-outs, no external CLIs. Each driver implements three operations: a cheap ping (server version / status), list_children (one lazy level of the schema tree), and run_query (a query/command producing a unified result).

Three more direct drivers ship in the engine for completeness:

• DynamoDB (document category) — browse tables and run PartiQL (SELECT * FROM "table", and INSERT/UPDATE/DELETE for writes).
• Apache Kafka / Redpanda / MSK (streaming category) — the tree is topics → partitions; a "query" is a topic (orders), a partition (orders/2), or an explicit start (orders/2/earliest) that the engine resolves to the latest N messages.
• MongoDB (document category) — a hand-rolled OP_MSG/SCRAM wire client, written because the upstream mongodb crate does not build on the pinned toolchain. It also resolves mongodb+srv addresses.

The engine normalizes many aliases to a driver key: postgresql/pg/cockroach/crdb/redshift → postgres; mariadb → mysql; valkey → redis; mongodb+srv → mongodb; redpanda/msk/warpstream → kafka; fs/file/local → filesystem.

The adaptive schema tree

The left panel is an adaptive object hierarchy that reshapes itself per engine. It is loaded lazily, one level at a time, through the ListChildren RPC: the browser sends the path trail from the root (empty = the connection root) and a per-driver interpreter returns that node's children. Each returned node carries a kind (database, schema, table, view, collection, key, label, reltype, bucket, object, dir, file, …), an icon hint, a detail string (row count, key type, file size), a has_children flag, and a runnable flag.

A filter box narrows the current level (passed as the filter argument, so filtering happens in the engine/connector, not just in the browser). For large listings — a Redis keyspace, an S3 bucket — the tree uses the streaming twin ListChildrenStream, which appends nodes as the source enumerates them so the panel paints progressively; a superseding filter aborts the in-flight stream.

Starter queries and query targeting

A runnable leaf (a table, collection, key, label, reltype, or file) can be double-clicked to drop a starter query into the playground — SELECT * FROM "schema"."table" LIMIT 100 for a SQL table, db.users.find({}) for a Mongo collection, GET key for a Redis key, MATCH (n:Label) RETURN n LIMIT 50 for a Neo4j label. Identifiers are quoted per engine (backticks for MySQL, double quotes elsewhere) so hyphens, mixed case and reserved words work.

For SQL, graph and document sources the tree's root segment is a database, so expanding a database node also makes it the active query target — a typed query then runs against the database you are browsing rather than the connection's default (which for Neo4j is often an empty neo4j graph).

The query playground and results

The right panel is a database-specific playground: a query/command editor with a Run action (⌘/Ctrl+Enter), an engine badge, an optional target-database field (for SQL, document and graph sources), a read-only indicator, and the results area. The editor's contents are shared through the store, so the tree, the ⌘K command palette, and the History view all write into the same editor. Every run is recorded to a local run history.

Queries run through RunQueryStream: the engine sends a QueryMeta header (columns, result kind, write-flag), then batches of rows / documents / graph data as they arrive, then a terminal summary — so results paint progressively. Batches amortize the per-frame seal (never one row per chunk). A default row cap of 1000 applies when the client sets none (hard-capped at 50,000) so a large result can neither blow the sealed frame nor the browser; a truncated result is flagged.

The unified result is rendered in the right shape by its result_kind:

The status bar reports execution time, rows affected (for writes), and truncation. Each statement is classified read-vs-write so the result is stamped is_write and, on a read-only connection, a write is refused before it runs.

Cloud connectors

A cloud connector lets you point the Data app at a database or object store that lives in a cloud you cannot reach directly. Instead of opening the database, the engine calls a small HTTPS service — a connector — that you deploy into your own cloud (an AWS Lambda, a GCP Cloud Function, or an Azure Function). The connector reaches the target inside your VPC and holds its credentials there; the engine only ever stores the connector's URL and your API key, and only ever sees the connector's trimmed JSON responses.

What a data connector does

A data connector speaks the OpenEng tools protocol over three HTTPS endpoints. The engine authenticates every request with an x-api-key header and an Authorization: Bearer header (the API key is reused as the bearer when you do not set a separate token):

GET  {base}/manifest        → { "engine": "postgres",
                                "readOnly": true,
                                "readOnlyCapable": true }

POST {base}/call            { "name": "<tool>", "arguments": { … } }
     x-api-key: <key>       → { "result": <value> }   |   { "error": "…" }

POST {base}/call-stream     NDJSON frames: {"t":"meta|chunk|end|error", …}

The /manifest tells the engine which database the connector fronts (so the UI picks the right panels) and whether it is currently read-only. Tree browsing and query execution map onto /call tool invocations; the engine prefers the NDJSON /call-stream endpoint so large listings and result sets stream, and transparently falls back to buffered /call when a connector predates streaming (a missing /call-stream route returns 404 with no error body). A 401/403 from the gateway is tagged with a sentinel so the UI can raise a re-authenticate dialog instead of surfacing a generic error.

Breadth and connectivity

The connector catalog is capability-abstracted: connectors are named for what they do, not for one vendor's product. There are 210 data-app connectors — most capabilities packaged for AWS, GCP and Azure — spanning far more than raw databases:

Each connector fronts the right native protocol behind the scenes. As one example, the Neo4j connector ships a pure-standard-library native Bolt client (the binary Bolt protocol on port 7687, with PackStream and Node/Relationship/Path decoding) alongside its HTTP client — so a bolt://-only deployment such as Neo4j Aura or a hardened cluster works, mapping its results into the same shape the HTTP path returns.

Read-only mode and write keys

A cloud connector is read-only by default. What grants write access is a plaintext write key, sent as an x-write-key header; the connector grants writes only when it matches the connector's configured WRITE_KEY hash. You generate a write key at connectors.openeng.app and enter it on the connect form; leave it blank and the connection stays read-only. The manifest's readOnly / readOnlyCapable flags drive the read-only badge and banner in the UI.

Configuration, scopes, and security

Data is intentionally almost knob-free. The engine is fixed to loopback 50057, self-update is mandatory and always on, and the only environment variable it reads is OPENENG_HOME, which relocates the root; all state lives under ~/.openeng (or that root).

• Credentials are local and 0600. Direct DSNs live under ~/.openeng/data/connections/ and connector URL + API key (+ optional bearer + write key) under ~/.openeng/data/connectors/, written with owner-only permissions on Unix.
• Credentials go back only to the app, to pre-fill a dialog. GetConnection returns a direct connection's DSN, and a connector's API key and bearer token, to the app over the same sealed loopback that entered them, to pre-fill the edit / re-auth dialog; they go nowhere else.
• No secret on the wire schema. A build-time schema test enforces that no RPC field carries a product secret; the DSN (connection-credential) and the connector's API key and bearer token (connector-credential) are the marked exceptions, and the write key is an unmarked credential field. All of them are your own credentials for your own sources.
• Read-only scope. Read-only is a connector concept: a connector connection refuses writes unless a valid write key is presented, and the engine additionally blocks any statement it cannot prove is a read. A direct connection runs whatever the underlying database user is permitted to do — scope it with a read-only database role if you want a hard guarantee.
• The engine's own cloud calls are the pairing exchange at boot, sign-in renewal and the certificate fetch (api.openeng.app), update checks at start and every 30 minutes (releases.openeng.app), and, when an update exists, its download from openeng.app. Apart from those, database traffic goes directly to your database or to your own connector.

Alongside connections, the engine persists two generic UI-owned stores that never carry credentials: a client-state key/value store (opaque JSON shared across every browser on the machine) and a stale-while-revalidate cache (last read result plus an engine-driven refresh timestamp), built so panels can render instantly from cache while the engine revalidates in the background. The current Data app does not use either store yet: its run history lives in the browser tab's memory, and its panels load fresh from the engine.