Connectors

A connector is a pure-Python, source-only function you deploy into your own cloud account — AWS Lambda, a GCP Cloud Function, or an Azure Function — that an OpenEng app engine calls over HTTPS to reach a resource the engine can't touch directly: a private Kubernetes cluster, a cloud database, an object store, a managed shell. Your cloud credentials stay in your account and nothing passes through OpenEng. Results come back to your engine over HTTPS. The engine holds the connector's URL, API key, bearer token and write key, and sends them only to your gateway. The marketplace at connectors.openeng.app ships 252 ready packages — 80 capabilities across AWS, GCP and Azure plus 12 standalone connectors (Kubernetes, Kafka, two object stores, and eight databases) — every one of them stdlib-only source you can read before you run.

What a connector is

The OpenEng apps (Terminal, Kubernetes, Data) run a local, loopback-sealed engine on your machine. That engine can reach anything on your laptop or LAN, but it deliberately cannot reach into your cloud — it holds no cloud credentials and sits behind a sealed boundary. A connector bridges that gap without ever handing a cloud secret to the app.

Concretely, a connector is:

• A single function that speaks a small tools-over-HTTP protocol (/healthz, /manifest, /tools, /call, /call-stream).
• Pure Python, source-only — no binaries, no .pyc, no vendored dependencies. Everything runs on the bare cloud runtime using only the standard library. Even the wire protocols it can't get from stdlib (SigV4 signing, RS256 JWT signing, RFC-6455 WebSocket framing, the Postgres/Redis/Mongo/Bolt wire protocols) are hand-rolled in Python. The packaging script skips byte-compiled files (.pyc and __pycache__) and syntax-checks every .py file without writing bytecode, so a published zip holds source only.
• Deployed in your cloud, behind your cloud's native API-key gateway (API Gateway, a Cloud Function with an API key, an Azure Function key). The gateway validates the x-api-key header before the request reaches your function — the connector itself performs no authentication.
• Positioned where the resource lives — e.g. the kubernetes connector runs inside the cluster's VPC and talks to the API server over raw HTTPS, so a cluster with a private endpoint becomes reachable from your local app without a VPN or a kubeconfig on your laptop.

Why bring-your-own-cloud

The connector model exists to keep a hard property: the OpenEng app engine never holds your cloud secret.

This is why connectors are distributed as source, not a hosted service: you can read every line before you deploy it, and you host it yourself. OpenEng publishes the code and the marketplace catalog; the running function, and everything it can touch, is yours.

Deploy a connector and use it

1. Pick a connector. Browse connectors.openeng.app. You can search it and filter by read or write type, by category, and by the OpenEng app a connector works with. A connector's page lists its tools and the environment variables it needs, and offers the download: a zip per cloud for a multi-cloud capability (Download for your cloud), or a single zip for a standalone connector. Each zip holds the Python source plus a README.md, requirements.txt, a Dockerfile, deploy.sh, and .env.example.
2. Deploy it in your cloud. Run the function where it can reach the resource (for example, attached to the same VPC), set the environment variables from its page, and put your cloud's API-key gateway in front of it: the connector does no authentication of its own. The README in the zip walks through the deploy.
3. Allow writes, if you want them. A connector is read-only until a request carries a valid write key. To create one, open connectors.openeng.app/write-key (Write key in the site's menu), set the encoded hash it generates as the function's WRITE_KEY environment variable, and keep the plaintext key for the app. See Read-only by default, per-request write key.
4. Add it to an app with the gateway URL and API key, plus the plaintext write key if you made one:
   • Kubernetes: Settings → Clusters → Add a cloud cluster. See Cloud clusters via connector.
   • Data: Settings → Connections → New connection → Cloud connector. See Data's cloud connectors.
   • Terminal: SSH → New connection → Cloud connector. See Terminal's cloud connectors.

If the connector later rejects the stored credentials (HTTP 401 or 403), the app asks you to re-enter them.

The connector protocol

Every connector — whatever ecosystem it fronts, whatever cloud it runs on — serves the same handful of endpoints:

A call looks like this:

POST {base}/call
  Headers:  x-api-key: <your connector API key>
            Authorization: Bearer <token>        # optional
            x-write-key: <plaintext write key>    # optional; grants write access
  Body:     {"name":"run_query","arguments":{"query":"SELECT 1"}}

  200  {"result": { ... }}
  404  {"error": "unknown tool 'foo'"}
  403  {"error": "<connector> is in read-only mode — 'delete_x' is a mutating action and is disabled"}
  502  {"error": "<handler exception message>"}

The capability contract

Data- and terminal-facing connectors converge on three tools so the engines can drive any of them uniformly:

• ping → {"version": "..."} — a connectivity/identity probe.
• list_children → {"items":[...]} — one level of a browsable tree (e.g. buckets → prefixes → objects; databases → tables; shell environments → status attributes). Each item carries a label, a kind, an icon hint, and a runnable flag.
• run_query → a normalized result (table / documents / graph / value / empty) — the connector's one command entry point. Read verbs are always allowed; write verbs are gated by the write key.

The Kubernetes connector is the deep exception: instead of three generic tools it exposes the full operator surface — list/get/delete across ~43 resource kinds, namespace overviews, pod logs, pod_exec, deployment scaling, rollout restarts, node cordon/uncordon, and a generic apply_manifest/delete_resource that works on any kind including CRDs.

Capability architecture & multi-cloud

Connectors are organized around two orthogonal axes, so one body of source serves every cloud.

Capability abstraction, not vendor names

A connector is named for the capability it provides, not the vendor product: object-storage, not "s3"; message-queue, not "SQS"; document-database, not "DynamoDB". Each capability defines one tool contract, and ships three interchangeable backends behind it:

impl/object-storage/
├── meta.py               # cloud-neutral env vars + static params
├── tools.py              # ping / list_children / run_query — the ONE contract
└── backends/
    ├── aws.py            # Amazon S3           (SigV4)
    ├── gcp.py            # Google Cloud Storage (OAuth2)
    └── azure.py          # Azure Blob Storage   (Shared Key / SAS / AAD)

The active backend is chosen at runtime from the cloud the package was stamped with (CONNECTOR_CLOUD, or a baked connector/_cloud.py). So "browse my object store" is one line of engine code whether the bytes live in S3, GCS, or Blob — the Azure container is even surfaced as a "bucket" so the vocabulary matches.

One source, three deploy targets

The second axis is where the function runs. Every package ships entry-point shims for all three managed runtimes plus a plain HTTP server, from one connector core:

The marketplace therefore offers a per-cloud download for each capability: object-storage-aws, object-storage-gcp, object-storage-azure are three catalog entries sharing one impl directory, packaged into three zips. That is how 80 capabilities × 3 clouds + 12 standalone connectors reach 252 packages.

Cloud auth cores (pure stdlib)

Each backend builds on a shared, dependency-free auth core in sdk/cloud/:

• aws.py — SigV4 request signing (hmac/hashlib), pinned byte-for-byte to the AWS get-vanilla official test vector. Signs any S3/SQS/DynamoDB/Secrets-Manager/CloudWatch/Lambda REST call, with STS session-token support.
• gcp.py — service-account OAuth2 via an RS256 signer written in pure Python (minimal DER parse, PKCS#1 v1.5 pad, big-integer pow), cross-checked byte-exact against openssl. Also accepts a caller-supplied access token or an API key.
• azure.py — Storage Shared-Key (HMAC-SHA256 canonicalization) plus AAD client-credentials, plus caller-supplied bearer or SAS. The HMAC primitive is cross-checked against openssl.

Streaming transport

Results paint incrementally instead of arriving as one blob. Alongside /call, every current connector serves POST /call-stream, which returns application/x-ndjson — one JSON frame per line:

{"t":"meta","seq":0,"kind":"query|list|logs|exec|generic","meta":{...}}  // 0..1, first
{"t":"chunk","seq":N,"data":{...}}                                       // 0..*  (batched)
{"t":"end","seq":M,"result":<value|null>,"trailer":{...}}                // success terminal
{"t":"error","seq":M,"error":"..."}                                      // failure terminal

A tool with a source-streaming generator (e.g. pod logs read in chunks, a live pod_exec stream) emits frames as data arrives. Every other tool is auto-reframed: its buffered result is inspected and re-emitted as batched query/list/logs/generic frames (200 rows or 64 KiB of text per chunk). So all connectors stream for free, and the lazy ones additionally win latency.

A hard transport fact shapes the design: AWS Lambda behind API Gateway and Azure Functions buffer the whole response (the Python managed runtimes can't wire-stream), while GCP Cloud Functions / Cloud Run and self-hosted serve_http truly flush frame-by-frame. The NDJSON protocol degrades cleanly: it wire-streams where the transport allows and delivers one buffered NDJSON body where it can't — and the engine parses frame-by-frame either way, so the code path is identical and a connector that moves to a streaming-capable runtime needs no change.

Authentication & access control

There are three independent credentials on a connector request, and they answer three different questions.

Read-only by default, per-request write key

Every connector is read-only unless a request presents a valid write key — there is no static "read-only" env var. The mechanism is password-vs-hash so a leaked function environment can't be replayed:

1. The operator generates a write key once (at connectors.openeng.app → Generate write key, or any 64-char secret). The generator returns two artifacts.
2. The encoded hash (pbkdf2_sha256$<iterations>$<salt>$<derived>, PBKDF2-HMAC-SHA256, 210,000 iterations) is stored on the function as the WRITE_KEY env var.
3. The plaintext key is entered in the OpenEng app and sent on every request as the x-write-key header.
4. On each request the connector verifies the presented plaintext against the stored hash (constant-time) and sets a per-request write mode. A match → mutating tools are allowed; anything else → read-only.

Because the function stores only the salted hash, someone who reads the function's env and replays that hash as x-write-key still fails (pbkdf2(hash) ≠ hash). The identical algorithm runs in the browser (Web Crypto PBKDF2), so the generator page and the function agree.

When a connector is read-only for a request, mutating tools are hidden from /tools and /manifest and 403-blocked at /call — defense in depth. An unknown tool is still a 404 (looked up in the full list, not the filtered one), so read-only mode never leaks tool existence differently.

Force-exec: read-only with named exceptions

A single tool can carry a force_exec exemption. On the Kubernetes connector, K8S_FORCE_EXEC holds comma-separated pod-name prefixes that remain pod_exec-able even in read-only mode — so an operator can keep a connector globally read-only yet still shell into a named set of pods (e.g. dv06 matches every pod whose name starts with it). The exemption is checked against the target pod name before the exec runs.

Per-connector command restrictions

The Kubernetes connector also honors K8S_RESTRICTED_COMMANDS — a comma-separated blocklist of command names pod_exec refuses to run, checked in the function before the exec reaches the cluster. It parses the command with real shell tokenization (shlex): it recurses into sh -c scripts, $(...) and backtick substitutions, and scans past wrapper programs (sudo/env/xargs/timeout/…) to the real command. This is defense-in-depth, not a sandbox — cluster RBAC remains the actual security boundary.

The manifest advertises the posture

GET /manifest makes the connector's access model inspectable before any call:

{
  "connector": "object-storage", "type": "write", "engine": "s3",
  "readOnly": true,              // this request's effective mode (x-write-key considered)
  "readOnlyCapable": true,       // has mutating tools the write key gates
  "writeKeyConfigured": false,   // a WRITE_KEY hash is set on the function
  "writeKeyRequired": true,      // writes need a valid x-write-key
  "readOnlyRequiresRole": true,  // DB connectors: the write-key gate is defense-in-depth;
  "readOnlyRoleAdvice": "...",   //   a HARD guarantee needs a least-privilege read-only role
  "forceExec": [], "forceExecCapable": false,
  "streaming": true,
  "envVars": [...], "staticParams": [...], "tools": [...]
}

Adding a connector to an app

Adding a connector is the same three inputs everywhere: the connector's base URL, its API key, and an optional write key (plus an optional bearer). The engine probes /manifest on add to learn read-only status, the engine family, and the tool surface, then stores the credentials locally. Which apps expose connectors:

In all three apps the app engine stores api_key, auth_token, and write_key on your machine and forwards them as x-api-key / Authorization / x-write-key on each request. Terminal keeps them write-only, in its encrypted vault. Kubernetes and Data return them to the app over the sealed loopback to pre-fill the edit and re-auth dialogs (Kubernetes the API key, bearer token and write key; Data the API key and bearer token), the same trust domain that entered them; they go nowhere else.

Capability reference

The 80 multi-cloud capabilities span thirteen categories. Each is shipped for AWS, GCP, and Azure behind one contract; the per-cloud service each fronts is listed in the marketplace.

Headline capabilities

Standalone connectors

Twelve connectors target a specific engine directly over its native wire protocol (hand-rolled, stdlib-only) rather than a cloud REST API: kubernetes, postgres, mysql, cockroach, redis, mongodb, cassandra, neo4j, kafka, s3, gcs, and dynamodb. The database connectors implement the Data-engine contract with three-layer read-only enforcement (transport guard, audited classifier defaulting to WRITE, and normalized server-side permission errors) and connect to self-hosted, managed, and cloud-compatible variants alike — e.g. neo4j speaks native Bolt (5.x down to 4.1) or the HTTP transaction API from whichever URL you paste.

Security model

Two properties are worth restating because they are load-bearing. First, the gateway authenticates, not the connector — the x-api-key check happens at your cloud's edge, so once you put an API-key gateway in front of the function, an unauthenticated request never reaches your code. The deploy script doesn't create that gateway, and the connector does no authentication of its own. Second, the write-key hash can't be replayed — only a PBKDF2 hash lives in the function env, and presenting that hash as the plaintext key fails verification.

Build your own connector

The marketplace ships a BYOC (Bring Your Own Code) starter — public/byoc/aws.zip, linked from the site — for writing a connector against a resource no published capability covers. It contains the full sdk/ framework verbatim plus a one-file stub:

from sdk import Connector, Tool, KIND_READ, obj, str_p

def example_tool(args):
    return {"echo": args.get("text", "")}   # TODO: your logic

CONNECTOR = Connector(
    id="my-connector",
    type=KIND_READ,                          # read-only; use KIND_WRITE for mutating tools
    tools=[
        Tool("example_tool", "Describe what this does.",
             obj({"text": str_p("Some input.")}), example_tool),
    ],
)

You get the whole protocol for free — /manifest, /tools, /call, /call-stream, the read-only + write-key gate, and the NDJSON auto-reframer — by declaring a Connector and its Tools. Mark a tool mutating=True and it becomes write-gated automatically. Provide a stream_handler generator to stream at the source; otherwise your buffered handler is auto-reframed. Deploy the same way as any published connector: as a Lambda in-VPC or a container, fronted by an API-key gateway. The stub ships with a README, Dockerfile, and requirements so the deploy path matches the reference connectors exactly.