Connectors

A connector is a pure-Python, source-only function you deploy into your own cloud account — AWS Lambda, a GCP Cloud Function, or an Azure Function — that an OpenEng app engine calls over HTTPS to reach a resource the engine can't touch directly: a private Kubernetes cluster, a cloud database, an object store, a managed shell. Your credentials and data never leave your account; the app engine holds only the connector's URL and API key. The marketplace at connectors.openeng.app ships 252 ready packages — 80 capabilities across AWS, GCP and Azure plus 12 standalone database connectors — every one of them stdlib-only source you can read before you run.

What a connector is

The OpenEng apps (Terminal, Kubernetes, Data) run a local, loopback-sealed engine on your machine. That engine can reach anything on your laptop or LAN, but it deliberately cannot reach into your cloud — it holds no cloud credentials and sits behind a sealed boundary. A connector bridges that gap without ever handing a cloud secret to the app.

Concretely, a connector is:

• A single function that speaks a small tools-over-HTTP protocol (/healthz, /manifest, /tools, /call, /call-stream).
• Pure Python, source-only — no binaries, no .pyc, no vendored dependencies. Everything runs on the bare cloud runtime using only the standard library. Even the wire protocols it can't get from stdlib (SigV4 signing, RS256 JWT signing, RFC-6455 WebSocket framing, the Postgres/Redis/Mongo/Bolt wire protocols) are hand-rolled in Python. CI enforces that no binary or byte-compiled file ever lands in a published zip.
• Deployed in your cloud, behind your cloud's native API-key gateway (API Gateway, a Cloud Function with an API key, an Azure Function key). The gateway validates the x-api-key header before the request reaches your function — the connector itself performs no authentication.
• Positioned where the resource lives — e.g. the kubernetes connector runs inside the cluster's VPC and talks to the API server over raw HTTPS, so a cluster with a private endpoint becomes reachable from your local app without a VPN or a kubeconfig on your laptop.

Why bring-your-own-cloud

The connector model exists to keep a hard property: the OpenEng app engine never holds your cloud secret.

This is why connectors are distributed as source, not a hosted service: you can read every line before you deploy it, and you host it yourself. OpenEng publishes the code and the marketplace catalog; the running function, and everything it can touch, is yours.

The connector protocol

Every connector — whatever ecosystem it fronts, whatever cloud it runs on — serves the same handful of endpoints:

A call looks like this:

POST {base}/call
  Headers:  x-api-key: <your connector API key>
            Authorization: Bearer <token>        # optional
            x-write-key: <plaintext write key>    # optional; grants write access
  Body:     {"name":"run_query","arguments":{"query":"SELECT 1"}}

  200  {"result": { ... }}
  404  {"error": "unknown tool 'foo'"}
  403  {"error": "<connector> is in read-only mode — 'delete_x' is a mutating action and is disabled"}
  502  {"error": "<handler exception message>"}

The capability contract

Data- and terminal-facing connectors converge on three tools so the engines can drive any of them uniformly:

• ping → {"version": "..."} — a connectivity/identity probe.
• list_children → {"items":[...]} — one level of a browsable tree (e.g. buckets → prefixes → objects; databases → tables; shell environments → status attributes). Each item carries a label, a kind, an icon hint, and a runnable flag.
• run_query → a normalized result (table / documents / graph / value / empty) — the connector's one command entry point. Read verbs are always allowed; write verbs are gated by the write key.

The Kubernetes connector is the deep exception: instead of three generic tools it exposes the full operator surface — list/get/delete across ~43 resource kinds, namespace overviews, pod logs, pod_exec, deployment scaling, rollout restarts, node cordon/uncordon, and a generic apply_manifest/delete_resource that works on any kind including CRDs.

Capability architecture & multi-cloud

Connectors are organized around two orthogonal axes, so one body of source serves every cloud.

Capability abstraction, not vendor names

A connector is named for the capability it provides, not the vendor product: object-storage, not "s3"; message-queue, not "SQS"; document-database, not "DynamoDB". Each capability defines one tool contract, and ships three interchangeable backends behind it:

impl/object-storage/
├── meta.py               # cloud-neutral env vars + static params
├── tools.py              # ping / list_children / run_query — the ONE contract
└── backends/
    ├── aws.py            # Amazon S3           (SigV4)
    ├── gcp.py            # Google Cloud Storage (OAuth2)
    └── azure.py          # Azure Blob Storage   (Shared Key / SAS / AAD)

The active backend is chosen at runtime from the cloud the package was stamped with (CONNECTOR_CLOUD, or a baked connector/_cloud.py). So "browse my object store" is one line of engine code whether the bytes live in S3, GCS, or Blob — the Azure container is even surfaced as a "bucket" so the vocabulary matches.

One source, three deploy targets

The second axis is where the function runs. Every package ships entry-point shims for all three managed runtimes plus a plain HTTP server, from one connector core:

The marketplace therefore offers a per-cloud download for each capability: object-storage-aws, object-storage-gcp, object-storage-azure are three catalogue entries sharing one impl directory, packaged into three zips. That is how 80 capabilities × 3 clouds + 12 standalone connectors reach 252 packages.

Cloud auth cores (pure stdlib)

Each backend builds on a shared, dependency-free auth core in sdk/cloud/:

• aws.py — SigV4 request signing (hmac/hashlib), pinned byte-for-byte to the AWS get-vanilla official test vector. Signs any S3/SQS/DynamoDB/Secrets-Manager/CloudWatch/Lambda REST call, with STS session-token support.
• gcp.py — service-account OAuth2 via an RS256 signer written in pure Python (minimal DER parse, PKCS#1 v1.5 pad, big-integer pow), cross-checked byte-exact against openssl. Also accepts a caller-supplied access token or an API key.
• azure.py — Storage Shared-Key (HMAC-SHA256 canonicalization) plus AAD client-credentials, plus caller-supplied bearer or SAS. The HMAC primitive is cross-checked against openssl.

Streaming transport

Results paint incrementally instead of arriving as one blob. Alongside /call, every current connector serves POST /call-stream, which returns application/x-ndjson — one JSON frame per line:

{"t":"meta","seq":0,"kind":"query|list|logs|exec|generic","meta":{...}}  // 0..1, first
{"t":"chunk","seq":N,"data":{...}}                                       // 0..*  (batched)
{"t":"end","seq":M,"result":<value|null>,"trailer":{...}}                // success terminal
{"t":"error","seq":M,"error":"..."}                                      // failure terminal

A tool with a source-streaming generator (e.g. pod logs read in chunks, a live pod_exec stream) emits frames as data arrives. Every other tool is auto-reframed: its buffered result is inspected and re-emitted as batched query/list/logs/generic frames (200 rows or 64 KiB of text per chunk). So all connectors stream for free, and the lazy ones additionally win latency.

A hard transport fact shapes the design: AWS Lambda behind API Gateway and Azure Functions buffer the whole response (the Python managed runtimes can't wire-stream), while GCP Cloud Functions / Cloud Run and self-hosted serve_http truly flush frame-by-frame. The NDJSON protocol degrades cleanly: it wire-streams where the transport allows and delivers one buffered NDJSON body where it can't — and the engine parses frame-by-frame either way, so the code path is identical and a connector that moves to a streaming-capable runtime needs no change.

Authentication & access control

There are three independent credentials on a connector request, and they answer three different questions.

Read-only by default, per-request write key

Every connector is read-only unless a request presents a valid write key — there is no static "read-only" env var. The mechanism is password-vs-hash so a leaked function environment can't be replayed:

1. The operator generates a write key once (at connectors.openeng.app → Generate write key, or any 64-char secret). The generator returns two artefacts.
2. The encoded hash (pbkdf2_sha256$<iterations>$<salt>$<derived>, PBKDF2-HMAC-SHA256, 210,000 iterations) is stored on the function as the WRITE_KEY env var.
3. The plaintext key is entered in the OpenEng app and sent on every request as the x-write-key header.
4. On each request the connector verifies the presented plaintext against the stored hash (constant-time) and sets a per-request write mode. A match → mutating tools are allowed; anything else → read-only.

Because the function stores only the salted hash, someone who reads the function's env and replays that hash as x-write-key still fails (pbkdf2(hash) ≠ hash). The identical algorithm runs in the browser (Web Crypto PBKDF2), so the generator page and the function agree.

When a connector is read-only for a request, mutating tools are hidden from /tools and /manifest and 403-blocked at /call — defense in depth. An unknown tool is still a 404 (looked up in the full list, not the filtered one), so read-only mode never leaks tool existence differently.

Force-exec: read-only with named exceptions

A single tool can carry a force_exec exemption. On the Kubernetes connector, K8S_FORCE_EXEC holds comma-separated pod-name prefixes that remain pod_exec-able even in read-only mode — so an operator can keep a connector globally read-only yet still shell into a named set of pods (e.g. dv06 matches every pod whose name starts with it). The exemption is checked against the target pod name before the exec runs.

Per-connector command restrictions

The Kubernetes connector also honours K8S_RESTRICTED_COMMANDS — a comma-separated blocklist of command names pod_exec refuses to run, checked in the function before the exec reaches the cluster. It parses the command with real shell tokenization (shlex): it recurses into sh -c scripts, $(...) and backtick substitutions, and scans past wrapper programs (sudo/env/xargs/timeout/…) to the real command. This is defense-in-depth, not a sandbox — cluster RBAC remains the actual security boundary.

The manifest advertises the posture

GET /manifest makes the connector's access model inspectable before any call:

{
  "connector": "object-storage", "type": "write", "engine": "s3",
  "readOnly": true,              // this request's effective mode (x-write-key considered)
  "readOnlyCapable": true,       // has mutating tools the write key gates
  "writeKeyConfigured": false,   // a WRITE_KEY hash is set on the function
  "writeKeyRequired": true,      // writes need a valid x-write-key
  "readOnlyRequiresRole": true,  // DB connectors: the write-key gate is defense-in-depth;
  "readOnlyRoleAdvice": "...",   //   a HARD guarantee needs a least-privilege read-only role
  "forceExec": [], "forceExecCapable": false,
  "streaming": true,
  "envVars": [...], "staticParams": [...], "tools": [...]
}

Adding a connector to an app

Adding a connector is the same three inputs everywhere: the connector's base URL, its API key, and an optional write key (plus an optional bearer). The engine probes /manifest on add to learn read-only status, the engine family, and the tool surface, then stores the credentials locally. Which apps expose connectors:

In all three apps the app engine stores api_key, auth_token, and write_key as write-only secrets (the Terminal engine keeps them in its encrypted vault) and forwards them as x-api-key / Authorization / x-write-key on each request. They never round-trip to the UI except to pre-fill a re-auth dialog, which stays on the loopback seal boundary — the same trust domain that entered them.

Capability reference

The 80 multi-cloud capabilities span thirteen categories. Each is shipped for AWS, GCP, and Azure behind one contract; the per-cloud service each fronts is listed in the marketplace.

Headline capabilities

Standalone connectors

Twelve connectors target a specific engine directly over its native wire protocol (hand-rolled, stdlib-only) rather than a cloud REST API: kubernetes, postgres, mysql, cockroach, redis, mongodb, cassandra, neo4j, kafka, s3, gcs, and dynamodb. The database connectors implement the Data-engine contract with three-layer read-only enforcement (transport guard, audited classifier defaulting to WRITE, and normalized server-side permission errors) and connect to self-hosted, managed, and cloud-compatible variants alike — e.g. neo4j speaks native Bolt (5.x down to 4.1) or the HTTP transaction API from whichever URL you paste.

Security model

Two properties are worth restating because they are load-bearing. First, the gateway authenticates, not the connector — the x-api-key check happens at your cloud's edge, so an unauthenticated request never reaches your code. Second, the write-key hash can't be replayed — only a PBKDF2 hash lives in the function env, and presenting that hash as the plaintext key fails verification.

Build your own connector

The marketplace ships a BYOC (Bring Your Own Code) starter — public/byoc/aws.zip, linked from the site — for writing a connector against a resource no published capability covers. It contains the full sdk/ framework verbatim plus a one-file stub:

from sdk import Connector, Tool, KIND_READ, obj, str_p

def example_tool(args):
    return {"echo": args.get("text", "")}   # TODO: your logic

CONNECTOR = Connector(
    id="my-connector",
    type=KIND_READ,                          # read-only; use KIND_WRITE for mutating tools
    tools=[
        Tool("example_tool", "Describe what this does.",
             obj({"text": str_p("Some input.")}), example_tool),
    ],
)

You get the whole protocol for free — /manifest, /tools, /call, /call-stream, the read-only + write-key gate, and the NDJSON auto-reframer — by declaring a Connector and its Tools. Mark a tool mutating=True and it becomes write-gated automatically. Provide a stream_handler generator to stream at the source; otherwise your buffered handler is auto-reframed. Deploy the same way as any published connector: as a Lambda in-VPC or a container, fronted by an API-key gateway. The stub ships with a README, Dockerfile, and requirements so the deploy path matches the reference connectors exactly.