RFC: Multi-tenant ingest gateway

Abstract

Inbound events today land on a single worker pool. This RFC proposes one ingest gateway in front of per-tenant workers so isolation, backpressure, and auth live at the edge.

Context

Shared workers mix tenant load, so a noisy tenant can stall every tenant. Auth is checked only after work is already queued, combining unverified traffic, queue growth, and worker pressure in one shared path.

Decision

Authenticate first, then dispatch each accepted event to a tenant-owned worker.

Architecture

The gateway owns authentication, admission control, and tenant routing. Each tenant worker owns processing and retry behavior. The gateway acknowledges an event only after the tenant worker accepts it.

Event path

Source

sequenceDiagram
  participant Client
  participant Gateway
  participant Auth
  participant Worker as Tenant worker
  Client->>Gateway: POST /events
  Gateway->>Auth: authenticate(tenant)
  Auth-->>Gateway: ok
  Gateway->>Worker: enqueue(event)
  Worker-->>Gateway: ack
  Gateway-->>Client: 202

Handler sketch

def handle_event(gateway, %{tenant: tenant, payload: payload}) do
  with :ok <- Auth.verify(tenant) do
    Worker.enqueue(tenant, payload)
  end
end

Trade-offs

The gateway becomes a deliberate choke point in the ingest path.
Capacity, fallback, and overload behavior must be explicit.
Tenant pressure becomes observable and enforceable before work reaches a queue.

Failure behavior

Under the proposed gateway design:

Invalid or expired credentials are rejected before the queue with 401; no work is enqueued.
A saturated tenant queue is rejected with 429 and Retry-After; the event is not enqueued.
An unavailable tenant worker returns 503 without gateway acknowledgement.
Clients retry with a stable event ID so accepted duplicates can be detected and handled idempotently.

Rollout

Mirror traffic through the gateway without changing dispatch.
Move one internal tenant to gateway-owned authentication and routing.
Expand in tenant cohorts while comparing queue delay and rejection logs.

Open questions

What is the idle timeout for a tenant worker with no events?
Do we drop poison messages or park them on a dead-letter stream?
Are auth tokens sticky per connection or verified per event?