Vertex AI Proxy

Vertex AI Proxy is an open-source gateway for developers who want a Gemini-compatible API backed by Google Vertex AI. An operator manages models, upstream credentials, keys, prices, and usage in a separate web console. Clients call the gateway with a key rather than handling the upstream credential themselves.

The problem

Sharing a cloud credential with every client makes access and spending difficult to control. A gateway needs to accept familiar model endpoints, restrict each key, reserve credit before a request, and record what the upstream actually used. Streaming responses add another accounting path because a client may disconnect before the stream finishes.

Key features

• Public model endpoints handle generation, streaming generation, token counting, embeddings, and model discovery.
• Administrators create, suspend, revoke, regenerate, and bulk-create gateway API keys.
• Keys can restrict models, endpoints, and IP addresses, with request and token limits.
• Prepaid balances use pricing rules and a credit ledger for usage accounting.
• The console exposes usage reports, CSV export, request metadata, and an audit log.
• Multiple Vertex AI upstreams can be configured and selected by the routing layer.
• Health endpoints and a protected metrics endpoint support operation.

Technical architecture

A NestJS API receives the Gemini-style request and authenticates its gateway key. Redis holds rate-limit counters and short-lived credit reservations. The API selects an upstream, calls Vertex AI, then settles usage into a PostgreSQL ledger. TypeORM maps the stored keys, pricing rules, accounts, and ledger records. A Next.js admin app calls separate administration endpoints. Docker Compose definitions include PostgreSQL, Redis, the API, the console, and nginx for self-hosting.

Engineering highlights

• Redis Lua scripts make credit holds and rate-limit decisions atomic across concurrent requests.
• Upstream credentials are encrypted with AES-256-GCM; administrator passwords use Argon2id.
• The gateway settles streamed requests on completion, failure, or cancellation, so each path reaches accounting.
• API keys are stored as hashes; the full value is returned when a key is created or regenerated.