Infrastructure · Advanced · 35 min read

AI Gateway on ECS Fargate: Per-Application Budgets, Retries, and Isolation Testing

AI Gateway на ECS Fargate: бюджеты приложений, retries и проверка изоляции
Temporary fallback cover; replace in editorial pass.

A shared model endpoint is easy to centralize and surprisingly difficult to control. One application can consume the budget, retry a provider outage into a traffic storm, or exhaust concurrency needed by every other application. This guide builds a reproducible control plane around an OpenAI-compatible gateway running on Amazon ECS Fargate, with Amazon Bedrock as the provider, separate application keys, explicit budgets, bounded retries, fallback routing, and tests for isolation.

Updated 17 November 2026 Advanced 35 min read

The concrete problem

Imagine a team with two internal applications. summarizer processes user-uploaded documents. support-assistant serves interactive requests. Both applications call one AI gateway, and the gateway calls Bedrock. The team wants the following behavior:

  • summarizer has a monthly spend ceiling and cannot consume the interactive application's allocation.
  • A temporary provider error produces a small, bounded number of retries rather than an unbounded retry loop.
  • If the primary model is unavailable, requests can use an explicitly approved fallback model.
  • One application's burst does not starve the other application's requests.
  • Operators can prove these properties with repeatable tests and inspect the evidence afterward.

The gateway is a policy boundary, not a billing system. It can reject requests based on configured keys, rate limits, budgets, and routing rules, but the authoritative spend record remains the provider and your own accounting pipeline. Treat gateway budget enforcement as a safety control and reconcile it against provider usage data.

Target architecture

The design uses one stateless gateway service behind an Application Load Balancer. The service runs on ECS Fargate in private subnets. It reaches Bedrock through the appropriate AWS network path and receives its gateway configuration from a secret store or injected configuration mechanism. Applications receive different gateway keys and use the same OpenAI-compatible endpoint.

application A ── key_summarizer ──┐
                                  ├── ALB ── ECS Fargate gateway ── Bedrock
application B ── key_support ─────┘
                                      │
                                      ├── logs and metrics
                                      └── secret/config store

The important isolation boundary is the application key. A shared key makes attribution, budget enforcement, and incident containment ambiguous. A key should identify an application or workload, not an individual employee. If a workload has materially different risk or budget classes, give it a separate key even when it uses the same model.

Control-plane responsibilities

Control Purpose What to verify
Application key Attributes traffic to one workload Each request carries only the intended key
Budget Stops or restricts a workload after its allowance One key is rejected while another remains usable
Rate and concurrency limits Protects service capacity and provider quotas A burst does not exhaust all gateway workers
Retry policy Recovers from transient failures without amplification Attempts are finite and observable
Fallback route Provides an approved alternative model Fallback occurs only for configured failure classes
IAM role Limits what the task can call and read The task role has only required Bedrock and secret permissions

Prerequisites and safe variables

Prepare an AWS account, an ECS cluster, a VPC with private subnets, an Application Load Balancer, and provider model access for the Bedrock models you intend to use. The exact model IDs and regional availability are account- and region-dependent, so keep them as variables rather than copying an example value.

export AWS_REGION="replace-with-your-region"
export AWS_ACCOUNT_ID="replace-with-your-account-id"
export ECS_CLUSTER="ai-gateway-cluster"
export ECS_SERVICE="ai-gateway"
export GATEWAY_HOST="gateway.example.internal"
export PRIMARY_MODEL="replace-with-approved-primary-model-id"
export FALLBACK_MODEL="replace-with-approved-fallback-model-id"

# Do not put real API keys in this file.
export SUMMARIZER_KEY="test-only-key-for-local-shell"
export SUPPORT_KEY="test-only-key-for-local-shell"

Use a shell session with a short lifetime, a password manager, or an approved secret manager for real credentials. The values above are placeholders. They are not valid credentials and do not represent a customer configuration.

Choose the gateway configuration

The example below uses a gateway that exposes an OpenAI-compatible API and supports virtual keys, per-key budgets, model aliases, and provider fallbacks. Pin the gateway image to a reviewed version. Do not use a floating latest tag in a production task definition.

The names of configuration fields vary by gateway version. Treat this as a configuration shape to adapt to the pinned version you have selected. Before deploying, validate the fields against that version's documentation and run a configuration check in CI.

# gateway-config.yaml — illustrative configuration
model_list:
  - model_name: primary
    litellm_params:
      model: bedrock/<PRIMARY_MODEL_ID>
      aws_region_name: <AWS_REGION>

  - model_name: fallback
    litellm_params:
      model: bedrock/<FALLBACK_MODEL_ID>
      aws_region_name: <AWS_REGION>

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 45
  fallbacks:
    - primary: ["fallback"]

general_settings:
  master_key: os.environ/GATEWAY_MASTER_KEY
  database_url: os.environ/GATEWAY_DATABASE_URL

# Exact key and budget syntax is version-specific.
# Keep the policy data in a protected store and render this file during deployment.
virtual_keys:
  - key: os.environ/SUMMARIZER_KEY
    alias: summarizer
    models: ["primary", "fallback"]
    max_budget: 100
    budget_duration: 30d
    rpm_limit: 30

  - key: os.environ/SUPPORT_KEY
    alias: support-assistant
    models: ["primary", "fallback"]
    max_budget: 100
    budget_duration: 30d
    rpm_limit: 120

The budget values are deliberately examples. They are not prices, forecasts, or recommendations. Select amounts from your own allocation policy and reconcile them with the provider's reported usage. If your gateway stores budget state in a database, use a durable database rather than local container storage; Fargate task replacement must not reset budget state.

Budget design rules

  • Define whether a budget is a hard stop, a soft alert, or a routing downgrade. Document the behavior visible to callers.
  • Define the time window explicitly. “Monthly” can mean a calendar month or a rolling duration.
  • Keep budget state outside the task filesystem.
  • Account for retries. A failed attempt may still consume provider capacity or incur usage depending on the failure point.
  • Use separate keys for production, staging, load tests, and incident experiments.
  • Do not assume a gateway budget is an exact provider invoice limit.

Package the gateway for ECS Fargate

The container should start one gateway process, listen on a health-check path, write structured logs to standard output, and receive configuration through environment variables, mounted secrets, or a protected configuration service.

# Dockerfile — adapt the image and startup command to the pinned gateway
FROM <reviewed-gateway-image>

COPY gateway-config.yaml /app/gateway-config.yaml

ENV PORT=4000
EXPOSE 4000

CMD ["<gateway-command>", "--config", "/app/gateway-config.yaml", "--port", "4000"]

Do not bake production keys into the image. A safer pattern is to render configuration at startup from secrets and non-secret variables, or use the gateway's supported secret references. Ensure that debug logging does not print authorization headers, prompt content, or full provider responses.

Task definition concerns

The ECS task definition should include a task execution role and a separate task role. The execution role pulls the image and writes logs. The task role grants only the actions needed by the gateway, such as calling the selected Bedrock APIs and reading the selected secret. Keep the policy resource scope as narrow as the service permits.

{
  "family": "ai-gateway",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "1024",
  "memory": "2048",
  "containerDefinitions": [
    {
      "name": "gateway",
      "image": "<account>.dkr.ecr.<region>.amazonaws.com/ai-gateway:<pinned-tag-or-digest>",
      "essential": true,
      "portMappings": [
        { "containerPort": 4000, "protocol": "tcp" }
      ],
      "environment": [
        { "name": "AWS_REGION", "value": "<region>" },
        { "name": "GATEWAY_CONFIG", "value": "/run/config/gateway-config.yaml" }
      ],
      "secrets": [
        {
          "name": "GATEWAY_MASTER_KEY",
          "valueFrom": "<secret-arn>:master_key::"
        }
      ],
      "healthCheck": {
        "command": [
          "CMD-SHELL",
          "curl -fsS http://localhost:4000/health/liveliness || exit 1"
        ],
        "interval": 30,
        "timeout": 5,
        "retries": 3,
        "startPeriod": 60
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-region": "<region>",
          "awslogs-group": "/ecs/ai-gateway",
          "awslogs-stream-prefix": "gateway"
        }
      }
    }
  ]
}

The health endpoint should answer whether the process is alive. A separate readiness check should represent whether the task has loaded its configuration and can accept traffic. Avoid making a provider call on every load-balancer health check; that can create hidden provider traffic and make a transient provider outage look like a crashed container.

Deploy the service

The exact AWS CLI commands depend on how your VPC, load balancer, secrets, and logging are managed. The following sequence shows the dependency order. Replace every placeholder and review the resulting resource ARNs before applying.

# Build and push a pinned image.
docker build -t ai-gateway:<version> .
docker tag ai-gateway:<version> \
  "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/ai-gateway:<version>"

aws ecr get-login-password --region "$AWS_REGION" |
  docker login --username AWS --password-stdin \
  "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com"

docker push \
  "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/ai-gateway:<version>"

# Register the reviewed task definition.
aws ecs register-task-definition \
  --cli-input-json file://task-definition.json

# Create or update the service with your existing networking identifiers.
aws ecs update-service \
  --cluster "$ECS_CLUSTER" \
  --service "$ECS_SERVICE" \
  --task-definition "ai-gateway:<revision>" \
  --desired-count 2 \
  --deployment-configuration "maximumPercent=200,minimumHealthyPercent=100"

Two tasks are useful for basic rolling deployment behavior, but they do not by themselves provide application isolation. A single gateway process can still accept traffic for both keys. Isolation is established by policy, durable budget state, concurrency controls, and tests.

Call the OpenAI-compatible endpoint

Keep application code pointed at the gateway's base URL. The application should not receive AWS credentials or call Bedrock directly.

curl --fail-with-body "$GATEWAY_HOST/v1/chat/completions" \
  -H "Authorization: Bearer $SUMMARIZER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "primary",
    "messages": [
      {
        "role": "user",
        "content": "Return one sentence explaining why request budgets need durable state."
      }
    ],
    "temperature": 0
  }'

Record a request identifier from the gateway response or response headers if available. Correlate it with gateway logs, ECS task logs, and provider-side usage records. Avoid logging full prompts by default; log metadata such as application alias, model alias, status class, latency, retry count, and token counts when those fields are safe and available.

Retries and fallbacks: make the policy explicit

A retry is useful only when the failure is likely to be temporary and another attempt will not create unacceptable load. Configure a small maximum attempt count, a request timeout, and backoff. Do not retry authentication failures, malformed requests, invalid model names, or budget rejections.

A fallback is a routing decision, not a promise that two models are interchangeable. The fallback may have different context limits, latency, safety behavior, or output quality. Document the approved fallback and test the application against its output contract.

retryable:
  - provider_timeout
  - connection_reset
  - throttling
  - selected_5xx

not_retryable:
  - invalid_request
  - authentication_failure
  - authorization_failure
  - budget_exceeded
  - unknown_model

retry:
  max_attempts: 3
  backoff: exponential
  initial_delay_ms: 250
  max_delay_ms: 2000
  request_timeout_seconds: 45

fallback:
  primary: fallback
  only_for:
    - provider_timeout
    - throttling
    - selected_5xx

The field names above are illustrative. The safety property is the important part: finite attempts, bounded waits, a narrow retryable error set, and an explicit fallback map. If both the gateway and the application implement retries, multiply the attempt counts mentally. A gateway limit of three attempts combined with an application limit of three attempts can produce up to nine provider calls for one user action.

Isolation test plan

Run these tests in a non-production environment with test-only keys and a deliberately small allocation. The examples describe expected assertions, not observed results. Mark each test pass or fail from your own logs and response data.

1. Key attribution

Send one request with each key and confirm that logs identify the expected application alias. Then send a request with an unknown key and confirm that it is rejected without a provider call.

curl -sS -o /tmp/unknown-response -w "%{http_code}\n" \
  "$GATEWAY_HOST/v1/chat/completions" \
  -H "Authorization: Bearer test-invalid-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"primary","messages":[{"role":"user","content":"test"}]}"

# Inspect only status and sanitized metadata.
jq '{status, error}' /tmp/unknown-response

Evidence to retain: HTTP status, sanitized gateway log entry, and confirmation that no provider request was emitted for the invalid key.

2. Budget boundary

Use a test key with a small configured allowance. Send requests until the gateway reports the documented budget-exceeded behavior. Then send a request with the support key and verify that it remains eligible.

This test must distinguish a gateway rejection from provider throttling. Compare the response body, gateway status fields, retry count, and provider-side evidence. A budget rejection should not trigger retries.

3. Rate and concurrency isolation

Generate a controlled burst for summarizer while sending a low-rate probe with support-assistant. The success criterion is not that every summarizer request succeeds. The criterion is that the support probe remains within its documented latency and admission policy while the summarizer key is throttled or queued according to its own limits.

# Example only. Use a load tool that you have approved for your environment.
for n in $(seq 1 20); do
  curl -sS "$GATEWAY_HOST/v1/chat/completions" \
    -H "Authorization: Bearer $SUMMARIZER_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model":"primary","messages":[{"role":"user","content":"short test"}]}' \
    > "summarizer-$n.json" &
done
wait

curl --fail-with-body "$GATEWAY_HOST/v1/chat/completions" \
  -H "Authorization: Bearer $SUPPORT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"primary","messages":[{"role":"user","content":"support probe"}]}'

For a meaningful result, collect timestamps, response classes, gateway queue or concurrency metrics, ECS CPU and memory, and provider throttling signals. A single successful probe is not proof of isolation; repeat the test with different burst sizes and task counts.

4. Retry count

Use a controlled fault mechanism that your test environment supports, such as a provider throttle response, a temporary route failure, or a gateway test hook. Do not simulate failure by corrupting production configuration.

Assert that the request produces no more than the configured number of attempts, that backoff is bounded, and that the final response identifies the terminal failure class. Verify that a non-retryable error produces one attempt.

5. Fallback behavior

Force a failure that is explicitly mapped to the fallback route. Confirm that the gateway uses the approved fallback model and that the response contains enough metadata to diagnose the route without exposing secrets. Then force an invalid-request error and confirm that it does not fall through to a different model.

6. Task replacement

Restart one task during a test window. Confirm that the service becomes healthy again, that durable budget state is preserved, and that in-flight requests fail or complete according to the documented behavior. A replacement task starting with a fresh local budget is an isolation defect.

Verification checklist

Use this checklist after deployment. The article does not claim these checks have been run; they are the minimum evidence to collect in the target environment.

  • Container image is pinned by reviewed tag or digest.
  • ECS task reaches healthy status through the load balancer health check.
  • Gateway accepts a valid test key and rejects an invalid key.
  • Each test key maps to the expected application alias.
  • Budget state survives a task replacement.
  • Budget rejection is not retried.
  • Retryable provider failures stop at the configured attempt limit.
  • Backoff and request timeout stay within the documented bounds.
  • Fallback routing occurs only for configured failure classes.
  • A burst from one key does not remove the other key's ability to make an admitted request.
  • Logs contain request metadata but not authorization headers or sensitive prompt content.
  • CloudWatch alarms or equivalent monitoring cover task health, error rate, latency, throttling, and budget events.
  • Provider usage can be reconciled to application identity using gateway records.

Failure cases and how to reason about them

Every application is blocked after one application exceeds budget

This usually indicates a global budget, a shared key, a gateway-wide rate limit, or a shared provider quota rather than a per-application policy. Check the key attached to each request, the budget scope, and whether the limit is applied before or after key resolution.

Budget resets after deployment

The gateway is probably storing state locally or using an ephemeral database. Move budget state to a durable store and test task replacement explicitly. Increasing the desired task count will not fix this problem.

Retries make an outage worse

Check for retries at every layer: client SDK, gateway, load balancer, provider SDK, and queue worker. Reduce the total attempt budget, add exponential backoff, and exclude deterministic errors. A timeout should also include a request deadline so attempts cannot overlap indefinitely.

Fallback hides a configuration error

A fallback map that catches every error can turn invalid credentials, malformed requests, and missing model access into misleading success or inconsistent output. Keep fallback conditions narrow and test non-retryable errors separately.

Support traffic still becomes slow during a summarizer burst

Key-level rate limits may not be enough if all requests share the same worker pool, CPU, memory, connection pool, or provider quota. Add per-key concurrency controls, reserve capacity for latency-sensitive traffic, or split workloads across services when the required isolation is stronger than a shared process can provide.

Logs cannot explain the result

Add structured fields for application alias, key identifier hash, model alias, provider model, attempt number, terminal status, latency, token counts, and request correlation ID. Hash or truncate identifiers where possible. Never solve observability gaps by logging raw secrets or full user content.

Security and operational limits

A gateway key is a credential. Store it like one, rotate it, and give each application a separate value. If a key leaks, revoke only that key where possible. Use TLS at the load balancer and private networking for the tasks. Restrict security groups so only the intended callers can reach the gateway.

Per-key budgets do not prevent a caller from creating many keys unless key issuance is also controlled. They do not replace provider account quotas, organizational budgets, or invoice reconciliation. They also cannot guarantee perfect cost isolation when provider pricing, token accounting, retries, or asynchronous processing are still being finalized.

Model fallback is an application decision. A cheaper or more available model may not preserve output quality, structured-output guarantees, context capacity, or safety behavior. Treat fallback responses as a separately tested contract.

Finally, a shared gateway is a shared failure domain. If one tenant requires hard CPU, memory, network, or data-plane isolation, deploy separate services or accounts. A key boundary is appropriate for many logical workloads; it is not equivalent to a dedicated runtime boundary.

Cleanup procedure

Remove temporary test traffic first, then revoke test keys and delete test secrets. Preserve logs and test evidence for the retention period required by your process. Before deleting infrastructure, identify whether the ECS service, load balancer, log group, ECR repository, database, secrets, or network components are shared by other workloads.

# Review before changing anything.
aws ecs describe-services \
  --cluster "$ECS_CLUSTER" \
  --services "$ECS_SERVICE"

aws ecs list-task-definitions \
  --family-prefix ai-gateway

aws logs describe-log-groups \
  --log-group-name-prefix /ecs/ai-gateway

# Example scale-down for an isolated test service.
aws ecs update-service \
  --cluster "$ECS_CLUSTER" \
  --service "$ECS_SERVICE" \
  --desired-count 0

# Delete only after confirming ownership and retention requirements.
# aws ecs delete-service --cluster "$ECS_CLUSTER" --service "$ECS_SERVICE"
# aws logs delete-log-group --log-group-name /ecs/ai-gateway
# aws ecr batch-delete-image --repository-name ai-gateway --image-ids imageTag=<test-tag>

The deletion commands are intentionally commented out. Destructive cleanup must be performed only after checking resource ownership, dependencies, and retention requirements. Remove unused IAM roles, security groups, secrets, and load-balancer target groups when they are confirmed to be test-only.

What “done” means

The deployment is ready for a controlled pilot when each application has its own key, budget state is durable, retries are finite and observable, fallbacks are explicit, and the isolation tests produce evidence that can be reviewed later. The strongest result is not a successful chat response. It is a predictable response under budget exhaustion, provider failure, task replacement, and competing traffic.

Keep the gateway configuration versioned, keep secrets outside source control, pin the runtime image, and rerun the isolation suite whenever you change gateway versions, provider models, retry settings, budget storage, or ECS capacity.

We publish what works for us—and implement the same solutions for your business. We design AI automation, Telegram bots, chats, and AI agents for real-world processes. Discuss your project →