← All articles

API Gateway vs. Service Mesh: A Practical Guide for Architects

API Gateway vs. Service Mesh: A Practical Guide for Architects

Hands connecting network cables at edge cluster

An API gateway handles external (north–south) traffic — the requests your clients, mobile apps, and third-party consumers send into your system. A service mesh handles internal (east–west) traffic — the service-to-service calls that happen once a request is already inside your cluster. Most production Kubernetes environments eventually need both, but they solve different problems and you rarely need to deploy them at the same time.

  • API gateway: sits at the edge, authenticates external callers, enforces rate limits, routes requests, and exposes a clean API surface to the outside world.
  • Service mesh: runs as a sidecar proxy (or eBPF kernel layer) alongside every service, securing and observing every internal call with mTLS, retries, circuit breaking, and distributed tracing.
  • Coexistence: gateways and meshes are complementary, not competing — a gateway terminates external TLS and enforces API policies; the mesh picks up from there and secures every hop inside the cluster.

For architects: if you have fewer than a dozen services and no zero-trust requirement, start with a gateway. Add a mesh when service-to-service observability, mTLS, or fine-grained traffic control becomes a real operational need, not a theoretical one.

Pro Tip: Don’t let the feature overlap between gateways and meshes (both can do rate limiting and circuit breaking) push you into duplicating policy enforcement. Assign each concern to one layer and document it — ambiguity here causes incidents.

Key Takeaways

An API gateway and a service mesh solve different problems at different layers — deploying both deliberately, with clear policy ownership at each layer, is the architecture that scales and satisfies compliance requirements.

Point Details
Traffic direction defines the split API gateways own north–south (external) traffic; service meshes own east–west (internal) traffic.
Avoid duplicating policies Both layers can enforce rate limits and circuit breakers — assign each policy to one layer and document it.
Sequence the rollout Deploy the gateway first, validate it, then add the mesh incrementally by namespace with latency baselines at each stage.
Measure sidecar overhead Sidecar proxies add measurable memory and latency per pod; test in staging before cluster-wide rollout and evaluate eBPF alternatives for latency-sensitive workloads.
Jundago for regulated environments Jundago centralizes API lifecycle, policy enforcement, and compliance evidence across gateway and mesh telemetry for HIPAA, PCI DSS, and similar frameworks.

Table of Contents

How do API gateways and service meshes compare?

The table below maps the eight dimensions that actually drive architectural decisions. Neither tool is universally “better” — the right answer depends on where the traffic originates and what you need to do with it.

Dimension API Gateway Service Mesh
Traffic direction North–south (client to service) East–west (service to service)
Architectural position Edge / cluster ingress Sidecar proxy or eBPF layer per service
Primary responsibilities Routing, auth (OAuth/JWT), rate limiting, caching, WAF, analytics mTLS, retries, circuit breaking, service discovery, load balancing
Observability API-level metrics, access logs, request tracing at ingress Per-service distributed tracing, golden-signal metrics, full mesh telemetry
Deployment model Single (or HA pair) gateway process; Kubernetes Ingress or Gateway API resource Sidecar injected into every pod; control plane manages config
Performance/latency Minimal overhead at edge; one extra hop per external request Per-request sidecar hop adds measurable latency; eBPF reduces this
Protocol support HTTP/2, gRPC, WebSocket, REST, GraphQL HTTP/2, gRPC, TCP; protocol-aware at L7
Best for Public APIs, partner integrations, developer portals, API monetization Zero-trust internal security, canary releases, resilience patterns

A few clarifications worth calling out:

  • Observability overlap: gateways give you ingress-level traces; meshes give you the full call graph. You need both to correlate an external request with every downstream service it touched.
  • Policy duplication risk: both layers can enforce rate limits and circuit breakers. Pick one layer per policy type and stick to it — running the same rule in two places creates inconsistency and debugging pain.
  • Gateway API convergence: the Kubernetes Gateway API standard is narrowing some gaps between ingress controllers and service mesh data planes, so watch that space when planning long-term infrastructure.

What does an API gateway actually do?

An API gateway is the single entry point for all external traffic hitting your services. It sits at the cluster edge (or in front of it), terminates TLS, and applies API-level policies before a request ever reaches a backend service. Think of it as the bouncer, the traffic cop, and the analytics collector rolled into one process.

API gateways handle north–south traffic and cover routing, authentication, rate limiting, caching, and API analytics as core responsibilities — not optional add-ons.

Core responsibilities:

  • Request routing and transformation: path-based and header-based routing, request/response rewriting, protocol translation (REST to gRPC, for example).
  • Authentication and authorization: OAuth 2.0, JWT validation, API key management, OIDC integration. External callers should never reach a backend service without passing through this layer.
  • Rate limiting and throttling: per-client, per-route, or global limits that protect backend services from traffic spikes and abuse.
  • Caching: response caching at the edge reduces backend load for read-heavy endpoints.
  • WAF and policy enforcement: block malicious payloads, enforce CORS, apply IP allowlists/denylists.
  • API analytics and monetization: usage metrics, quota tracking, and billing hooks for developer portals and partner APIs.

In Kubernetes, an API gateway typically runs as an Ingress controller or as a Gateway API implementation. The Kubernetes Gateway API is worth understanding here: it is a more expressive, role-oriented replacement for the older Ingress resource, and several gateway implementations (including Envoy-based ones) now support it natively. For teams building on Kubernetes, Gateway API is the direction the community is moving, and designing around it now avoids a painful migration later.

What does a service mesh actually do?

A service mesh is an infrastructure layer that manages every service-to-service call inside your cluster, without requiring any changes to application code. That last part matters: the mesh injects a sidecar proxy (typically Envoy) into each pod, and that proxy intercepts all inbound and outbound traffic transparently. The application thinks it is talking directly to another service; the proxy handles encryption, retries, and telemetry behind the scenes.

A service mesh provides mTLS, distributed tracing, fine-grained traffic control, and observability without modifying application code — the control plane pushes configuration to each sidecar, and the data plane (the proxies) enforces it.

Core responsibilities:

  • Mutual TLS (mTLS): every service-to-service call is encrypted and both sides are authenticated. This is the foundation of zero-trust networking inside a cluster.
  • Retries and circuit breaking: automatic retry logic and circuit breakers prevent cascading failures without any code changes in the services themselves.
  • Service discovery and load balancing: the mesh knows which instances of a service are healthy and routes traffic accordingly, often with more sophisticated algorithms than a basic Kubernetes Service.
  • Distributed tracing and metrics: every request generates spans that feed into tools like Jaeger or Zipkin, giving you a full call graph across dozens of services.
  • Fine-grained traffic control: canary deployments, A/B testing, traffic mirroring, and weighted routing — all configurable through the mesh control plane.

A service mesh shifts security and resilience from application code into infrastructure. Teams that previously embedded retry logic, circuit breakers, and TLS configuration in every service library can offload all of it to the mesh — and get consistent behavior across every language and framework in the cluster.

Pro Tip: Before committing to a sidecar-based mesh in production, measure the memory and latency overhead of the sidecar proxy in a staging environment that mirrors your production pod density. For latency-sensitive workloads, evaluate sidecar-less alternatives that use eBPF at the kernel level instead of a per-pod proxy.

What are the four key differences between a gateway and a mesh?

The north–south vs. east–west framing is useful but incomplete. Here are the four differences that actually shape architectural decisions.

  1. Communication direction and scope. A gateway processes traffic that originates outside your system — a mobile client calling /api/orders, a partner consuming a webhook, a developer testing an endpoint. A mesh processes traffic that originates inside your system — the order service calling the inventory service, the payment service calling the fraud-detection service. These are fundamentally different trust boundaries. External callers are untrusted by default; internal services in a mesh can be mutually authenticated.

  2. Position in the stack. The gateway lives at the perimeter. It is the first thing an external request hits, and it is the last thing that sees the response before it leaves the cluster. The mesh lives everywhere inside the cluster simultaneously — every pod has a proxy, and the control plane coordinates them all. This means a gateway failure affects all external traffic at once, while a mesh failure (or misconfiguration) can be scoped to specific services or namespaces.

  3. Observability and security focus. Gateways give you API-level visibility: which clients are calling which endpoints, how often, with what response codes. Meshes give you service-level visibility: which services are calling which other services, what the latency distribution looks like, where retries are happening. For compliance and incident response, you need both layers. A gateway log tells you a request came in; a mesh trace tells you what happened to it afterward.

  4. Deployment and management model. A gateway is typically one (or a few, for HA) processes you manage explicitly. A mesh is a distributed system — a control plane plus a sidecar in every pod. The operational surface is orders of magnitude larger. Both layers can enforce policies like rate limiting and circuit breaking, which creates a real risk of duplication. Assign each policy type to exactly one layer and document the decision.

Where overlap exists, the practical rule is: enforce API-consumer-facing policies (rate limits, auth, quotas) at the gateway; enforce service-to-service resilience and security policies (mTLS, retries, circuit breaking) at the mesh. Crossing those lines without a deliberate reason adds complexity without adding safety.

Pros and cons of each approach

API gateway strengths and trade-offs

Pros:

  • Centralizes the entire external API surface in one place — easier to audit, version, and document.
  • Handles authentication for all external callers before a request reaches any backend, reducing the attack surface significantly.
  • Supports developer portals, API monetization, and partner onboarding workflows that a mesh has no concept of.
  • Operationally simpler than a mesh: one component to deploy, configure, and monitor.

Cons:

  • A misconfigured or overloaded gateway becomes a single point of failure for all external traffic. High-availability deployment and careful capacity planning are non-negotiable.
  • Plugin ecosystems (Kong plugins, for example) can grow into a sprawling, hard-to-audit security surface if teams add capabilities without governance.
  • Limited visibility into what happens after a request passes through the gateway — you see the entry point, not the full call chain.

Service mesh strengths and trade-offs

Pros:

  • mTLS across every service-to-service call with no application code changes — this is the most practical path to zero-trust networking inside a cluster.
  • Distributed tracing and golden-signal metrics for every service, not just the ones your developers remembered to instrument.
  • Fine-grained traffic control (canary, A/B, mirroring) that lets you ship changes safely without deploying new gateway rules.

Cons:

  • Operational complexity is real. A mesh control plane (especially Istio) has a steep learning curve, and misconfigured policies can silently drop traffic.
  • Sidecar proxies introduce measurable memory and latency overhead — this compounds as pod count grows. A cluster with 200 pods has 200 sidecar processes consuming memory and adding per-request latency.
  • Debugging mesh-related issues (certificate rotation failures, policy conflicts, proxy version mismatches) requires specialized knowledge most teams build slowly.

Pro Tip: Run a sidecar overhead test in staging before rolling out a mesh cluster-wide. Measure memory per pod and p99 latency with and without the sidecar under realistic load. The numbers will tell you whether eBPF-based alternatives are worth evaluating for your workload.

How do API gateways and service meshes work together?

The most common production pattern is straightforward: the gateway handles everything at the edge, and the mesh handles everything inside. But the details of how they hand off traffic, where TLS terminates, and how observability data correlates across both layers are where architects spend real time.

Three integration patterns worth knowing:

  • Gateway at edge, mesh for internal traffic (most common). The gateway terminates external TLS, authenticates the caller, and forwards the request to a backend service. Inside the cluster, the mesh proxy intercepts that forwarded request, establishes mTLS to the target service, and handles retries and tracing. The gateway and mesh operate independently — they share no configuration, but their logs and traces need to be correlated using a common trace ID propagated in request headers (W3C Trace Context or B3 headers work well here).

  • Gateway delegating internal routing to the mesh. The gateway routes to a mesh-managed virtual service rather than directly to a pod IP. This lets the mesh control canary splits and traffic weights while the gateway focuses on auth and rate limiting. The boundary is clean: the gateway owns the external contract; the mesh owns the internal routing logic.

  • Hybrid API management plus mesh. For regulated enterprises or large platform teams, a full API management layer (lifecycle, versioning, developer portal, monetization) sits above the gateway, which in turn sits above the mesh. This is the architecture that makes sense when you have dozens of teams publishing APIs and need governance across all of them.

Observability correlation is the hardest part. Gateway logs tell you a request arrived at 14:03:22 with a 503 response. Mesh traces tell you the downstream service returned a 503 because the database connection pool was exhausted. Without a shared trace ID flowing through both layers, you are correlating two separate data sources by timestamp — which works until it doesn’t.

Pro Tip: Standardize on W3C Trace Context headers at the gateway and configure your mesh (Istio, Linkerd) to propagate them. This single decision makes post-incident analysis dramatically faster and is required for any meaningful compliance audit trail.

The Kubernetes Gateway API standard is also worth watching here — it is designed to express both ingress and mesh routing in a unified resource model, which could eventually reduce the configuration gap between the two layers.

How do you decide: gateway, mesh, or both?

Work through these questions with your team before committing to either layer.

  1. How large is your external API surface? If you expose APIs to external clients, partners, or a developer portal, you need a gateway. No mesh replaces this.
  2. Do you have a zero-trust requirement for internal traffic? If compliance (HIPAA, PCI DSS) or security policy requires encrypted, authenticated service-to-service calls, a mesh is the practical path. Implementing mTLS manually in every service is fragile.
  3. How mature is your observability practice? A mesh generates a large volume of telemetry. If you don’t have a metrics and tracing stack ready to consume it, the mesh’s observability value is unrealized and the overhead is pure cost.
  4. What is your team’s operational maturity with Kubernetes? Istio in particular requires solid Kubernetes knowledge to operate safely. Linkerd is significantly simpler but has a narrower feature set.
  5. Are you already on Kubernetes? Both gateways and meshes are designed for Kubernetes-first environments. Running either on bare VMs or legacy infrastructure is possible but adds friction.
Scenario Recommended approach
Small service count, external API needed Gateway only — add mesh when service count and complexity grow
Many services, zero-trust required, mature Kubernetes team Both — gateway at edge, mesh for internal traffic
Many services, limited ops capacity Gateway now, lightweight mesh (Linkerd) later
Regulated environment (HIPAA, PCI DSS) Both — mesh for mTLS evidence, gateway for access control audit logs
Kubernetes-native, Gateway API adoption Evaluate Gateway API implementations that support both ingress and mesh routing

Concrete next steps for rollout:

  1. Deploy the gateway first and validate external auth, rate limiting, and routing in staging.
  2. Instrument your services with distributed tracing before adding a mesh — this gives you a baseline.
  3. Roll out the mesh namespace by namespace using a canary approach, not cluster-wide on day one.
  4. Measure p99 latency and memory per pod before and after sidecar injection at each stage.
  5. Validate that trace IDs propagate end-to-end before declaring the integration production-ready.

Pro Tip: Treat the mesh rollout as a phased infrastructure change, not a feature flag. Namespace-by-namespace adoption lets you catch policy misconfigurations before they affect the whole cluster.

Which tools implement gateways and meshes?

These are the implementations you will encounter most often in production Kubernetes environments.

  • Istio is the most feature-complete service mesh available. It uses Envoy as its data plane sidecar and provides mTLS, fine-grained traffic management, rich observability, and a powerful (if complex) policy engine. Istio’s control plane has matured significantly, but its resource consumption and configuration surface remain a real operational consideration — sidecar memory overhead is a documented pitfall worth measuring before cluster-wide rollout.

  • Envoy is the proxy that powers much of the cloud-native networking ecosystem. It runs as the data plane in Istio, as the core of several API gateways, and as a standalone edge proxy. Understanding Envoy’s filter chain and xDS API gives you transferable knowledge across multiple tools. Many service mesh implementations use Envoy as their data plane precisely because of its extensibility and performance.

  • Kong is one of the most widely deployed API gateways, available as open source and as an enterprise platform. It runs on top of Nginx/OpenResty (Kong Gateway) or as a Kubernetes-native Ingress controller. Kong’s plugin ecosystem covers authentication, rate limiting, logging, and transformations. It also has a mesh product (Kuma), though most teams use Kong specifically for its gateway capabilities.

  • Linkerd is the CNCF-graduated lightweight alternative to Istio. It uses a Rust-based micro-proxy (not Envoy) that has a notably smaller memory footprint and simpler operational model. Linkerd covers the core mesh use cases — mTLS, retries, circuit breaking, observability — without Istio’s full traffic management feature set. For teams that need zero-trust internal networking without the Istio learning curve, Linkerd is worth serious evaluation.

  • Ambassador (Emissary-Ingress) is an Envoy-based Kubernetes-native API gateway and ingress controller. It maps Kubernetes resources directly to Envoy configuration, making it a natural fit for teams already comfortable with Envoy semantics. Ambassador handles north–south routing, auth, rate limiting, and TLS termination, and it integrates well with service meshes running Envoy in the data plane.

When choosing between a lightweight mesh like Linkerd and a full-featured control plane like Istio, the deciding factor is usually operational capacity, not features. Istio earns its complexity when you need advanced traffic management (fault injection, mirroring, multi-cluster) or a rich policy engine.

Operational and compliance checklist for regulated environments

Regulated industries — healthcare (HIPAA), finance (PCI DSS), manufacturing (IEC 62443) — have specific requirements that go beyond what a gateway or mesh provides out of the box. The architecture needs to be designed for auditability from the start.

Compliance checklist:

  • Access controls: implement RBAC at the gateway (who can call which APIs) and ABAC at the mesh level (which services can communicate with which other services). Both layers need to be configured and audited independently.
  • Audit logging retention: gateway access logs and mesh telemetry must be retained for the period your compliance framework requires (HIPAA: six years; PCI DSS: one year minimum with three months immediately available). Configure log forwarding to a tamper-evident store from day one.
  • Encryption in transit: mTLS inside the mesh covers service-to-service traffic; TLS termination at the gateway covers external traffic. Verify that no unencrypted paths exist between any two services — mesh policy can enforce this automatically.
  • Policy enforcement points: document which policies live at the gateway (external auth, rate limits, WAF) and which live at the mesh (internal mTLS, retry budgets, circuit breakers). Auditors will ask.

Operational considerations:

  • Size sidecar resource requests and limits explicitly — don’t rely on defaults. Measure actual memory consumption per pod in staging under production-representative load.
  • Set tracing and metrics retention policies before going live. Mesh telemetry volume grows with service count and request rate.
  • Write incident playbooks for the most likely failure modes: gateway certificate expiry, mesh control plane unavailability, sidecar injection failures.
  • Monitor p99 latency at both the gateway and mesh layers continuously. A latency regression at the mesh layer often surfaces before it appears in application-level SLAs.

Jundago’s API lifecycle and governance platform centralizes policy management, automates compliance checks, and correlates gateway and mesh telemetry into a unified audit trail — which matters when you need to produce evidence for a HIPAA or PCI DSS audit without manually assembling logs from three different systems.

Pro Tip: For regulated environments, treat your gateway and mesh configuration as code in version control. Every policy change should go through a review and approval workflow, and the diff should be part of your audit evidence.

Operational and compliance checklist for regulated environments — overview diagram

What most teams get wrong about this choice

The conventional wisdom — “use a gateway for north–south, a mesh for east–west” — is correct as far as it goes. What it misses is the sequencing problem. Most teams I have seen deploy both layers simultaneously on a tight deadline, then spend months untangling duplicated policies, inconsistent observability, and undocumented ownership of shared concerns like rate limiting.

The smarter path is deliberate sequencing. Start with the gateway. Get your external API surface clean, authenticated, and observable. Then, when service-to-service complexity actually demands it (not when it theoretically might), add the mesh incrementally. The teams that do this well treat the mesh rollout as a separate infrastructure project with its own success metrics — p99 latency baseline, mTLS coverage percentage, trace propagation validation — rather than bolting it onto an existing delivery sprint.

There is also a tendency to over-index on Istio because it is the most-discussed option. For most teams, Linkerd’s operational simplicity is a better fit until the use cases that justify Istio’s complexity actually materialize. Choosing Istio because it has more features is like choosing a full enterprise database for a service that runs three queries — the features are real, but the operational cost is also real, and it compounds.

The compliance angle is where the combined architecture genuinely earns its keep. A gateway log plus a mesh trace plus a governance platform that correlates them is a materially stronger audit artifact than either layer alone. For regulated enterprises, that combination is not optional — it is the architecture.

Jundago fits where gateways and meshes leave off

Gateways and meshes handle traffic. What they don’t handle is the API lifecycle: generating compliant APIs from intent, enforcing schema governance across teams, automating policy checks before deployment, and producing the audit evidence your compliance team actually needs.

Jundago

Jundago is built for regulated enterprises that need all of that in one place. API Studio generates REST, GraphQL, gRPC, and SOAP APIs from natural language intent. Command Center governs every API across AWS, Azure, GCP, and Oracle Cloud with RBAC and ABAC controls built in. The platform correlates gateway logs and mesh telemetry into a unified compliance record — so when a HIPAA or PCI DSS auditor asks for evidence, you are not assembling it manually from three separate systems. For teams running Kubernetes-based microservices in regulated industries, Jundago sits above the gateway and mesh layers and ties them together into a governed, auditable API platform. Request a demo at Jundago to see how it fits your architecture.

Sources