Service Mesh: Hardening inter-service communication security in Kubernetes

TL;DR

This article explores why communication between services inside a Kubernetes cluster is not inherently secure, how plaintext inter-pod traffic can be intercepted, and how TLS and mutual TLS (mTLS) help protect against these threats. Building onto this concept, I will explain a way of minimizing the complexity of implementing and maintaining a TLS-hardened system using a service mesh.

It is important to note before we begin that the attack vectors of different configurations and deployments are not the same, and the scope of this article may not apply to your system. A completely air-gapped or offline system is, of course, not as susceptible to the kinds of attacks as an online and publicly accessible system.

We begin with a practical introduction to TLS and mTLS, including how secure connections are established and why authentication between services matters in distributed systems. From there, we look at the operational complexity introduced by managing certificates and trust across many services, and how service mesh architectures address this problem by transparently handling secure service-to-service communication.

Alongside the theoretical concepts, the article includes practical demonstrations showing:

• plaintext packet sniffing inside a Kubernetes deployment,
• the effect of enabling TLS/mTLS,
• and how a service mesh can simplify secure communication between services.

Introduction

If you are familiar with deploying applications with more than one parts which need to communicate together, you know the importance of communication between services. This article will focus on hardening security around inter-service communication, specifically for Kubernetes deployments. Therefore, it is assumed you are familiar with the basics of Kubernetes such as pods, containers, and using the internal DNS for addressing a specific service.

One of the great selling features for Kubernetes is that regardless of on which node a pod is running, by default, pods can reach each other over the cluster network. Kubernetes deployments are often designed to handle running multiple instances of the same pod (even with autoscaling as an option), and, therefore, inter-service communication leverages an internal DNS service which means you do not have to consider ephemeral IP addresses for services—only the automatically assigned service or pod DNS name. While this kind of communication is a great feature, it is a common misconception that communication between pods is innately secure since it is running within a closed system. However, a single compromised pod may allow an attacker to observe inter-pod communication, depending on network access and configuration. This can include credentials or sensitive content.

The Problem: Unencrypted pod-to-pod communication

Plaintext communication between pods is the natural first step to building an application with more than one running parts—even if it is as simple as a monolith application communicating with its database. The problem with this is that TCP, which underlies most service-to-service communication, does not provide encryption by default, and vulnerabilities in an application’s design, implementation or dependencies can open an attack vector for malicious actors to exploit.

A common attack for gaining access to network traffic is packet sniffing: capturing packets sent between applications. A common tool for this is the Linux command-line tool “tcpdump”. This exploit generally requires access to the computer where the applications being sniffed are running.

Other attack types can include impersonation via a man-in-the-middle attack where communication is intercepted and relayed between the two parties allowing the attacker to obtain private data as well as alter requests in transit. This can, for example, happen with ARP spoofing (impersonating another device to have traffic routed through the compromised device) and only requires access to the same network.

It is, of course, important mention before we go any further that a system’s specific attack vectors are strongly predicated by its associated threat model. An on-premise, fully air-gapped deployment will not have the same kind of risks as a deployment which is readily accessible via the public internet.

Regardless of how secure or private a network is, it is always best practice to not assume that another device or application is trusted.

Demo: Plaintext packet sniffing via vulnerable pod

To demonstrate a specific attack, I have put together a sample Kubernetes deployment made up of 3 simple parts:

• Backend: a slim HTTP API server
• Frontend: a HTTP client (running as a Python app) calling the backend regularly, providing authorization credentials
• “Vulnerable app”: a creatively named app which can be compromised

This demo takes vulnerabilities to the next level, but that is to keep the complexity of attack low. Normally, the assumptions and decisions taken would be different to close these obvious security holes, but the concepts still apply.

The specific vulnerabilities include are as follows:

• Deployment parameters:
  • Network mode “host”: runs the pod on the same network as the node/host. This is generally not recommended to isolate each service from each other.
• Application:
  • System command injection: an endpoint which has a vulnerability in its design allowing an attacker to run commands on the host system (container)

Combining these vulnerabilities, an attacker is able to get a foothold into an environment where communication between other pods can be compromised. Once inside the compromised pod, tcpdump can be used to retrieve the entire request being sent from the client pod to the server pod looking something like this:

15:04:47.403700 veth0f831cde In  IP 10.244.0.5.39062 > 10.96.60.56.5000: Flags [P.], seq 0:189, ack 1, win 502, options [nop,nop,TS val 1628014161 ecr 1042053975], length 189
E...A-@.@..I
...
`<8.....9..........Rt.....
a	.Q>.{WGET /data HTTP/1.1
Host: backend:5000
User-Agent: python-requests/2.33.1
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Authorization: Bearer SUPER_SECRET_TOKEN

Likewise, the response from the server pod to the client pod:

15:13:35.425171 veth0f831cde Out IP 10.96.60.56.5000 > 10.244.0.5.36608: Flags [P.], seq 1:167, ack 190, win 508, options [nop,nop,TS val 1042581997 ecr 1628542181], length 166
E.....@.?...
`<8
.......:O.y..-.....R].....
>$..a...HTTP/1.1 200 OK
Server: Werkzeug/3.1.8 Python/3.11.15
Date: Wed, 20 May 2026 15:13:35 GMT
Content-Type: application/json
Content-Length: 52
Connection: close

15:13:35.425237 vethf040d456 In  IP 10.244.0.6.5000 > 10.244.0.5.36608: Flags [P.], seq 167:219, ack 190, win 508, options [nop,nop,TS val 1042581997 ecr 1628542183], length 52
E..h..@.@.N.
...
.......:O....-......M.....
>$..a...{"auth":"Bearer SUPER_SECRET_TOKEN","message":"ok"}

As you can see, secret authorization headers and sensitive response bodies are able to be sniffed.

Transport-layer security

The immediate answer to the problem I outlined is encryption: transport-layer encryption (TLS). TLS uses a combination of symmetric and asymmetric key cryptography to allow parties that have never communicated before to securely establish a shared, session-specific encryption key.

In simple terms, the client reaches out to the server and says hello and presents its available encryption methods. The server answers hello and presents its certificate and chosen encryption method. When the client receives this, it verifies that it trusts the certificate: among other things, it is not expired, it is issued by an authority which the client trusts, and it belongs to the server the client intended to call. Finally, an encryption key must be agreed on, but if this is simply sent as plaintext over the network, an attacker could just get a hold of it and defeat the purpose of encryption. While there are multiple ways of carrying this out, the core concept is each party generates a private value and shares a corresponding public value. Using these, both sides independently derive the same shared secret without ever transmitting it directly.

Subsequently, all communication is encrypted using this new key, and the receiving party uses the same key to decrypt it. This generally solves the problem of an attacker intercepting communication in transit as they lack the necessary private key parts to decrypt the messages.

This is not a not a crypto article, so I promise to end this TLS crash course right after I explain this last part.

We’ve covered TLS and how it solves vulnerabilities in plaintext communication, but there is still one important element in securing communication between services: authentication. In its simplest form, TLS only helps one party (the client) in ensuring that it is communicating with whom it intended to and not an impostor. However, in a distributed deployment, when we are securing communication between services, we are definitely interested in verifying in each service that the other service it is communicating with is the intended service and not an impostor. This is where “mutual TLS” (mTLS) comes in. What mTLS adds is that the client must also present its certificate when the server requests it, and the server performs the same verification on the client’s certificate as the client normally performs on the server’s certificate.

Demo: TLS-encrypted packet sniffing rendered useless

Now we will build on the demo outlined earlier in this article of our Kubernetes deployment with communication between pods alongside a third vulnerable pod where an attacker has managed to exfiltrate sensitive data by running the tcpdump tool.

What we add is TLS to the HTTP communication between the client and server pods, and the outcome immediately apparent. Running the same tcpdump tool yields the expected results:

15:32:17.258077 vetha2c792fb In  IP 10.244.0.8.37958 > 10.244.0.9.5000: Flags [P.], seq 1876:2814, ack 1607, win 708, options [nop,nop,TS val 302429824 ecr 519595561], length 938
E.....@.@..w
...
..	.F...3T[...............
......f).......\...o.....D.K.[..(..H.h....f........
X.C.....e..}c...e. . ..K....n..]c..a.7.^..He%...@W%....36..q@\.d.._......Doufme......j....'..&Tr......}....
....Q.s...ie,.V.}S].!....|...uQ.).A...w...W...RWM.O*..bq........q0.nB...$..
q.L.*..I..*@.....%=.I..{w.}.5...AM.Q...q..m.gF|s./..G..V......)..Y..o...su..>......7.......!({.X.u|M.T.Gt...."	.u...@0.......td....q....I...9.!..=......#..3&..Y..u......Z5.7.;.....Zm...B"..&......
.t.t.1...-e.,...%...n.... OJR...!...d...).}U.....}nbIhD.....9A.]....O./".
.[.A...uKE..<...UR...r.<...ztBQ....&.<.....fM.C..{v.....r.1.T.......HX.	....k.	......8.KC5....=..."m<...."...a..=F.Lg..j..vZuw.O......i.......U.....o..f9...}.X$.!&.*......j...Z<)..^.-.MI.Q.	  ./x.@...6-.....-.... .q......4.4....E.Q...9.;xoq...wuEyt&...@..!A[...e...f......~.....C]i .....%..|...t..%./|n.D........{.0......A.U.Q..M....`....k...$R..l....p..qE..w...g....#D...........r
..~..w.a..a.....?.ZL.	e...6.]..:G0.......G...>....f...N
Y.$

Both the request and response are entirely encrypted: headers and bodies.

While this does not solve the vulnerability which allowed the attacker the foothold in the first place, it does limit how much sensitive information can be exfiltrated.

Next problem: TLS introduces complexity

Compared to the original architecture of the deployment, introducing mTLS between services simply adds a degree of complexity. This is because of the dynamic of trust required to use TLS.

Much of the overhead added by TLS is in the infrastructure needed to provision and maintain certificates and establish the chain of trust. While it is not a great problem to implement for a handful of services, implementing in a great number of inter-connected services becomes a pain point which is not made any easier with the nature of most Kubernetes environments where services scale dynamically and instances are short-lived.

Just to name a few, the following are necessary components for each service running mTLS:

• Add a certificate authority to its trust chain
• Get issued a certificate and private key
• Rotate the certificate as it expires
• Distribute and maintain trust between services
• Design services such that each ephemeral instance can be individually identified and trusted by all other services

Yes, there likely exist helper libraries to make this process easier but there is always the risk of misconfiguration and mismatch between service TLS configuration. Each language and framework introduces its own TLS configuration model, increasing the likelihood of inconsistent or incorrect implementations. Some services you deploy may not even natively support TLS which only adds to the complexity of implementing TLS for them. In some cases, it would be necessary to write a dedicated wrapper which handles TLS termination for communication to and from the service.

The Solution: Service mesh

As the title of the section very neatly gives it away, a solution to the problem outlined in this article is the use of a service mesh. A service mesh is a separation of concerns to remove the networking concerns out of our services and consolidate the configuration for enabling mTLS between services.

Most service mesh implementations rely on the concept of sidecars in Kubernetes (a sidecar in Kubernetes is an extra container running in a pod which shares the same file system and networking of the other containers in the pod).

From a birds-eye view, what happens when using a service mesh is a redirection of direct communication between two services to passing through a proxy sidecar in each service’s pod. This way, pods think they are communicating directly with each other when, in reality, each is talking to its proxy sidecar which is applying TLS termination and routing the traffic to the other pod’s proxy sidecar which performs the inverse operation.

 

Each service communicates as if it were using plain TCP (often HTTP but not limited to it), without implementing TLS itself. The sidecar proxy captures this traffic, establishes a TLS connection with the other service’s sidecar proxy, and sends the original request over this secure channel. The receiving sidecar proxy then receives the TLS request, converts it back to its original unencrypted form, and sends the request to the receiving service. This happens transparently through network interception (typically via iptables rules) configured by the service mesh, rather than changes to application code.

The result is that unencrypted communication is limited to within the pod, while all inter-pod traffic (which may need to pass between physical nodes) is encrypted. Additionally, all configuration, certificate rotation, and certificate authority trust setup is centralized to the configuration of the service mesh. TLS is no longer an application concern; it is enforced by the platform. Besides establishing a secure connection with TLS, the service mesh applies mTLS, hardening the security of the communication even further.

Demo: Enabling service mesh in K8s

In this last demo section, I am going to show you the general idea of what is required to set up a service mesh in the Kubernetes deployment we have used as a demo throughout this article. For this specific demo, I have chosen to use Istio as the implementation of service mesh, but there are other options out there.

While production deployments often involve additional policy, observability, and certificate configuration, the core workload changes required to enable service mesh are often surprisingly small. The changes to the individual pod deployment in the current demo are quite minimal: a single label added to indicate to Istio that it needs to inject a sidecar (deployment spec redacted for brevity):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ...
  namespace: mesh-demo
  labels:
    ...
spec:
  replicas: 1
  selector:
    ...
  template:
    metadata:
      labels:
        ...
        # The following line is added to enable service mesh sidecar injection
        sidecar.istio.io/inject: "true"
    ...

Then there is a security policy to ensure mTLS authentication between peers:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: mesh-demo-strict-mtls
  namespace: mesh-demo
spec:
  mtls:
    mode: STRICT

That’s about it for deployment specification changes. It is then required to install the Istio service mesh helm chart (the istiocli tool can be used alternatively), and roll out the changes to the existing pods. Once that has happened, communication between pods is redirected between proxy sidecars.

There is admittedly more configuration required if you need to change the CA or trust root which is utilized for this, but the TLS used internally in the service mesh is not the same as used for publicly accessible user-facing entrypoints where the certificate presented should be trusted through the installed trust store in their browser or operating system.

Conclusion

Communication between services inside a Kubernetes cluster is not inherently secure simply because it happens on an internal network. Without encryption and authentication, inter-pod traffic may be observable or manipulable by an attacker who gains sufficient access to the cluster or surrounding network infrastructure.

TLS and mutual TLS solve this problem by encrypting communication and allowing services to verify each other’s identity. However, implementing and maintaining mTLS across many distributed services introduces operational complexity in the form of certificate management, trust distribution, rotation, and consistent configuration between services.

A service mesh addresses much of this complexity by moving networking concerns out of the applications themselves and into dedicated infrastructure components. TLS stops being an application concern and becomes a platform concern. By transparently handling mTLS between services, a service mesh allows applications to communicate securely without each service individually implementing and maintaining TLS logic.

At the same time, a service mesh is not without trade-offs. It introduces additional infrastructure, operational overhead, resource consumption, and debugging complexity. In some environments, simpler approaches such as node-to-node encryption using technologies like WireGuard or IPSec may provide sufficient protection with significantly less operational effort, though without the stronger workload identity guarantees provided by mTLS.

Finally, it is important to note that service mesh is not a concept exclusive to Kubernetes. While Kubernetes is a particularly natural environment for implementing sidecar-based networking patterns, the underlying idea of separating communication concerns from application logic can be applied in many types of distributed systems and deployment environments.