How to intercept your Kubernetes cluster traffic to your local environment with Telepresence
Jan 29, 2023
8 min read
Today, I want to tell you about a powerful tool that I use at my job to intercept Kubernetes traffic to my local environment, so I can test or debug my code locally when I need to: Telepresence.
I want to honorably mention some of my colleagues like Javi, Christian and Rubén. They have implemented this tool in our company and today this article is possible thanks that they have shared their knowledge with me.
What is Telepresence?
Telepresence is an open source tool that allows you to set up remote development environments for Kubernetes, where you can still use all of your favorite local tools like IDEs, debuggers, and profilers. Telepresence allows you to:
- Connect to a test environment and directly access the pods with DNS resolution (without explicitly having to port-forward any of its dependencies).
- Intercept traffic temporarily, replacing a deployment with an instance running locally in your machine so that you can debug internal calls from inside the cluster in your developer machine.
How can Telepresence help me?
Sometimes, when you are working with environments deployed with Kubernetes, you need to:
- Debug or test a component in your local environment. This is difficult sometimes because you need a lot of required dependencies: data sources, other services, environment variables, etc.
- Know why a component (as part of a complex flow) is failing when deployed but it works as expected locally.
You might be using
kubectl port-forward along with
kubefwd for that, and these are really good options for connecting purpose, but not for traffic interception, as their set up is very tedious and involves a lot of manual steps.
However, Telepresence helps you to:
- Accelerate the inner dev loop. You can make changes to your service as if you're developing locally, without having to run all the dependencies on your local machine.
- Shift-left testing. It lets you connect the copy of your service locally to your remote dependencies, so you can test like your local machine is in your cluster.
- Use your existing workflow. Anything that runs on your local machine works with Telepresence. So you can use your favorite tools and IDEs.
How to use it
You can follow the instructions here to install and start using the Telepresence CLI in your local machine and Telepresence in your Kubernetes cluster. Note that, if you are using a MacOS as local machine, there are different versions of the CLI, one for Intel and one for Apple Silicon (MacBooks with M1/M2 chips and later). Choose the right one for your machine.
Once all the dependencies are installed, you can start using Telepresence to intercept your cluster traffic to your local environment as follows:
In our case, we are using kubeconfig and AWS, so first, we need to configure kubeconfig to address to the cluster whose traffic we want to intercept:
aws eks --region eu-west-1 update-kubeconfig --name <cluster_name> --profile <profile_name>
Now, we have configured our environment to point to the cluster we want to test. Next, lets intercept the traffic for
serviceA. For that, we need first to connect to the cluster (running
telepresence connect) and then intercept the service (running
telepresence intercept indicating the namespace):
$> telepresence connect Connected to context arn:aws:eks:eu-west-1:1234567890:cluster/cluster-name
Then, let's intercept the traffic for the service in the cluster (listening on port 80) to our local instance running in port 8080:
$> telepresence intercept serviceA -p 8080:80 -n serviceA Using Deployment serviceA intercepted Intercept name : serviceA-serviceA State : ACTIVE Workload kind : Deployment Destination : 127.0.0.1:8080 Service Port Identifier: 80 Volume Mount Error : sshfs is not installed on your local machine Intercepting : all TCP requests
From now on, all traffic sent to
serviceA will be rerouted to our local (and debuggable)
serviceA instance listening in 8080 port in our local machine. That means that, for example, if
serviceA is a webapp, accessing it using the cluster URL will show the traffic in our local development console. For example, for a node webapp, accessing it through our web browser in the cluster URL (like, for example,
https://testing.webapp.com/) will show in our local console:
info: <Your-public-IP> - - [29/Jan/2023:09:55:30 +0000] "GET / HTTP/1.1" 200 13105 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/220.127.116.11 Safari/537.36"
When you are connected to the cluster, you can directly access the pods with DNS resolution. That means, that, for example
serviceA can communicate with
You can check the status of your interceptions running
$> telepresence status Using Deployment serviceA intercepted Intercept name : serviceA-serviceA State : ACTIVE Workload kind : Deployment Destination : 127.0.0.1:8080 Service Port Identifier: 80 Volume Mount Error : sshfs is not installed on your local machine Intercepting : all TCP requests
When you are done... remember to stop the interception!
Always remember to stop the interception once you are done, otherwise, you will continue intercepting the traffic for
serviceA in your local machine, so you can overload your machine, or generate errors in the cluster if your local instance is down or the code is not stable. To stop the interception run
telepresence leave <name>
$> telepresence leave serviceA-serviceA
To quit the telepresence connection run
$> telepresence quit Telepresence Daemons disconnecting...done
Check the Telepresence CLI help for more specific uses, like types of intercepts allowing be selective about which traffic to intercept, or sharing development environments.
Error: connector.CreateIntercept: A workstation cannot have simultaneous intercepts in different namespaces.
There is a limitation in Telepresence. You only can have interceptors defined for a single namespace. You need to leave all the interceptors before creating the new one.
telepresence leave <interceptor-name>
Sometimes the traffic-agent injected is stuck to our service after running
2023-01-29 11:01:03.8317 info agent-injector/conn=10.0.10.48:443 : Injecting traffic-agent into pod serviceB-56dd5749db-.serviceB 2023-01-29 11:01:03.8331 info agent-injector/conn=10.0.10.48:443 : Injecting 7 patches into pod serviceB-56dd5749db-.serviceB 2023-01-29 11:01:04.8316 info agent-injector/conn=10.0.10.48:443 : Injecting traffic-agent into pod serviceA-b8f5dd5-.serviceA 2023-01-29 11:01:04.8320 info agent-injector/conn=10.0.10.48:443 : Injecting 7 patches into pod serviceA-b8f5dd5-.serviceA
Or receive a timeout trying to connect to it. In those case you can run
telepresence uninstall --all-agents to restore the status of the cluster previous to the use of Telepresence.
Error: telepresence: error: connector.Connect: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"
If you get the following error after running
telepresence: error: connector.Connect: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1" See logs for details (1 error found): "/Users/username/Library/Logs/telepresence/connector.log" If you think you have encountered a bug, please run `telepresence gather-logs` and attach the telepresence_logs.zip to your github issue or create a new one: https://github.com/telepresenceio/telepresence/issues/new?template=Bug_report.md .
Then you need to update the version of
aws-cli. After the upgrade you could get some warnings (opening k9s in our case) like this:
Kubeconfig user entry is using deprecated API version client.authentication.k8s.io/v1alpha1. Run 'aws eks update-kubeconfig' to update.
You could solve it easily updating the kubeconfig file as suggested:
Telepresence intercept: error: connector.CreateIntercept: request timed out while waiting for agent serviceA.serviceA to arrive
Clean all possible changes created in previous sessions running
telepresence uninstall --all-agents
Conflicts with "tcp" mechanism
This kind of error happens when two (or more) local machines are trying to intercept the same service at the same time.
telepresence intercept serviceB -p 8080:80 -n serviceA telepresence intercept: error: connector.CreateIntercept: Failed to establish intercept: intercept in error state AGENT_ERROR: 7d386a54-b870-45cb-b5ea-04a8e7b6ee95:serviceB-serviceB conflicts with "tcp" mechanism of intercept 0c5291fa-9f66-4e4a-901e-e71d0431b279:serviceB-serviceB
You must wait or communicate with your peer to coordinate with him/her.
If there are two or more people working on the same environment, when running
telepresence intercept you can receive the next message (only for glogal intercept):
There are 3 clients connected to this traffic manager. We recommend you switch Traffic Manager to team mode. Team mode changes the default intercept type from global to personal intercepts, allowing more than one developer to intercept the same service and enabling the easier sharing of work-in-progress via secure preview URLs. Enable team mode with the command: telepresence helm upgrade --team-mode
Telepresence provide different types of interceptions. By default, without any kind of filter, running
telepresence intercept injects a global interceptor of the service to your local machine.
Enabling team mode, changes the default intercept injected to be of personal type.
Volume Mount Error : sshfs is not installed on your local machine
It's no necessary to have it to use Telepresence.
An update of telepresence from version X.X.X to Y.Y.Y is available
The Telepresence CLI will periodically check for new versions and notify you when an upgrade is available. Running the same commands used for installation will replace your current binary with the latest version. You can run
telepresence version to check your current client/traffic manager version. You can check more details about upgrading in the official docs.
Before upgrading your CLI, you must stop any live Telepresence processes by issuing
telepresence quit -s to avoid a
telepresence intercept: error: version mismatch error.
I hope my article has helped you, or at least, that you have enjoyed reading it. I do this for fun and I don't need money to keep the blog running. However, if you'd like to show your gratitude, you can pay for my next coffee(s) with a one-time donation of just $1.00. Thank you!