Blog Cover

How to intercept your Kubernetes cluster traffic to your local environment with Telepresence

Author profile image
Aitor Alonso

Jan 29, 2023

8 min read

Today, I want to tell you about a powerful tool that I use at my job to intercept Kubernetes traffic to my local environment, so I can test or debug my code locally when I need to: Telepresence.

I want to honorably mention some of my colleagues like Javi, Christian and Rubén. They have implemented this tool in our company and today this article is possible thanks that they have shared their knowledge with me.

What is Telepresence?

Telepresence is an open source tool that allows you to set up remote development environments for Kubernetes, where you can still use all of your favorite local tools like IDEs, debuggers, and profilers. Telepresence allows you to:

  • Connect to a test environment and directly access the pods with DNS resolution (without explicitly having to port-forward any of its dependencies).
  • Intercept traffic temporarily, replacing a deployment with an instance running locally in your machine so that you can debug internal calls from inside the cluster in your developer machine.

How can Telepresence help me?

Sometimes, when you are working with environments deployed with Kubernetes, you need to:

  • Debug or test a component in your local environment. This is difficult sometimes because you need a lot of required dependencies: data sources, other services, environment variables, etc.
  • Know why a component (as part of a complex flow) is failing when deployed but it works as expected locally.

You might be using kubectl port-forward along with kubefwd for that, and these are really good options for connecting purpose, but not for traffic interception, as their set up is very tedious and involves a lot of manual steps.

However, Telepresence helps you to:

  • Accelerate the inner dev loop. You can make changes to your service as if you're developing locally, without having to run all the dependencies on your local machine.
  • Shift-left testing. It lets you connect the copy of your service locally to your remote dependencies, so you can test like your local machine is in your cluster.
  • Use your existing workflow. Anything that runs on your local machine works with Telepresence. So you can use your favorite tools and IDEs.

How to use it

You can follow the instructions here to install and start using the Telepresence CLI in your local machine and Telepresence in your Kubernetes cluster. Note that, if you are using a Mac as local machine, there are different versions of the CLI, one for Intel and one for Apple Silicon (MacBooks with M1/M2 chips and later). Choose the right one for your machine.

Once all the dependencies are installed, you can start using Telepresence to intercept your cluster traffic to your local environment as follows:

In our case, we are using kubeconfig and AWS, so first, we need to configure kubeconfig to address to the cluster whose traffic we want to intercept:

aws eks --region eu-west-1 update-kubeconfig --name <cluster_name> --profile <profile_name>

Now, we have configured our environment to point to the cluster we want to test. Next, lets intercept the traffic for serviceA. For that, we need first to connect to the cluster (running telepresence connect) and then intercept the service (running telepresence intercept indicating the namespace):

$> telepresence connect

Connected to context arn:aws:eks:eu-west-1:1234567890:cluster/cluster-name

Then, let's intercept the traffic for the service in the cluster (listening on port 80) to our local instance running in port 8080:

$> telepresence intercept serviceA -p 8080:80 -n serviceA

Using Deployment serviceA
intercepted
    Intercept name         : serviceA-serviceA
    State                  : ACTIVE
    Workload kind          : Deployment
    Destination            : 127.0.0.1:8080
    Service Port Identifier: 80
    Volume Mount Error     : sshfs is not installed on your local machine
    Intercepting           : all TCP requests

From now on, all traffic sent to serviceA will be rerouted to our local (and debuggable) serviceA instance listening in 8080 port in our local machine. That means that, for example, if serviceA is a webapp, accessing it using the cluster URL will show the traffic in our local development console. For example, for a node webapp, accessing it through our web browser in the cluster URL (like, for example, https://testing.webapp.com/) will show in our local console:

info: <Your-public-IP> - - [29/Jan/2023:09:55:30 +0000] "GET / HTTP/1.1" 200 13105 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"

When you are connected to the cluster, you can directly access the pods with DNS resolution. That means, that, for example serviceA can communicate with serviceB.

You can check the status of your interceptions running telepresence status:

$> telepresence status

Using Deployment serviceA
intercepted
   Intercept name         : serviceA-serviceA
   State                  : ACTIVE
   Workload kind          : Deployment
   Destination            : 127.0.0.1:8080
   Service Port Identifier: 80
   Volume Mount Error     : sshfs is not installed on your local machine
   Intercepting           : all TCP requests

When you are done... remember to stop the interception!

Always remember to stop the interception once you are done, otherwise, you will continue intercepting the traffic for serviceA in your local machine, so you can overload your machine, or generate errors in the cluster if your local instance is down or the code is not stable. To stop the interception run telepresence leave <name>

$> telepresence leave serviceA-serviceA

To quit the telepresence connection run telepresence quit:

$> telepresence quit

Telepresence Daemons disconnecting...done

Check the Telepresence CLI help for more specific uses, like types of intercepts allowing be selective about which traffic to intercept, or sharing development environments.

Troubleshooting

Error: connector.CreateIntercept: A workstation cannot have simultaneous intercepts in different namespaces.

There is a limitation in Telepresence. You only can have interceptors defined for a single namespace. You need to leave all the interceptors before creating the new one.

    telepresence leave <interceptor-name>

Sometimes the traffic-agent injected is stuck to our service after running telepresence leave:

2023-01-29 11:01:03.8317 info    agent-injector/conn=10.0.10.48:443 : Injecting traffic-agent into pod serviceB-56dd5749db-.serviceB
2023-01-29 11:01:03.8331 info    agent-injector/conn=10.0.10.48:443 : Injecting 7 patches into pod serviceB-56dd5749db-.serviceB
2023-01-29 11:01:04.8316 info    agent-injector/conn=10.0.10.48:443 : Injecting traffic-agent into pod serviceA-b8f5dd5-.serviceA
2023-01-29 11:01:04.8320 info    agent-injector/conn=10.0.10.48:443 : Injecting 7 patches into pod serviceA-b8f5dd5-.serviceA

Or receive a timeout trying to connect to it. In those case you can run telepresence uninstall --all-agents to restore the status of the cluster previous to the use of Telepresence.

Error: telepresence: error: connector.Connect: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"

If you get the following error after running telepresence connect:

telepresence: error: connector.Connect: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"

See logs for details (1 error found): "/Users/username/Library/Logs/telepresence/connector.log"
If you think you have encountered a bug, please run `telepresence gather-logs` and attach the telepresence_logs.zip to your github issue or create a new one: https://github.com/telepresenceio/telepresence/issues/new?template=Bug_report.md .

Then you need to update the version of aws-cli. After the upgrade you could get some warnings (opening k9s in our case) like this:

Kubeconfig user entry is using deprecated API version client.authentication.k8s.io/v1alpha1. Run 'aws eks update-kubeconfig' to update.

You could solve it easily updating the kubeconfig file as suggested:

Telepresence intercept: error: connector.CreateIntercept: request timed out while waiting for agent serviceA.serviceA to arrive

Clean all possible changes created in previous sessions running telepresence uninstall --all-agents

Conflicts with "tcp" mechanism

This kind of error happens when two (or more) local machines are trying to intercept the same service at the same time.

telepresence intercept serviceB -p 8080:80 -n serviceA

telepresence intercept: error: connector.CreateIntercept: Failed to establish intercept: intercept in error state AGENT_ERROR:      7d386a54-b870-45cb-b5ea-04a8e7b6ee95:serviceB-serviceB conflicts with "tcp" mechanism of intercept  0c5291fa-9f66-4e4a-901e-e71d0431b279:serviceB-serviceB

You must wait or communicate with your peer to coordinate with him/her.

Team mode

If there are two or more people working on the same environment, when running telepresence intercept you can receive the next message (only for glogal intercept):

There are 3 clients connected to this traffic manager.

We recommend you switch Traffic Manager to team mode.
Team mode changes the default intercept type from global to personal intercepts, allowing more than one developer to intercept the same service and enabling the easier sharing of work-in-progress via secure preview URLs.
Enable team mode with the command: telepresence helm upgrade --team-mode

Telepresence provide different types of interceptions. By default, without any kind of filter, running telepresence intercept injects a global interceptor of the service to your local machine.

Enabling team mode, changes the default intercept injected to be of personal type.

Volume Mount Error : sshfs is not installed on your local machine

It's no necessary to have it to use Telepresence.

An update of telepresence from version X.X.X to Y.Y.Y is available

The Telepresence CLI will periodically check for new versions and notify you when an upgrade is available. Running the same commands used for installation will replace your current binary with the latest version. You can run telepresence version to check your current client/traffic manager version. You can check more details about upgrading in the official docs.

Before upgrading your CLI, you must stop any live Telepresence processes by issuing telepresence quit -s to avoid a telepresence intercept: error: version mismatch error.


I hope my article has helped you, or at least, that you have enjoyed reading it. I do this for fun and I don't need money to keep the blog running. However, if you'd like to show your gratitude, you can pay for my next coffee(s) with a one-time donation of just $1.00. Thank you!