Updated: Jun 16
Microservices have multiple benefits for development organizations. They make applications easily scalable, highly resilient, and easier to maintain, and keeping track of hundreds of microservices can be challenging.
As many organizations move towards microservices architecture to boost developer productivity, they are also facing new operational challenges. "We can't fix something which we can't observe," so it is important to have a strong monitoring and observability solution in place to track the performance of microservices applications. That’s where observability comes into the picture. Observability platforms help businesses understand system behavior and predict outages or problems before they occur. Observability tools are used by businesses to take preventive action and stop system issues. In this post, we will see how to use OpenTelemetry and Jaeger to trace the data of applications.
OpenTelemetry (OTel) is a set of open-source libraries and tools that enable the collection, processing, and export of telemetry data from cloud-native software applications. It provides a vendor-agnostic approach to instrumenting applications and collecting telemetry data.
Instrumenting is the process of adding monitoring and telemetry code to software applications. This code collects data about the application's performance, behavior, and usage. This data can then be used to gain insights into how the application is performing, identify issues and errors, and optimize its performance.
OpenTelemetry provides both manual and automatic instrumentation capabilities. Manual instrumentation allows developers to add instrumentation code to their applications themselves. Automatic instrumentation allows developers to let OpenTelemetry automatically add instrumentation code to their applications.
OpenTelemetry can be used to monitor and trace applications in a distributed environment. Distributed tracing allows developers to see how requests flow through a distributed system, which can be helpful for identifying performance bottlenecks and errors.
Manual Instrumentation Manual instrumentation involves developers explicitly creating and recording spans using the Opentelemetry SDK API. It allows developers to trace specific parts of their application code and provides greater control over the instrumentation process.
Automatic Instrumentation Automatic instrumentation utilizes pre-built integrations to instrument an application automatically, eliminating the need for developers to write any code. Opentelemetry provides automatic instrumentation for popular frameworks and libraries like Spring, Django, and Node.js.
Here, we will find out how you can use automatic instrumentation in your application.
You must have the Kubernetes cluster up and running
You must have Kubectl access to your Kubernetes cluster
Make sure to install cert-manager
Let's assume that you have all the prerequisites and now you are ready for the installation.
Jaeger is an open-source distributed tracing system that provides end-to-end tracing, allowing developers to track the flow of requests through a complex system of microservices. To start visualizing data, we need to set up Jaeger.
Jaeger installation contains two steps:
Jaeger Operator Installation
Deploying the AllInOne image
Jaeger Operator Installation
The Jaeger Operator is an open-source Kubernetes operator that simplifies the deployment and management of Jaeger. Firstly, we will create an observability namespace and deploy the Jaeger Operator in that namespace. By default, the operator will monitor all namespaces.
$ kubectl create namespace observability $ kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.44.0/jaeger-operator.yaml -n observability
Once it is deployed, you can view it by running the following command.
$ kubectl get deployment jaeger-operator -n observability NAME READY UP-TO-DATE AVAILABLE AGE jaeger-operator 1/1 1 1 27m
$ kubectl get svc -n observability NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) jaeger-operator-metrics ClusterIP 10.64.15.107 <none> 8443/TCP jaeger-operator-webhook-service ClusterIP 10.64.10.106 <none> 443/TCP
Deploying the AllInOne image
The Jaeger All-In-One (AIO) image is a pre-built Docker image that includes all the necessary components of Jaeger, including the Jaeger collector, Jaeger query, Jaeger agent, Jaeger UI, as well as the necessary dependencies, all packaged in a single image and this is the simplest possible way to create a Jaeger instance is by creating a YAML file.
Create a file named “jeager-instance.yaml” and add the following YAML content to the file.
apiVersion: jaegertracing.io/v1 kind: Jaeger metadata: name: jaeger namespace: observability spec: ingress: enabled: true
Deploy Jeager all-in-one image by running the following:
$ Kubectl apply -f jeager-instance.yaml
Verify the deployment by running the command:
$ kubectl get jaegers -n observability NAME STATUS VERSION STRATEGY STORAGE AGE jaeger Running 1.44.0 allinone memory 43m
When the Jeager all-in-one image is deployed, you can access the Jaeger UI at a specific ingress URL, allowing you to explore the traces generated by your instrumentation.
For the installation, we need to follow below steps:
Add annotations to the deployment manifest
The OpenTelemetry Operator is designed to deliver auto-instrumentation to export traces and metrics in new and existing applications without any code changes (Source). Now, we would introduce OpenTelemetry Operator which makes it very easy to set up OpenTelemetry collector and instrument workloads deployed on Kubernetes.
To install the OpenTelemetry Operator in an existing Kubernetes Cluster, make sure you have the cert-manager installed and running.
$ kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
The above command creates a new namespace called opentelemetry-operator-system and deploys the OpenTelemetry Operator within that namespace.
You can verify the deployment of the OpenTelemetry operator by running the following command:
$ kubectl get all -n opentelemetry-operator-system NAME READY STATUS RESTARTS AGE pod/opentelemetry-operator-controller-manager-b9f568969-lpzc5 2/2 Running 0 3h8m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/opentelemetry-operator-controller-manager-metrics-service ClusterIP 10.104.3.204 <none> 8443/TCP 3h8m service/opentelemetry-operator-webhook-service ClusterIP 10.104.15.157 <none> 443/TCP 3h8m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/opentelemetry-operator-controller-manager 1/1 1 1 3h8m NAME DESIRED CURRENT READY AGE replicaset.apps/opentelemetry-operator-controller-manager-b9f568969 1 1 1 3h8m
Once the opentelemetry-operator deployment is ready, define a custom Opentelemetry configuration by creating a new file “otelcol.yaml” and adding the desired configuration. Here is the following configuration file that enables the collection and export of traces to a backend. As you know that we are using Jaeger as the backend.
apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: otelcol spec: config: | receivers: otlp: protocols: grpc: http: processors: exporters: logging: jaeger: endpoint: "jaeger-collector.observability.svc.cluster.local:14250" tls: insecure: true service: pipelines: traces: receivers: [otlp] processors:  exporters: [logging, jaeger]
Run the following command to confirm the deployment of the collector:
$ kubectl get deploy otelcol-collector NAME READY UP-TO-DATE AVAILABLE AGE otelcol-collector 1/1 1 1 178m
The operator can inject and configure OpenTelemetry auto-instrumentation libraries for various programming languages and frameworks. Configure an Instrumentation resource with the SDK and instrumentation configurations in order to use auto-instrumentation.
The following YAML will create a basic Auto-instrumentation resource:
apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: demo-instrumentation spec: exporter: endpoint: http://otelcol-collector:4317 propagators: - tracecontext - baggage sampler: type: parentbased_traceidratio argument: "1"
Verify the instrumentation by running the below-mentioned command:
$ kubectl get instrumentation NAME AGE ENDPOINT SAMPLER SAMPLER ARG otel-operator-instrumentation 32s http://otel-collector:4317 parentbased_traceidratio 1
Add Annotations To The Deployment Manifest
The final step is to add an annotation to a pod to enable injection. The annotation can be added to a namespace, so that all pods within that namespace will get instrumentation. Alternatively, the annotation can be added to individual PodSpec objects, available as part of a Deployment.
As we are going to demo a Nodejs application, the annotation which we will have to use here is:
Here, we are utilizing an open-source two-tier Node.js application. We have added annotations to the Node.js application to enable injection, as shown in the Kubernetes manifest below.
apiVersion: v1 kind: Service metadata: name: knote spec: selector: app: knote ports: - port: 80 targetPort: 3000 type: LoadBalancer --- apiVersion: apps/v1 kind: Deployment metadata: name: knote spec: replicas: 1 selector: matchLabels: app: knote template: metadata: labels: app: knote annotations: instrumentation.opentelemetry.io/inject-nodejs: "true" instrumentation.opentelemetry.io/container-names: "knote" spec: containers: - name: knote image: learnk8s/knote-js:1.0.0 ports: - containerPort: 3000 env: - name: MONGO_URL value: mongodb://mongo:27017/dev imagePullPolicy: Always
You can verify the deployment through the following command:
$ kubectl get pod NAME READY STATUS RESTARTS AGE knote-58fdfcc89d-b9tnn 1/1 Running 0 18m mongo-586867d494-wgf8d 1/1 Running 0 18m
To generate traces, you can navigate through all pages of the website and access the Jaeger UI. By selecting the specified service name and operation, you will be able to view the traces generated by those requests.
Below, you will find all the traces captured within a specific time window and their associated spans.
Distributed tracing is invaluable in modern software systems and application architectures. It aids in understanding and troubleshooting complex, distributed systems by tracking the flow of requests across various services and components.
In this blog post, we have explored how to instrument our application without making changes to the code using the OpenTelemetry Auto-Instrumentation method. This method allows you to monitor transactions, conduct root cause analysis, optimize performance and latency, and visualize service dependencies.
Are you looking to improve your organization's DevOps capabilities?
At Vikasietum, we have a team of experienced DevOps engineers who can help you improve your organization's DevOps capabilities. Contact us today to learn more about our DevSecOps services!