Python OpenTelemetry Instrumentation

This document contains instructions on how to set up OpenTelemetry instrumentation in your Python applications and view your application traces in SigNoz.

OpenTelemetry helps to generate and collect telemetry data from your application which is then sent to an observability backend like SigNoz
OpenTelemetry helps generate and collect telemetry data from Python applications which can then be sent to SigNoz for storage, visualization, and analysis.

Requirements

  • Python 3.8 or newer

Send Traces to SigNoz Cloud

Based on your application environment, you can choose the setup below to send traces to SigNoz Cloud.

From VMs, there are two ways to send data to SigNoz Cloud.

Send traces directly to SigNoz Cloud

Step 1. Create a virtual environment

python3 -m venv .venv
source .venv/bin/activate

Step 2. Install the OpenTelemetry dependencies

pip install opentelemetry-distro==0.43b0
pip install opentelemetry-exporter-otlp==1.22.0

Step 3. Add automatic instrumentation

opentelemetry-bootstrap --action=install

Step 4. Run your application

OTEL_RESOURCE_ATTRIBUTES=service.name=<service_name> \
OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<region>.signoz.cloud:443" \
OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key=<your-ingestion-key>" \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
opentelemetry-instrument <your_run_command>
  • Set the <region> to match your SigNoz Cloud region
  • Replace <your-ingestion-key> with your SigNoz ingestion key.
  • <service_name> is name of your service

Step 5. Validate if your application is sending traces to SigNoz cloud by following the instructions here.

In case you encounter an issue where all applications do not get listed in the services section then please refer to the troubleshooting section.


Send traces via OTel Collector binary

Step 1. Install OTel Collector binary

OTel Collector binary helps to collect logs, hostmetrics, resource and infra attributes.

You can find instructions to install OTel Collector binary here in your VM.

Step 2. Install the OpenTelemetry dependencies

pip install opentelemetry-distro==0.43b0
pip install opentelemetry-exporter-otlp==1.22.0

Step 3. Add automatic instrumentation

opentelemetry-bootstrap --action=install

Step 4. To run your application and send data to collector in same VM:

OTEL_RESOURCE_ATTRIBUTES=service.name=<service_name> \
OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
opentelemetry-instrument <your_run_command>

where,

  • <service_name> is name of your service
  • <your_run_command> can be python3 app.py or flask run

In case you have OtelCollector Agent in different VM, replace localhost:4317 with <IP Address of the VM>:4317.

Step 4. You can validate if your application is sending traces to SigNoz cloud by following the instructions here.

In case you encounter an issue where all applications do not get listed in the services section then please refer to the troubleshooting section.

Validating instrumentation by checking for traces

With your application running, you can verify that you’ve instrumented your application with OpenTelemetry correctly by confirming that tracing data is being reported to SigNoz.

To do this, you need to ensure that your application generates some data. Applications will not produce traces unless they are being interacted with, and OpenTelemetry will often buffer data before sending. So you need to interact with your application and wait for some time to see your tracing data in SigNoz.

Validate your traces in SigNoz:

  1. Trigger an action in your app that generates a web request. Hit the endpoint a number of times to generate some data. Then, wait for some time.
  2. In SigNoz, open the Services tab. Hit the Refresh button on the top right corner, and your application should appear in the list of Applications.
  3. Go to the Traces tab, and apply relevant filters to see your application’s traces.

You might see other dummy applications if you’re using SigNoz for the first time. You can remove it by following the docs here.

Python Application in the list of services being monitored in SigNoz
Python Application in the list of services being monitored in SigNoz

Running applications with Gunicorn, uWSGI

For application servers which are based on pre fork model like Gunicorn, uWSGI you have to add a post_fork hook or a @postfork decorator in your configuration.

Check this documentation from OpenTelemetry on how to set it up.

Here's a working example configured with a gunicorn server with post_fork hook.

Troubleshooting your SigNoz installation

Application servers such as Uvicorn, Hypercorn, etc.

  • Uvicorn with --workers flag is not supported. The work around for this is to use gunicorn with uvicorn as the worker class gunicorn -k uvicorn.workers.UvicornWorker.
  • Hypercorn is not supported. There is no workaround for this. Please follow the issue https://github.com/pgjones/hypercorn/issues/215

Spans are not being reported

If spans are not being reported to SigNoz, try enabling debug exporter which writes the JSON formatted trace data to the console by setting env var OTEL_TRACES_EXPORTER=console.

OTEL_RESOURCE_ATTRIBUTES=service.name=python_app OTEL_TRACES_EXPORTER=console opentelemetry-instrument <your run command>
{
    "name": "alice",
    "context": {
        "trace_id": "0xedb7caf0c8b082a9578460a201759193",
        "span_id": "0x57cf7eee198e1fed",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2022-03-27T14:55:18.804758Z",
    "end_time": "2022-03-27T14:55:18.804805Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "telemetry.sdk.language": "python",
        "telemetry.sdk.name": "opentelemetry",
        "telemetry.sdk.version": "1.10.0",
        "service.name": "my-service"
    }
}

Sample Application

  • Working example where we have configured a gunicorn server with post_fork hook.

Frequently Asked Questions

  1. How to find what to use in IP of SigNoz if I have installed SigNoz in Kubernetes cluster?

    Based on where you have installed your application and where you have installed SigNoz, you need to find the right value for this. Please use this grid to find the value you should use for IP of SigNoz

  2. I am sending data from my application to SigNoz, but I don't see any events or graphs in the SigNoz dashboard. What should I do?

    This could be because of one of the following reasons:

    1. Your application is generating telemetry data, but not able to connect with SigNoz installation

      Please use this troubleshooting guide to find if your application is able to access SigNoz installation and send data to it.

    2. Your application is not actually generating telemetry data

      Please check if the application is generating telemetry data first. You can use Console Exporter to just print your telemetry data in console first. Join our Slack Community if you need help on how to export your telemetry data in console

    3. Your SigNoz installation is not running or behind a firewall

      Please double check if the pods in SigNoz installation are running fine. docker ps or kubectl get pods -n platform are your friends for this.

What Cloud Endpoint Should I Use?

The primary method for sending data to SigNoz Cloud is through OTLP exporters. You can either send the data directly from your application using the exporters available in SDKs/language agents or send the data to a collector agent, which batches/enriches telemetry and sends it to the Cloud.

My Collector Sends Data to SigNoz Cloud

Using gRPC Exporter

The endpoint should be ingest.{region}.signoz.cloud:443, where {region} should be replaced with in, us, or eu. Note that the exporter endpoint doesn't require a scheme for the gRPC exporter in the collector.

# Sample config with `us` region
exporters:
    otlp:
        endpoint: "ingest.us.signoz.cloud:443"
        tls:
            insecure: false
        headers:
            "signoz-ingestion-key": "<SIGNOZ_INGESTION_KEY>"

Using HTTP Exporter

The endpoint should be https://ingest.{region}.signoz.cloud:443, where {region} should be replaced with in, us, or eu. Note that the endpoint includes the scheme https for the HTTP exporter in the collector.

# Sample config with `us` region
exporters:
    otlphttp:
        endpoint: "https://ingest.us.signoz.cloud:443"
        tls:
            insecure: false
        headers:
            "signoz-ingestion-key": "<SIGNOZ_INGESTION_KEY>"

My Application Sends Data to SigNoz Cloud

The endpoint should be configured either with environment variables or in the SDK setup code.

Using Environment Variables

Using gRPC Exporter

Examples with us region

  • OTEL_EXPORTER_OTLP_PROTOCOL=grpc OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.us.signoz.cloud:443 OTEL_EXPORTER_OTLP_HEADERS=signoz-ingestion-key=<SIGNOZ_INGESTION_KEY>
Using HTTP Exporter
  • OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.us.signoz.cloud:443 OTEL_EXPORTER_OTLP_HEADERS=signoz-ingestion-key=<SIGNOZ_INGESTION_KEY>

Configuring Endpoint in Code

Please refer to the agent documentation.

Sending Data from a Third-Party Service

The endpoint configuration here depends on the export protocol supported by the third-party service. They may support either gRPC, HTTP, or both. Generally, you will need to adjust the host and port. The host address should be ingest.{region}.signoz.cloud:443, where {region} should be replaced with in, us, or eu, and port 443 should be used.

Was this page helpful?