Skip to content

Telemetry#

Daft exports OpenTelemetry (OTLP) metrics for query execution. Export is off by default and is enabled by setting an OTLP endpoint environment variable. Metrics can be sent to any OTLP-compatible backend, such as Prometheus with OTLP receiver enabled.

Configuration#

Set OTLP environment variables to configure telemetry. Daft enables OpenTelemetry when any OTLP endpoint is set.

Examples#

Configure telemetry with environment variables:

1
2
3
4
5
# 1) Set an OTLP endpoint (enables telemetry)
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# 2) Run your Daft pipeline
python my-daft-script.py

For metrics-only export to Prometheus (with the OTLP receiver enabled):

1
2
3
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://localhost:9090/api/v1/otlp/v1/metrics
python my-daft-script.py

Configuration details#

Variable Description Default Allowed values
OTEL_EXPORTER_OTLP_ENDPOINT Base OTLP endpoint used for metrics/logs/traces unless signal-specific endpoints are set. Unset OTLP endpoint URL, for example http://localhost:4317
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT Metrics-only OTLP endpoint. Overrides OTEL_EXPORTER_OTLP_ENDPOINT for metrics. Unset OTLP endpoint URL
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT Logs-only OTLP endpoint. Overrides OTEL_EXPORTER_OTLP_ENDPOINT for logs. Unset OTLP endpoint URL
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT Traces-only OTLP endpoint. Overrides OTEL_EXPORTER_OTLP_ENDPOINT for traces. Unset OTLP endpoint URL
OTEL_EXPORTER_OTLP_PROTOCOL OTLP exporter protocol. grpc grpc, http/protobuf, http/json
OTEL_METRIC_EXPORT_INTERVAL Metrics export interval in milliseconds. 500 Positive integer (milliseconds)
DAFT_DEV_OTEL_EXPORTER_OTLP_ENDPOINT Deprecated alias for OTEL_EXPORTER_OTLP_ENDPOINT. Unset OTLP endpoint URL

Info

OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT take precedence over OTEL_EXPORTER_OTLP_ENDPOINT for their respective signal.

Warning

For metrics, http/protobuf is supported and http/json is not currently supported. For logs and traces, Daft currently exports via gRPC. Use HTTP-style endpoint URLs (http:// or https://) for OTLP exporters; protocol is controlled by OTEL_EXPORTER_OTLP_PROTOCOL.

Metrics#

Daft exports the following metrics.

Metric Type Labels Description
daft.bytes.read Counter node.id, node.type Total bytes read by source/scan operators.
daft.bytes.written Counter node.id, node.type Total bytes written by write sinks.
daft.duration Counter node.id, node.type Accumulated operator CPU time in microseconds.
daft.rows.in Counter node.id, node.type Total rows consumed by an operator.
daft.rows.out Counter node.id, node.type Total rows emitted by an operator.
daft.rows.written Counter node.id, node.type Total rows written by write sinks.
daft.task.active UpDownCounter node.id, node.type Number of currently active tasks.
daft.task.cancelled Counter node.id, node.type Total cancelled tasks.
daft.task.completed Counter node.id, node.type Total completed tasks.
daft.task.failed Counter node.id, node.type Total failed tasks.

Info

The following metrics are emitted only in distributed (Ray) execution: daft.bytes.read, daft.task.active, daft.task.cancelled, daft.task.completed, daft.task.failed.