haystack

1
---
2
features:
3
  - |
4
    Added option to instrument pipeline and component runs.
5
    This allows users to observe their pipeline runs and component runs in real-time via their chosen observability
6
    tool. Out-of-the-box support for OpenTelemetry and Datadog will be added in separate contributions.
7

8
    Example usage for [OpenTelemetry](https://opentelemetry.io/docs/languages/python/):
9

10
    1. Install OpenTelemetry SDK and exporter:
11
      ```bash
12
      pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
13
      ```
14

15
    2. Configure OpenTelemetry SDK with your tracing provider and exporter:
16
      ```python
17
      from opentelemetry.sdk.resources import SERVICE_NAME, Resource
18

19
      from opentelemetry import trace
20
      from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
21
      from opentelemetry.sdk.trace import TracerProvider
22
      from opentelemetry.sdk.trace.export import BatchSpanProcessor
23

24
      # Service name is required for most backends
25
      resource = Resource(attributes={
26
          SERVICE_NAME: "haystack"
27
      })
28

29
      traceProvider = TracerProvider(resource=resource)
30
      processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
31
      traceProvider.add_span_processor(processor)
32
      trace.set_tracer_provider(traceProvider)
33

34
      tracer = traceProvider.get_tracer("my_application")
35

36

37
    3. Create tracer
38
      ```python
39
         import contextlib
40
         from typing import Optional, Dict, Any, Iterator
41

42
         from opentelemetry import trace
43
         from opentelemetry.trace import NonRecordingSpan
44

45
         from haystack.tracing import Tracer, Span
46
         from haystack.tracing import utils as tracing_utils
47
         import opentelemetry.trace
48

49

50
         class OpenTelemetrySpan(Span):
51
             def __init__(self, span: opentelemetry.trace.Span) -> None:
52
                 self._span = span
53

54
             def set_tag(self, key: str, value: Any) -> None:
55
                 coerced_value = tracing_utils.coerce_tag_value(value)
56
                 self._span.set_attribute(key, coerced_value)
57

58

59
         class OpenTelemetryTracer(Tracer):
60
             def __init__(self, tracer: opentelemetry.trace.Tracer) -> None:
61
                 self._tracer = tracer
62

63
             @contextlib.contextmanager
64
             def trace(self, operation_name: str, tags: Optional[Dict[str, Any]] = None) -> Iterator[Span]:
65
                 with self._tracer.start_as_current_span(operation_name) as span:
66
                     span = OpenTelemetrySpan(span)
67
                     if tags:
68
                         span.set_tags(tags)
69

70
                     yield span
71

72
             def current_span(self) -> Optional[Span]:
73
                 current_span = trace.get_current_span()
74
                 if isinstance(current_span, NonRecordingSpan):
75
                     return None
76

77
                 return OpenTelemetrySpan(current_span)
78

79
        ```
80

81
    4. Use the tracer with Haystack:
82
      ```python
83
      from haystack import tracing
84

85
      haystack_tracer = OpenTelemetryTracer(tracer)
86
      tracing.enable_tracing(haystack_tracer)
87
      ```
88

89
    5. Run your pipeline
90