ray-llm

1
# This is a lightweight fork of Opentelemetry's FastAPI instrmentor that fixes
2
# an issue where they create a `meter` from a `meter_provider` inside the
3
# instrumentor. Unfortunately the `meter` contains a threading.Lock, so it
4
# isn't safe to serialize which explodes when Ray Serve tries to serialize it.
5
# This is totally unncessary since if they didn't do that, it does it anyways on
6
# the other side. Forking to quick fix instead of waiting for upstream to fix.
7

8
# Copyright The OpenTelemetry Authors
9
#
10
# Licensed under the Apache License, Version 2.0 (the "License");
11
# you may not use this file except in compliance with the License.
12
# You may obtain a copy of the License at
13
#
14
#     http://www.apache.org/licenses/LICENSE-2.0
15
#
16
# Unless required by applicable law or agreed to in writing, software
17
# distributed under the License is distributed on an "AS IS" BASIS,
18
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
19
# See the License for the specific language governing permissions and
20
# limitations under the License.
21

22
"""
23
Usage
24
-----
25

26
.. code-block:: python
27

28
    import fastapi
29
    from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
30

31
    app = fastapi.FastAPI()
32

33
    @app.get("/foobar")
34
    async def foobar():
35
        return {"message": "hello world"}
36

37
    FastAPIInstrumentor.instrument_app(app)
38

39
Configuration
40
-------------
41

42
Exclude lists
43
*************
44
To exclude certain URLs from tracking, set the environment variable ``OTEL_PYTHON_FASTAPI_EXCLUDED_URLS``
45
(or ``OTEL_PYTHON_EXCLUDED_URLS`` to cover all instrumentations) to a string of comma delimited regexes that match the
46
URLs.
47

48
For example,
49

50
::
51

52
    export OTEL_PYTHON_FASTAPI_EXCLUDED_URLS="client/.*/info,healthcheck"
53

54
will exclude requests such as ``https://site/client/123/info`` and ``https://site/xyz/healthcheck``.
55

56
You can also pass comma delimited regexes directly to the ``instrument_app`` method:
57

58
.. code-block:: python
59

60
    FastAPIInstrumentor.instrument_app(app, excluded_urls="client/.*/info,healthcheck")
61

62
Request/Response hooks
63
**********************
64

65
This instrumentation supports request and response hooks. These are functions that get called
66
right after a span is created for a request and right before the span is finished for the response.
67

68
- The server request hook is passed a server span and ASGI scope object for every incoming request.
69
- The client request hook is called with the internal span and an ASGI scope when the method ``receive`` is called.
70
- The client response hook is called with the internal span and an ASGI event when the method ``send`` is called.
71

72
.. code-block:: python
73

74
    def server_request_hook(span: Span, scope: dict):
75
        if span and span.is_recording():
76
            span.set_attribute("custom_user_attribute_from_request_hook", "some-value")
77

78
    def client_request_hook(span: Span, scope: dict):
79
        if span and span.is_recording():
80
            span.set_attribute("custom_user_attribute_from_client_request_hook", "some-value")
81

82
    def client_response_hook(span: Span, message: dict):
83
        if span and span.is_recording():
84
            span.set_attribute("custom_user_attribute_from_response_hook", "some-value")
85

86
   FastAPIInstrumentor().instrument(server_request_hook=server_request_hook, client_request_hook=client_request_hook, client_response_hook=client_response_hook)
87

88
Capture HTTP request and response headers
89
*****************************************
90
You can configure the agent to capture specified HTTP headers as span attributes, according to the
91
`semantic convention <https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/http.md#http-request-and-response-headers>`_.
92

93
Request headers
94
***************
95
To capture HTTP request headers as span attributes, set the environment variable
96
``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST`` to a comma delimited list of HTTP header names.
97

98
For example,
99
::
100

101
    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST="content-type,custom_request_header"
102

103
will extract ``content-type`` and ``custom_request_header`` from the request headers and add them as span attributes.
104

105
Request header names in FastAPI are case-insensitive. So, giving the header name as ``CUStom-Header`` in the environment
106
variable will capture the header named ``custom-header``.
107

108
Regular expressions may also be used to match multiple headers that correspond to the given pattern.  For example:
109
::
110

111
    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST="Accept.*,X-.*"
112

113
Would match all request headers that start with ``Accept`` and ``X-``.
114

115
To capture all request headers, set ``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST`` to ``".*"``.
116
::
117

118
    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_REQUEST=".*"
119

120
The name of the added span attribute will follow the format ``http.request.header.<header_name>`` where ``<header_name>``
121
is the normalized HTTP header name (lowercase, with ``-`` replaced by ``_``). The value of the attribute will be a
122
single item list containing all the header values.
123

124
For example:
125
``http.request.header.custom_request_header = ["<value1>,<value2>"]``
126

127
Response headers
128
****************
129
To capture HTTP response headers as span attributes, set the environment variable
130
``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE`` to a comma delimited list of HTTP header names.
131

132
For example,
133
::
134

135
    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE="content-type,custom_response_header"
136

137
will extract ``content-type`` and ``custom_response_header`` from the response headers and add them as span attributes.
138

139
Response header names in FastAPI are case-insensitive. So, giving the header name as ``CUStom-Header`` in the environment
140
variable will capture the header named ``custom-header``.
141

142
Regular expressions may also be used to match multiple headers that correspond to the given pattern.  For example:
143
::
144

145
    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE="Content.*,X-.*"
146

147
Would match all response headers that start with ``Content`` and ``X-``.
148

149
To capture all response headers, set ``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE`` to ``".*"``.
150
::
151

152
    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER_RESPONSE=".*"
153

154
The name of the added span attribute will follow the format ``http.response.header.<header_name>`` where ``<header_name>``
155
is the normalized HTTP header name (lowercase, with ``-`` replaced by ``_``). The value of the attribute will be a
156
single item list containing all the header values.
157

158
For example:
159
``http.response.header.custom_response_header = ["<value1>,<value2>"]``
160

161
Sanitizing headers
162
******************
163
In order to prevent storing sensitive data such as personally identifiable information (PII), session keys, passwords,
164
etc, set the environment variable ``OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS``
165
to a comma delimited list of HTTP header names to be sanitized.  Regexes may be used, and all header names will be
166
matched in a case-insensitive manner.
167

168
For example,
169
::
170

171
    export OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SANITIZE_FIELDS=".*session.*,set-cookie"
172

173
will replace the value of headers such as ``session-id`` and ``set-cookie`` with ``[REDACTED]`` in the span.
174

175
Note:
176
    The environment variable names used to capture HTTP headers are still experimental, and thus are subject to change.
177

178
API
179
---
180
"""
181
import logging
182
import typing
183
from typing import Collection
184

185
import fastapi
186
from opentelemetry.instrumentation.asgi import OpenTelemetryMiddleware
187
from opentelemetry.instrumentation.fastapi.package import _instruments
188
from opentelemetry.instrumentation.instrumentor import BaseInstrumentor
189
from opentelemetry.semconv.trace import SpanAttributes
190
from opentelemetry.trace import Span
191
from opentelemetry.util.http import get_excluded_urls, parse_excluded_urls
192
from starlette.routing import Match
193

194
_excluded_urls_from_env = get_excluded_urls("FASTAPI")
195
_logger = logging.getLogger(__name__)
196

197
_ServerRequestHookT = typing.Optional[typing.Callable[[Span, dict], None]]
198
_ClientRequestHookT = typing.Optional[typing.Callable[[Span, dict], None]]
199
_ClientResponseHookT = typing.Optional[typing.Callable[[Span, dict], None]]
200

201

202
class FastAPIInstrumentor(BaseInstrumentor):
203
    """An instrumentor for FastAPI
204

205
    See `BaseInstrumentor`
206
    """
207

208
    _original_fastapi = None
209

210
    @staticmethod
211
    def instrument_app(
212
        app: fastapi.FastAPI,
213
        server_request_hook: _ServerRequestHookT = None,
214
        client_request_hook: _ClientRequestHookT = None,
215
        client_response_hook: _ClientResponseHookT = None,
216
        tracer_provider=None,
217
        meter_provider=None,
218
        excluded_urls=None,
219
    ):
220
        """Instrument an uninstrumented FastAPI application."""
221
        if not hasattr(app, "_is_instrumented_by_opentelemetry"):
222
            app._is_instrumented_by_opentelemetry = False
223

224
        if not getattr(app, "_is_instrumented_by_opentelemetry", False):
225
            if excluded_urls is None:
226
                excluded_urls = _excluded_urls_from_env
227
            else:
228
                excluded_urls = parse_excluded_urls(excluded_urls)
229

230
            app.add_middleware(
231
                OpenTelemetryMiddleware,
232
                excluded_urls=excluded_urls,
233
                default_span_details=_get_default_span_details,
234
                server_request_hook=server_request_hook,
235
                client_request_hook=client_request_hook,
236
                client_response_hook=client_response_hook,
237
                tracer_provider=tracer_provider,
238
                meter_provider=meter_provider,
239
            )
240
            app._is_instrumented_by_opentelemetry = True
241
            if app not in _InstrumentedFastAPI._instrumented_fastapi_apps:
242
                _InstrumentedFastAPI._instrumented_fastapi_apps.add(app)
243
        else:
244
            _logger.warning(
245
                "Attempting to instrument FastAPI app while already instrumented"
246
            )
247

248
    @staticmethod
249
    def uninstrument_app(app: fastapi.FastAPI):
250
        app.user_middleware = [
251
            x for x in app.user_middleware if x.cls is not OpenTelemetryMiddleware
252
        ]
253
        app.middleware_stack = app.build_middleware_stack()
254
        app._is_instrumented_by_opentelemetry = False
255

256
    def instrumentation_dependencies(self) -> Collection[str]:
257
        return _instruments
258

259
    def _instrument(self, **kwargs):
260
        self._original_fastapi = fastapi.FastAPI
261
        _InstrumentedFastAPI._tracer_provider = kwargs.get("tracer_provider")
262
        _InstrumentedFastAPI._server_request_hook = kwargs.get("server_request_hook")
263
        _InstrumentedFastAPI._client_request_hook = kwargs.get("client_request_hook")
264
        _InstrumentedFastAPI._client_response_hook = kwargs.get("client_response_hook")
265
        _excluded_urls = kwargs.get("excluded_urls")
266
        _InstrumentedFastAPI._excluded_urls = (
267
            _excluded_urls_from_env
268
            if _excluded_urls is None
269
            else parse_excluded_urls(_excluded_urls)
270
        )
271
        _InstrumentedFastAPI._meter_provider = kwargs.get("meter_provider")
272
        fastapi.FastAPI = _InstrumentedFastAPI
273

274
    def _uninstrument(self, **kwargs):
275
        for instance in _InstrumentedFastAPI._instrumented_fastapi_apps:
276
            self.uninstrument_app(instance)
277
        _InstrumentedFastAPI._instrumented_fastapi_apps.clear()
278
        fastapi.FastAPI = self._original_fastapi
279

280

281
class _InstrumentedFastAPI(fastapi.FastAPI):
282
    _tracer_provider = None
283
    _meter_provider = None
284
    _excluded_urls = None
285
    _server_request_hook: _ServerRequestHookT = None
286
    _client_request_hook: _ClientRequestHookT = None
287
    _client_response_hook: _ClientResponseHookT = None
288
    _instrumented_fastapi_apps: typing.Set[fastapi.FastAPI] = set()
289

290
    def __init__(self, *args, **kwargs):
291
        super().__init__(*args, **kwargs)
292
        self.add_middleware(
293
            OpenTelemetryMiddleware,
294
            excluded_urls=_InstrumentedFastAPI._excluded_urls,
295
            default_span_details=_get_default_span_details,
296
            server_request_hook=_InstrumentedFastAPI._server_request_hook,
297
            client_request_hook=_InstrumentedFastAPI._client_request_hook,
298
            client_response_hook=_InstrumentedFastAPI._client_response_hook,
299
            tracer_provider=_InstrumentedFastAPI._tracer_provider,
300
            meter_provider=_InstrumentedFastAPI._meter_provider,
301
        )
302

303
        self._is_instrumented_by_opentelemetry = True
304
        _InstrumentedFastAPI._instrumented_fastapi_apps.add(self)
305

306
    def __del__(self):
307
        if self in _InstrumentedFastAPI._instrumented_fastapi_apps:
308
            _InstrumentedFastAPI._instrumented_fastapi_apps.remove(self)
309

310

311
def _get_route_details(scope):
312
    """
313
    Function to retrieve Starlette route from scope.
314

315
    TODO: there is currently no way to retrieve http.route from
316
    a starlette application from scope.
317
    See: https://github.com/encode/starlette/pull/804
318

319
    Args:
320
        scope: A Starlette scope
321
    Returns:
322
        A string containing the route or None
323
    """
324
    app = scope["app"]
325
    route = None
326

327
    for starlette_route in app.routes:
328
        match, _ = starlette_route.matches(scope)
329
        if match == Match.FULL:
330
            route = starlette_route.path
331
            break
332
        if match == Match.PARTIAL:
333
            route = starlette_route.path
334
    return route
335

336

337
def _get_default_span_details(scope):
338
    """
339
    Callback to retrieve span name and attributes from scope.
340

341
    Args:
342
        scope: A Starlette scope
343
    Returns:
344
        A tuple of span name and attributes
345
    """
346
    route = _get_route_details(scope)
347
    method = scope.get("method", "")
348
    attributes = {}
349
    if route:
350
        attributes[SpanAttributes.HTTP_ROUTE] = route
351
    if method and route:  # http
352
        span_name = f"{method} {route}"
353
    elif route:  # websocket
354
        span_name = route
355
    else:  # fallback
356
        span_name = method
357
    return span_name, attributes
358