Repository navigation
RFC: Support for external observability providers - Tracer #2030
Description
Is this related to an existing feature request or issue?
Issue: #1433
Logger RFC: #2014
Metrics RFC: #2015
Which AWS Lambda Powertools utility does this relate to?
Tracer
Summary
This RFC is one of the three that defines the format when setting up loggers, metrics and traces for better integration with other observability providers.
This RFC is specifically for the Tracer. Currently, we have undocumented BaseProvider for Tracer, but we need to decide more on what minimum features the BaseProvider should support. The RFC discusses on the features that could be a part of custom tracer for users to integrate other Observability providers easily.
Use case
The use case for this utility would be for developers who want to use other observability providers to trace their application, other than AWS X-Ray.
Proposal
Current tracer experience
The Powertools’ tracer utility is essentially a wrapper for the AWS X-Ray SDK. Some key features of this utility include auto capturing cold start as annotation, auto capturing responses or full exceptions as metadata, and auto-disabling when not running in AWS Lambda environment. Tracer also auto patches supported modules by AWS X-Ray.
from aws_lambda_powertools import Tracer
from aws_lambda_powertools.utilities.typing import LambdaContext
# Current experience in using metrics should not change
tracer = Tracer(service="ServerlessAirline")
def collect_payment(charge_id: str) -> str:
return f"dummy payment collected for charge: {charge_id}"
@tracer.capture_lambda_handler
def handler(event: dict, context: LambdaContext) -> str:
charge_id = event.get("charge_id", "")
return collect_payment(charge_id=charge_id)JSON output
{
"trace_id": "1-5e367daf-6c7f6d9f6c3a6e5800c7d42d",
"id": "e986a861d4590d97",
"name": "payment",
"start_time": 1580441546.023,
"end_time": 1580441552.983,
"http": {
"request": {
"method": "GET",
"url": "https://api.example.com/",
"client_ip": "192.168.1.1",
"user_agent": "Mozilla/5.0",
},
"response": {
"status": 200,
"content_length": 1024,
"headers": {
"Content-Type": "application/json"
}
}
},
"subsegments": [
{
"id": "3b3b3d8ba74fa7fe",
"name": "my-subsegment",
"start_time": 1580441548.023,
"end_time": 1580441551.983,
"http": {
"request": {
"method": "POST",
"url": "https://api.example.com/submit",
"headers": {
"Content-Type": "application/json",
"Authorization": "Bearer abc123"
},
"body": "{\"data\": \"example\"}"
},
"response": {
"status": 200,
"content_length": 128,
"headers": {
"Content-Type": "application/json"
}
}
},
"annotations": {
"example": "annotation"
}
}
],
"annotations": {
"example": "annotation"
},
"metadata": {
"example": "metadata"
}
}Tracer proposal
We propose a new parameter to the existing tracer utility that developers can use to specify which observability provider they would like their traces to be pushed to. The below code snippet is a rudimentary look at how this utility can be used and how it will function. Out of the box, we will support DataDog. Other providers TBD
from aws_lambda_powertools import Tracer
from aws_lambda_powertools.utilities.typing import LambdaContext
tracer = Tracer(service="ServerlessAirline", format=Tracer.DATADOG)
def collect_payment(charge_id: str) -> str:
return f"dummy payment collected for charge: {charge_id}"
@tracer.capture_lambda_handler
def handler(event: dict, context: LambdaContext) -> str:
charge_id = event.get("charge_id", "")
return collect_payment(charge_id=charge_id)JSON output
{
"trace_id": "3541457326329954564",
"span_id": "467508042476235233",
"parent_id": "3541457326329954564",
"name": "payment",
"resource": "GET /api",
"start": 1647370203.4475,
"duration": 0.0325,
"service": "serverlessAirline",
"type": "web",
"meta": {
"http": {
"method": "GET",
"url": "http://localhost:8000/api",
"status_code": 200
}
}
}Bring your own provider
If you would like to use an observability provider not supported out of the box, or define their own tracer functions, we will define an interface that the customer can implement and pass in to the Tracer class.
classDiagram
class BaseProvider {
+start_span() -> Span
+end_span() -> Span
+put_annotation(key: str, value: Union[str, numbers.Number, bool]) -> None
+put_metadata(key: str, value: Any, namespace: str = "default") -> None
}
class CustomTracerProvider {
+start_span() -> Span
+end_span() -> Span
+put_annotation(key: str, value: Union[str, numbers.Number, bool]) -> None
+put_metadata(key: str, value: Any, namespace: str = "default") -> None
}
BaseProvider <|-- CustomTracerProvider
Example
from aws_lambda_powertools.tracing.tracer import Tracer
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.tracing.base import BaseProvider
from contextlib import contextmanager
import threading
import asyncio
class CustomTracerProvider(BaseProvider):
_thread_local = threading.local()
@contextmanager
def trace_context(self):
# when we enter this context, we start a new span and store its context
if not hasattr(self._thread_local, "trace_id"):
self._thread_local.trace_id = self.start_span()
try:
yield
finally:
self.end_span(self._thread_local.trace_id)
@contextmanager
def trace(self, span:"Span",parent_context: Optional[context_api.Context] = None) -> None:
# when we enter this we start a child span with the given parent context
try:
self.start_span(self, span, parent_context=parent_context)
yield
finally:
self.end_span(self, span)
def start_span(self, span, parent_context: Optional[context_api.Context] = None) -> span:
"""This method is proposed as a solution as it exists for other providers
This method is responsible for starting the trace. This might involve initializing some data structures,
connecting to an external service, or performing some other setup work"""
return span()
def end_span(self, span):
"""This method is proposed as a solution as it exists for other providers.
This method is responsible for ending the tracing of a span. This might involve finalizing data structures,
sending data to an external service, or performing some other cleanup work"""
def put_annotation(self, key: str, value: Union[str, numbers.Number, bool]) -> None:
"""Annotate current active trace entity with a key-value pair."""
def put_metadata(self, key: str, value: Any, namespace: str = "default") -> None:
"""Add metadata to the current active trace entity."""
def add_exception(self, exception):
"""Add an exception to trace entities."""
def ignore_endpoint(self, hostname: Optional[str] = None, urls: Optional[List[str]] = None):
"""To ignore the endpoints you don't want requests to be traced,
perhaps due to the volume of calls or sensitive URLs. """
def inject_context(self,context):
"""To inject missing context/information like service name"""
def capture_method_async(self, method:Callable, capture_response: Optional[Union[bool, str]] = None, capture_error: Optional[Union[bool, str]] = None):
"""To capture async method"""
tracer = Tracer({serviceName: 'serverlessAirline'},provider=CustomTracerProvider())
@tracer.capture_method_async
async def collect_payment_async(charge_id: str) -> str:
tracer.put_annotation(key="PaymentId", value=charge_id)
await asyncio.sleep(0.5)
return f"dummy payment collected for charge: {charge_id}"
@tracer.capture_method
def collect_payment(charge_id: str) -> str:
return f"dummy payment collected for charge: {charge_id}"
@tracer.capture_lambda_handler
def handler(event: dict, context: LambdaContext) -> str:
charge_id = event.get("charge_id", "")
with tracer.provider.trace(span="charge"):
return collect_payment(charge_id=charge_id)The five methods defined above are a combination of methods that already exist in the BaseProvider and the ones that are most common in other observability providers.
The current BaseProvider does support most of the features used in the major observability providers. There are couple of differences I noticed while researching through the other Observability providers.
- There is difference in nomenclature used to define data that gets received from services, Powertools call them segments whereas other observability providers call them span.
- Observability providers like Datadog, Lumigo, NewRelic provides an option to start and end tracing through their start_span and end_span methods, whereas in Powertools we do not have such methods. The possible reason could be that AWS X-Ray anyway keeps track of how the request flows within the application from the start and if it doesn’t support any service, it gives an option to add a subsegment there to keep track of it. Whereas for most of the other providers we need to mention it to start and end tracing. We will add those methods so people can utilize them when using other providers with this capability.
Out of scope
Sending traces from Powertools to the customer's desired observability platform will not be in the scope of this project. The implementation should only support modifying the output of the Tracer so that the customer can push them to their platform of choice.
Potential challenges
We need to determine which platforms we want to support out-of-the-box (apart from Datadog).
Dependencies and Integrations
We will have to integrate with (and thus, have a dependency on) Datadog and any other platforms we decide to support out-of-the-box.
Alternative solutions
No response
Acknowledgment
- This feature request meets Lambda Powertools Tenets
- Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript
Activity
boring-cyborg commentedon Mar 21, 2023
Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link
heitorlessa commentedon Mar 23, 2023
got dragged in meetings and will reply to you properly tomorrow. It'll be along these lines I posted on Metrics: #2015 (comment)
TracerProvideris good, butformatwill confuse customers.Vandita2020 commentedon Mar 29, 2023
Thank you @heitorlessa
I see how it can be a bit confusing to the customers. I have revised the RFC and tried to make it simple by removing the
formatpart while keeping theBaseProvideras a parent class toCustomTraceProvider.heitorlessa commentedon Mar 31, 2023
I underestimated how much feedback I needed to write for Metrics RFC so I ran out of time (apologies!) - I've blocked time on Monday afternoon (morning PST) to go through this.
At a quick glance, the piece I'm missing in the contract is
patching- a custom provider will have to own the responsibility to patch one or all supported libraries.heitorlessa commentedon Apr 3, 2023
hey @Vandita2020 that's a great start!! I took liberty to address some low hanging fruits and made a list of suggestions similar to what I made to Metrics.
There are some minor changes like using the
spanterminology instead of X-Ray specificsegmentto ease authoring custom providers. Two major ones are considering that we need a Span representation, and what strategies should we consider to support threading without leaking too much.Changes
(e.g., Tracer(service=""))Custom tracer usagetoBring your own providerAsks
providerinstead offormat(leftover)tracecontext manager in the BaseProvider signature. This acts as a shortcut so customers can create/close spans within parts of their code instead of the entire function.capture_method,capture_lambda_handlercapture_method_async(something we should move towards given the complexity of having under a single methodsegmenttospanterminologies to ease integration with partners, e.g.:start_span. We can then create a X-Ray Provider and use the equivalent methods/terminologies with the added benefit of demonstrating how other providers could do it.inject_context.start_spanshould return a Span. A Span can start another span (parent/child), end, receive annotations/metadata, etc. This part needs more research. We need to strike a balance of keeping operations simple since the provider does all the hard work of keeping tracing context, asyncio support, etc., and allowing extension due to Liskov substitution principle.tracer.start_segment(),tracer.end_segment()in the Bring your own provider section, I suspect you meant something else entirely.Vandita2020 commentedon Apr 6, 2023
Hey @heitorlessa,
Thanks so much for the review. I have made the changes in the RFC accordingly, also for couple of comments, I have provided come comments/context below.
Currently async methods are handled using
@tracer.capture_method, which uses couple of if-else conditions to check if it is sync or async. To simplify the execution of async methods, we create a new method to specifically handle async methods.I have corrected it, earlier I was using
tracer.start_segment()to start the tracing for everything andtracer.end_segment()to end the tracing, but it got more clearer to me now. As now we are usingcapture_lambda_handlerto trace the handler,capture_methodto trace any method, andstart_spanend_spanto trace any particular span.For threading, the concept that I used is that the BaseProvider class needs to implement methods to create and manage thread-local storage for each trace. When a new child thread is created, the trace context can be copied to the new thread-local storage, allowing the new thread to continue the trace without interfering with the parent thread’s trace. When the thread finishes, the trace context can be removed from the thread-local storage.
I've implemented context manager for it now, however still need to provide an example of how to use it. But before I need to make sure if this sounds a good way to support threading?
heitorlessa commentedon Apr 19, 2023
Great!! As for threading, before you dive in the implementation per se, take a look at how DataDog, NewRelic, and OpenTelemetry Tracers handle threading + asyncio.
Reason I ask that is that we wouldn't necessarily need to implement threadlocal or contextvars, because the Provider would be a wrapper on top of the actual implementation (e.g., DataDog Tracer, OpenTelemetry Tracer, etc.).
What's missing in the RFC is a section with a comparison on how observability providers out there handle threading. Then call out whether we need additional public methods in the BaseProvider that will be implemented by the actual provider who already handle the threadlocal or contextvars.
As this is a complex topic, to recap, our Tracer Observability Provider is a thin wrapper on top of the actual Observability Provider SDK (e.g., DataDog SDK, OpenTelemetry SDK, New Relic SDK, etc.). This helps customers use the same Powertools DX across Providers, and each provider simply implement our interface while bringing their various flavours of SDK already handling this use case.
Please do let me know if I can make this any clearer.
Tks a lot!!
7 remaining items
roger-zhangg commentedon Apr 11, 2024
Hello everyone! I'm updating this issue with some decisions we made regard the Tracer provider
Naming
We decide the follow the naming convention of OpenTelemetry tracing. Comparing to the current provider:
BaseSegment->BaseSpanBaseProvidersignature of
set_attributeThe current proposed signature
powertools-lambda-python/aws_lambda_powertools/tracing/provider/base.py
Lines 10 to 14 in ee869b3
We decided to accept
Anyvalue here, but the actual supported data type will be decided in the specific providerDeprecation of current
BaseSegmentandBaseProviderThese two classes will remain for backwards compatibility until Powertools V3
docstring change in the current
Subsegment.close:https://github.com/aws-powertools/powertools-lambda-python/pull/2342/files#diff-af72ce002a5d3a9cfd406fddde5c809367a781bae77dad2bef6009d79dde6502R12-R18
We come to the conclusion that using float in epoch seconds in this case is more appropriate. I also created a aws/aws-xray-sdk-python#424 for this in X-Ray python SDK as I believe the docstring here is referencing the X-Ray Python SDK.
Backwards compatibility of X-Ray provider
We support all Backwards compatibility except escape hatch usage directly on X-Ray recorder. For example:
We didn't have
capturefunction in the currentBaseProviderthus this behavior will not be supported in the new provider. For existing function in the currentBaseProviderlikein_subsegment, they will still be supported.