[BUG] Worker fails when azure-core-tracing-opencensus is installed

[helsaawy]

Investigative information

Please provide the following:

• Timestamp: n/a
• Function App name: n/a
• Function name(s) (as appropriate): n/a
• Core Tools version: 3.0.3568 Commit hash: e30a0ede85fd498199c28ad699ab2548593f759b (64-bit)

Repro steps

1. pip install azure-core-tracing-opencensus
2. import part of the above package, ie: from azure.core.tracing.ext.opencensus_span import OpenCensusSpan
3. Call the function:

import azure.functions as func
from azure.core.tracing.ext.opencensus_span import OpenCensusSpan

def main(req: func.HttpRequest) -> func.HttpResponse:
    return func.HttpResponse(f"Python HTTP trigger function processed a request.")

Expected behavior

Proper tracing of Azure SDK calls

Actual behavior

Function crashes with the following error

Executed 'Functions.HttpTrigger1' (Failed, Id=b5e974a0-575b-4c52-b60d-65d18f41758a, Duration=135ms)
System.Private.CoreLib: Exception while executing function: Functions.HttpTrigger1. System.Private.CoreLib: Result: Failure
Exception: AttributeError: 'NoopTracer' object has no attribute 'sampler'
Stack:   File "/usr/lib/azure-functions-core-tools-3/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 398, in _handle__invocation_request
    call_result = await self._loop.run_in_executor(
  File "/anaconda/lib/python3.8/asyncio/base_events.py", line 783, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/anaconda/lib/python3.8/site-packages/opencensus/ext/threading/trace.py", line 126, in call
    wrapped_kwargs["sampler"] = _tracer.sampler

Known workarounds

Inject current tracing RuntimeContext into Tasks created by the dispatcher by modifying the task factory before functions run (ie, in an extension)

from azure_functions_worker.dispatcher import DispatcherMeta, ContextEnabledTask 
from opencensus.trace.tracer import Tracer
from opencensus.common.runtime_context import RuntimeContext

tracer = Tracer(...)
dispatcher = DispatcherMeta.__current_dispatcher__
dispatcher._loop.set_task_factory(
    RuntimeContext.with_current_context(
        lambda loop, coro: ContextEnabledTask(coro, loop=loop)
    )
)

Contents of the requirements.txt file:

azure-core-tracing-opencensus == 1.0.0b8
azure-functions
azure-functions-durable
azure-identity
azure-mgmt-containerinstance
opencensus-ext-azure
opencensus-ext-logging

[Hazhzeng]

Hi @helsaawy, thanks for reporting this issue.

The Azure Monitoring team and the Azure Functions team are cooperating and recently we introduces a new feature to support this scenario. Please follow the guideline at https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference-python?tabs=azurecli-linux%2Capplication-level#log-custom-telemetry. You can still use the opencensus-ext-azure and opencensus-ext-logging along with context.tracer in the example.

[helsaawy]

Hello,
This solves my issue, thank you

[DRV2SI]

Hello,

I ran into the exact same issue as documented by @helsaawy . According to my research, the methodology as described at the referred link does not lead to the Azure SDK being instrumented.
This is because the decorator @distributed_trace will only receive a valid span if and only if this property exists - which is exactly what azure-core-tracing-opencensus provides.

Leaving this library out and following the linked tutorial leads to custom spans being emitted, but not Azure SDK spans.
Including this library does lead to emission of SDK spans, but also the error as described above.

Could you please shed some light on how you managed to resolve this? Thank you in advance!
@helsaawy @Hazhzeng