Provide a better way to debug of why fetch request is not instrumented

[gajus]

I've been trying to add tracing using 8.0.0-alpha.4 to our codebase.

I have this code in our codebase:

// We are doing this to demonstrate how traces are propagated across services.
// The request that goes to the 'generate-random-number' route will have the same trace ID as the request that hit this route.
// Note: `?traceId=` is added for debugging purposes only and does not affect the trace propagation.
const randomNumber = await (
  await fetch(
    `https://local.contra.dev:8081/generate-random-number?traceId=${traceId}`,
  )
).text();

This is in the context of Node.js SDK.

Below are the logs at the time when the request is issued and when it is received:

Sentry Logger [log]: [Tracing] Inheriting parent's sampled decision for HTTP GET: true
Sentry Logger [log]: [Tracing] Starting span "HTTP GET" (8ca0fdc1efa7baf9)
[18:34:48.451]   3ms info  %@contra/contra-api: incoming request
reqId:   7eb88144-2f2c-4cdb-8f1e-d467bbe38428
request:
  headers:
    host:            local.contra.dev:8081
    connection:      keep-alive
    accept:          */*
    accept-language: *
    sec-fetch-mode:  cors
    user-agent:      node
    accept-encoding: br, gzip, deflate
  hostname:      local.contra.dev:8081
  method:        GET
  remoteAddress: 127.0.0.1
  remotePort:    61210
  url:           /generate-random-number?traceId=9cfafc0c6ce22d9599bbcad391c248c9
Sentry Logger [debug]: @opentelemetry/instrumentation-fastify Patching fastify reply.send function

The key thing to notice is that sentry-trace and baggage headers are not added to the request, and I cannot tell why.

My best guess is that it is somehow related to tracePropagationTargets. However, I've tried every value I could think of and I am still unable to make it work.

Here is our config:

init({
  debug: true,
  normalizeDepth: 5,
  tracePropagationTargets: [
    'contra',
    'local.contra.dev',
    '/.*/',
    '.*',
    /.*/u,
    'https://local.contra.dev',
    'https://local.contra.dev:8081',
    '/generate-random-number',
  ],
  tracesSampleRate: 1,
});

It would be nice if tracePropagationTargets accept a function, so I could at least confidently tell if that's the issue.

For what it is worth, these are the initialization logs:

Sentry Logger [log]: Integration installed: InboundFilters
Sentry Logger [log]: Integration installed: FunctionToString
Sentry Logger [log]: Integration installed: LinkedErrors
Sentry Logger [log]: Integration installed: RequestData
Sentry Logger [log]: Integration installed: Console
Sentry Logger [log]: Integration installed: Http
Sentry Logger [log]: Integration installed: NodeFetch
Sentry Logger [log]: Integration installed: OnUncaughtException
Sentry Logger [log]: Integration installed: OnUnhandledRejection
Sentry Logger [log]: Integration installed: ContextLines
Sentry Logger [log]: Integration installed: LocalVariables
Sentry Logger [log]: Integration installed: Context
Sentry Logger [log]: Integration installed: Express
Sentry Logger [log]: Integration installed: Fastify
Sentry Logger [log]: Integration installed: Graphql
Sentry Logger [log]: Integration installed: Mongo
Sentry Logger [log]: Integration installed: Mongoose
Sentry Logger [log]: Integration installed: Mysql
Sentry Logger [log]: Integration installed: Mysql2
Sentry Logger [log]: Integration installed: Postgres
Sentry Logger [log]: Integration installed: Prisma
Sentry Logger [log]: Integration installed: Nest
Sentry Logger [log]: Integration installed: Hapi
Sentry Logger [log]: Integration installed: Koa
Sentry Logger [debug]: @opentelemetry/api: Registered a global for diag v1.7.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for trace v1.7.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for context v1.7.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for propagation v1.7.0.
Sentry Logger [debug]: @opentelemetry/instrumentation-pg Applying patch for [email protected]
Sentry Logger [debug]: @opentelemetry/instrumentation-pg Applying patch for [email protected]
Sentry Logger [debug]: @opentelemetry/instrumentation-pg Patching pg.Client.prototype.query
Sentry Logger [debug]: @opentelemetry/instrumentation-http Applying patch for [email protected]
Sentry Logger [debug]: @opentelemetry/instrumentation-http Applying patch for [email protected]
Sentry Logger [debug]: @opentelemetry/instrumentation-http Applying patch for [email protected]
Sentry Logger [debug]: @opentelemetry/instrumentation-http Applying patch for [email protected]
Sentry Logger [debug]: @opentelemetry/instrumentation-graphql Applying patch for [email protected] parse
Sentry Logger [debug]: @opentelemetry/instrumentation-graphql Applying patch for [email protected] validate
Sentry Logger [debug]: @opentelemetry/instrumentation-graphql Applying patch for [email protected] execute
Sentry Logger [debug]: @opentelemetry/instrumentation-fastify Applying patch for [email protected]
Sentry Logger [debug]: @opentelemetry/instrumentation-fastify Patching fastify constructor function

It appears that NodeFetch is patched correctly, so I expect it to just work.

[gajus]

CC @mydea

[mydea]

Hello, this all looks correct as far as I can tell. What Node version are you using? I'll try to make some time to reproduce this, maybe it is more of a propagation issue 🤔

[mydea]

So after looking some more into this, turns out we were actually not respecting tracePropagationTargets at all in node so far (see #11285) - so that can't be the problem there, it must mean we are not injecting this at all.

To clarify, is the fetch span being recorded correctly? Could you share the link to a transaction event that has such a fetch http.client span in it, maybe?

[gajus]

I believe this is one of the traces https://contra.sentry.io/performance/trace/9cfafc0c6ce22d9599bbcad391c248c9/

[mydea]

OK, I think the issue is then that fetch is not instrumented at all! (The NodeFetch you see does not tell you if it actually instrumented it, though I should probably add a log message for this case there). What Node version are you using? And how exactly are you using fetch, how do you import it etc.?

[gajus]

https://github.com/gajus/sentry-tracing is a mirror of our setup.

Also more details https://gajus.com/blog/how-to-add-sentry-tracing-to-your-node-js-app.

Using Node.js v21.

[mydea]

I opened a PR, tried it with latest alpha and it seems to be working for me there! Give it a try :) We bumped the underlying fetch instrumentation, maybe that helped!

[mydea]

Is this still a problem in the current version of the SKD? Otherwise, we may close this?