Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

java.lang.UnsatisfiedLinkError: Unable to execute or load jffi binary stub from (TMP dir) - /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/ibm/Sterling95GA/bin/jffi4477209065662703492.so) #7717

Closed
ecout opened this issue Oct 2, 2024 · 4 comments
Closed

java.lang.UnsatisfiedLinkError: Unable to execute or load jffi binary stub from (TMP dir) - /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/ibm/Sterling95GA/bin/jffi4477209065662703492.so) #7717

ecout opened this issue Oct 2, 2024 · 4 comments

Comments

@ecout
Copy link

ecout commented Oct 2, 2024 •
edited
Loading

We've been troubleshooting an issue with dd trace apm version 1.37 on RHEL6.10 Santiago.

Datadog supports this OS with its agent up to version 7.51 but we've found the exact same issue that was closed here:
#4702

With the following code commit (the jffi libraries are excluded from the relocate call)

b46d096

And we're using version 1.37.1 of the dd-trace-java library, which still carries the same code:
datadog-apm-library-java-1.37.1-1.x86_64
https://github.com/DataDog/dd-trace-java/blob/release/v1.37.x/dd-trace-ot/build.gradle

Our Java version is:
Java(TM) SE Runtime Environment (build 1.8.0_202-b08).

We're seeing the exact same error as on the issue above. We already opened a support case and looked into the noexec portion of the error to verify that volumes where the temp folder is mounted do not have the noexec flag. We verified that's not the case and then narrowed it down to the GLIBC portion of the error, which is related to jnr-jffi, as mentioned on the issue above:
jnr/jffi#138

Please address this as soon possible as possible as it is impacting our visibility for our company's business peak season during the holidays.
Here's the support case for your reference:
https://help.datadoghq.com/hc/en-us/requests/1865303?page=1

This are our trace debug logs with the exception Stack trace details.

[dd.trace 2024-10-01 11:34:19:629 -0700] [main] DEBUG datadog.trace.agent.tooling.context.FieldBackedContextRequestRewriter - Rewriting context-store map fetch - instrumentation.class=java_concurrent instrumentation.target.context=java.lang.Runnable->datadog.trace.bootstrap.instrumentation.java.concurrent.State
[dd.trace 2024-10-01 11:34:19:630 -0700] [main] DEBUG datadog.trace.agent.tooling.AgentInstaller$TransformLoggingListener - Transformed - instrumentation.target.class=java.util.concurrent.ThreadPoolExecutor$AbortPolicy instrumentation.target.classloader=null
[dd.trace 2024-10-01 11:34:19:651 -0700] [main] DEBUG datadog.communication.http.OkHttpUtils - Using UnixDomainSocket as http transport
[dd.trace 2024-10-01 11:34:19:682 -0700] [main] DEBUG datadog.common.container.ContainerInfo - Proc file is empty
[dd.trace 2024-10-01 11:34:19:685 -0700] [main] DEBUG datadog.common.container.ContainerInfo - Unable to read cgroup inode of /proc/self/ns/cgroup because of class java.nio.file.NoSuchFileException
[dd.trace 2024-10-01 11:34:19:956 -0700] [main] DEBUG datadog.remoteconfig.ConfigurationPoller - Started remote config poller every 5000 ms
[dd.trace 2024-10-01 11:34:20:215 -0700] [main] EXCLUDE_TELEMETRY datadog.communication.ddagent.DDAgentFeaturesDiscovery - Error querying info at http://localhost:8126/
java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider
	at jnr.ffi.provider.InvalidProvider$1.loadLibrary(InvalidProvider.java:49)
	at jnr.ffi.LibraryLoader.load(LibraryLoader.java:420)
	at jnr.unixsocket.Native.<clinit>(Native.java:80)
	at jnr.unixsocket.UnixSocketChannel.<init>(UnixSocketChannel.java:101)
	at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:60)
	at datadog.common.socket.UnixDomainSocketFactory.createSocket(UnixDomainSocketFactory.java:27)
	at datadog.okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:241)
	at datadog.okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167)
	at datadog.okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
	at datadog.okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
	at datadog.okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
	at datadog.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
	at datadog.okhttp3.RealCall.execute(RealCall.java:93)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.doDiscovery(DDAgentFeaturesDiscovery.java:150)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discoverIfOutdated(DDAgentFeaturesDiscovery.java:135)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discover(DDAgentFeaturesDiscovery.java:119)
	at datadog.communication.ddagent.SharedCommunicationObjects.featuresDiscovery(SharedCommunicationObjects.java:102)
	at datadog.trace.agent.common.writer.WriterFactory.createWriter(WriterFactory.java:81)
	at datadog.trace.agent.common.writer.WriterFactory.createWriter(WriterFactory.java:42)
	at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:662)
	at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:122)
	at datadog.trace.agent.core.CoreTracer$CoreTracerBuilder.build(CoreTracer.java:483)
	at datadog.trace.agent.tooling.TracerInstaller.installGlobalTracer(TracerInstaller.java:26)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at datadog.trace.bootstrap.Agent.installDatadogTracer(Agent.java:609)
	at datadog.trace.bootstrap.Agent.access$400(Agent.java:67)
	at datadog.trace.bootstrap.Agent$InstallDatadogTracerCallback.execute(Agent.java:494)
	at datadog.trace.bootstrap.Agent.start(Agent.java:323)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at datadog.trace.bootstrap.AgentBootstrap.agentmain(AgentBootstrap.java:71)
	at datadog.trace.bootstrap.AgentBootstrap.premain(AgentBootstrap.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.UnsatisfiedLinkError: could not get native definition for type `POINTER`, original error message follows: java.lang.UnsatisfiedLinkError: Unable to execute or load jffi binary stub from `/opt/ibm/Sterling95GA/tmp`. Set `TMPDIR` or Java property `java.io.tmpdir` to a read/write path that is not mounted "noexec".
/opt/ibm/Sterling95GA/bin/jffi4477209065662703492.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/ibm/Sterling95GA/bin/jffi4477209065662703492.so)
	at com.kenai.jffi.internal.StubLoader.tempLoadError(StubLoader.java:563)
	at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:462)
	at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:338)
	at com.kenai.jffi.internal.StubLoader.<clinit>(StubLoader.java:626)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at com.kenai.jffi.Init.load(Init.java:68)
	at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:50)
	at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:46)
	at com.kenai.jffi.Foreign.getInstance(Foreign.java:104)
	at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:242)
	at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
	at com.kenai.jffi.Type.resolveSize(Type.java:155)
	at com.kenai.jffi.Type.size(Type.java:138)
	at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:198)
	at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:77)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:49)
	at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:73)
	at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:60)
	at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
	at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
	at jnr.ffi.LibraryLoader.create(LibraryLoader.java:89)
	at jnr.unixsocket.Native.<clinit>(Native.java:76)
	at jnr.unixsocket.UnixSocketChannel.<init>(UnixSocketChannel.java:101)
	at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:60)
	at datadog.common.socket.UnixDomainSocketFactory.createSocket(UnixDomainSocketFactory.java:27)
	at datadog.okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:241)
	at datadog.okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167)
	at datadog.okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
	at datadog.okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
	at datadog.okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
	at datadog.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
	at datadog.okhttp3.RealCall.execute(RealCall.java:93)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.doDiscovery(DDAgentFeaturesDiscovery.java:150)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discoverIfOutdated(DDAgentFeaturesDiscovery.java:135)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discover(DDAgentFeaturesDiscovery.java:119)
	at datadog.communication.ddagent.SharedCommunicationObjects.featuresDiscovery(SharedCommunicationObjects.java:102)
	at datadog.trace.agent.common.writer.WriterFactory.createWriter(WriterFactory.java:81)
	at datadog.trace.agent.common.writer.WriterFactory.createWriter(WriterFactory.java:42)
	at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:662)
	at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:122)
	at datadog.trace.agent.core.CoreTracer$CoreTracerBuilder.build(CoreTracer.java:483)
	at datadog.trace.agent.tooling.TracerInstaller.installGlobalTracer(TracerInstaller.java:26)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at datadog.trace.bootstrap.Agent.installDatadogTracer(Agent.java:609)
	at datadog.trace.bootstrap.Agent.access$400(Agent.java:67)
	at datadog.trace.bootstrap.Agent$InstallDatadogTracerCallback.execute(Agent.java:494)
	at datadog.trace.bootstrap.Agent.start(Agent.java:323)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at datadog.trace.bootstrap.AgentBootstrap.agentmain(AgentBootstrap.java:71)
	at datadog.trace.bootstrap.AgentBootstrap.premain(AgentBootstrap.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)

	at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:253)
	at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
	at com.kenai.jffi.Type.resolveSize(Type.java:155)
	at com.kenai.jffi.Type.size(Type.java:138)
	at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:198)
	at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:77)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:49)
	at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:73)
	at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:60)
	at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
	at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
	at jnr.ffi.LibraryLoader.create(LibraryLoader.java:89)
	at jnr.unixsocket.Native.<clinit>(Native.java:76)
	... 51 more
[dd.trace 2024-10-01 11:34:20:218 -0700] [main] DEBUG datadog.communication.ddagent.DDAgentFeaturesDiscovery - Falling back to probing, client dropping will be disabled
[dd.trace 2024-10-01 11:34:20:221 -0700] [main] ERROR datadog.trace.bootstrap.Agent - Throwable thrown while installing the Datadog Tracer
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at datadog.trace.bootstrap.Agent.installDatadogTracer(Agent.java:609)
	at datadog.trace.bootstrap.Agent.access$400(Agent.java:67)
	at datadog.trace.bootstrap.Agent$InstallDatadogTracerCallback.execute(Agent.java:494)
	at datadog.trace.bootstrap.Agent.start(Agent.java:323)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at datadog.trace.bootstrap.AgentBootstrap.agentmain(AgentBootstrap.java:71)
	at datadog.trace.bootstrap.AgentBootstrap.premain(AgentBootstrap.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.UnsatisfiedLinkError: could not load FFI provider jnr.ffi.provider.jffi.Provider
	at jnr.ffi.provider.InvalidProvider$1.loadLibrary(InvalidProvider.java:49)
	at jnr.ffi.LibraryLoader.load(LibraryLoader.java:420)
	at jnr.unixsocket.Native.<clinit>(Native.java:80)
	at jnr.unixsocket.UnixSocketChannel.<init>(UnixSocketChannel.java:101)
	at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:60)
	at datadog.common.socket.UnixDomainSocketFactory.createSocket(UnixDomainSocketFactory.java:27)
	at datadog.okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:241)
	at datadog.okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167)
	at datadog.okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
	at datadog.okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
	at datadog.okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
	at datadog.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
	at datadog.okhttp3.RealCall.execute(RealCall.java:93)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.doDiscovery(DDAgentFeaturesDiscovery.java:150)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discoverIfOutdated(DDAgentFeaturesDiscovery.java:135)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discover(DDAgentFeaturesDiscovery.java:119)
	at datadog.communication.ddagent.SharedCommunicationObjects.featuresDiscovery(SharedCommunicationObjects.java:102)
	at datadog.trace.agent.common.writer.WriterFactory.createWriter(WriterFactory.java:81)
	at datadog.trace.agent.common.writer.WriterFactory.createWriter(WriterFactory.java:42)
	at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:662)
	at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:122)
	at datadog.trace.agent.core.CoreTracer$CoreTracerBuilder.build(CoreTracer.java:483)
	at datadog.trace.agent.tooling.TracerInstaller.installGlobalTracer(TracerInstaller.java:26)
	... 20 more
Caused by: java.lang.UnsatisfiedLinkError: could not get native definition for type `POINTER`, original error message follows: java.lang.UnsatisfiedLinkError: Unable to execute or load jffi binary stub from `/opt/ibm/Sterling95GA/tmp`. Set `TMPDIR` or Java property `java.io.tmpdir` to a read/write path that is not mounted "noexec".
/opt/ibm/Sterling95GA/bin/jffi4477209065662703492.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/ibm/Sterling95GA/bin/jffi4477209065662703492.so)
	at com.kenai.jffi.internal.StubLoader.tempLoadError(StubLoader.java:563)
	at com.kenai.jffi.internal.StubLoader.loadFromJar(StubLoader.java:462)
	at com.kenai.jffi.internal.StubLoader.load(StubLoader.java:338)
	at com.kenai.jffi.internal.StubLoader.<clinit>(StubLoader.java:626)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at com.kenai.jffi.Init.load(Init.java:68)
	at com.kenai.jffi.Foreign$InstanceHolder.getInstanceHolder(Foreign.java:50)
	at com.kenai.jffi.Foreign$InstanceHolder.<clinit>(Foreign.java:46)
	at com.kenai.jffi.Foreign.getInstance(Foreign.java:104)
	at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:242)
	at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
	at com.kenai.jffi.Type.resolveSize(Type.java:155)
	at com.kenai.jffi.Type.size(Type.java:138)
	at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:198)
	at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:77)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:49)
	at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:73)
	at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:60)
	at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
	at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
	at jnr.ffi.LibraryLoader.create(LibraryLoader.java:89)
	at jnr.unixsocket.Native.<clinit>(Native.java:76)
	at jnr.unixsocket.UnixSocketChannel.<init>(UnixSocketChannel.java:101)
	at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:60)
	at datadog.common.socket.UnixDomainSocketFactory.createSocket(UnixDomainSocketFactory.java:27)
	at datadog.okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:241)
	at datadog.okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167)
	at datadog.okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
	at datadog.okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
	at datadog.okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
	at datadog.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at datadog.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
	at datadog.okhttp3.RealCall.execute(RealCall.java:93)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.doDiscovery(DDAgentFeaturesDiscovery.java:150)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discoverIfOutdated(DDAgentFeaturesDiscovery.java:135)
	at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discover(DDAgentFeaturesDiscovery.java:119)
	at datadog.communication.ddagent.SharedCommunicationObjects.featuresDiscovery(SharedCommunicationObjects.java:102)
	at datadog.trace.agent.common.writer.WriterFactory.createWriter(WriterFactory.java:81)
	at datadog.trace.agent.common.writer.WriterFactory.createWriter(WriterFactory.java:42)
	at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:662)
	at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:122)
	at datadog.trace.agent.core.CoreTracer$CoreTracerBuilder.build(CoreTracer.java:483)
	at datadog.trace.agent.tooling.TracerInstaller.installGlobalTracer(TracerInstaller.java:26)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at datadog.trace.bootstrap.Agent.installDatadogTracer(Agent.java:609)
	at datadog.trace.bootstrap.Agent.access$400(Agent.java:67)
	at datadog.trace.bootstrap.Agent$InstallDatadogTracerCallback.execute(Agent.java:494)
	at datadog.trace.bootstrap.Agent.start(Agent.java:323)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at datadog.trace.bootstrap.AgentBootstrap.agentmain(AgentBootstrap.java:71)
	at datadog.trace.bootstrap.AgentBootstrap.premain(AgentBootstrap.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)

	at com.kenai.jffi.Type$Builtin.lookupTypeInfo(Type.java:253)
	at com.kenai.jffi.Type$Builtin.getTypeInfo(Type.java:237)
	at com.kenai.jffi.Type.resolveSize(Type.java:155)
	at com.kenai.jffi.Type.size(Type.java:138)
	at jnr.ffi.provider.jffi.NativeRuntime$TypeDelegate.size(NativeRuntime.java:198)
	at jnr.ffi.provider.AbstractRuntime.<init>(AbstractRuntime.java:48)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:77)
	at jnr.ffi.provider.jffi.NativeRuntime.<init>(NativeRuntime.java:49)
	at jnr.ffi.provider.jffi.NativeRuntime$SingletonHolder.<clinit>(NativeRuntime.java:73)
	at jnr.ffi.provider.jffi.NativeRuntime.getInstance(NativeRuntime.java:60)
	at jnr.ffi.provider.jffi.Provider.<init>(Provider.java:29)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.getInstance(FFIProvider.java:68)
	at jnr.ffi.provider.FFIProvider$SystemProviderSingletonHolder.<clinit>(FFIProvider.java:57)
	at jnr.ffi.provider.FFIProvider.getSystemProvider(FFIProvider.java:35)
	at jnr.ffi.LibraryLoader.create(LibraryLoader.java:89)
	at jnr.unixsocket.Native.<clinit>(Native.java:76)
	... 51 more
This was referenced Oct 2, 2024
Closed
Closed
@headius
Copy link

headius commented Oct 10, 2024

As discussed at jnr/jffi#138 (comment), I believe DataDog just needs to update the jnr-unixsocket dependency to use the latest version, which will pull in latest jnr-ffi and jffi. Recent releases of jffi have fixed this problem by not using code that depends on the specific glibc version.

@mcculls
Copy link
Contributor

mcculls commented Oct 11, 2024 •
edited
Loading

We upgraded to jnr_posix 3.1.19 and jnr_unixsocket 0.38.22 in #6749 which went into dd-java-agent v1.31.0, so we should already be including that fix

Also in this particular case the RHEL6 image is providing GLIBC_2.12, while JFFI still needs at least 2.14

@ecout the recommended workaround, if you are not able to update the base image to a version that provides at least GLIBC 2.14, is to switch to use ports instead of unix domain sockets:

DD_TRACE_AGENT_URL=http://localhost:8126

@ecout
Copy link
Author

ecout commented Oct 21, 2024 •
edited
Loading

@headius and @mcculls That's exactly what we did(see: jnr/jffi#138 (comment)). We switched the datadog trace agent connection from the unix socket to the http socket listening on port 8126 and started receiving traces for these servers:
DD_TRACE_AGENT_URL=http://localhost:8126

And disabled the unix socket altogether as datadog had issues starting it.

Given that is the blocking issue:

Also in this particular case the RHEL6 image is providing GLIBC_2.12, while JFFI still needs at least 2.14

Thanks!!

@ecout ecout closed this as completed Oct 21, 2024
@ecout
Copy link
Author

ecout commented Oct 21, 2024

@mcculls We also worked out with datadog support that Single Step APM trace had to be disabled due to the glibc 2.12 limitation causing the unhandled exception above. Please keep that in mind for future references.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@headius @mcculls @ecout