std.zig.system.NativeTargetInfo: improve glibc version and dynamic linker detection #12788

andrewrk · 2022-09-09T01:38:15Z

Previously, this code would fail to detect glibc version because it
relied on libc.so.6 being a symlink which revealed the answer. On modern
distro versions, this is no longer the case.

This new strategy finds the path to libc.so.6 from /usr/bin/env, then
inspects the .dynstr section of libc.so.6, looking for symbols that
start with "GLIBC_2.". It then parses those as semantic versions and
takes the maximum value as the system-native glibc version.

Before, native glibc and dynamic linker detection attempted to use the
executable's own binary if it was dynamically linked to answer both the
C ABI question and the dynamic linker question. However, this could be
problematic on a system that uses a RUNPATH for the compiler binary,
locking it to an older glibc version, while system binaries such as
/usr/bin/env use a newer glibc version. The problem is that libc.so.6
glibc version will match that of the system while the dynamic linker
will match that of the compiler binary. Executables with these versions
mismatching will fail to run.

Therefore, this pull request changes the logic to be the same regardless of
whether the compiler binary is dynamically or statically linked. It
inspects /usr/bin/env as an ELF file to find the answer to these
questions, or if there is a shebang line, then it chases the referenced
file recursively. If that does not provide the answer, then the function
falls back to defaults.

This pull request also solves a TODO to remove an Allocator parameter to the
detect() function.

closes #6469
closes #12567

@Vexu can you check if this solves #11137 for you?

BratishkaErik · 2022-09-09T02:50:16Z

Doesn't work on my Gentoo system :( rpath_offset is null

andrewrk · 2022-09-09T02:52:41Z

@BratishkaErik thank you for checking. I will poke around on Gentoo in a Docker container and then push another commit for you to try shortly.

topolarity · 2022-09-09T02:53:57Z

lib/std/zig/system/NativeTargetInfo.zig

+    // Empirically, glibc 2.34 libc.so .dynstr section is 32441 bytes on my system.
+    // Here I use this value plus some headroom. This makes it only need
+    // a single read syscall here.
+    var buf: [40000]u8 = undefined;


It seems like if we just wait for enough glibc releases this buffer will be overrun eventually, since new glibc versions will essentially only ever add new symbols

As a datapoint, .dynstr on my system glibc 2.34 is 35062 bytes and 2.12 is 22098 bytes

Fair point, I will update this to account for any size.

Previously, this code would fail to detect glibc version because it relied on libc.so.6 being a symlink which revealed the answer. On modern distros, this is no longer the case. This new strategy finds the path to libc.so.6 from /usr/bin/env, then inspects the .dynstr section of libc.so.6, looking for symbols that start with "GLIBC_2.". It then parses those as semantic versions and takes the maximum value as the system-native glibc version. closes #6469 see #11137 closes #12567

Before, native glibc and dynamic linker detection attempted to use the executable's own binary if it was dynamically linked to answer both the C ABI question and the dynamic linker question. However, this could be problematic on a system that uses a RUNPATH for the compiler binary, locking it to an older glibc version, while system binaries such as /usr/bin/env use a newer glibc version. The problem is that libc.so.6 glibc version will match that of the system while the dynamic linker will match that of the compiler binary. Executables with these versions mismatching will fail to run. Therefore, this commit changes the logic to be the same regardless of whether the compiler binary is dynamically or statically linked. It inspects `/usr/bin/env` as an ELF file to find the answer to these questions, or if there is a shebang line, then it chases the referenced file recursively. If that does not provide the answer, then the function falls back to defaults. This commit also solves a TODO to remove an Allocator parameter to the detect() function.

This commit removes the check that takes advantage of when the dynamic linker is a symlink. Instead, it falls back on the same directory as the dynamic linker as a de facto runpath. Empirically, this gives correct results on Gentoo and NixOS. Unfortunately it is still falling short for Debian, which has libc.so.6 in a different directory as the dynamic linker.

This is a partial revert of the previous commit, fixing a regression on Debian. However, the commit additionally improves the detectAbiAndDynamicLinker function to read more than 1 byte at a time when detecting a shebang line.

andrewrk · 2022-09-09T03:56:17Z

@BratishkaErik I have pushed more commits that use an additional heuristic that solves Gentoo. It looks like unfortunately a future enhancement may require a full integration with /etc/ld.so.cache, however every distro I tested, these heuristics gave the correct answer. Tested distros include: NixOS, Gentoo, Archlinux, Debian

BratishkaErik · 2022-09-09T04:05:57Z

@BratishkaErik I have pushed more commits that use an additional heuristic that solves Gentoo. It looks like unfortunately a future enhancement may require a full integration with /etc/ld.so.cache, however every distro I tested, these heuristics gave the correct answer. Tested distros include: NixOS, Gentoo, Archlinux, Debian

Yes. it works fine! Thank you!

pub const os = std.Target.Os{
    .tag = .linux,
    .version_range = .{ .linux = .{
        .range = .{
            .min = .{
                .major = 5,
                .minor = 19,
                .patch = 6,
            },
            .max = .{
                .major = 5,
                .minor = 19,
                .patch = 6,
            },
        },
        .glibc = .{
            .major = 2,
            .minor = 35,
            .patch = 0,
        },
    }},
};

topolarity · 2022-09-09T04:15:57Z

a future enhancement may require a full integration with /etc/ld.so.cache, however every distro I tested, these heuristics gave the correct answer. Tested distros include: NixOS, Gentoo, Archlinux, Debian

I'm seeing bad results on Debian bookworm (testing) - glibc 2.34 but Zig reports 2.19

It looks like the latest glibc updates removed the versioned symlinks for Debian too, unfortunately

After failing to find RUNPATH in the ELF of /usr/bin/env, not finding the answer in a symlink of the dynamic interpreter, and not finding libc.so.6 in the same directory as the dynamic interpreter, Zig will check `/lib/$triple`. This fixes incorrect native glibc version detected on Debian bookworm.

andrewrk · 2022-09-09T07:11:32Z

Alright another commit pushed, now Debian bookworm is handled. If anyone wants to test the logic without compiling anything, you can do this:

$ wget https://andrewkelley.me/temp/test4
$ chmod +x test4
$ ./test4
ver: 2.34 dl: /lib64/ld-linux-x86-64.so.2

Any more distros not working still?

Vexu · 2022-09-09T10:01:49Z

Can confirm that this detects the correct glibc version for me and I can build the example in #11137 if I explicitly specify an earlier version that Zig can provide.

andrewrk · 2022-09-09T16:24:13Z

The failure affects master branch. I will address it in master branch by adding more debug printing to help when the problem happens again.

andrewrk mentioned this pull request Sep 9, 2022

std.zig.system.NativeTargetInfo: use getconf for detecting glibc version on glibc systems #12567

Closed

andrewrk changed the title ~~std.zig.NativeTargetInfo: improve glibc version and dynamic linker detection~~ std.zig.system.NativeTargetInfo: improve glibc version and dynamic linker detection Sep 9, 2022

This was referenced Sep 9, 2022

zig detects wrong libc version #6469

Closed

-lc doesn't link c_nonshared #11137

Closed

andrewrk force-pushed the detect-native-glibc branch from 25489c2 to 0d70c04 Compare September 9, 2022 02:04

This comment was marked as resolved.

Sign in to view

topolarity reviewed Sep 9, 2022

View reviewed changes

andrewrk added 5 commits September 8, 2022 20:52

stage2: fix building for 32-bit targets

1b6fa19

std.zig.system.NativeTargetInfo: restore symlink logic

9f40f34

This is a partial revert of the previous commit, fixing a regression on Debian. However, the commit additionally improves the detectAbiAndDynamicLinker function to read more than 1 byte at a time when detecting a shebang line.

andrewrk force-pushed the detect-native-glibc branch from 0d70c04 to 9f40f34 Compare September 9, 2022 03:52

std.zig.system.NativeTargetInfo: more headroom for libc.so.6 .dynstr

68e61bb

andrewrk merged commit 9e070b6 into master Sep 9, 2022

andrewrk deleted the detect-native-glibc branch September 9, 2022 16:28

BratishkaErik mentioned this pull request Sep 9, 2022

zig cannot build against a native glibc greater than the latest cross-compilation-supported version of glibc #12797

Closed

BratishkaErik mentioned this pull request Sep 22, 2022

for detecting the native dynamic linker path, zig should prefer /usr/bin/env to its own executable #5280

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

std.zig.system.NativeTargetInfo: improve glibc version and dynamic linker detection #12788

std.zig.system.NativeTargetInfo: improve glibc version and dynamic linker detection #12788

andrewrk commented Sep 9, 2022

This comment was marked as resolved.

BratishkaErik commented Sep 9, 2022

andrewrk commented Sep 9, 2022

topolarity Sep 9, 2022

andrewrk Sep 9, 2022

andrewrk commented Sep 9, 2022

BratishkaErik commented Sep 9, 2022

topolarity commented Sep 9, 2022 •

edited

Loading

andrewrk commented Sep 9, 2022

Vexu commented Sep 9, 2022

andrewrk commented Sep 9, 2022

std.zig.system.NativeTargetInfo: improve glibc version and dynamic linker detection #12788

std.zig.system.NativeTargetInfo: improve glibc version and dynamic linker detection #12788

Conversation

andrewrk commented Sep 9, 2022

This comment was marked as resolved.

BratishkaErik commented Sep 9, 2022

andrewrk commented Sep 9, 2022

topolarity Sep 9, 2022

Choose a reason for hiding this comment

andrewrk Sep 9, 2022

Choose a reason for hiding this comment

andrewrk commented Sep 9, 2022

BratishkaErik commented Sep 9, 2022

topolarity commented Sep 9, 2022 • edited Loading

andrewrk commented Sep 9, 2022

Vexu commented Sep 9, 2022

andrewrk commented Sep 9, 2022

topolarity commented Sep 9, 2022 •

edited

Loading