[question] Upgrade to conan2 resolves symbols differently

[hoisunng]

What is your question?

Hello!

I am trying to upgrade our code to use conan2, but I am stuck with this problem.
We build a python extension module, so it's a shared library, but the following minimal example also demonstrates my problem.

# main.cpp
#include "spdlog/spdlog.h"
int main()
{
    spdlog::info("Hello!");
    return 0;
}

# CMakeLists.txt
cmake_minimum_required(VERSION 3.19 FATAL_ERROR)

project(spdlog_test)

find_package(spdlog CONFIG REQUIRED)

add_executable(spdlog_test main.cpp)
target_link_libraries(spdlog_test PRIVATE spdlog::spdlog)

# conanfile.py
from conan import ConanFile
from conan.tools.cmake import cmake_layout

class Test(ConanFile):
    settings = "os", "compiler", "build_type", "arch"
    # package_type = "shared-library"
    generators = "CMakeToolchain", "CMakeDeps"

    def requirements(self):
        self.requires("spdlog/1.10.0", transitive_headers = True, transitive_libs=True)

    def layout(self):
        cmake_layout(self)

The conan profiles are similar, but for conan 2 I try to use the new tool_requires which makes it very easy to manage the compiler.

# conan 1
[settings]
os=Linux
os_build=Linux
arch=x86_64
arch_build=x86_64
compiler=clang
compiler.version=19
compiler.libcxx=libstdc++11
build_type=Release
[options]
[build_requires]
[env]
CC=/home/hoisun/.conan/data/clang/19.1.0/synsense/stable/package/f153394db194e4f468f20116e911379c34a765b0/bin/clang
CXX=/home/hoisun/.conan/data/clang/19.1.0/synsense/stable/package/f153394db194e4f468f20116e911379c34a765b0/bin/clang++

# conan 2
[settings]
os=Linux
arch=x86_64
build_type=Release
compiler=clang
compiler.cppstd=20
compiler.version=19
compiler.libcxx=libstdc++11
[tool_requires]
clang/19.1.0@synsense/stable

The executable depends on spdlog, which in turn depends on fmt. There are some symbols that spdlog uses that are weak symbols in fmt. So far so good and in both versions I seem to get the more or less the same static libraries. It's not binary identical, but the symbols I'm interested in look the same, for example:

# conan 1
$ nm /home/hoisun/.conan/data/fmt/8.1.1/_/_/package/b7974c2d7a9c0e26f4ab9c5a49ca0e4e10ba3480/lib/libfmt.a | grep _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_
0000000000000000 W _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_
$ nm /home/hoisun/.conan/data/spdlog/1.10.0/_/_/package/b172c514242aa3953b6f24b81e1d93a4adfe439d/lib/libspdlog.a | grep _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_
                 U _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_

# conan 2
$ nm /home/hoisun/.conan2/p/b/fmt4d22e3832e921/p/lib/libfmt.a | grep _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_
0000000000000000 W _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_
$ nm /home/hoisun/.conan2/p/b/spdlo13ab8f5cc4e3e/p/lib/libspdlog.a | grep _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_
                 U _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_

However, the symbol in the final executable differs in visibility. In conan 1 it was still an external (weak) symbol but when I use conan 2 it has become local (and strong).

# conan 1
$ nm spdlog_test | grep _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_
00000000000365c0 W _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_

# conan 2
$ nm spdlog_test | grep _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_
0000000000036d70 t _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_

Like I mentioned above, our real application is actually a shared library, but it also has this change of visibility of the symbol like the minimal example. This is a problem because it also contains JIT compilation support and now the JIT compiled code reports missing symbols, because it can't find it in the shared library.

I know that conan 2 has some changes that made things more private (as it should be), so less is leaked to dependants. I think I am just overlooking some small configuration, but I can't find how I can control this last linker step. Unfortunately this is blocking us from upgrading to conan 2.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @hoisunng

Thanks for your question.

I have a quick question. Why are you using?

self.requires("spdlog/1.10.0", transitive_headers = True, transitive_libs=True)

The transitive_header=True is not always necessary, only in those cases that the public headers of the dependency are included in the public headers of the current package. But it happens sometimes.
The transitive_libs=True is even more exceptional, it only happens in some occassions that the transitive_headers=True visibility produces some linking needs in the final consumer.

Have you tried with a plain self.requires("spdlog/1.13.0")?

[hoisunng]

Hi @memsharded

Thanks for your quick response. I ran it again without the extra arguments.
transitive_headers was added because I thought it was necessary to use fmt in my library, but I misunderstood that it's actually for my dependants. transitive_libs was probably just added out of desparation.

Anyway the nm output is the same with just self.requires("spdlog/1.10.0").
I am using an older version, I hope that doesn't matter. But we're not ready to upgrade both conan and all the dependencies at the same time.

# conan 1
~/work/minimal_example/build$ nm spdlog_test | grep _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_
00000000000365c0 W _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_

# conan 2
~/work/minimal_example/build2/build$ nm spdlog_test | grep _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_
0000000000036d70 t _ZN3fmt2v86detail9dragonbox10to_decimalIfEENS2_10decimal_fpIT_EES5_

[memsharded]

Something that might be worth trying, if in the latest Conan: Use the incubating CMakeDeps new generator by adding -c tools.cmake.cmakedeps:new=will_break_next, to see how the new incubating CMakeDeps behave too, because this new generator solves some of the issues that the existing one has (and are difficult to change without breaking). You can read more about it in https://docs.conan.io/2/incubating.html

[hoisunng]

Good point, I forgot to mention the versions. I am trying to upgrade from 1.65 to 2.10 (I also upgraded to 2.11 now).

After regenerating everything from conan with conan install -pr:h=clang19 -pr:b=clang19 -c:a tools.cmake.cmakedeps:new=will_break_next ../.. --build="fmt*" --build="spdlog*" -of build2
Unfortunately, there is no effect on the nm output, so the symbol is still local.

Was there maybe an (un)intentional change in conan 2 that I'm not aware of that influences the final binary and not the dependencies?

[memsharded]

Yes, the Conan 2 dependency propagation model is more complete and advanced compared to the Conan 1 one. So Conan 2 knows and model the package_type of every package (it can be shared/static/header/application), and it can also model the traits of the requirements to specify what is propagated and what not. Those are connected, and based on the package_type, the traits values are defaulted to different values.

This improves different aspects, for example it can avoid overlinkage that was way more frequent in Conan 1, and also provides better hiding of things like transitive headers, supporting better SW engineering defaults. So it is expected that the linkage of an application to transitive (not direct) shared libraries.

I still don't fully grasp the details of the linkage issue (I am also not expert in Linux shared libs symbol visibility), so the first thing that I am trying to do is to reduce to something minimal that we can fully reproduce in both sides. This is what I have:

def test_shared_symbols():
    c = TestClient()
    c.run("new cmake_lib -d name=fmt")
    c.run("export .")
    c.run("new cmake_lib -d name=spdlog -d requires=fmt/0.1 --force")
    test = textwrap.dedent("""\
        from conan import ConanFile
        from conan.tools.cmake import CMake

        class spdlogTestConan(ConanFile):
            settings = "os", "compiler", "build_type", "arch"
            generators = "CMakeDeps", "CMakeToolchain"

            def requirements(self):
                self.requires(self.tested_reference_str)

            def build(self):
                cmake = CMake(self)
                cmake.configure()
                cmake.build()

            def test(self):
                self.run("./example", env="conanrun")
                self.run("nm example")
            """)
    c.save({"test_package/conanfile.py": test})
    c.run("create . -o *:shared=True --build=missing")
    print(c.out)

It is not fully clear to me if the fmt dependency is a static library or shared one. If it is static, we should change the *:shared=True.

Then I see some symbols like:

_Z19spdlog_print_vectorRKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS5_EE

but I don't see the basic fmt() and spdlog() function call symbols.

[hoisunng]

I'm not sure if it's possible to reproduce this with just the conan commands, because of the type of symbols involved. I have to admit I'm not very familiar with them except for the basic consumer ones, nor am I an expert in shared libraries either.
I will also try to demonstrate the behaviour based on your current test example.

Just to clarify on your comment. In my case both spdlog and fmt are static libraries and their symbols get copied into my shared library/executable.

[memsharded]

I'm not sure if it's possible to reproduce this with just the conan commands, because of the type of symbols involved. I have to admit I'm not very familiar with them except for the basic consumer ones, nor am I an expert in shared libraries either.
I will also try to demonstrate the behaviour based on your current test example.

For that case it might be worth to provide some .cpp code that adds some of the potentially problematic symbols. It can also be added in the test. I am trying to reduce the scope and remove other possible moving pieces like the spdlog project itself, that might have some specifics in its build scripts or something like that.

Likewise it would be great to reproduce first with some more standard compiler like the gcc mainstream in the Linux distro. It is possible that different compilers/linkers might also behave slightly different.

[hoisunng]

Short update: I haven't been able to reproduce it without using the real libraries. I still don't really understand how the symbols work.

However, I did try with the default compiler and conan profile on ubuntu 22.04 (in WSL) and the difference also exists.