[question] [docs] Rules about upgrading the requirements

[prince-chrismc]

Given that a recipe has many dependencies, it's very common that one of it's requirements can have a higher version available in CCI then whats written in the recipe.

For header only there is no issue as the requirement can be overridden by the consuming conanfile, however compiled libs may suffer from API/ABI incompatibilities. This makes resolving conflicts very painful when two graphs overlap. Typically we get stuck building the package locally.

Are we permitted to open PR's where the only change it to the version of a required dependency?

Ideally as a consumer, it's preferable to pull everything from CCI rather than building things locally.

[jgsogo]

Yes, the main problem is when there is a conflict because two graphs require a different version of some requirements. We would really want to avoid this situation, so it is perfectly ok to open a PR to upgrade only a requirement.

What else can be done?

• Can CCI trigger a job with every new version of a library (or daily), check for other libraries consuming it, modify the conanfile.py and submit a PR? Some challenges here:

  • "Find other libraries consuming it": we store the reverse graph, but only for requirements that are consumed with the default option values. Probably this is good enough, getting a conflict after modifying a default option is not so painful (you would need to build from sources).
  • "Modify the conanfile.py": probably a regex replace would work. It can fail if the requirement name/version is not written in plain text, but... 99% it will work.
  • What if the job fails because the new version is incompatible? Who fixes the PR? Better to have a failing PR than nothing.

• Can we have a set of master versions and other recipes should require those ones?

  • It would be challenging to bump one of them and rebuild all affected packages
  • How to deal with consumers that need other versions than the master one?

Any other alternative?

[SSE4]

let's try to illustrate the situation on the hypothetical example. for instance, we have a library boost, which did release 1.72.0. at the time of release, boost had a dependency on another library, named zlib, and the latest release of zlib was 1.2.11.
it means in conan we would have a relationship like: boost/1.72.0 -> zlib/1.2.11.
okay, we have conan packages for both boost and zlib. now imagine that zlib suddenly made a new release, e.g. 1.2.12. should we upgrade boost release 1.72.0 right away to reference new zlib release?
it might look that obvious decision is to upgrade, but is it really so simple?
when boost maintainers did 1.72.0 release they run their regression tests with zlib 1.2.11, they tested various configurations (different compiler versions, operations systems, etc.) to ensure stability.
in conan we don't run so complete testing matrix, we only ensure that packages successfully build and pass simple test package. we never run test suites, and we never run the same set of supported configurations.
the same probably applies to every package.
I understand zlib example might be not so illustrative, as it's pretty stable and has a very stable interface, but in general - does it mean we always have to keep these references on bleeding edge to ensure there are no conflicts?

[uilianries]

Most of projects don't use a specific version for dependencies, it's hard for us tracking it. Indeed should reasonable keeping them aligned due the compatibility, but would need to check which version the author used, the latest version available when that release was made (by date)?

[SSE4]

@uilianries yes, there a projects like that. on the opposite side, there are cases then versions of dependencies are very hard requirements. for instance, your favorite protobuf - in case the project includes sources generated with protoc, it will need to link with the same version of protobuf, as generated sources contain hard version check.

[prince-chrismc]

My particular pain is fmt, which has a very aggressive release cycle, much more than it's dependents. Juggling 4 projects which require it is a growing pain. Version 6.0, 6.2, and 7.0 are requirements but all support 7.0 since they are not impacted by the API changes.

I would have to agree that automatically upgrading the requirements would be a sticky situation. To add to SSE$'s points, some recipes require a different version depending on the version (but these could be changed).

I have to test my code base when I changing dependencies, I may submit a few PRs after I am confident there's no issue. There will have to be a case by case review.

Would it be possible for the bot to fill an issue that shows all the conflicts?

[SpaceIm]

Auto updating recipes doesn't scale. What about recipes with a lot of dependencies (and big binaries)?
Look at GDAL. Do you really want to create another revision each time one of its many dependencies release a new version?
I think that CCI recipes should support version range, and eventually a "preferred" version if reproducibility is an issue (but a lock file should be what ensure reproducibility, not specific hardcoded version in recipes).
Consumers lose a lot of information without version range, they have to figure out themself compatible versions, while it is basically what a package manager should provide.

[timblechmann]

looking at this from a different angle: as i've came across this when trying to resolve dependency conflicts in my project. i certainly understand the combinatoric explosion of configurations and problems that arise from dependency management, but i tend to run into issues where different dependencies pull in different versions of other dependencies.

libcurl is a great example:

11:35 $ git grep libcurl\/
recipes/cfitsio/all/conanfile.py:            self.requires("libcurl/7.73.0")
recipes/cpr/all/conanfile.py:        self.requires("libcurl/{}".format("7.67.0" if not self._supports_openssl else "7.69.1"))
recipes/czmq/all/conanfile.py:            self.requires("libcurl/7.71.1")
recipes/date/all/conanfile.py:            self.requires("libcurl/7.69.1")
recipes/gdal/all/conanfile.py:            self.requires("libcurl/7.73.0")
recipes/kcov/all/conanfile.py:                "libcurl/7.64.1",
recipes/libgit2/0.27.x/conanfile.py:            self.requires("libcurl/7.67.0")
recipes/libmediainfo/all/conanfile.py:        "libcurl/7.69.1",
recipes/mariadb-connector-c/all/conanfile.py:            self.requires("libcurl/7.73.0")
recipes/poppler/all/conanfile.py:            self.requires("libcurl/7.73.0")
recipes/proj/all/conanfile.py:            self.requires("libcurl/7.73.0")
recipes/sentry-native/all/conanfile.py:            self.requires("libcurl/7.71.0")
recipes/tgbot/all/conanfile.py:        self.requires("libcurl/7.72.0")

different recipes depend on different libcurl versions. typically it seems to be the "most recent version when writing a packge".

openssl is another, where recipes tend to depend on 1.1.1d, 1.1.1g, 1.1.1h and 1.1.1i. in the case of openssl where 1.1 has a stable ABI, but 'letter' releases tend to be security fixes, it seems to be rather important to actually pull in the "latest 1.1" rather than the "latest at the time of writing the recipe".

this makes me wonder if there should be some kind of recommendation to require with version ranges rather than a specific version? in the world of package managers like debian packages this seems to be quite common

[prince-chrismc]

The challenge is the ABI correctness is not always guaranteed. with libcurl https://abi-laboratory.pro/index.php?view=timeline&l=curl every minor release is an ABI break, so your range would be >=7.73.0 && <7.74.0 but then you tradeoff on reproducibility, and you would be generating different package_id... you would still need to solve the combinatric hell that we already live with =D

[timblechmann]

with libcurl https://abi-laboratory.pro/index.php?view=timeline&l=curl every minor release is an ABI break

this is certainly not true: if you consult the "Backward Compat.", the only change ABI break listed here was 7.62.0 where the value of an enum was changed. all other changes are backwards compatible. of course ABIs can break, but in these cases one would have to decide on case-by-case basis. considering that a package could be built from sources, one may not even worry about ABI compatibility, but merely about API compatibility.

then you tradeoff on reproducibility

well, conan already has issues when it comes to reproducibility: since recipes aren't versioned one build of libcurl=7.73.0 might be quite different than another version of libcurl=7.73.0 😬

---

out of curiosity: in what way is conan different than debian packages? after all version ranges work quite well for debian, which is a rather mature project with a huge number of packages

[prince-chrismc]

There's a reason the squares are colored... orange, yellow, green have meaning.

You might find backwards compatible with a sprinkle of surprise behavior ! https://abi-laboratory.pro/index.php?view=compat_report&l=curl&v1=7.69.1&v2=7.70.0&obj=19c9c&kind=abi#c_1

last != last ... Oops!

---

since recipes aren't versioned

They are, CCI fully supports revisions check https://conan.io/center/libcurl you need to enable you local client for this feature to be enabled. Check this blog out https://blog.conan.io/2019/03/12/New-conan-release-1-13.html

[timblechmann]

You might find backwards compatible with a sprinkle of surprise behavior ! https://abi-laboratory.pro/index.php?view=compat_report&l=curl&v1=7.69.1&v2=7.70.0&obj=19c9c&kind=abi#c_1

reading this it is a little paranoid:

• adding struct members without reordering is the typical approach for keeping ABI compatibility
• changing a VERSION_LATEST enum is not exactly an ABI change: it really matters how the value is used in the implementation.
• also renaming parameters from len to len_in in openssl 1.1.1g is in no way an ABI change. once can even argue that it's not an API change, as c doesn't have keyword arguments

since recipes aren't versioned

They are, CCI fully supports revisions check https://conan.io/center/libcurl you need to enable you local client for this feature to be enabled. Check this blog out https://blog.conan.io/2019/03/12/New-conan-release-1-13.html

interesting. i need to have a look how to integrate this into my project. though i wonder why this is per-local-client rather than per-project 🤔

---

though again: in which way are conan recipes different to debian packages?

[uilianries]

Closing this issue due the new redesigned ConanCenterIndex CI service.

Please, read #25772 for further information.