[question] Parallel jobs and independent caches

[maitrey]

What is your question?

Dear Conan Folks,

First of all Happy New Year!
I have a question, in the documentation I have read: " that the cache is still not concurrent, so parallel jobs or tasks should use independent caches."
If I have a package with multiple platform and variant combination and if I run these cobinations parallely on CI infrastructure(using seperate caches), how shall I collect and upload all the different combinations together?

Right now, I do them sequentially, and so it takes time but its easy to upload.
Is there an example that I can follow ?

Thanks!
Br,
Maitrey

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @maitrey happy new year to you too!

If I have a package with multiple platform and variant combination and if I run these cobinations parallely on CI infrastructure(using seperate caches), how shall I collect and upload all the different combinations together?

There are a couple of strategies here:

• Upload them to a "build" server repo, and once all combination finish, then run a promotion, see the new CI tutorial: https://docs.conan.io/2/ci_tutorial/packages_pipeline/multi_configuration.html
• Aggregate the packages in CI, using conan cache save/restore and the CI specific mechanism to transfer files between jobs, and then use conan upload after the job that has collected all binaries from all configurations.

Both have advantages and disadvantages, for example the later is a bit more fragile in terms of random errors, as it might need to repeat all builds for all configurations if something crashes, while the former will allow to relaunch only the jobs that are minimally necessary as it already stored in a more permanent location the binary packages.

[maitrey]

Understood, I am anyhow trying to understand the CI Tutorial. I use conan build and conan export-pkg from the game application example. I use python-3.12.4 and in Windows context. I use in multi configuration context following the guidelines as in: https://docs.conan.io/2/ci_tutorial/packages_pipeline/multi_configuration.html. The only difference is I use conan build/conan export-pkg instead of conan create.
But in both the cases when I do a conan list I get:
'utf-8' codec can't decode byte 0xff in position 0: invalid start byte.
I am using cross compilation meaning build and host profile are different
graph.json from the first command looks like this:

{
    "graph": {
        "nodes": {
            "0": {
                "ref": "conanfile",
                "id": "0",
                "recipe": "Cli",
                "package_id": null,
                "prev": null,
                "rrev": null,
                "rrev_timestamp": null,
                "prev_timestamp": null,
                "remote": null,
                "binary_remote": null,
                "build_id": null,
                "binary": null,
                "invalid_build": false,
                "info_invalid": null,
                "name": null,
                "user": null,
                "channel": null,
                "url": null,
                "license": null,
                "author": null,
                "description": null,
                "homepage": null,
                "build_policy": null,
                "upload_policy": null,
                "revision_mode": "hash",
                "provides": null,
                "deprecated": null,
                "win_bash": null,
                "win_bash_run": null,
                "default_options": null,
                "options_description": null,
                "version": null,
                "topics": null,
                "package_type": "unknown",
                "languages": [],
                "settings": {
                    "os": "RTOS",
                    "arch": "small",
                    "compiler": "ppc-vle-gcc",
                    "compiler.version": "4.9.9",
                    "build_type": "Release"
                },
                "options": {},
                "options_definitions": {},
                "generators": [],
                "python_requires": null,
                "system_requires": {},
                "recipe_folder": null,
                "source_folder": null,
                "build_folder": null,
                "generators_folder": null,
                "package_folder": null,
                "immutable_package_folder": null,
                "cpp_info": {
                    "root": {
                        "includedirs": [
                            "include"
                        ],
                        "srcdirs": null,
                        "libdirs": [
                            "lib"
                        ],
                        "resdirs": null,
                        "bindirs": [
                            "bin"
                        ],
                        "builddirs": null,
                        "frameworkdirs": null,
                        "system_libs": null,
                        "frameworks": null,
                        "libs": null,
                        "defines": null,
                        "cflags": null,
                        "cxxflags": null,
                        "sharedlinkflags": null,
                        "exelinkflags": null,
                        "objects": null,
                        "sysroot": null,
                        "requires": null,
                        "properties": null,
                        "exe": null,
                        "type": null,
                        "location": null,
                        "link_location": null,
                        "languages": null
                    }
                },
                "conf_info": {},
                "label": "cli",
                "vendor": false,
                "dependencies": {
                    "1": {
                        "ref": "ai/1.0",
                        "run": false,
                        "libs": true,
                        "skip": false,
                        "test": false,
                        "force": false,
                        "direct": true,
                        "build": false,
                        "transitive_headers": null,
                        "transitive_libs": null,
                        "headers": true,
                        "package_id_mode": null,
                        "visible": true
                    },
                    "2": {
                        "ref": "mathlib/1.0",
                        "run": false,
                        "libs": true,
                        "skip": false,
                        "test": false,
                        "force": false,
                        "direct": false,
                        "build": false,
                        "transitive_headers": null,
                        "transitive_libs": null,
                        "headers": false,
                        "package_id_mode": null,
                        "visible": true
                    }
                },
                "context": "host",
                "test": false
            },
            "1": {
                "ref": "ai/1.0#01c428141fcd8d23202bda964cb2b06d",
                "id": "1",
                "recipe": "Cache",
                "package_id": "b9d3b44ab0c90c4fd1cee18cbdd8ee8cd328bfb7",
                "prev": "bd87c9dc5ef19b5a3a787e321aebf3bc",
                "rrev": "01c428141fcd8d23202bda964cb2b06d",
                "rrev_timestamp": 1736330981.3136816,
                "prev_timestamp": null,
                "remote": null,
                "binary_remote": null,
                "build_id": null,
                "binary": "Build",
                "invalid_build": false,
                "info_invalid": null,
                "name": "ai",
                "user": null,
                "channel": null,
                "url": null,
                "license": null,
                "author": null,
                "description": null,
                "homepage": null,
                "build_policy": null,
                "upload_policy": null,
                "revision_mode": "hash",
                "provides": null,
                "deprecated": null,
                "win_bash": null,
                "win_bash_run": null,
                "default_options": {
                    "shared": false
                },
                "options_description": null,
                "version": "1.0",
                "topics": null,
                "package_type": "static-library",
                "languages": [],
                "settings": {
                    "os": "RTOS",
                    "arch": "small",
                    "compiler": "ppc-vle-gcc",
                    "compiler.version": "4.9.9",
                    "build_type": "Release"
                },
                "options": {
                    "shared": "False"
                },
                "options_definitions": {
                    "shared": [
                        "True",
                        "False"
                    ]
                },
                "generators": [],
                "python_requires": null,
                "system_requires": {},
                "recipe_folder": "C:\\Users\\.conan2\\p\\aiaed5d824120f7\\e",
                "source_folder": "C:\\Users\\.conan2\\p\\b\\ai2e48dfa4b9957\\b",
                "build_folder": "C:\\Users\\.conan2\\p\\b\\ai2e48dfa4b9957\\b\\build\\Release",
                "generators_folder": "C:\\Users\\.conan2\\p\\b\\ai2e48dfa4b9957\\b\\build\\Release\\generators",
                "package_folder": "C:\\Users\\.conan2\\p\\b\\ai2e48dfa4b9957\\p",
                "immutable_package_folder": "C:\\Users\\.conan2\\p\\b\\ai2e48dfa4b9957\\p",
                "cpp_info": {
                    "root": {
                        "includedirs": [
                            "C:\\Users\\.conan2\\p\\b\\ai2e48dfa4b9957\\p\\include"
                        ],
                        "srcdirs": null,
                        "libdirs": [
                            "C:\\Users\\.conan2\\p\\b\\ai2e48dfa4b9957\\p\\lib"
                        ],
                        "resdirs": [],
                        "bindirs": [
                            "C:\\Users\\.conan2\\p\\b\\ai2e48dfa4b9957\\p\\bin"
                        ],
                        "builddirs": null,
                        "frameworkdirs": null,
                        "system_libs": null,
                        "frameworks": null,
                        "libs": [
                            "ai"
                        ],
                        "defines": null,
                        "cflags": null,
                        "cxxflags": null,
                        "sharedlinkflags": null,
                        "exelinkflags": null,
                        "objects": null,
                        "sysroot": null,
                        "requires": [],
                        "properties": null,
                        "exe": null,
                        "type": null,
                        "location": null,
                        "link_location": null,
                        "languages": null
                    }
                },
                "conf_info": {},
                "label": "ai/1.0",
                "info": {
                    "settings": {
                        "os": "RTOS",
                        "arch": "small",
                        "compiler": "ppc-vle-gcc",
                        "compiler.version": "4.9.9",
                        "build_type": "Release"
                    },
                    "options": {
                        "shared": "False"
                    },
                    "requires": [
                        "mathlib/1.0.Z"
                    ]
                },
                "vendor": false,
                "dependencies": {
                    "2": {
                        "ref": "mathlib/1.0",
                        "run": false,
                        "libs": true,
                        "skip": false,
                        "test": false,
                        "force": false,
                        "direct": true,
                        "build": false,
                        "transitive_headers": null,
                        "transitive_libs": null,
                        "headers": true,
                        "package_id_mode": "minor_mode",
                        "visible": true
                    }
                },
                "context": "host",
                "test": false
            },
            "2": {
                "ref": "mathlib/1.0#5e85077c7ff99f0029052b34d551303c",
                "id": "2",
                "recipe": "Cache",
                "package_id": "a83f4f9d82e06f26dd16d26baeeefd7d9cc26447",
                "prev": "9d1a30c9b6f9b4bc15157bc2a16221a0",
                "rrev": "5e85077c7ff99f0029052b34d551303c",
                "rrev_timestamp": 1736326682.4685566,
                "prev_timestamp": 1736326683.1204329,
                "remote": null,
                "binary_remote": null,
                "build_id": null,
                "binary": "Cache",
                "invalid_build": false,
                "info_invalid": null,
                "name": "mathlib",
                "user": null,
                "channel": null,
                "url": null,
                "license": null,
                "author": null,
                "description": null,
                "homepage": null,
                "build_policy": null,
                "upload_policy": null,
                "revision_mode": "hash",
                "provides": null,
                "deprecated": null,
                "win_bash": null,
                "win_bash_run": null,
                "default_options": {
                    "shared": false
                },
                "options_description": null,
                "version": "1.0",
                "topics": null,
                "package_type": "static-library",
                "languages": [],
                "settings": {
                    "os": "RTOS",
                    "arch": "small",
                    "compiler": "ppc-vle-gcc",
                    "compiler.version": "4.9.9",
                    "build_type": "Release"
                },
                "options": {
                    "shared": "False"
                },
                "options_definitions": {
                    "shared": [
                        "True",
                        "False"
                    ]
                },
                "generators": [],
                "python_requires": null,
                "system_requires": {},
                "recipe_folder": "C:\\Users\\.conan2\\p\\mathl336f931840dd0\\e",
                "source_folder": null,
                "build_folder": null,
                "generators_folder": null,
                "package_folder": "C:\\Users\\.conan2\\p\\b\\mathl074b034811a54\\p",
                "immutable_package_folder": "C:\\Users\\.conan2\\p\\b\\mathl074b034811a54\\p",
                "cpp_info": {
                    "root": {
                        "includedirs": [
                            "C:\\Users\\.conan2\\p\\b\\mathl074b034811a54\\p\\include"
                        ],
                        "srcdirs": null,
                        "libdirs": [
                            "C:\\Users\\.conan2\\p\\b\\mathl074b034811a54\\p\\lib"
                        ],
                        "resdirs": null,
                        "bindirs": [
                            "C:\\Users\\.conan2\\p\\b\\mathl074b034811a54\\p\\bin"
                        ],
                        "builddirs": null,
                        "frameworkdirs": null,
                        "system_libs": null,
                        "frameworks": null,
                        "libs": [
                            "mathlib"
                        ],
                        "defines": null,
                        "cflags": null,
                        "cxxflags": null,
                        "sharedlinkflags": null,
                        "exelinkflags": null,
                        "objects": null,
                        "sysroot": null,
                        "requires": [],
                        "properties": null,
                        "exe": null,
                        "type": null,
                        "location": null,
                        "link_location": null,
                        "languages": null
                    }
                },
                "conf_info": {},
                "label": "mathlib/1.0",
                "info": {
                    "settings": {
                        "os": "RTOS",
                        "arch": "small",
                        "compiler": "ppc-vle-gcc",
                        "compiler.version": "4.9.9",
                        "build_type": "Release"
                    },
                    "options": {
                        "shared": "False"
                    }
                },
                "vendor": false,
                "dependencies": {},
                "context": "host",
                "test": false
            }
        },
        "root": {
            "0": "None"
        },
        "overrides": {},
        "resolved_ranges": {
            "mathlib/[>=1.0 <2]": "mathlib/1.0"
        },
        "replaced_requires": {},
        "error": null
    }
}

What have I udnerstood wrong or doing it incorrectly?

[memsharded]

But in both the cases when I do a conan list I get:
'utf-8' codec can't decode byte 0xff in position 0: invalid start byte.

This is unexpected, I don't know where it could come from. This seems like some file with an unexpected encoding is being loaded. Maybe running with -vvv to show the full trace could help. If you can please try to provide more details to reproduce that, it would be useful.

[maitrey]

There are two issues meanwhile, it looks like with VScode + powershell-5.1 terminal conan commands are not working with the redirect command.

 [System.Text.Encoding]::Default


IsSingleByte      : True
BodyName          : iso-8859-1
EncodingName      : Western European (Windows)
HeaderName        : Windows-1252
WebName           : Windows-1252
WindowsCodePage   : 1252
IsBrowserDisplay  : True
IsBrowserSave     : True
IsMailNewsDisplay : True
IsMailNewsSave    : True
EncoderFallback   : System.Text.InternalEncoderBestFitFallback
DecoderFallback   : System.Text.InternalDecoderBestFitFallback
IsReadOnly        : True
CodePage          : 1252

With windows command prompt it works with conan create + conan list but not with conan build and conan export-pkg which is our use-case . It seems to come from the fact that the graph.json generated during conan build point to -of instead of cache. conan list seems to return a empty list with detecting nothing from local cache.

[maitrey]

And I forgot the trace output from the conan list command:

Traceback (most recent call last):
  File "D:\Python\Python312\Lib\site-packages\conan\api\model.py", line 104, in load_graph
    graph = json.loads(load(graphfile))
                       ^^^^^^^^^^^^^^^
  File "D:\Python\Python312\Lib\site-packages\conans\util\files.py", line 143, in load
    tmp = handle.read()
          ^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Python\Python312\Lib\site-packages\conan\cli\cli.py", line 192, in run
    command.run(self._conan_api, args[0][1:])
  File "D:\Python\Python312\Lib\site-packages\conan\cli\command.py", line 180, in run
    info = self._method(conan_api, parser, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Python\Python312\Lib\site-packages\conan\cli\commands\list.py", line 251, in list
    pkglist = MultiPackagesList.load_graph(graphfile, args.graph_recipes, args.graph_binaries)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Python\Python312\Lib\site-packages\conan\api\model.py", line 109, in load_graph
    raise ConanException(f"Graph file broken: {graphfile}\n{e}")
conan.errors.ConanException: Graph file broken: C:\Projects\ai\graph.json
'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

ERROR: Graph file broken: D:\ai\graph.json
'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

[memsharded]

There are two issues meanwhile, it looks like with VScode + powershell-5.1 terminal conan commands are not working with the redirect command.

Thanks for the feedback, this explains it:

CodePage          : 1252

The default powershell encoding is not utf-8, which Conan expects it to be. I think this can be configured in powershell like with Out-File and the -Encoding.

Another alternative would be to use the new --out-file merged in #17507, that will be in next Conan 2.12, but it will have to wait until the next Conan 2.12 release.

[maitrey]

I could get it working with conan build and conan export-pkg as well. I was using it wrongly but figured out the correct commands. Since, we use conan build with -of the generated graph contains references to the output folder. The correct way is to generate the graphs when using conan export-pkg. So like this:

conan build . -of ndp --deployer=full_deploy --update -pr:b=default -pr:h=ndp
conan export-pkg . -of ndp -pr:b=default -pr:h=ndp --no-remote --format=json > graph.json
conan list --graph=graph.json --graph-binaries=build --format=json > built.json

I think this time I am doing it correctly.

[memsharded]

conan export-pkg . -of ndp -pr:b=default -pr:h=ndp --no-remote --format=json > graph.json

Yes, this is how it should be. So this is not failing now with the encoding issue?

[maitrey]

conan export-pkg . -of ndp -pr:b=default -pr:h=ndp --no-remote --format=json > graph.json

Yes, this is how it should be. So this is not failing now with the encoding issue?

It fails if I use powershell , with windows command prompt it works!

[memsharded]

Ok, yes, this is the encoding issue. As the default powershell encoding is cp1512, then the > graph.json is creating a cp1512 encoded file, instead of the expected utf-8 one.

This is what can be fixed by some of the above suggestions.

[maitrey]

But one point is the error message is not very inituitive :(. But may be it will be handled better when written to out file instead of redirecting.

[memsharded]

But one point is the error message is not very inituitive :(. But may be it will be handled better when written to out file instead of redirecting.

I agree that a more complete error message could help. It would be useful to have the full output of the command that is producing that stack trace with the -vvv, to check how the message could be improved.

[maitrey]

And I have the next problem , when I use conan build/conan export-pkg , it complains that the Lockfile doesnot exist

conan export-pkg . -of ndp -pr:b=default -pr:h=ndp --no-remote --format=json --lockfile=conan_ndp.lock > graph.json

Also the same with conan build the lockfile does not exist.

[memsharded]

And I have the next problem , when I use conan build/conan export-pkg , it complains that the Lockfile doesnot exist

I'd need the exact commands and the output of those commands. I guess that you have some conan lock create --lockfile-out=conan_ndp.lock before the build and export-pkg commands?

[maitrey]

So here is the commands that I tried:
Fails:

conan build . -of ndp --deployer=full_deploy --update -pr:b=default -pr:h=ndp --lockfile=conan_ndp.lock

Without specifying lock for conan build but specifying it for conan export-pkg also fails:

conan export-pkg . -of ndp -pr:b=default -pr:h=ndp --no-remote --format=json --lockfile=conan_ndp.lock > graph.json

I am following the documentation as in : https://docs.conan.io/2/ci_tutorial/packages_pipeline/multi_configuration_lockfile.html
But we use conan build/conan export-pkg

[maitrey]

Sorry, I need to run the conan lock creat ecommand before build/export-pkg.

[memsharded]

Sorry, I need to run the conan lock creat ecommand before build/export-pkg.

Yes, this might be important. Think that you might conan build with some dependencies versions, but if for some reason a new version of those dependencies comes in, and the export-pkg somehow picks it (--update or any other similar remote check), then the exported binary would have a different and incorrect version of the dependency than the one that was actually used for the build. Using the same lockfile for the conan build and conan export-pkg operations make this flow more robust under external concurrent changes of dependencies versions.

[maitrey]

After reading the tutorial, I have a few questions:

1. It seems that one needs to know the top level dependencies meaning the "products" pipeline. In this case, mapviewer and game. From there, one needs to generate the graph build order. But as soon as I have a new ai/1.1.0 version, how shall game/mapviewer know that it now needs to get the trigger that something new was introduced in "products" version.
   It is not clear in the tutorial if it is cron based or event based.

2. We stopped to use user and channel but still use set version to determine the branch or tag on which the package was building. Our developers are typically not software developers. We need to hide complexity. Pkgid, pkg revisions and recipe revisions are sha ids and this would already look complex to our users someof them have never worked on git. We currently plan to use the package versions like this: pkg/0.0.0-feature4567upgradeconan.23 where feature4567upgradeconan is the feature branch name or PR name and 23 is the jenkins build id. For tag builds the version is same as the tag build.
   Do you find any problems with this approach?

3. We are planning to use fixed versions only from the tag versions as this is easy for our developers to understand. In the tutorial it is using floating versions. However this means that we need to create implementation for creating a branch, edit conan recipes on the dependencies in order to get them tested.

[memsharded]

It seems that one needs to know the top level dependencies meaning the "products" pipeline. In this case, mapviewer and game. From there, one needs to generate the graph build order. But as soon as I have a new ai/1.1.0 version, how shall game/mapviewer know that it now needs to get the trigger that something new was introduced in "products" version.
It is not clear in the tutorial if it is cron based or event based.

The integration strategy might differ from org to org (or team to team), depending on the project size, number of developers, build times, CI capabilities and cost, etc. For example:

• One team might decide to run the "products" pipeline as a pre-requisite to merge any PR that modify any package. Then the products pipeline will be trigger automatically after the package pipeline has built ai/1.1.0. And the changes can be rejected if the new version doesn't integrate cleanly (and maybe require a major ai/2.0.0 version bump)
• Another team might decide to run the "products" pipeline nightly, trying to integrate all the new package versions that have been published during the day. This means that potentially ai/1.1.0 might break some products, but still it is safe as it hasn't been moved to the develop repo yet. The nightly build will report the failures to developers and they can produce the next day a new ai/1.1.1 that tries to fix the issues, and keep iterating until the nightly passes
• Other teams might decide to run the products pipeline after the PR is merged. This is an intermediate strategy between the two above, it will not do excessive building in every PR rebuilding products, because PRs might get new commits as results of reviews, etc, and the CI cost would be higher. But it will check immediately after the PR is merged, notifying as soon as possible that the "products" pipeline would be broken, so developers can act on it before waiting for the next day. It will use more CI than the nightly approach, which basically just do 1 build per day for the products.

We stopped to use user and channel but still use set version to determine the branch or tag on which the package was building. Our developers are typically not software developers. We need to hide complexity. Pkgid, pkg revisions and recipe revisions are sha ids and this would already look complex to our users someof them have never worked on git. We currently plan to use the package versions like this: pkg/0.0.0-feature4567upgradeconan.23 where feature4567upgradeconan is the feature branch name or PR name and 23 is the jenkins build id. For tag builds the version is same as the tag build.
Do you find any problems with this approach?

It depends on how you define your consumers requires() to use that version. That version is a prerelease, so in order to be able to use it, there is a conf core.version_ranges:resolve_prereleases that activates prereleases resolution. But even for prereleases, there is an order. package/0.0.0-feature2 will be resolved if existing instead of package/0.0.0-feature1, because it is "latest" for self.requires("package/0.0.0") or self.requires("package/[>=0.0 <1.0]") or something like that. When the -featureXXX prerelease versions are done concurrently, by multiple developers doing changes and pull requests and CI building them, is where lockfiles would also be very important, and you want the CI jobs and the product pipelines to use a lockfile that contains the exact -featureXXX that you are testing. But with the correct usage of lockfile, I think this is a valid strategy.

We are planning to use fixed versions only from the tag versions as this is easy for our developers to understand. In the tutorial it is using floating versions. However this means that we need to create implementation for creating a branch, edit conan recipes on the dependencies in order to get them tested.

With "floating versions" you mean version ranges?
In general, needing automation for modifying/editing Conan recipes to update them to require some specific dependency is not recommended. Note this changes recipe revisions, and this can also be an extra problem. So the general approach would be the one outlined there, keep the recipes without modifications with the reasonable version ranges, use pre-releases activation to opt-in into using prereleases and use lockfiles to solve potential concurrency issues.

[maitrey]

• One team might decide to run the "products" pipeline as a pre-requisite to merge any PR that modify any package. Then the products pipeline will be trigger automatically after the package pipeline has built ai/1.1.0. And the changes can be rejected if the new version doesn't integrate cleanly (and maybe require a major ai/2.0.0 version bump)

It is unclear How does products pipeline get automatic notification that a new version of ai is available in the products repository. The nightly build is like a cron job that I thought of.