From adcd24470983c5784f5019106263b884d71bd4fb Mon Sep 17 00:00:00 2001 From: Katrina Prosise Date: Thu, 16 Nov 2023 05:18:54 -0500 Subject: [PATCH] Cleanup of factory reference manual section Style guide lines applied. Removed link to the Yocto Project layers section which is part of the LmP reference manual; this was done to make navigation less confusing, and to keep sections focused. The section on Factory Sources was placed before the Factory Definition, for better explanatory flow. QA steps: Ran linter and addressed issues where possible/appropriate. Checked html output in browser, no obvious rendering issues discovered. Ran linkcheck. This commit addresses FFTK-2790 This commit addresses FFTK-1880 This commit applies to FFTK-2732 This commit applies to FFTK-2730 This commit applies to FFTK-988 Signed-off-by: Katrina Prosise --- .../reference-manual/factory/api-access.rst | 105 ++++++------ .../reference-manual/factory/ci-webhooks.rst | 85 ++++------ .../factory/data-retention.rst | 8 +- .../reference-manual/factory/event-queues.rst | 100 ++++++----- .../factory/factory-definition.rst | 157 +++++++++--------- .../factory/factory-sources.rst | 134 ++++++--------- source/reference-manual/factory/factory.rst | 26 ++- source/reference-manual/factory/sboms.rst | 26 +-- 8 files changed, 300 insertions(+), 341 deletions(-) diff --git a/source/reference-manual/factory/api-access.rst b/source/reference-manual/factory/api-access.rst index 830fe9f93..14ac3bd74 100644 --- a/source/reference-manual/factory/api-access.rst +++ b/source/reference-manual/factory/api-access.rst @@ -3,9 +3,9 @@ API Access ========== -FoundriesFactory APIs can be accessed with two different methods: +FoundriesFactory® APIs can be accessed via two methods: - #. OAuth2_ tokens managed in the `Application Credentials`_ interface. + #. `OAuth2`_ tokens managed in the `Application Credentials`_ interface. #. API Tokens managed in the `API tokens`_ interface. These credentials allow users to access: @@ -18,70 +18,81 @@ These credentials allow users to access: password to Git clone and fetch operations. * `Factory containers`_. Access is granted by passing an API token as the password to ``docker login hub.foundries.io``. - * Fioctl uses OAuth2 by default, but can also use API Tokens. + * Fioctl® uses OAuth2 by default, but can also use API Tokens. All tokens are created with scopes to help limit what they can do. .. note:: - Fioctl® has a :ref:`docker-credential-helper` which simplifies access to - hub.foundries.io. -Common scopes + Fioctl has a :ref:`docker-credential-helper` which simplifies access to + ``hub.foundries.io``. + +Common Scopes ------------- -Some common scopes users may find handy include: +Some common scopes you may find useful include: - * ``source:read-update`` - Useful for Git. - * ``targets:read, devices:read, ci:read`` - read-only access - for fioctl or REST API - * ``targets:read-update, devices:read-update, ci:read`` - read-update - access for fioctl. - * ``containers:read`` - Useful for running docker commands on - factory containers. + * ``source:read-update``: For Git. + * ``targets:read, devices:read, ci:read``: Read-only access for Fioctl or REST API. + * ``targets:read-update, devices:read-update, ci:read``: Read-update access for Fioctl. + * ``containers:read``: For running docker commands on Factory containers. .. _ref-scopes: Token Scopes ------------ -Scopes define what resources a given token may perform operations on. The -following scopes are supported: +Scopes define what resources a given token may perform operations on. +The following scopes are supported: + +Source +^^^^^^ -source:read +``source:read`` Can perform git clone/fetch/pull operations. -source:read-update +``source:read-update`` Can perform git push operations. -source:delete - Can delete a reference (git push --delete ...) and force-push (git push -f). -source:create +``source:delete`` + Can delete a reference (``git push --delete ...``) and force-push (``git push -f``). +``source:create`` Can create a new references (tags and branches). -containers:read - Can docker pull. -containers:read-update - Can docker push. - -ci:read - Can access CI builds `https://api.foundries.io/projects//lmp/`. -ci:read-update - This isn’t needed normally because ``source:read-update`` triggers CI. - However, certain custom use-cases that trigger CI builds via - `https://api.foundries.io/projects//lmp/builds/` can use this. - -devices:read - Can view device(s) `https://api.foundries.io/ota/devices/`. -devices:read-update - Can update configuration on a device - `https://api.foundries.io/ota/devices//config/` -devices:create - Can create a device (lmp-device-register with an API token). -devices:delete - Can delete a device `https://api.foundries.io/ota/devices//` - -targets:read - Can view targets.json `https://api.foundries.io/ota/factories//targets/`. -targets:read-update - Can update targets.json `https://api.foundries.io/ota/factories//targets/`. +Containers +^^^^^^^^^^ + +``containers:read`` + Can ``docker pull``. +``containers:read-update`` + Can ``docker push``. + +CI +^^ + +``ci:read`` + Can access CI builds ``https://api.foundries.io/projects//lmp/``. +``ci:read-update`` + This is not usually needed as ``source:read-update`` triggers the CI. + However, custom use-cases that trigger CI builds via ``https://api.foundries.io/projects//lmp/builds/`` may use this. + +Devices +^^^^^^^ + +``devices:read`` + Can view device(s) ``https://api.foundries.io/ota/devices/``. +``devices:read-update`` + Can update configuration on a device ``https://api.foundries.io/ota/devices//config/`` +``devices:create`` + Can create a device (``lmp-device-register`` with an API token). +``devices:delete`` + Can delete a device ``https://api.foundries.io/ota/devices//`` + +Targets +^^^^^^^ + +``targets:read`` + Can view ``targets.json`` ``https://api.foundries.io/ota/factories//targets/``. +``targets:read-update`` + Can update ``targets.json`` ``https://api.foundries.io/ota/factories//targets/``. .. _API Tokens: https://app.foundries.io/settings/tokens diff --git a/source/reference-manual/factory/ci-webhooks.rst b/source/reference-manual/factory/ci-webhooks.rst index 683ed977f..a9cf45438 100644 --- a/source/reference-manual/factory/ci-webhooks.rst +++ b/source/reference-manual/factory/ci-webhooks.rst @@ -3,13 +3,11 @@ CI Webhooks =========== -By default each factory is configured to send -:ref:`email notifications ` when CI builds complete. -Customers also have the option of receiving webhooks when builds -complete. Webhooks provide a way to do things like launching -internal workflows. +By default, each Factory is configured to send :ref:`email notifications ` when CI builds complete. +You also have the option of receiving *webhooks* when builds complete. +Webhooks provide a way to do things such as launching internal workflows. -Webhook data sent by CI looks like:: +Webhook data sent by the CI looks like:: { "build_id": 67, @@ -28,17 +26,6 @@ Webhook data sent by CI looks like:: "tests": "https://api.foundries.io/projects/andy-corp/lmp/builds/67/runs/build-amd64-partner/tests/" }, { - "name": "build-armhf", - "url": "https://api.foundries.io/projects/andy-corp/lmp/builds/67/runs/build-armhf/", - "status": "PASSED", - "log_url": "https://api.foundries.io/projects/andy-corp/lmp/builds/67/runs/build-armhf/console.log", - "web_url": "https://ci.foundries.io/projects/andy-corp/lmp/builds/67/build-armhf/", - "created": "2021-07-26T18:10:39+00:00", - "completed": "2021-07-26T18:11:10+00:00", - "host_tag": "armhf", - "tests": "https://api.foundries.io/projects/andy-corp/lmp/builds/67/runs/build-armhf/tests/" - }, - { "name": "build-aarch64", "url": "https://api.foundries.io/projects/andy-corp/lmp/builds/67/runs/build-aarch64/", "status": "PASSED", @@ -66,9 +53,7 @@ Webhook data sent by CI looks like:: "completed": "2021-07-26T18:19:24+00:00" } -This message includes an HMAC-SHA256_ HTTP header from the -Foundries `CI engine`_, ``X-JobServ-Sig`` to help authenticate the -sender. +This message includes an `HMAC-SHA256`_ HTTP header from the FoundriesFactory® `CI engine`_, ``X-JobServ-Sig`` to help authenticate the sender. .. _HMAC-SHA256: https://en.wikipedia.org/wiki/HMAC @@ -78,63 +63,61 @@ sender. Prerequisites ------------- -The customer will need a server with an HTTP(S) port exposed to the -internet. HTTPS is recommended, but the HMAC signing does help -prevent malicious 3rd parties from tampering with the message body. +You will need a server with an HTTP(S) port exposed to the internet. +HTTPS is recommended, but the HMAC signing also helps prevent malicious 3rd parties from tampering with the message body. -.. note:: Ngrok_ can be a useful tool for prototype work. +.. note:: + `Ngrok`_ can be a useful tool for prototype work. .. _Ngrok: https://ngrok.com/ -Configuring webhooks +Configuring Webhooks -------------------- Define a webhook secret for HMAC signing:: $ fioctl secrets update webhook-docs-example=UseSomethingGoodHere -This command informs the Foundries backend about a new CI related -secret. +This informs the FoundriesFactory backend about a new CI related secret. Configuring the Factory ----------------------- -The `factory-config.yml` file in ci-scripts.git will need something -like:: +``factory-config.yml`` from ``ci-scripts.git`` will need something like: + +.. code-block:: YAML notify: webhooks: - url: https://29171f519f0a.ngrok.io/webhook secret_name: webhook-docs-example +``secret_name`` ties things together so that the backend will use the correct key when signing the HTTP message. -The ``secret_name`` ties things together so that the backend will -know what key to use when signing the HTTP message. +Once this change is pushed, all builds will send webhooks to the configured server. -Once this change is pushed, all future builds will send webhooks to -the configured server. +Example +------- -A quick example ---------------- +A Docker-compose based project has been written to help prototype. +This example uses Ngrok_ so that it can be tested behind firewalls. -A simple project docker-compose based project has been written to -help users prototype. This example uses Ngrok_ so that it can be -tested behind firewalls. +.. note:: Ngrok changes URLs every time it restarts. + This requires the ``factory-config.yml`` file to change as well. -.. note:: Ngrok changes URLs every time it restarts. This requires - the factory-config.yml file to change as well. - -Prepare the app +Prepare the App ~~~~~~~~~~~~~~~ + :: $ git clone https://github.com/foundriesio/jobserv-webhook-example $ cd jobserv-webhook-example $ docker-compose build -Create the secret +Create the Secret ~~~~~~~~~~~~~~~~~ + :: # Set secret in backend: @@ -142,27 +125,27 @@ Create the secret # Set secret for compose app: $ echo UseSomethingGoodHere > webhook-secret -Launch the app +Launch the App ~~~~~~~~~~~~~~ + :: $ docker-compose up -At this point, close attention must be paid to the docker-compose -logs. Ngrok will print a message like:: +At this point, pay attention to the Docker-compose logs. +Ngrok will print a message like:: ngrok-proxy_1 | t=2021-07-26T17:51:15+0000 lvl=info msg="started tunnel" obj=tunnels name=command_line addr=http://webhook:5000 url=https://29171f519f0a.ngrok.io That tells the URL its proxying with. Take this URL and configure the factory-config.yml's ``notify.webhooks[0].url`` value. -Push a change +Push a Change ~~~~~~~~~~~~~ -Go to a branch in containers.git like "devel" and push and empty -change with:: +Go to a branch in ``containers.git`` such as ``devel`` if you have it, and push an empty change with:: $ git commit --allow-empty -m "bump to test webhooks" -Now sit and wait for CI to complete and webhook to be delivered to -the webhook app. It will print the contents of the webhook data. +Wait for it to complete and the webhook to be delivered to the webhook app. +It will print the contents of the webhook data. diff --git a/source/reference-manual/factory/data-retention.rst b/source/reference-manual/factory/data-retention.rst index 1a83e6703..426ab7d68 100644 --- a/source/reference-manual/factory/data-retention.rst +++ b/source/reference-manual/factory/data-retention.rst @@ -3,7 +3,7 @@ Data Retention Policies ======================= -Understanding data retention policies within a FoundriesFactory can help with the compliance processes for a product. +Understanding data retention policies for Foundries.io™ FoundriesFactory® can help with the compliance processes for a product. This page explains how data is managed within a Factory. FoundriesFactory consists of several tightly integrated projects: @@ -28,7 +28,7 @@ Customer Data source.foundries.io ~~~~~~~~~~~~~~~~~~~ Source code like ``containers.git`` lives in Digital Ocean's SFO2 data center where it is periodically backed up. -Backups —but not the source itself— older than 6 months get pruned. +Backups—but not the source itself—older than 6 months get pruned. Source code gets deleted when a Factory is deleted. ci.foundries.io @@ -55,10 +55,12 @@ This data get pruned periodically, and gets deleted when a Factory is deleted. api.foundries.io ~~~~~~~~~~~~~~~~ + A reverse proxy to other services, and stores no data. app.foundries.io ~~~~~~~~~~~~~~~~ + This service has two components: * A web view to ``api.foundries.io`` @@ -108,4 +110,4 @@ Backups over 6 months old get pruned. ostree.foundries.io ~~~~~~~~~~~~~~~~~~~ Devices pull LmP updates down from this service. -It is managed in a similar way to hub.foundries.io. +It is managed in a similar way to ``hub.foundries.io``. diff --git a/source/reference-manual/factory/event-queues.rst b/source/reference-manual/factory/event-queues.rst index 1244bb264..d492b7c50 100644 --- a/source/reference-manual/factory/event-queues.rst +++ b/source/reference-manual/factory/event-queues.rst @@ -3,61 +3,52 @@ Event Queues ============ -The FoundriesFactory API_ exposes everything about a Factory. In fact, -app.foundries.io and fioctl_ are both built on top of that API. While -this API is fully functional, it can be hard for certain operators to -build their custom tooling for large numbers of devices if they -must constantly poll the API for information. Event queues were -designed to solve this issue. +The FoundriesFactory® API_ exposes everything about a Factory. +In fact, ``app.foundries.io`` and Fioctl_ ® are both built on top of that API. +While this API is fully functional, it can be difficult to build custom tooling for a large numbers of devices if constantly polling the API. +Event queues were designed to solve this issue. -Event queues provide a way for customers to receive messages about -important events that are not easy to poll for. These include: +Event queues provide a way to receive messages about important events not easy polled for. +These include: * The first time a device connects to the :ref:`device gateway `. * When a device applies a configuration change. * When an OTA update starts and completes. -Event queues are implemented using Google PubSub_ to provide -a well understood and tested framework. There are two types -of event queues that can be created: +Event queues are implemented using Google PubSub_ to provide a well understood and tested framework. +There are two types of event queues: - * Push - Works as a webhook_ service. Events are sent - to a managed URL where they can be processed. + * Push: Works as a webhook_ service. + Events are sent to a managed URL where they can be processed. - * Pull - Works like a typical message queue system where one - can write their own client to receive and process events. + * Pull: Works like a typical message queue system where one can write their own client to receive and process events. -PubSub documentation includes a very useful guide_ to help decide -which approach will work best for you. They also include a wide -range of `client libraries`_ for consuming the Pull API. PubSub -subscriptions are created with default retention, expiration, -and acknowledgement values_. +The PubSub documentation includes a very useful guide_ for deciding which approach will work best for you. +They also include a wide range of `client libraries`_ for consuming the Pull API. +PubSub subscriptions are created with default retention, expiration, and acknowledgement values_. Implementation Details ---------------------- -Each FoundriesFactory is given a single PubSub Topic. Each Push -and Pull queue created by a customer results in the creation of -a PubSub Subscription. The FoundriesFactory API provides a -thin, multi-tenant friendly wrapper to manage everything. +Each FoundriesFactory is given a single PubSub Topic. +Each Push and Pull queue created by a customer results in the creation of a PubSub subscription. +The FoundriesFactory API provides a thin, multi-tenant friendly wrapper to manage everything. .. note:: - For performance reasons, new push queues can take up to five - minutes before they start receiving events. + For performance reasons, new push queues can take up to five minutes before they start receiving events. Creating a Pull Queue --------------------- -A pull queue can be created using fioctl:: +A pull queue can be created using Fioctl:: - # fioctl event-queues mk-pull - $ fioctl event-queues mk-pull docs-example $HOME/.fio-pull-queue.creds + fioctl event-queues mk-pull + fioctl event-queues mk-pull docs-example $HOME/.fio-pull-queue.creds -Fioctl® can also monitor this queue:: +Fioctl can also monitor this queue:: - $ fioctl event-queues listen docs-example $HOME/.fio-pull-queue.creds + fioctl event-queues listen docs-example $HOME/.fio-pull-queue.creds -This command also serves as a reference example_ of how to implement -a pull queue listener. +This command also serves as a reference example_ on implementing a pull queue listener. Creating a Push Queue --------------------- @@ -69,31 +60,33 @@ A push queue requires a little up front work: Quick Start Example ~~~~~~~~~~~~~~~~~~~ -A great way to prototype a push queue is by using ngrok_. This tool allows -a service running on your laptop to be exposed via an ngrok reverse -proxy. With ngrok installed, you can play with the -:download:`example push queue<../../_static/push_queue_example.py>` -code from your laptop:: + +A great way to prototype a push queue is by using ngrok_. +This tool allows a service running on your laptop to be exposed via an ngrok reverse proxy. +With ngrok installed, you can play with the :download:`example push queue <../../_static/push_queue_example.py>` code from your computer: + +.. code-block:: bash # Start ngrok - $ ngrok http 8080 - # make note of the "Forwarding https:..." + ngrok http 8080 + # make note of the "Forwarding https:..." # This URL is required for the JWT_AUDIENCE below # From another terminal: # Install required python dependencies: - $ python3 -m venv /tmp/venv - $ /tmp/venv/bin/pip3 install cryptography requests pyjwt - $ JWT_AUDIENCE= /tmp/venv/bin/python3 push_queue_example.py + python3 -m venv /tmp/venv + /tmp/venv/bin/pip3 install cryptography requests pyjwt + JWT_AUDIENCE= /tmp/venv/bin/python3 push_queue_example.py Once the server is running, you can create a push queue with:: - $ fioctl event-queues mk-push docs-push + fioctl event-queues mk-push docs-push At this point events will start showing up in the example server. Push Queue Payloads ~~~~~~~~~~~~~~~~~~~ + Incoming HTTP requests will look similar to:: { @@ -112,19 +105,19 @@ Incoming HTTP requests will look similar to:: Push Queue Security ~~~~~~~~~~~~~~~~~~~ + Incoming requests will include a header, ``Authorization: Bearer ``. -This JWT is signed with one of Google's own private keys. The -`public keys`_ are published online so that users can validate the -signatures. -The JWT audience header is set to the URL you specified when creating -the push queue. The :download:`example push queue<../../_static/push_queue_example.py>` -includes logic for validating this header. +This JWT is signed with one of Google's own private keys. +The `public keys`_ are published online so that users can validate the signatures. +The JWT audience header is set to the URL you specified when creating the push queue. +The :download:`example push queue<../../_static/push_queue_example.py>` includes logic for validating this header. Event Types ----------- DEVICE_FIRST_SEEN ~~~~~~~~~~~~~~~~~ + :: { @@ -134,6 +127,7 @@ DEVICE_FIRST_SEEN DEVICE_CONFIG_APPLIED ~~~~~~~~~~~~~~~~~~~~~ + :: { @@ -144,6 +138,7 @@ DEVICE_CONFIG_APPLIED DEVICE_OTA_STARTED ~~~~~~~~~~~~~~~~~~ + :: { @@ -155,6 +150,7 @@ DEVICE_OTA_STARTED DEVICE_OTA_COMPLETED ~~~~~~~~~~~~~~~~~~~~ + :: { @@ -167,6 +163,7 @@ DEVICE_OTA_COMPLETED DEVICE_OTA_APPS_STATE_CHANGED ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + :: { @@ -194,6 +191,7 @@ DEVICE_OTA_APPS_STATE_CHANGED DEVICE_PUBKEY_CHANGE ~~~~~~~~~~~~~~~~~~~~ + :: { @@ -206,7 +204,7 @@ DEVICE_PUBKEY_CHANGE .. _API: https://api.foundries.io/ota/ -.. _fioctl: +.. _Fioctl: https://github.com/foundriesio/fioctl .. _PubSub: diff --git a/source/reference-manual/factory/factory-definition.rst b/source/reference-manual/factory/factory-definition.rst index e6e8ef2ca..2edcb0853 100644 --- a/source/reference-manual/factory/factory-definition.rst +++ b/source/reference-manual/factory/factory-definition.rst @@ -3,16 +3,15 @@ Factory Definition ================== -Each Factory can be customized to control how CI handles it. This is managed in -the "Factory Definition" which is located in a factory's **ci-scripts.git** -repository in the **factory-config.yml** file. +Each Factory can be customized to control how CI handles it. +This is managed in the "Factory Definition" , located in a Factory's ``ci-scripts.git`` repo, in ``factory-config.yml``. .. _def-notify: notify ------ -Configures the users who receive an email with the build notification. +Configures who receives an email with build notifications. .. sidebar:: ``notify:`` Section Example @@ -26,10 +25,10 @@ Configures the users who receive an email with the build notification. notify: email: users: ``,,<...>`` - **Required:** Comma separated list of addresses to email after each CI build. + **Required:** A Comma separated list of email addresses to email after each CI build. failures_only: ```` - **Optional:** If set to ``true`` users will only be notified of CI failures. + **Optional:** If set to ``true``, users will only be notified of CI failures. **Default:** ``false`` @@ -38,12 +37,12 @@ notify: **Default:** ``Disabled`` - - url: ``https://example.com/customer-webhook-endpoint`` - **Required:** Customer owned HTTP(s) endpoint to send webhooks + url: ``https://example.com/your-webhook-endpoint`` + **Required:** A HTTP(s) endpoint you own to send webhooks secret_name: ``my-secret-name`` **Required:** See :ref:`ref-ci-webhooks` for details. failures_only: ```` - **Optional:** If set to ``true`` users will only be notified of CI failures. + **Optional:** If set to ``true``, users will only be notified of CI failures. **Default:** ``false`` @@ -52,8 +51,8 @@ notify: lmp --- -Configures the LmP aspects to this factory, including images, distro and machine names. -Variables to be used with the metadata and artifacts. +Configures the LmP aspects of the Factory, including images, distro, and machine names. +Variables to be used with metadata and artifacts. .. sidebar:: ``lmp:`` Section Example @@ -66,22 +65,22 @@ Variables to be used with the metadata and artifacts. params: EXAMPLE_VARIABLE_1: hello_world machines: - - imx8mmevk + - imx8mm-lpddr4-evk image_type: lmp-factory-image ref_options: - refs/heads/devel: + refs/heads/main: machines: - - raspberrypi3-64 + - raspberrypi4-64 params: - IMAGE: lmp-mini + IMAGE: lmp-factory-image EXAMPLE_VARIABLE_1: foo tagging: refs/heads/main: - tag: postmerge - refs/heads/devel: - - tag: devel + refs/heads/feature1: + - tag: feature1 mfg_tools: - - machine: imx8mmevk + - machine: imx8mm-lpddr4-evk image_type: mfgtool-files params: DISTRO: lmp-mfgtool @@ -90,49 +89,44 @@ Variables to be used with the metadata and artifacts. lmp: preloaded_images: enabled: ```` - **Optional:** Whether to preload docker images into the system-image as - part of a platform build. + **Optional:** Whether to preload Docker images into the system-image as part of a platform build. **Default:** ``false`` **Inherits:** ``containers`` shortlist: ``,,<...>`` - **Optional:** Comma separated list of apps to preload. If it is not specified - or its value is empty, then all Target's apps are preloaded. + **Optional:** Comma separated list of apps to preload. + If not specified, or its value is empty, then all of a Target's apps are preloaded. **Default:** None params: EXAMPLE_VARIABLE_1: ```` - **Optional:** Define an arbitrary environment variable to be passed into - the LmP build. You can define as many as you want. + **Optional:** Define an arbitrary environment variable to be passed into the LmP build. + You can define as many as you want. - **Default:** This variable is user defined and does not exist unless - instantiated. + **Default:** This variable is user defined and does not exist unless instantiated. machines:|br| ``- `` |br| ``- `` - **Required**: Specify the list of :ref:`Supported LmP Machines - ` to build for by their ``MACHINE`` name. A Factory's - subscription is generally only good for a single machine. - - **Default:** Set by user during :ref:`gs-signup` + **Required**: Specify the list of :ref:`Supported LmP Machines ` to build for, using the ``MACHINE`` name. + A Factory's subscription is generally only for a single machine. - .. note:: + **Default**: Set during :ref:`gs-signup`. - The CI is configured to decline changes in the ``machines:`` parameter. - If needed, ask a support engineer to update the machine definition for your - FoundriesFactory. + .. note:: + + The CI is configured to decline changes to the ``machines:`` parameter. + If needed, `ask a support engineer `_ to update the machine definition for your Factory. image_type:```` - **Optional:** Set the LmP image type to produce by recipe name. For - example, ``lmp-mini-image``, ``lmp-base-console-image`` from meta-lmp_. + **Optional:** Set the LmP image type by recipe name. + For example, ``lmp-mini-image``, ``lmp-base-console-image`` from meta-lmp_. - **Default:** ``lmp-factory-image`` |br| (from - **recipes-samples/images/lmp-factory-image.bb** in your - **meta-subscriber-overrides.git** repo) + **Default:** ``lmp-factory-image`` |br| + from ``recipes-samples/images/lmp-factory-image.bb`` in your ``meta-subscriber-overrides.git`` repo. - ref_options: +ref_options: refs/heads/````: **Optional:** Override options when specific git references are updated @@ -140,36 +134,37 @@ lmp: .. code-block:: yaml - # In the below example, when the branch named "devel" is built by our - # CI system, it will have its option values for "machine" and - # "params" overriden by what is specified after "refs/heads/devel:". - # In the "devel" build, IMAGE will now equal "lmp-mini" rather than - # "lmp-factory-image" as initially defined. - - lmp: - params: - IMAGE: lmp-factory-image - machines: - - imx8mmevk - ref_options: - refs/heads/devel: - machines: - - raspberrypi3-64 + # In the below example, when the branch named "feature1" is built by our + # CI system, it will have its option values for "machine" and + # "params" overriden by what is specified after "refs/heads/feature1:". + # In the "feature1" build, IMAGE will now equal "lmp-mini-image" rather than + # "lmp-factory-image" as initially defined. + + lmp: params: - IMAGE: lmp-mini + IMAGE: lmp-factory-image + machines: + - imx8mn-ddr4-evk + ref_options: + refs/heads/feature1: + machines: + - imx8mn-ddr4-evk + params: + IMAGE: lmp-mini-image + tagging: refs/heads/````:|br| ``-tag: `` **Optional:** Control how OTA_LITE tags are handled. See :ref:`ref-advanced-tagging` for more details. mfg_tools:|br| ``- machine: `` - **Optional:** Do an OE build to produce manufacturing tooling for a given - ``MACHINE``. This is used to facilitate the manufacturing process and to ensure - secure boot on devices. Currently only NXP tools are supported.** + **Optional:** Do an OE build to produce manufacturing tooling for a given ``MACHINE``. + This is used to facilitate the manufacturing process, and to ensure secure boot on devices. + Currently, only NXP® tools are supported. **Default:** None - image_type: ```` + image_type: ```` **Optional:** Sets the name of the recipe to use to build mfg_tools. **Default:** ``mfgtool-files`` |br| (from `meta-lmp-base/recipes-support/mfgtool-files/mfgtool-files_0.1.bb `_) @@ -178,24 +173,22 @@ Variables ^^^^^^^^^ * **BUILD_SDK**: - With this variable set to ``1``, the SDK artifact will be part - of the build. Reference: :ref:`ref-building-sdk`. + With this variable set to ``1``, the SDK artifact will be part of the build. + Reference: :ref:`ref-building-sdk`. * **DEV_MODE**: - This is a flexible variable used to configure the source code - into development mode. The development mode should be defined - by the user. Reference: :ref:`ref-dev-mode`. + This is a flexible variable used to configure the source code into development mode. + The development mode should be defined by you. + Reference: :ref:`ref-dev-mode`. * **DISABLE_GPLV3**: - When set to ``1``, this variable configures the source code - to avoid the LmP default packages under GPLv3. + When set to ``1``, this variable configures the source code to avoid the LmP default packages under GPLv3. Reference: :ref:`ref-remove-gplv3`. * **DISTRO**: - Defines the distro being used. Reference: :ref:`ref-linux-distro`. + Defines the distro being used. + Reference: :ref:`ref-linux-distro`. * **SSTATE_CACHE_MIRROR**: Defaults to the directory mounted on the SDK build container. - If this directory exists, it is used as the source for the - shared state cache (sstate-cache) mirror. When the directory does - not exist, the ``lmp-manifest`` value is used (currently it points - to the public HTTP shared state cache). + If this directory exists, it is used as the source for the shared state cache (``sstate-cache``) mirror. + When the directory does not exist, the ``lmp-manifest`` value is used (currently points to the public HTTP shared state cache). * **TUF_TARGETS_EXPIRE**: Is used to change the default target expiration date (default 1y). @@ -230,14 +223,13 @@ Defines the container's configuration, including some image configuration and ta containers: preloaded_images: enabled: ```` - **Optional:** Whether to preload docker images into the system-image as - part of a containers build. + **Optional:** Whether to preload Docker images into the system-image as part of a containers build. **Default:** ``false`` shortlist: ``,,<...>`` - **Optional:** Comma separated list of apps to preload. If it is not specified - or its value is empty, then all Target's apps are preloaded. + **Optional:** Comma separated list of apps to preload. + If it is not specified or its value is empty, then all Target's apps are preloaded. **Default:** None @@ -245,7 +237,7 @@ containers: **Optional:** Specify a list of architectures to build containers for. Containers are only built for the specified list. - **Default:** ``arm,arm64,amd64`` + **Default:** ``arm,arm64,amd64``. tagging: refs/heads/````:|br| ``-tag: `` @@ -259,18 +251,17 @@ containers: **Default:** false -container_registries: ---------------------- +container_registries +-------------------- container_registries: type: |br| ``aws|azure|gar`` - **Optional:** Authenticate with :ref:`third-party registries - ` during container builds. + **Optional:** Authenticate with :ref:`third-party registries ` during container builds. **Default:** none ci_scripts ---------- -Optionally use a custom version of ci-scripts_ to perform CI builds. +Optionally, use a custom version of ci-scripts_ to perform CI builds. ci_scripts: url: diff --git a/source/reference-manual/factory/factory-sources.rst b/source/reference-manual/factory/factory-sources.rst index 7ad9debb6..ead176e56 100644 --- a/source/reference-manual/factory/factory-sources.rst +++ b/source/reference-manual/factory/factory-sources.rst @@ -3,10 +3,8 @@ Factory Source Code =================== -The FoundriesFactory provides you with a private git sandbox which allows you -to maintain and customize your platform. - -Navigate to https://source.foundries.io/factories// +FoundriesFactory® provides you with a private git sandbox, allowing you to maintain and customize your platform. +Navigate to ``https://source.foundries.io/factories//``: .. figure:: /_static/factory-cgit.png :alt: Source code navigation @@ -15,114 +13,96 @@ Navigate to https://source.foundries.io/factories// CGit browser -You will find four git repositories, below is a brief description of each one. +You will find four git repositories: .. Glossary:: - meta-subscriber-overrides.git - This OE layer defines what is included into your factory image. You can add - board specific customizations and override, add and remove packages provided - in the default Linux microPlatform base. - - lmp-manifest.git - The repo manifest for the platform build. It defines which layer versions - are included in your platform image. The ``default.xml`` file is the latest - released manifest of our Linux microPlatform, and the ``.xml`` - includes your factory changes which allows you to customize your image - against our common base. - - containers.git - This is where containers and docker-compose apps are defined. It allows you - to define what containers to build, and how to orchestrate them on the - platform. By default it will build containers for amd64, aarch64, and - armhf architectures. - - ci-scripts.git - Defines your platform and container build job to our continuous integration system - which uses the data from ``master`` branch. - - The **ci-scripts.git** repository prevents a commit changing the ``lmp:machines:`` - stanza as well as any changes altering the history (force push is disabled). + ``meta-subscriber-overrides.git`` + This OE layer defines what is included in your Factory image. + You can add board specific customizations and overrides, add and remove the packages provided in the default Linux microPlatform (LmP) base. + + ``lmp-manifest.git`` + The repo manifest for the platform build. + This defines which layer versions are included in your platform image. + The ``default.xml`` file is the latest released manifest of the LmP. + ``.xml`` includes your Factory changes, allowing you to customize your image against our common base. + + ``containers.git`` + This is where containers and Docker-compose apps are defined. + It allows you to define what containers to build, and how to orchestrate them on the platform. + + ``ci-scripts.git`` + Defines your platform and container build job to our continuous integration system which uses the data from ``master`` branch. + Note that you are prevented from making a commit changing the ``lmp:machines:`` stanza , as well as any changes altering the history (force push is disabled). Factories are created to support specific machines. - If you need to alter this behavior after starting a FoundriesFactory, - please open a `support ticket `_. + If you need to alter this behavior after starting a Factory, please open a `support ticket `_. Triggering Builds ~~~~~~~~~~~~~~~~~ -If you push changes to either ``lmp-manifest.git`` or ``meta-subscriber-overrides.git``, -a new platform build will be triggered, and if successful will deploy the -update to any registered devices. +If you push changes to either ``lmp-manifest.git`` or ``meta-subscriber-overrides.git``, a new platform build will be triggered. +If successful, the update will deploy to your registered devices. -Any changes pushed to ``containers.git`` will trigger a container build job, and -any containers defined will be pushed to your factory’s private Docker -registry at: +Changes pushed to ``containers.git`` will trigger a container build job. +Defined containers will be pushed to your Factory’s private Docker registry at: ``https://hub.foundries.io//:latest`` - .. note:: - Commit messages that include ``[skip ci]`` or ``[ci skip]`` will not - trigger CI builds. + Commit messages that include ``[skip ci]`` or ``[ci skip]`` will not trigger CI builds. -Configuring CI to Build New Branches -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Configuring the CI to Build New Branches +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -By default, ``meta-subscriber-overrides``, ``lmp-manifest`` and ``containers`` -have the single branch, ``main``. -For these repos it is recommended to also have a ``devel`` branch for non-production purposes. +By default, ``meta-subscriber-overrides``, ``lmp-manifest`` and ``containers`` have the single branch, ``main``. +For these repos, it is recommended to also have a ``devel`` branch for non-production purposes. ``ci-scripts`` has the single default branch ``master``. Platform Branches ^^^^^^^^^^^^^^^^^ -To create new buildable platform branches, first enable the new branch in -``ci-scripts``, for example: +To create new buildable platform branches, first enable the new branch in ``ci-scripts``. +For example, if you wanted to add a development branch named ``feature1``: -.. code-block:: +.. code-block:: YAML lmp: tagging: refs/heads/main: - tag: main - refs/heads/devel: - - tag: devel - refs/heads/new_branch: - - tag: new_branch + refs/heads/feature1: + - tag: feature1 -Then branch out from the wanted branches in ``meta-subscriber-overrides`` and -``lmp-manifest``. For example, using ``devel`` as a base for the new branch: +Then branch out from the wanted branches in ``meta-subscriber-overrides`` and ``lmp-manifest``. +For example, using ``main`` as a base for the new branch: .. prompt:: bash host:~$ cd meta-subscriber-overrides - git checkout devel - git checkout -b new_branch - git commit -m "[skip ci] create new branch" --allow-empty - git push --set-upstream origin new_branch + git checkout main + git checkout -b feature1 + git commit -m "[skip ci] create devel branch" --allow-empty + git push --set-upstream origin feature1 -The ``lmp-manifest`` repo change is similar as above, but includes an additional -change to point to the correct ``meta-subscriber-overrides`` branch: +The ``lmp-manifest`` repo change is similar as above, but includes an additional change to point to the correct ``meta-subscriber-overrides`` branch: .. prompt:: bash host:~$ cd lmp-manifest - git checkout devel - git checkout -b new_branch - sed -i 's/devel/new_branch/' .xml + git checkout main + git checkout -b feature1 + sed -i 's/main/feature1/' .xml git add .xml git commit -m "point meta-subscriber-overrides to correct branch" - git push --set-upstream origin new_branch + git push --set-upstream origin feature1 -After the last step, a platform build for the ``new_branch`` is triggered in the -factory. +After the last step, a platform build for the ``feature1`` branch is triggered for your Factory. Container Branches ^^^^^^^^^^^^^^^^^^ -To create new buildable container branches, first enable the new branch in -``ci-scripts``, for example: +To create new buildable container branches, first enable the new branch in ``ci-scripts``, for example: .. code-block:: @@ -130,20 +110,16 @@ To create new buildable container branches, first enable the new branch in tagging: refs/heads/main: - tag: main - refs/heads/devel: - - tag: devel - refs/heads/new_branch: - - tag: new_branch + refs/heads/feature1: + - tag: feature1 -Then branch out from the wanted branch in ``containers``, for example using -``devel``: +Then branch out from the wanted branch in ``containers``, for example using ``main``: .. prompt:: bash host:~$ cd containers - git checkout devel - git checkout -b new_branch - git push --set-upstream origin new_branch + git checkout main + git checkout -b feature1 + git push --set-upstream origin feature1 -After the last step, a container build for the ``new_branch`` is triggered in -the factory. +After the last step, a container build for the ``feature1`` is triggered for your Factory. diff --git a/source/reference-manual/factory/factory.rst b/source/reference-manual/factory/factory.rst index 9eeb423bc..f423da135 100644 --- a/source/reference-manual/factory/factory.rst +++ b/source/reference-manual/factory/factory.rst @@ -3,39 +3,33 @@ FoundriesFactory ================ -**FoundriesFactory**: The set of tools, services, and support that enable a Factory and assist it -throughout the device lifecycle. - +**FoundriesFactory®**: The set of tools, services, and support that enable a Factory and assist it throughout the device lifecycle. This section provides support for working with and customizing FoundriesFactory and the LmP. -Topics that deal with how CI functions include :ref:`ref-factory-definition` and :ref:`ref-ci-webhooks`. +* Topics that deal with how CI functions include :ref:`ref-factory-definition` and :ref:`ref-ci-webhooks`. -:ref:`ref-fioctl` provides a helpful CLI tool for :ref:`ref-api-access`, -while using :ref:`event queues` helps with managing a large number of devices. +* :ref:`ref-fioctl` provides a helpful CLI tool for :ref:`ref-api-access`, + while using :ref:`event queues` helps with managing a large number of devices. -For customizing the Linux Micro Platform (LmP) image, -or if you are curious in knowing the "ingredients" included in a build, -The :ref:`ref-linux-layers` page covers the collections of recipes used. -Each Factory has a set of git repositories used for customizing the LmP platform, defining container -and Docker Compose apps, and managing CI build jobs. +Each Factory has a set of git repositories used for customizing the LmP platform, defining container and Docker Compose apps, and managing CI build jobs. These repositories are explored under :ref:`ref-factory-sources`. -For compliance concerns, there is an overview of the FoundriesFactory :ref:`ref-data-retention` policies, -which covers customer (Factory) data and device data. +For compliance concerns, there is an overview of the FoundriesFactory :ref:`ref-data-retention` policies, which covers customer (Factory) data and device data. The FoundriesFactory :ref:`sbom` (SBOM) feature can assist with for license compliance. .. toctree:: :maxdepth: 1 + Factory Sources factory-definition Fioctl - Factory Sources sboms api-access - ../linux/linux-layers ci-webhooks event-queues data-retention .. seealso:: - :ref:`account-management` for managing a FoundriesFactory subscription and access control settings. + :ref:`account-management` covers managing a FoundriesFactory subscription and access control settings. + + :ref:`lmp-customization` explores ways of customizing the LmP platform. diff --git a/source/reference-manual/factory/sboms.rst b/source/reference-manual/factory/sboms.rst index 211cd86d2..c73a29c1d 100644 --- a/source/reference-manual/factory/sboms.rst +++ b/source/reference-manual/factory/sboms.rst @@ -6,23 +6,27 @@ Software Bill of Materials A Software Bill of Materials(SBOM) declares *the list of software packages used to build a Target*. SBOMs are foundational to understanding: - * Inventory management — the packages a Target uses. - * License compliance — the software licenses of the packages. - * Vulnerability management — the package versions. + * Inventory management—the packages a Target uses. + * License compliance—the software licenses of the packages. + * Vulnerability management—the package versions. -The FoundriesFactory SBOM feature extracts the SBOM data and analyzes it according to your needs. +The FoundriesFactory® SBOM feature extracts the SBOM data and analyzes it according to your needs. .. important:: - `Per our terms and conditions `_: FoundriesFactory build SBOMs (“the SBOM data”) are provided for your use and are generated from SPDX metadata in all project source code files. Responsibility for open source license compliance rests with you. In no event shall Foundries.io Limited be liable for any claim, damages or other liability, whether in an action of contract, tort or other legal theory, arising from, out of, or in connection with the use of the SBOM data. + `Per our terms and conditions `_: + FoundriesFactory build SBOMs (“the SBOM data”) are provided for your use and are generated from SPDX metadata in all project source code files. + Responsibility for open source license compliance rests with you. + In no event shall Foundries.io Limited be liable for any claim, damages or other liability, + whether in an action of contract, tort or other legal theory, arising from, out of, or in connection with the use of the SBOM data. SBOMs and Builds ---------------- The FoundriesFactory CI generates SBOM artifacts whenever there a change happens in a Factory build. -this happens for two kinds of builds: +This happens for two kinds of builds: - * Yocto Project — `Software Package Data Exchange`_ (SPDX) artifacts using built-in tooling. - * Container — produce SDPX artifacts using Syft_. + * Yocto Project: `Software Package Data Exchange`_ (SPDX) artifacts using built-in tooling. + * Container: produce SDPX artifacts using Syft_. You can download them from the web UI when viewing a Target. Both artifacts go into the ``sboms`` directory. @@ -72,7 +76,7 @@ For example:: Notice how: * The Target SBOMs come from container build 222 and Yocto build 262. - * The Yocto build has 3 different SBOMs, available as `tar.zst` files. Two of note: + * The Yocto build has 3 different SBOMs, available as ``tar.zst`` files. Two of note: * ``initramfs-...``; runtime packages * ``lmp-factory-image-...``; packages required for boot. @@ -101,9 +105,9 @@ These tend to work with two competing SBOM formats: * SPDX_ * CycloneDX_ -Customers may want to just export their SBOM data into spreadsheets for quick, custom processing. +You may want to just export their SBOM data into spreadsheets for quick, custom processing. While the native storage format for Factory SBOMs is SPDX, -the Foundries.io API provides a best-effort conversion to both CycloneDX and CSV. +the Foundries.io™ API provides a best-effort conversion to both CycloneDX and CSV. This allows users to export data from their Factory and into their tool of choice. To view an SBOM in a given format::