Secure boot failure if silent=1 specified in CONFIG_EXTRA_ENV_SETTINGS

[MMG-DG]

Using an iMX8MP with kirkstone yocto build (based on tdx-minimal-reference-image)...

Having added silent=1 loglevel=0 udev.log_level=0 settings to the u-boot environment, Should any critical errors be written to the console.

I expect the following line should be changed to print error and not just normal printf message to console?
0008-toradex-integrate-bootargs-protection-downstream.patch

I assume there is something that can allow this to still be printed as a critical error message?
At least I believe critical errors should still be surfaced? (I have seen Linux kernel errors when they are raised, but nothing from u-boot; maybe the silent setting can be removed if the loglevel one(s) will silence all output except for errors?).

[MMG-DG]

Just an update from my local testing....

If I remove the silent=1 from the u-boot commands then it would appear I still get full output (which shows a lot of memory addresses and information to a potential attacker).
This means that for my case I will need to continue with setting this value and just hope that nothing else goes wrong when setting the required fuses.

On a positive note, it does appear that the issue I was seeing in my previous build was likely not related to bootargs and may have been an error in one of my systemd services causing a boot loop due to watchdog timeouts.

[MMG-DG]

Addition to above from local testing...

In my latest testing, it would seem that if I add the silent=1 to the CONFIG_EXTRA_ENV_SETTINGS in include/configs/verdin-imx8mp.h, that once I set the secure boot fuse, my device now ends up in an infinite boot loop with no output to determine why this is happening.
This is the only difference I have between a SoM that works (but shows a lot of information that shows memory addresses that could be used to attack the system) and a SoM the gets into a boot loop (which does not show any details.

This is my current patch being applied that doesn't work.

From 7ce11a7ac92bf4cae930f761ae0645f9dda6a1ff Mon Sep 17 00:00:00 2001
From: John Doe <[email protected]>
Date: Sat, 24 Aug 2024 16:14:49 +0100
Subject: [PATCH 12/12] Silent uboot args

---
 include/configs/verdin-imx8mp.h | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/include/configs/verdin-imx8mp.h b/include/configs/verdin-imx8mp.h
index 194c2b4364..d9e1c03b04 100644
--- a/include/configs/verdin-imx8mp.h
+++ b/include/configs/verdin-imx8mp.h
@@ -88,16 +88,19 @@
 
 /* Initial environment variables */
 #define CONFIG_EXTRA_ENV_SETTINGS \
+	"silent=1\0" \                           <<<< if this is set to `"" \` then everything works but with the output shown below
+	"loglevel=0\0" \
+	"udev.log_level=0\0" \
 	BOOTENV \
 	MEM_LAYOUT_ENV_SETTINGS \
 	"boot_file=Image\0" \
 	"boot_scripts=" BOOT_SCRIPT "\0" \
 	BOOT_SCRIPT_DHCP \
-	"console=ttymxc2\0" \
+	"console=null\0" \
 	"fdt_board=dev\0" \
 	"initrd_addr=0x43800000\0" \
 	"initrd_high=0xffffffffffffffff\0" \
-	"setup=setenv setupargs console=tty1 console=${console},${baudrate} " \
+	"setup=setenv setupargs console=tty1 console=null quiet loglevel=0 udev.log_level=0 " \
 		"consoleblank=0 earlycon\0" \
 	"update_uboot=askenv confirm Did you load flash.bin (y/N)?; " \
 		"if test \"$confirm\" = \"y\"; then " \
-- 
2.34.1

I also have the following addition to the u-boot configs:

CONFIG_SILENT_CONSOLE=y
CONFIG_SILENT_CONSOLE_UPDATE_ON_SET=y
CONFIG_SILENT_CONSOLE_UPDATE_ON_RELOC=y
CONFIG_SYS_DEVICE_NULLDEV=y
CONFIG_SILENT_U_BOOT_ONLY=n

Broken SoM Output:

U-Boot SPL 2022.04-6.7.0-devel+git.7588eb559ca2 (May 28 2024 - 11:19:14 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
DDR configured as single rank
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Boot Stage: Primary boot
Find img info 0x&48026c00, size 20480
Need continue download 19456
Download 884224, Total size 904352

Authenticate image from DDR location 0x401fadc0...
NOTICE:  BL31: v2.6(release):lf_v2.6-g3c1583ba0a
NOTICE:  BL31: Built : 11:00:38, Nov 21 2022

U-Boot SPL 2022.04-6.7.0-devel+git.7588eb559ca2 (May 28 2024 - 11:19:14 +0000)
##### repeated every 1-2 seconds #####

Output of working non-silent SoM:

U-Boot SPL 2022.04-6.7.0-devel+git.7588eb559ca2 (May 28 2024 - 11:19:14 +0000)
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done
DDR configured as single rank
SEC0:  RNG instantiated
Normal Boot
WDT:   Started watchdog@30280000 with servicing (60s timeout)
Boot Stage: Primary boot
Find img info 0x&48026c00, size 20480
Need continue download 19456
Download 884224, Total size 904320

Authenticate image from DDR location 0x401fadc0...
NOTICE:  BL31: v2.6(release):lf_v2.6-g3c1583ba0a
NOTICE:  BL31: Built : 11:00:38, Nov 21 2022


U-Boot 2022.04-6.7.0-devel+git.7588eb559ca2 (May 28 2024 - 11:19:14 +0000)

CPU:   i.MX8MP[8] rev1.1 1600 MHz (running at 1200 MHz)
CPU:   Industrial temperature grade (-40C to 105C) at 29C
Reset cause: POR
DRAM:  4 GiB
Core:  91 devices, 25 uclasses, devicetree: separate
WDT:   Started watchdog@30280000 with servicing (60s timeout)
MMC:   FSL_SDHC: 1, FSL_SDHC: 2
Loading Environment from MMC... ## Warning: Unknown environment variable type 'i'
## Warning: Unknown environment variable type 'i'
## Warning: Unknown environment variable type 'i'
OK
In:    serial
Out:   serial
Err:   serial
Model: Toradex 0058 Verdin iMX8M Plus Quad 4GB WB IT V1.1A
Serial#: 1500XXXX
get_tdx_eeprom: cannot find EEPROM by node
MISSING TORADEX CARRIER CONFIG BLOCKS
get_tdx_eeprom: cannot find EEPROM by node
## Warning: Unknown environment variable type 'm'
## Warning: Unknown environment variable type 'm'
SEC0:  RNG instantiated

 BuildInfo:
  - ATF 3c1583b

Setting variant to wifi
flash target is MMC:2
Fastboot: Normal
Normal Boot
## U-Boot CLI access is disabled due to Secure Boot
MMC: no card present
switch to partitions #0, OK
mmc2(part 0) is current device
Scanning mmc 2:1...
Found U-Boot script /boot.scr
5904 bytes read in 0 ms
## Executing script at 50280000
54 bytes read in 0 ms
11413494 bytes read in 42 ms (259.2 MiB/s)
Bootargs: root=/dev/mmcblk2p2 rootfstype=ext4 ro rootwait console=tty1 console=null quiet loglevel=0 udev.log_level=0 consoleblank=0 earlycon
## Loading kernel from FIT Image at 50300000 ...
   Using 'conf-freescale_imx8mp-verdin-wifi-dev.dtb' configuration
   Verifying Hash Integrity ... sha256,rsa2048:dev+ OK
   Trying 'kernel-1' kernel subimage
     Description:  Linux kernel
     Type:         Kernel Image
     Compression:  gzip compressed
     Data Start:   0x5030010c
     Data Size:    10604625 Bytes = 10.1 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x48200000
     Entry Point:  0x48200000
     Hash algo:    sha256
     Hash value:   <<BLANK>>
   Verifying Hash Integrity ... sha256+ OK
## Loading fdt from FIT Image at 50300000 ...
   Using 'conf-freescale_imx8mp-verdin-wifi-dev.dtb' configuration
   Verifying Hash Integrity ... sha256,rsa2048:dev+ OK
   Trying 'fdt-freescale_imx8mp-verdin-wifi-dev.dtb' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x50d8e4d8
     Data Size:    92629 Bytes = 90.5 KiB
     Architecture: AArch64
     Load Address: 0x50200000
     Hash algo:    sha256
     Hash value:   <<BLANK>>
   Verifying Hash Integrity ... sha256+ OK
   Loading fdt from 0x50d8e4d8 to 0x50200000
## Loading fdt from FIT Image at 50300000 ...
   Using 'conf-verdin-imx8mp-secboot-kargs_overlay.dtbo' configuration
   Verifying Hash Integrity ... sha256,rsa2048:dev+ OK
   Trying 'fdt-verdin-imx8mp-secboot-kargs_overlay.dtbo' fdt subimage
     Description:  Flattened Device Tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x50ddc9f0
     Data Size:    474 Bytes = 474 Bytes
     Architecture: AArch64
     Load Address: 0x50240000
     Hash algo:    sha256
     Hash value:  <<BLANK>>
   Verifying Hash Integrity ... sha256+ OK
   Booting using the fdt blob at 0x50200000
   Uncompressing Kernel Image
   Loading Device Tree to 00000000ffaf3000, end 00000000ffb0ca92 ... OK
## Validation of bootargs succeeded.

Starting kernel ...

I know there have been quite a few changes recently and I am currently pinned to an older manifest (as we are in qualification stage and unable to pull in latest changes until next release cycle).
However, is there something I have done wrong here? Or is this expected behaviour?

On another note, is the information shown from the working SoM image a cause for concern (I ask as it was raised at our previous security review, but I am wondering if this is actually an issue when we have the hardening enabled in u-boot?
The only thing our image does not do is sign the rootfs at present.

[rborn-tx]

@MMG-DG Let me try to answer most of your questions:

Having added silent=1 loglevel=0 udev.log_level=0 settings to the u-boot environment, Should any critical errors be written to the console.

I don't think adding loglevel=0 and udev.log_level=0 makes sense. I imagine you're trying to set the loglevel variable that is parsed by the Linux kernel so this value should be passed to the kernel as part of the kernel command line which in U-Boot is defined by variable bootargs. The same goes for udev.log_level that seems to be relevant to systemd-udevd.

AFAIU, variable silent is the only one relevant to U-Boot so it should the be only one you'd possibly add to the environment (in your patch).

I expect the following line should be changed to print error and not just normal printf message to console? 0008-toradex-integrate-bootargs-protection-downstream.patch

I think it would be appropriate to switch from printf to eprintf to send the messages to U-Boot's stderr but this wouldn't solve the issue of the messages not showing up because according to the documentation of this silent variable, when it's set both stdout and stderr are redirected to the null device. Perhaps you could try to set only the stdout to the nulldev and leave stderr untouched. Apparently this can be done by setting stderr=nulldev in the environment (instead of using silent). I've never tried that myself though.

I assume there is something that can allow this to still be printed as a critical error message? At least I believe critical errors should still be surfaced? (I have seen Linux kernel errors when they are raised, but nothing from u-boot; maybe the silent setting can be removed if the loglevel one(s) will silence all output except for errors?).

Most (if not all) error and warning messages in U-Boot are printed via printf or puts so that if you redirect or disable the console they won't show up. Even though they are important messages I believe it's assumed that they are relevant most during development or deployment tests where you could have a different setup...

If I remove the silent=1 from the u-boot commands then it would appear I still get full output (which shows a lot of memory addresses and information to a potential attacker). This means that for my case I will need to continue with setting this value and just hope that nothing else goes wrong when setting the required fuses.

I don't think the information shown on the logs would help an attacker. First because those addresses could be easily obtained from an existing image or even known beforehand from any other image than an attacker could build themselves. Also because the security of the system is not based on memory addresses being unknown. Besides this, once the OS kernel starts running processes cannot normally access physical memory addresses unless they have elevated access and if an attacker has such access it means the system is already compromised.

In my latest testing, it would seem that if I add the silent=1 to the CONFIG_EXTRA_ENV_SETTINGS in include/configs/verdin-imx8mp.h, that once I set the secure boot fuse, my device now ends up in an infinite boot loop with no output to determine why this is happening.
This is the only difference I have between a SoM that works (but shows a lot of information that shows memory addresses that could be used to attack the system) and a SoM the gets into a boot loop (which does not show any details.

If you add silent=1 it seems the kernel command line will be affected which I'd expect to badly interact with the kernel command-line protection feature. This is what U-Boot's README.silent says:

 - When booting a linux kernel, the "bootargs" are fixed up so that
   the argument "console=" will be in the command line, no matter how
   it was set in "bootargs" before. If you don't want the linux command
   line to be affected, define CONFIG_SILENT_U_BOOT_ONLY in your board
   config file as well, and this part of the feature will be disabled.

So, to avoid changing the kernel's command line you should probably set CONFIG_SILENT_U_BOOT_ONLY=y and then explicitly silence the kernel messages yourself by setting the bootargs (as you're already doing by explicitly setting the variable setupargs which is used by the boot script to construct bootargs).

I know there have been quite a few changes recently and I am currently pinned to an older manifest (as we are in qualification stage and unable to pull in latest changes until next release cycle). However, is there something I have done wrong here? Or is this expected behaviour?

There are couple things I'd say are probably wrong that I already mentioned above but I'm not sure which of them would cause the device to reboot. An extra complication is that since you're silence the Linux console messages, it's hard to tell if the reboot is happening during U-Boot's execution of afterwards after the kernel started running (which might have dozens of reasons to reboot).

On another note, is the information shown from the working SoM image a cause for concern (I ask as it was raised at our previous security review, but I am wondering if this is actually an issue when we have the hardening enabled in u-boot?

Mostly answered previously. However let me add here that those various messages are printed during the execution of the command that boots the kernel which is already a point where an attacker wouldn't have much more to do since control is about to be passed to the kernel.

The only thing our image does not do is sign the rootfs at present.

Without protecting the rootfs all of the previous protections are pretty much irrelevant since an attacker would simply attack the unprotected parts of the system...

Let me also say that these various warnings (marked with <==) are concerning:

Loading Environment from MMC... ## Warning: Unknown environment variable type 'i' <==
## Warning: Unknown environment variable type 'i' <==
## Warning: Unknown environment variable type 'i' <==
OK
In:    serial
Out:   serial
Err:   serial
Model: Toradex 0058 Verdin iMX8M Plus Quad 4GB WB IT V1.1A
Serial#: 1500XXXX
get_tdx_eeprom: cannot find EEPROM by node <==
MISSING TORADEX CARRIER CONFIG BLOCKS <==
get_tdx_eeprom: cannot find EEPROM by node <==
## Warning: Unknown environment variable type 'm' <==
## Warning: Unknown environment variable type 'm' <==
SEC0:  RNG instantiated

I wouldn't close a device before ensuring those are solved.

[MMG-DG]

@rborn-tx - Thanks for the detailed info here. Very useful to help understand what is going on.

I have updated my patches to use stdout=nulldev instead of silent=1 as I think that would achieve what I need so I can still see errors.
I have also applied the CONFIG_SILENT_U_BOOT_ONLY=y as although this won't be used now, it is possibly something we may investigate later and I would not want anyone to change this without reviewing our KB wiki (our code changes are linked to wiki decision logs).

RE the EEPROM messages:

get_tdx_eeprom: cannot find EEPROM by node <==
MISSING TORADEX CARRIER CONFIG BLOCKS <==
get_tdx_eeprom: cannot find EEPROM by node <==

We are on a custom carrier board which doesn't have an EEPROM (and for our v1 carrier this will not change).
Is there a way to disable this? or is this something that is required by the SoM for correct configurations?

RE the other warnings about variable 'i' and 'm', I have not got a clue where these are coming from.
Is there somewhere I can look to see what is being set in the environment so I can locate what is causing these?
FYI, we have already blown 12 SoMs with this output and they seem to be working fine, however, I agree and don't like any warnings/errors, but sadly I ran out of time to investigate it further in the current iteration of the project.

This is probably a different tangent, but the other "warning"/"error" that I don't like to see is the following:

DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
Training FAILED                                 <==
DDRINFO: start DRAM init
DDRINFO: DRAM rate 4000MTS
DDRINFO:ddrphy calibration done
DDRINFO: ddrmix config done

Is there something that can be done here so that training succeeds first time? Or is this just an expected thing to happen?

Regards,
MMG

[rborn-tx]

@MMG-DG

I have updated my patches to use stdout=nulldev instead of silent=1 as I think that would achieve what I need so I can still see errors.

I'll see about changing the warning messages about kernel command line mismatch to use stderr instead, but keep in mind that all the other hundreds of possible warning messages produced by U-Boot will still not show up since they are sent to stdout.

RE the other warnings about variable 'i' and 'm', I have not got a clue where these are coming from.

I've never seen those messages so I can only imagine this came from some addition of yours to the environment (maybe udev.log_level which as I said does not belong to U-Boot's environment). You could consider enabling CMD_ENV_FLAGS and try printing these variable flags.

About the other topics, I'll kindly ask you to go to Toradex' support since they are better prepared to answer them.

BR

[MMG-DG]

Happy Halloween All,

So.....
It appears to be that setting stdout=nulldev for u-boot, that renders the full console useless. As there is no ability to enter u-boot console to perform any fuse prog ... commands.

In the short term I have reverted back to the previous format, and have an patch for my device lined up to re-introduce the silent=1 instead. This will be using the CONFIG_SILENT_U_BOOT_ONLY=y config instead and leaving the bootargs to my other recipe change to set the console=null.

However, I have 2 remaining questions...

1. Is it possible to have something in u-boot that can set required fuses on first boot? - Ideally using the commands from the fuse_cmd.txt file that is created in the deploy directory (not sure if this file is actually included anywhere in the image itself).
2. (probably not a question for here, but will ask anyway...) In the default boot script I can see that there are 2 consoles defined. tty...2 and the one for the UART tty... (I am currently only setting the UART one to null). I am wondering if both of these can be removed without issue and replaced with a single console=null?

Regards,
MMG

[sergioprado]

Is it possible to have something in u-boot that can set required fuses on first boot? - Ideally using the commands from the fuse_cmd.txt file that is created in the deploy directory (not sure if this file is actually included anywhere in the image itself).

Yes, it is certainly possibly to change U-Boot to do this. But U-Boot source code woud need to be changed. About fuse_cmd.txt, it is not included in the image.

(probably not a question for here, but will ask anyway...) In the default boot script I can see that there are 2 consoles defined. tty...2 and the one for the UART tty... (I am currently only setting the UART one to null). I am wondering if both of these can be removed without issue and replaced with a single console=null?

This is the default configuration that comes from Toradex BSP. You can changed if needed to match your own needs.

[rborn-tx]

Regarding:

Is it possible to have something in u-boot that can set required fuses on first boot?

We are currently designing a feature that will do exactly this so as to simplify production fusing. I can't say exactly when it will be available though.