On 09/05/2023 20:42, Ulf Samuelsson wrote:> Den 2023-05-03 kl. 14:48, skrev David Brown:>> It makes sense to use an 8-bit CRC on small telegrams, 16-bit CRC on >> bigger things, 32-bit CRC on flash images, and 64-bit CRC when you >> want to use the CRC as an identifying hash (and malicious tampering is >> non-existent). There can also be benefits of particular choices of >> CRC for particular use-cases, in terms of detection of certain error >> patterns for certain lengths of data. > > Flash images larger than X kB may need a 64-bit CRC. > I don't remember exactly when to start considering it, > but something between 64kB-256kB is probably correct. > > It is all to do with Hamming Distance, and this is also affected by the > polynome. > /Ulf > >"Need" is too strong a word here. A CRC will guarantee detection of certain kinds of error (such as a single bit error), regardless of the length of the data. Some kinds of error are limited by length. If you plot a graph with guaranteed Hamming distance on the vertical scale and length of data on the horizontal scale, each CRC will drop off in steps. For the same CRC size, some will hold a high Hamming distance for longer and then drop off sharply, others will hold a lower Hamming distance for very large data. And in general, a bigger CRC will be better here. But Hamming distance is not everything. It is important in situations where there is an approximately independent risk of corruption for each bit individually - such as during radio transmission. Programming images into flash has a completely different error risk pattern. A little Hamming is nice to guarantee that any single cell failure in the flash will be be found, but the more realistic flash problems involve large scale effects - failure to erase a block fully, or software flaws. For this kind of thing, pretty much any valid CRC polynomial works the same - a 32-bit polynomial gives you a 1 in 2 ^ 32 chance of the error going undetected. Yes, a 1 in 2 ^ 64 chance is better, but it's rarely something to get excited about. Note that if you are sending the image to a board via a potentially flawed mechanism, you'll want appropriate checks during the transfers. Ethernet, Wifi, Bluetooth, USB - they will all have suitable checksums for each packet. And for some of those, Hamming distance and particular choice of polynomial /is/ an important consideration.
Embedding a Checksum in an Image File
Started by ●April 19, 2023
Reply by ●May 10, 20232023-05-10
Reply by ●May 10, 20232023-05-10
On 09/05/2023 20:34, Ulf Samuelsson wrote:> Den 2023-04-30 kl. 16:19, skrev David Brown: >> On 29/04/2023 23:03, Ulf Samuelsson wrote: >>> Den 2023-04-28 kl. 15:04, skrev David Brown: >>>> On 28/04/2023 10:50, Ulf Samuelsson wrote: >>>>> Den 2023-04-28 kl. 09:38, skrev David Brown: >> >>>>>> >>>>>> Or for my preferences, the CRC "DIGEST" would be put at the end of >>>>>> the image, rather than near the start. Then the "from, to" range >>>>>> would cover the entire image except for the final CRC. But I'd >>>>>> have a similar directive for the length of the image at a specific >>>>>> area near the start. >>>>>> >>>>> >>>>> I really do not see a benefit of splitting the meta information >>>>> about the image to two separate locations. >>>>> >>>>> The bootloader uses the struct for all checks. >>>>> It is a much simpler implementation once the tools support it. >>>>> >>>>> You might find it easier to write a tool which adds the CRC at the >>>>> end, but that is a different issue. >>>>> >>>>> Occam's Razor! >>>>> >>>> >>>> There are different needs for different projects - and more than one >>>> way to handle them. I find adding a CRC at the end of the image >>>> works best for me, but I have no problem appreciating that other >>>> people have different solutions. >>>> >>>> >>>> >>>> >>> I'd be curious to know WHY it works best for you. >>> /Ulf >> >> I regularly do not have a bootloader - I am not free to put a CRC at >> the start of the image. And if the bootloader itself needs to be >> updatable, it is again impossible to have the CRC (or any other >> metadata) at the start of the image. I want most of the metadata to >> be at a fixed location as close to the start as reasonably practical >> (such as after the vector table, or other microcontroller-specific >> information that might be used for flash security, early chip setup, >> etc.). If I am to have one single checksum for the image, which is >> what I prefer, then it has to be at the end of the image. For >> example, there might be : >> >> 0x00000000 : vectors >> 0x00000400 : external flash configuration block >> 0x00000600 : program info metadata >> 0x00001000 : main program >> : CRC >> >> There is no way to have the metadata or CRC at the start of the image, >> so the CRC goes at the end. > > For the Bootloader, I keep the CRC right after the vectors. > I keep a copy of the vectors right after the CRC, and compare > the two vector tables. > This is to always know the location of the CRC.Fair enough - that is an entirely reasonable alternative. I have a knee-jerk reaction against duplication as a check, having cut my teeth on microcontrollers where 16 KB devices were "big", but of course a duplication of the vector table is not going to be a noticeable waste on a more modern device. It does, however, mean extra steps in checking, compared to a simpler CRC run over the entire image.> >> >> It would be possible to have two CRCs - one that covers the vectors, >> configuration information, and metadata and is placed second last in >> the metadata block. A second CRC placed last in the metadata block >> would cover the main program - everything after the CRCs. That would >> let me have a single metadata block and no CRC at the end of the >> image. However, it would mean splitting the check in two, rather than >> one check for the whole image. I don't see that as a benefit. >> >> >> When making images that are started from a bootloader, I certainly >> /could/ put the CRC at the start. But I see no particular reason to >> do so - it makes a lot more sense to keep a similar format. >> > You want more metadata like entry point and length, as well as text > information about the image. Putting things in a header means that > location is fixed. > There are a number of checks in my bootloader to ensure that the > information in the header makes sense. >I do have all that kind of thing too. It's only the CRC itself that is put at the end, and it is easily found since the length of the image is in the metadata. (We are talking about one pointer access more than having it at a fixed address - it's not hard to find it!).>> (Bootloaders don't often have to check their own CRC - after all, even >> if the CRC fails there is usually little you can do about it, except >> charge on and hope for the best. But if the bootloader is updatable >> in system, then you want a CRC during the download procedure to check >> that you have got a good download copy before updating the flash.) > > In functional safety applications you regularily check the flash > contents and refuse to boot if there is a mismatch. >Yes, that is a possibility. I've worked on safety-certified systems which required things like regular checks of flash while running (not just at bootup). A lot of the so-called "safety requirements" were directly detrimental. I believe many of these kinds of requirements were made by people who understood the "Swiss cheese" model of risks and safety, but not the more realistic "Hot cheese" model. And they seem more concerned about box-ticking and legal arse-covering than actual risk reduction.
Reply by ●May 10, 20232023-05-10
Den 2023-05-10 kl. 10:06, skrev David Brown:> On 09/05/2023 20:42, Ulf Samuelsson wrote: >> Den 2023-05-03 kl. 14:48, skrev David Brown: > >>> It makes sense to use an 8-bit CRC on small telegrams, 16-bit CRC on >>> bigger things, 32-bit CRC on flash images, and 64-bit CRC when you >>> want to use the CRC as an identifying hash (and malicious tampering >>> is non-existent). There can also be benefits of particular choices >>> of CRC for particular use-cases, in terms of detection of certain >>> error patterns for certain lengths of data. >> >> Flash images larger than X kB may need a 64-bit CRC. >> I don't remember exactly when to start considering it, >> but something between 64kB-256kB is probably correct. >> >> It is all to do with Hamming Distance, and this is also affected by >> the polynome. >> /Ulf >> >> > "Need" is too strong a word here. A CRC will guarantee detection of > certain kinds of error (such as a single bit error), regardless of the > length of the data. Some kinds of error are limited by length. If you > plot a graph with guaranteed Hamming distance on the vertical scale and > length of data on the horizontal scale, each CRC will drop off in steps. > For the same CRC size, some will hold a high Hamming distance for > longer and then drop off sharply, others will hold a lower Hamming > distance for very large data. And in general, a bigger CRC will be > better here. > > But Hamming distance is not everything. It is important in situations > where there is an approximately independent risk of corruption for each > bit individually - such as during radio transmission. Programming > images into flash has a completely different error risk pattern. A > little Hamming is nice to guarantee that any single cell failure in the > flash will be be found, but the more realistic flash problems involve > large scale effects - failure to erase a block fully, or software flaws. > For this kind of thing, pretty much any valid CRC polynomial works the > same - a 32-bit polynomial gives you a 1 in 2 ^ 32 chance of the error > going undetected. Yes, a 1 in 2 ^ 64 chance is better, but it's rarely > something to get excited about.Programming a flash memory can flip bits in parts of the flash memory which is not programmed. Bit errors can also be introduced by radiation. Some applications require better security than others. Functional Safety may require CRC size based on code size. /Ulf> > Note that if you are sending the image to a board via a potentially > flawed mechanism, you'll want appropriate checks during the transfers. > Ethernet, Wifi, Bluetooth, USB - they will all have suitable checksums > for each packet. And for some of those, Hamming distance and particular > choice of polynomial /is/ an important consideration. > >
Reply by ●August 5, 20232023-08-05
On 5/3/2023 5:48 AM, David Brown wrote:>>>> Give me the sources for Windows (Linux, *BSD, etc.) and I can >>>> subvert all the state-of-the-art digital signing used to ensure >>>> binaries aren't altered. Nothing *outside* the box is involved >>>> so, by definition, everything I need has to reside *in* the box. >>> >>> No, you can't. The sources for Linux and *BSD /are/ all freely available. >>> The private signing keys used by, for example, Red Hat or Debian, are /not/ >>> freely available. You cannot make changes to a Red Hat or Debian package >>> that will pass the security checks - you are unable to sign the packages. >> >> Sure I can! If you are just signing a package to verify that it hasn't >> been tampered with BUT THE CONTENTS ARE NOT ENCRYPTED, then all you have >> to do is remove the signature check -- leaving the signature in the >> (unchecked) executable. > > Woah, you /really/ don't understand this stuff, do you? Here's a clue - ask > yourself what is being signed, and what is doing the checking.Exactly. You don't attack the signature or the keys. You BUILD A NEW KERNEL THAT DOESN'T CHECK THE SIGNATURE. You attack (replace if you have access to the sources -- as I stipulated above) the "what is doing the checking". This is c.a.e; you likely have physical access and control of the device (unlike trying to attack a remote system) The binary is exposed UNENCRYPTED in the signed executable (please note my stipulation to that, too, above). The only thing preventing its execution (if tampered -- or unlicensed!) is the signature check. Bypass that in any way and the code executes AS IF signed. I design "devices". Don't you think if there was a foolproof way (by resorting to "school boy techniques") to protect them from counterfeiting and tampering that I would have already embraced that? That EVERY computer-based product would be INHERENTLY SECURED?? [There are ways that are far from theoretical yet considerably more effective. A signature check is easy to detect in an executing device and, thus, elided. Lather, rinse, repeat for each precursor level of such protection.] Please read what I've written more carefully, lest you look foolish. Or, spend a few years playing red-blue games and actually trying to subvert hardware and software protection mechanisms in REAL products. (Hint: you will need to think down BELOW the hardware level to do so successfully so you can bypass the hardware mechanisms that folks keep trying to embed in their products)> Perhaps also ask yourself if /all/ the people involved in security for Linux or > BSD - all the companies such as Red Hat, IBM, Intel, etc., - ask if /all/ of > them have got it wrong, and only /you/ realise that digital signatures on open > source software is useless?The signature is only of use if the mechanism verifying it is tamperproof. That's not possible on most (all?) devices sold. SOMEONE has physical access to the device so all of the mechanisms you put in place can be subverted. *Ask* the Linux and BSD crowds if they can GUARANTEE that ALTERED signed code can't be executed on a system where the adversary can build and install their own kernel. Or, probe the innerworkings of such a device AT THEIR LEISURE/> /Very/ occasionally, there is a lone genius that > understands something while all the other experts are wrong - but in most > cases, the loner is the one that is wrong.In this case, you have clearly failed to understand what was being said. So, don't count yourself in with the "experts". If the kernel loading the executable doesn't contain code to validate the signature (and, if I have the sources for said kernel/OS then I can easily *make* such a kernel) then the signature is just another unused "section" in the BLOB. Just like debug symbols or copyright information.
Reply by ●August 5, 20232023-08-05
On 5/10/2023 3:03 AM, Ulf Samuelsson wrote:> Programming a flash memory can flip bits in parts of the flash memory which is > not programmed. > Bit errors can also be introduced by radiation.Executing code can also introduce write *and* read disturb events. Ask yourself how to protect a design that allows arbitrary code to be executed (even if in a sandbox) in the presence of potential side-channel exploits. Or, as a "simpler" problem: how to detect if such an exploit has been invoked (even possibly unintentionally)! [Imagine devices that "run forever"...]> Some applications require better security than others.Exactly.> Functional Safety may require CRC size based on code size.
