Is there a process for secure firmware install/upgrade for device made offshore?

Hi
Recently more and more companies want to add security (authentication and/or encryption) to their devices firmware install/update process. Typically this is done by storing a secret encryption key in bootloader or elsewhere in internal MCU flash. This should work if bootloader is installed in secure facility by trusted people. But then manufacturing is outsourced/offshored and then what? I do not want to send my precious key to China. So, I wonder whether it is possible to design an algorithm or process for secure firmware installation and updates while initial firmware is installed by a factory in China. Typically my devices have JTAG, some other port (UART, etc) and often wireless (WiFi or Bluetooth). Note: moving all newly manufactured devices to a secure location and reflashing via JTAG would be too expensive. This problem seem to be very common now, there must be some common solutions, are there? If pure software solution is not possible, are there some hardware assisted solutions? I guess if a chip would include a hardcoded inaccessible private key and assymetric decryption module, this would solve this problem, would it? Are there such chips?

Thank you

On 6/23/2017 9:34 PM, jhnlmn@gmail.com wrote:
> Hi Recently more and more companies want to add security (authentication
> and/or encryption) to their devices firmware install/update process.
> Typically this is done by storing a secret encryption key in bootloader or
> elsewhere in internal MCU flash. This should work if bootloader is installed
> in secure facility by trusted people. But then manufacturing is
> outsourced/offshored and then what? I do not want to send my precious key to
> China. So, I wonder whether it is possible to design an algorithm or process
> for secure firmware installation and updates while initial firmware is
> installed by a factory in China. Typically my devices have JTAG, some other
> port (UART, etc) and often wireless (WiFi or Bluetooth). Note: moving all
> newly manufactured devices to a secure location and reflashing via JTAG
> would be too expensive. This problem seem to be very common now, there must
> be some common solutions, are there? If pure software solution is not
> possible, are there some hardware assisted solutions? I guess if a chip
> would include a hardcoded inaccessible private key and assymetric decryption
> module, this would solve this problem, would it? Are there such chips?

What are you trying to guard/protect against?  "Factory" being able to
create unlicensed updates for your product?

Are you expecting to distribute unencrypted (though *signed*) updates
to your users?

How complex is the device (i.e., you;ll be giving your manufacturer the
binary image of the code to install in the virgin devices)?

Reflashing via JTAG (at a "secure location") isn't the *only* option.
(Hint:  how will your end users install updates?)

All you need (to allow you to NOT trust your manufacturing source)
is a "secret" that isn't available to them, or your users.

> What are you trying to guard/protect against?

1. Reverse engineering
2. Hacking
In resent years there were too many scary news about baby monitors and teddy bears, usb dongles and hard drives, networks routers and switches being hacked and used to spy on their users, steal their data or infect computers. Many of these hacks became possible because original firmware was reverse engineered and/or hacked firmware was installed. Companies that developed those insecure devices got very bad publicity. People are afraid to use devices. We need to find some best practices and follow them.

> Are you expecting to distribute unencrypted (though *signed*) updates to your users?

Requirements vary. Recently many of my clients began demanding encryption (at least those who understand the threat). And I advise them to use both authentication and encryption as much as microcontroller capabilities allow. Luckily, recently more and more microcontrollers include hardware encryption modules. The question is where to hide the secret key to use for encryption/decryption.

> How complex is the device (i.e., you;ll be giving your manufacturer the binary image of the code to install in the virgin devices)?

Again it varies, but recently I see more and more devices that only include bootloader and some basic firmware and then perform firmware upgrade upon first connection. Of course, this firmware upgrade can be done by anybody - end user or manufacturer or hacker.

> All you need (to allow you to NOT trust your manufacturing source) is a "secret" that isn't available to them, or your users.

Yes, this was my original question, how to store a secret on a device that is manufactured in China?

On 6/24/2017 3:27 PM, jhnlmn@gmail.com wrote:
>> What are you trying to guard/protect against?
> 
> 1. Reverse engineering

If your product is "trivial", there is no need to "decompile the code";
just reinvent the design from its apparent specifications/functionality.

If your product is complex, you probably won't be able to do much more
than *copy* the design as it quickly becomes impractical to reverse
engineer *big*/complex designs (even with knowledge of the toolchain
used in their development).

There are "things" you can do to confound counterfeiters (folks that
nominally copy a design with minimal changes -- like changing the
copyright notice to reflect "their" ownership of it, etc.).  But,
you always have to balance the effort (and inconvenience) of
introducing these as they don't *directly* contribute to the
overall functionality/reliability of your product (indeed, they
may *worsen* reliability!)

Moral of story:  make a great product that people WANT to copy!
(whether that is by cloning the implementation *or* by duplicating
the functionality)

> 2. Hacking In resent years there were too many scary news about baby
> monitors and teddy bears, usb dongles and hard drives, networks routers and
> switches being hacked and used to spy on their users, steal their data or
> infect computers. Many of these hacks became possible because original
> firmware was reverse engineered and/or hacked firmware was installed.
> Companies that developed those insecure devices got very bad publicity.
> People are afraid to use devices. We need to find some best practices and
> follow them.

The problem, there, is an attitude towards "security" as a "bolt-on"
aspect of a product's design instead of an inherent, "first-class" design
goal.

HP made a "secure web console" (I draw your attention to the word "secure"
in THEIR choice of product name!).  This was essentially a one-port terminal
server that you'd connect to the serial (console!) port of the system to
be administered.  An outward facing network interface on the device hosted
a web interface that allowed "administrators" to "log in" to the device and
have the sorts of capabilities afforded to "operators" sitting in the
data center alongside the machine being controled (i.e., the PHYSICAL acccess
that is usually essential to secure administration:  control who has
access to the physical machine to limit the attack surface).

So, from a browser, you could type on a virtual console AS IF you were seated
at a TTY a few feet from the machine.

Unfortunately, the implementation relied on a simple substitution (Caesar)
cipher to send the user's keystrokes from the browser to the remote SECURE
web console device.

Thus, when the administered machine sent it's "login: " prompt out to the
secure web console, that was forwarded to the remote user seated at the web
browser interface.  When he typed "root" followed by "mysecretpassword",
that text was sent to the SECURE web console as "sppu" and "nztfdsfuqbttxpse"
(for example).  Anyone sniffing packets would see this and trivially
decode the plaintext to be "root" and "mysecretpassword".

I.e., DESPITE the product being marketed with security in mind, the focus was
more on getting the web server in the device to operate with a reasonably
portable browser interface.  Security ended up sacrificed to meet those other
goals, instead.

Having default passwords has got to be the stupidest idea that ever came
to this market.  REQUIRE the user to set up an initial password BEFORE
allowing the device to operate.  If the user forgets the password, RESET
erases the password and restores the device to its "as shipped" configuration
(deliberately destroying any "sensitive" information that the user may have
forgotten was present on the device -- therefore, "Press RESET before
disposing of this device to safeguard its contents!") rendering it unsable
until a NEW password is specified.

[I suspect default passwords and account names represent the biggest
attack vector in most embedded devices.  Even MS got smart and realized
"Administrator" is a bad name for an ENABLED account!]

Beyond this, you have exploits that are consequential to sloppy or
poorly tested code (buffer overruns, stack overflows, etc.).  Adding
a secure update mechanism won't protect against these sorts of exploits.
And, an exploit that can hide "in RAM" (or in some aspect of the machine's
state) doesn't need to alter the "ROM image" -- it just needs to be able
to persist for some amount of time (which is usually easily done with
24/7/365 devices)

[Once an exploitable device is discovered, it can be targeted for periodic
"reinfection" even if it happens to cycle power (or otherwise purge the
exploit) often.]

>> Are you expecting to distribute unencrypted (though *signed*) updates to 
>> your users?
> 
> Requirements vary. Recently many of my clients began demanding encryption 
> (at least those who understand the threat). And I advise them to use both 
> authentication and encryption as much as microcontroller capabilities
> allow. Luckily, recently more and more microcontrollers include hardware
> encryption modules. The question is where to hide the secret key to use for 
> encryption/decryption.

Are you willing to impose a particular choice of devices on your customers
(i.e., all of your designs)?  Or, are you happier imposing a particular
*process* on them (thereby giving you more leeway in how you approach
the designs)?

>> How complex is the device (i.e., you;ll be giving your manufacturer the 
>> binary image of the code to install in the virgin devices)?
> 
> Again it varies, but recently I see more and more devices that only include 
> bootloader and some basic firmware and then perform firmware upgrade upon 
> first connection. Of course, this firmware upgrade can be done by anybody - 
> end user or manufacturer or hacker.

That's how I have approached this.  I distribute a "ROM image" that allows
the manufacturer (domestic or foreign) to ensure the device HARDWARE will
perform as I intend.  This exercises ALL of the hardware (in concert with
a suitably designed test fixture) even if the "release software" doesn't
(currently) use some portion thereof.

I.e., if the devices pass the final inspection, I have some high degree
of confidence that the manufacturer has built them properly.  And, the
manufacturer can likewise be reassured (no fear of me rejecting a lot due
to some criteria that I impose *after* delivery).

A side-effect of this is the manufacturer never sees my production code.
Instead, he has code (a binary image) that is designed to test a particular
piece of hardware in a particular test fixture.  It may have *NO* value
when used with a modified test fixture or on different hardware.

[Note that there is nothing to prevent said manufacturer from counterfeiting
these devices!  He knows what parts they contain, how they are wired,
what "software" resides in them, etc.]

I receive the devices and subject them to the same "acceptance test" that the
manufacturer (theoretically!) applied as his "final test".  If devices pass
the final test (my "acceptance test") but fail to meet some other aspect
that I require of the design, then (clearly) my test procedure needs to
be revised/upgraded.

A "passing" device is then reprogrammed, serialized and recorded as "ready
(for sale)".

After the sale, the user "introduces" the device to his system at which point
it is tailored to his needs, keys/certificates installed, etc.

All of the "post manufacture" processes use protocols that are typically
NOT routed -- so, they require the device to be present on the same subnet
as the companion/programming device.  Additionally, the individual initiating
the action must "press a button" to activate this process.

So, a hostile actor can't usually gain virtual access to the device (unless
he suborns another host on that subnet and installs an appropriate set of
services to mimic the legitimate "programming" ones).  Also, he needs to have
those in place to anticipate the "button press" AND has to hope the /bonafide/
services aren't running (as they would detect this conflict).

This isn't foolproof.  But, it greatly reduces the attack surface to one that
SHOULD be more manageable:  keep a secure "programming station" (even if
that means booting off R/O media, etc.)

>> All you need (to allow you to NOT trust your manufacturing source) is a 
>> "secret" that isn't available to them, or your users.
> 
> Yes, this was my original question, how to store a secret on a device that 
> is manufactured in China?

Add it after the manufacturing process is complete.  If it isn't a "shared"
secret common to ALL of your devices, then having one instance of it exposed
(accidentally, social engineering, etc.) doesn't compromise ALL of your
products (of that particular model).

> If your product is "trivial"
Personally I have no chance of working on "trivial" devices since they are developed entirely in China. Of course, it will be nice if makers of those "trivial" devices (like those USB dongles that got hacked few years ago) will follow good secure processes as well.

> make a great product that people WANT to copy!
I am not sure my clients want that. I heard of many innovating companies that died as soon as chineese outfits managed to copy their design.

Also defense against reverse engineering is needed not only to prevent copying, but to protect some other secrets, for example, an encryption or authentication key or some secret method that give them business advantage (detection of counterfeit printer cartridges comes to mind).

> REQUIRE the user to set up an initial password BEFORE allowing the device to operate.

And how can we securely send this password? It is easy to send password to a Web site (using HTTPs and private key on the server), but a lot of devices allow local connection between devices or between a device and PC or smartphone. And here we come back to the same original question: how can we securely store a private key on a device?

> you have exploits that are consequential to sloppy or poorly tested code (buffer overruns, stack overflows, etc.).
> Adding a secure update mechanism won't protect against these sorts of exploits.

Yes, I know that security through obscurity is a bad security, still it is much easier to find code susceptible to buffer overruns by reading disassembly than just by black box testing.

> Are you willing to impose a particular choice of devices on your customers (i.e., all of your designs)?
>  Or, are you happier imposing a particular *process* on them (thereby giving you more leeway in how you approach
the designs)?

No, I cannot force them to use a different microcontroller only because of security (we are not making ATMs machines).
Usually I am writing the code and writing a doc on how to install it. If a device is low volume and per-unit cost is  not so important I can require that bootloader (along with secret key) along with JTAG programmer  is only installed on more or less secure PC on company premises. But for more or less high volume devices they want to do the whole process in China.

> A "passing" device is then reprogrammed, serialized and recorded as "ready (for sale)".

This is what I called "moving all newly manufactured devices to a secure location and reflashing via JTAG " in my original question (replace JTAG by WiFi, BT, Serial, does not matter).  Alas, many startups have no resources to do that. They want to outsource everything.

On 6/24/2017 7:26 PM, jhnlmn@gmail.com wrote:
>> A "passing" device is then reprogrammed, serialized and recorded as "ready
>> (for sale)".
> 
> This is what I called "moving all newly manufactured devices to a secure
> location and reflashing via JTAG " in my original question (replace JTAG by
> WiFi, BT, Serial, does not matter).  Alas, many startups have no resources
> to do that. They want to outsource everything.

Then my solution won't work for you.

Having worked in industries where counterfeiting was BLATANT and very
costly (~25% of your market share, thousands to tens of thousands of
dollars per lost sale), I can tell you that you will find yourself
spending a fair bit of time chasing dragons -- and, unless you can grow
WINGS, they'll always be a step ahead of you!

[Unless, of course, you're designing products that NO ONE WANTS!]

You'll have PROOF of a supplier having explicitly copied your product.
By the time you get yourself in front of a judge to slap an injunction
on them or have their cargo seized at the local port (assuming you know
where it is!), they'll have folded up shop and restarted under the
name "Joe's Garage Shop #x" -- where <x> increases by one each iteration.

*YOU* will be devoting your energies to trying to thwart and detect
their infringements while they'll be diverting their energies into learning
how to copy your designs FASTER.

You outsource "everything" then you also have implicitly outsourced
your product security/integrity/quality -- and with it, your reputation.

Good luck!

Don, I appreciate your pessimism and despair.
Still, imagine that more and more devices are made just like I said: on a small budged, with almost everything outsourced except (hopefully) engineering. So, we must find some ways to retain control, otherwise everything will be hacked (which is not hacked already) and we will not be able to make or use any electronics. I really wish a solution will be found. Anybody?
Thank you

On 6/24/2017 9:59 PM, jhnlmn@gmail.com wrote:
> Don, I appreciate your pessimism and despair.

It's not pessimism; it's a realistic attitude given the capabilities
available to even "small players", nowadays.  "Kids" de-encapsulate
chips in college labs and microprobe die, etc.

> Still, imagine that more and more devices are made just like I said: on a
> small budged, with almost everything outsourced except (hopefully)
> engineering.

Imagine you have a device that lets you "hide" a key in it.  Who's going to
push the buttons, copy the files, or <whatever> to actually *hide* the key?
You're outsourcing EVERYTHING!

Do you trust the supplier to do the "secret key encoding" -- but NOT trust him
to actually *see* the key he is encoding into the devices?

Do you find a second firm to do the keying -- and hope these two firms never
"compare notes"?

What if you add a component containing the key to the design AFTER it has
been manufactured.  Who "adds the component"?  A third firm?

Do you buy custom silicon with the key already hidden inside?  Then, hope that
secret is never leaked/revealed/sold/etc. (i.e., have BETTER security over
your IP than the likes of the NSA, Sony, many multinational banks, etc.) by
a disgruntled employee, etc.?

And, WHEN it is disclosed/discovered (by some technique that you've not yet
heard about), do you get NEW silicon with a different key?  And, at the same
time, allow clients to be able to PICK the microcontroller core that they want
embedded in that silicon?

Do you suddenly stop making products when someone "cracks" your system?
And, not resume until you've come up with another "uncrackable" approach?

And, of course, you no doubt expect all of this to be inexpensive as these
"startups" don't have the re$ource$ for that, either?

You either have to trust *some* supplier OR be willing to take on some
of the trusted activities yourself.  How does your client know that
*YOU* can be trusted?  Maybe you hid a backdoor in THEIR product?!

> So, we must find some ways to retain control, otherwise
> everything will be hacked (which is not hacked already) and we will not be
> able to make or use any electronics. I really wish a solution will be found.

Invest your time in making better products -- so no one wants the last
generation product.  Design your products with security in mind from the
start, not bolted on as an afterthought.  Be an Engineer, not a Rent-a-Cop.

IFind people that you trust -- if you need them to be trustworthy -- in
much the same way that you hire people to be *competent* (at some skill)
when THAT is your need!

> Anybody? Thank you

Add a Dallas chip with a unique id in every product?

On 6/25/2017 2:37 AM, Bill Davy wrote:
> Add a Dallas chip with a unique id in every product?

That just gives you a serial number.  Do you plan on releasing
software updates (images) *tied* to specific serial numbers?
I.e., what is there that causes S/N 12345 to work but not 54321?

[I.e., you can use the device's MAC for a S/N]

What's to stop someone from downloading images for two different
serial numbers (perhaps because he legitimately OWNS two devices
each with different S/N's) and comparing the images to isolate the
location of the serial number and any other diff's (e.g., CRC)?
Armed with that, how hard would it be to create a NEW image
for some other S/N?  (Or, elide the S/N check entirely?)