Bootstrapping secure communications

Hi,

I'm deploying a prototype of a distributed system to
benchmark some algorithms before committing to final
hardware.

I have a large supply of 1GHz, 1GB diskless workstations
that I'll use (headless) as they are freely available,
small, reasonably (ha!) low power, etc.  Almost everything
is written in portable languages so the fact that this
is x86 doesn't impact my codebase.

For the prototype system, I'll deploy 30 of these.
Most will be 100BaseTX ethernet.  Some will be 802.11g
or n.  A single server hosts images.

Since these are COTS boxes (and *diskless*), they are
indistinguishable from each other besides MAC.  And,
MAC can be spoofed  :-/

I was originally thinking of PXE booting each box.  Then,
letting that bootstrap bring up the *secure* network for
the rest of IPL (key exchange, etc.).

But, I don't see any way that I can do this *without*
controlling the loading of the initial (PXE) image!
I.e., an attacker could intercept the PXE boot
request and force arbitrary code to be executed.
So, even if I added some local key storage, that
code could freely examine the key, reset the
processor and let the *next* PXE boot request go
through "as normal".

It seems like I *must* make the PXE sequence secure
if I want to have any hope of the rest of the process
being secure (?).  I.e., boot locally (add some sort
of medium) and implement my own form of PXE boot.  :<

However, the number of "tricks" that these folks
come up with for working in spite of adversary leaves
me never quite sure if someone hasn't already come
up with a workaround for this sort of situation -- ?

Any ideas?

Thx,
--don

On 2.7.10 6:19 , D Yuniskis wrote:
> Hi,
>
> I'm deploying a prototype of a distributed system to
> benchmark some algorithms before committing to final
> hardware.
>
> I have a large supply of 1GHz, 1GB diskless workstations
> that I'll use (headless) as they are freely available,
> small, reasonably (ha!) low power, etc. Almost everything
> is written in portable languages so the fact that this
> is x86 doesn't impact my codebase.
>
> For the prototype system, I'll deploy 30 of these.
> Most will be 100BaseTX ethernet. Some will be 802.11g
> or n. A single server hosts images.
>
> Since these are COTS boxes (and *diskless*), they are
> indistinguishable from each other besides MAC. And,
> MAC can be spoofed :-/
>
> I was originally thinking of PXE booting each box. Then,
> letting that bootstrap bring up the *secure* network for
> the rest of IPL (key exchange, etc.).
>
> But, I don't see any way that I can do this *without*
> controlling the loading of the initial (PXE) image!
> I.e., an attacker could intercept the PXE boot
> request and force arbitrary code to be executed.
> So, even if I added some local key storage, that
> code could freely examine the key, reset the
> processor and let the *next* PXE boot request go
> through "as normal".
>
> It seems like I *must* make the PXE sequence secure
> if I want to have any hope of the rest of the process
> being secure (?). I.e., boot locally (add some sort
> of medium) and implement my own form of PXE boot. :<
>
> However, the number of "tricks" that these folks
> come up with for working in spite of adversary leaves
> me never quite sure if someone hasn't already come
> up with a workaround for this sort of situation -- ?
>
> Any ideas?
>
> Thx,
> --don


PXE, and BOOTP / DHCP, is pretty promiscuous. The machine
being booted up is not selective to the boot image sent to it
nor to the server sending the image.

There are two kinds of attack to consider:

  a) An unauthorized client attempts to get an authorized image,
  b) An unauthorized server attempts to feed a fake image.

The MAC spoofing applies to a).

The problem is much the same as in HTTPS. You may need to
verify both the server and the client. To do this, you need
to install in a client both an individual client key and
the common server key, so the basic PXE boot code is not
sufficient.

The keys have to be exchanged in a secure way, either with
asymmetric encryption or challenge-response exchange with
symmetric encryption.

Just my .02 EUR.

-- 

Tauno Voipio
tauno voipio (at) iki fi

D Yuniskis <not.going.to.be@seen.com> wrote:
> Hi,
> 
> I'm deploying a prototype of a distributed system to
> benchmark some algorithms before committing to final
> hardware.
> 
> I have a large supply of 1GHz, 1GB diskless workstations
> that I'll use (headless) as they are freely available,
> small, reasonably (ha!) low power, etc.  Almost everything
> is written in portable languages so the fact that this
> is x86 doesn't impact my codebase.
> 
> For the prototype system, I'll deploy 30 of these.
> Most will be 100BaseTX ethernet.  Some will be 802.11g
> or n.  A single server hosts images.
> 
> Since these are COTS boxes (and *diskless*), they are
> indistinguishable from each other besides MAC.  And,
> MAC can be spoofed  :-/
> 
> I was originally thinking of PXE booting each box.  Then,
> letting that bootstrap bring up the *secure* network for
> the rest of IPL (key exchange, etc.).
> 
> But, I don't see any way that I can do this *without*
> controlling the loading of the initial (PXE) image!
> I.e., an attacker could intercept the PXE boot
> request and force arbitrary code to be executed.
> So, even if I added some local key storage, that
> code could freely examine the key, reset the
> processor and let the *next* PXE boot request go
> through "as normal".
> 
> It seems like I *must* make the PXE sequence secure
> if I want to have any hope of the rest of the process
> being secure (?).  I.e., boot locally (add some sort
> of medium) and implement my own form of PXE boot.  :<
> 
> However, the number of "tricks" that these folks
> come up with for working in spite of adversary leaves
> me never quite sure if someone hasn't already come
> up with a workaround for this sort of situation -- ?
> 
> Any ideas?
> 
> Thx,
> --don

The good, old, chicken and egg prob

Are the devices physically secure?
If not the there is always a risk

Do a PXE boot and once up and running use a smartcard or other physical
dev to auth the connection to the net?

What is your secure network anyway?
Do you have 802.1x or anything?

It's a prototype anyway, so how extreme are the lengths you need to go
to?

Glyn

Op Fri, 02 Jul 2010 05:19:33 +0200 schreef D Yuniskis  
<not.going.to.be@seen.com>:
> I'm deploying a prototype of a distributed system to
> benchmark some algorithms before committing to final
> hardware.
>
> I have

"I have" is the keyword here.  You have control over the network.   
Physical control for wired communications is easy; for any wireless-only  
boxen you could put them in a shielded room.


-- 
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/
(remove the obvious prefix to reply by mail)

D Yuniskis wrote:
> Hi,
>
> I'm deploying a prototype of a distributed system to
> benchmark some algorithms before committing to final
> hardware.
>
> I have a large supply of 1GHz, 1GB diskless workstations
> that I'll use (headless) as they are freely available,
> small, reasonably (ha!) low power, etc. Almost everything
> is written in portable languages so the fact that this
> is x86 doesn't impact my codebase.
>
> For the prototype system, I'll deploy 30 of these.
> Most will be 100BaseTX ethernet. Some will be 802.11g
> or n. A single server hosts images.
>
> Since these are COTS boxes (and *diskless*), they are
> indistinguishable from each other besides MAC. And,
> MAC can be spoofed :-/
>
> I was originally thinking of PXE booting each box. Then,
> letting that bootstrap bring up the *secure* network for
> the rest of IPL (key exchange, etc.).
>
> But, I don't see any way that I can do this *without*
> controlling the loading of the initial (PXE) image!
> I.e., an attacker could intercept the PXE boot
> request and force arbitrary code to be executed.
> So, even if I added some local key storage, that
> code could freely examine the key, reset the
> processor and let the *next* PXE boot request go
> through "as normal".

> It seems like I *must* make the PXE sequence secure
> if I want to have any hope of the rest of the process
> being secure (?). I.e., boot locally (add some sort
> of medium) and implement my own form of PXE boot. :<
>
> However, the number of "tricks" that these folks
> come up with for working in spite of adversary leaves
> me never quite sure if someone hasn't already come
> up with a workaround for this sort of situation -- ?

I've gotta ask, is this a theoretical problem or are
you really concerned with someone hacking your prototype?

What kind of a prototype environment would you have that
you'd be worried about spoofed MAC addresses?

As others have pointed out, physical security should
be the first goal.

Assuming you really are concerned with someone hacking
your prototype network, wouldn't it be enough just to
*detect* a rouge PXE server?  That should be easy enough.

Likewise, it should be possible to detect that your PXE
server has downloaded and validated one and only one good
image to each MAC address.  With the existence of a
spoofed MAC, there should ultimately be more than one.

Finally, you could have a honeypot node.  It could be
running an app that would look like a node wanting to
PXE boot, then validating the boot code it gets and
sounding an alarm if it's suspect.  In fact, the PXE
server should be able to run a process that detects
other suspicious PXE servers.  You could even bait
them with an inert boot and see who responds...

Hi Tauno,

Tauno Voipio wrote:
> On 2.7.10 6:19 , D Yuniskis wrote:
>>
>> I'm deploying a prototype of a distributed system to
>> benchmark some algorithms before committing to final
>> hardware.
>>
>> I have a large supply of 1GHz, 1GB diskless workstations
>> that I'll use (headless) as they are freely available,
>> small, reasonably (ha!) low power, etc. Almost everything
>> is written in portable languages so the fact that this
>> is x86 doesn't impact my codebase.
>>
>> For the prototype system, I'll deploy 30 of these.
>> Most will be 100BaseTX ethernet. Some will be 802.11g
>> or n. A single server hosts images.
>>
>> Since these are COTS boxes (and *diskless*), they are
>> indistinguishable from each other besides MAC. And,
>> MAC can be spoofed :-/
>>
>> I was originally thinking of PXE booting each box. Then,
>> letting that bootstrap bring up the *secure* network for
>> the rest of IPL (key exchange, etc.).
>>
>> But, I don't see any way that I can do this *without*
>> controlling the loading of the initial (PXE) image!
>> I.e., an attacker could intercept the PXE boot
>> request and force arbitrary code to be executed.
>> So, even if I added some local key storage, that
>> code could freely examine the key, reset the
>> processor and let the *next* PXE boot request go
>> through "as normal".
>>
>> It seems like I *must* make the PXE sequence secure
>> if I want to have any hope of the rest of the process
>> being secure (?). I.e., boot locally (add some sort
>> of medium) and implement my own form of PXE boot. :<
> 
> PXE, and BOOTP / DHCP, is pretty promiscuous. The machine
> being booted up is not selective to the boot image sent to it
> nor to the server sending the image.

Exactly.

I.e., I can do exactly what I want *iff* I can control the
code that (always) executes on the node.  If I open the door
for "anything" to creep in, then there is no (?) way to
make any guarantees thereafter.


> There are two kinds of attack to consider:
>  a) An unauthorized client attempts to get an authorized image,
>  b) An unauthorized server attempts to feed a fake image.
> The MAC spoofing applies to a).

In my understanding of PXE, MAC spoofing only applies to a.
But, relying on MAC as an authentication mechanism for *anything*
leaves you open to spoofing.  E.g., if you wrote a PXE replacement
that "hard coded" the MAC of the server from which it expects
to retrieve it's boot image, you're just as vulnerable.

It's like delivering keys in sealed envelopes -- but, handing them
to some guy you met on the street immediately outside their
intended destination (i.e., *he* can peek inside the envelope
and thwart all your security to date)

> The problem is much the same as in HTTPS. You may need to
> verify both the server and the client. To do this, you need
> to install in a client both an individual client key and
> the common server key, so the basic PXE boot code is not
> sufficient.

Yes.  The boot code does this.  The problem lies in my
trying to treat a PXE loaded image in the same way as a
"locally resident" image.  I.e., trying to get something
for nothing  :-/

The fact that there is *nothing* that I can do in this
environment to keep *any* secrets.  :<

> The keys have to be exchanged in a secure way, either with
> asymmetric encryption or challenge-response exchange with
> symmetric encryption.
> 
> Just my .02 EUR.

Hi Glyn,

Glyn Davies wrote:
> D Yuniskis <not.going.to.be@seen.com> wrote:
>
>> I'm deploying a prototype of a distributed system to
>> benchmark some algorithms before committing to final
>> hardware.
>>
>> I have a large supply of 1GHz, 1GB diskless workstations
>> that I'll use (headless) as they are freely available,
>> small, reasonably (ha!) low power, etc.  Almost everything
>> is written in portable languages so the fact that this
>> is x86 doesn't impact my codebase.
>>
>> For the prototype system, I'll deploy 30 of these.
>> Most will be 100BaseTX ethernet.  Some will be 802.11g
>> or n.  A single server hosts images.
>>
>> Since these are COTS boxes (and *diskless*), they are
>> indistinguishable from each other besides MAC.  And,
>> MAC can be spoofed  :-/
>>
>> I was originally thinking of PXE booting each box.  Then,
>> letting that bootstrap bring up the *secure* network for
>> the rest of IPL (key exchange, etc.).
>>
>> But, I don't see any way that I can do this *without*
>> controlling the loading of the initial (PXE) image!
>> I.e., an attacker could intercept the PXE boot
>> request and force arbitrary code to be executed.
>> So, even if I added some local key storage, that
>> code could freely examine the key, reset the
>> processor and let the *next* PXE boot request go
>> through "as normal".
>>
>> It seems like I *must* make the PXE sequence secure
>> if I want to have any hope of the rest of the process
>> being secure (?).  I.e., boot locally (add some sort
>> of medium) and implement my own form of PXE boot.  :<
> 
> The good, old, chicken and egg prob
> 
> Are the devices physically secure?
> If not the there is always a risk

No, they aren't secure but the "local" threat isn't
a problem.  I am mainly concerned about the nodes
that operate wirelessly as it would be easy for an
outsider (literally) to inject traffic at IPL
that would *invisibly* compromise a node and/or
the system.

> Do a PXE boot and once up and running use a smartcard or other physical
> dev to auth the connection to the net?

Yeah, but anything that can intercept that PXE boot
request and substitute some *other* executable image
then has unhindered access to that physical/secure
device (unless you rely on an organic memory module
being nearby to participate in the process  :> )

> What is your secure network anyway?
> Do you have 802.1x or anything?
> 
> It's a prototype anyway, so how extreme are the lengths you need to go
> to?

The prototype will be deployed in a production environment
for alpha and beta testing.  Thereafter, it will be used
to gather metrics on the various algorithms deployed within
it.  As well as to see what sort of attacks are waged
against it (including "red team" activities) and how it
copes in those scenarios.

I figure it will sit in place for the better part of a year
before production hardware is committed.  Thereafter, it
may survive another year or two as a "well instrumented
test bed".

Hi Boudewijn,

Boudewijn Dijkstra wrote:
> Op Fri, 02 Jul 2010 05:19:33 +0200 schreef D Yuniskis 
> <not.going.to.be@seen.com>:
>> I'm deploying a prototype of a distributed system to
>> benchmark some algorithms before committing to final
>> hardware.
>>
>> I have
> 
> "I have" is the keyword here.  You have control over the network.  
> Physical control for wired communications is easy; for any wireless-only 
> boxen you could put them in a shielded room.

The wireless stuff is the problem.  I'd have to get the
clients in the same faraday cage as the antenna (not
going to happen  :< ).

I think I just have to abandon the idea of using a stock
PXE boot loader and write something from scratch.  Maybe
steal a few "CMOS" locations for the "secret".  Or, add
a CF device as a temporary measure (I really don't want
to spend much effort on a throw-away design...)

Hi Jim,

Jim Stewart wrote:
> D Yuniskis wrote:
>>
>> I'm deploying a prototype of a distributed system to
>> benchmark some algorithms before committing to final
>> hardware.
>>
>> I have a large supply of 1GHz, 1GB diskless workstations
>> that I'll use (headless) as they are freely available,
>> small, reasonably (ha!) low power, etc. Almost everything
>> is written in portable languages so the fact that this
>> is x86 doesn't impact my codebase.
>>
>> For the prototype system, I'll deploy 30 of these.
>> Most will be 100BaseTX ethernet. Some will be 802.11g
>> or n. A single server hosts images.
>>
>> Since these are COTS boxes (and *diskless*), they are
>> indistinguishable from each other besides MAC. And,
>> MAC can be spoofed :-/
>>
>> I was originally thinking of PXE booting each box. Then,
>> letting that bootstrap bring up the *secure* network for
>> the rest of IPL (key exchange, etc.).
>>
>> But, I don't see any way that I can do this *without*
>> controlling the loading of the initial (PXE) image!
>> I.e., an attacker could intercept the PXE boot
>> request and force arbitrary code to be executed.
>> So, even if I added some local key storage, that
>> code could freely examine the key, reset the
>> processor and let the *next* PXE boot request go
>> through "as normal".
> 
>> It seems like I *must* make the PXE sequence secure
>> if I want to have any hope of the rest of the process
>> being secure (?). I.e., boot locally (add some sort
>> of medium) and implement my own form of PXE boot. :<
> 
> I've gotta ask, is this a theoretical problem or are
> you really concerned with someone hacking your prototype?

I worry about theory only to the extent that it affects
*practice*.  :>

> What kind of a prototype environment would you have that
> you'd be worried about spoofed MAC addresses?

The fact that all *runtime* communications are encrypted
should suggest that there is value to the information
being exchanged between nodes.  :>  And, there is value
in knowing that the information is unalterable.

> As others have pointed out, physical security should
> be the first goal.

I'm not worried about the *wired* nodes.  There, the only
threat is "insiders".  Aside from a soon-to-be-fired
employee, the insiders have more to lose than to *gain*
(when offset with "risk")

The wireless nodes are the problem areas.

> Assuming you really are concerned with someone hacking
> your prototype network, wouldn't it be enough just to
> *detect* a rouge PXE server?  That should be easy enough.

How do you do that -- since the nodes have no local store?
You can't rely on the server being "in earshot" of the
rogue server.  Any other node can be compromised, reset
and returned to operation without anyone being the wiser
for it...

> Likewise, it should be possible to detect that your PXE
> server has downloaded and validated one and only one good
> image to each MAC address.  With the existence of a
> spoofed MAC, there should ultimately be more than one.

An image needs to be downloaded each time the device is
reset/rebooted.  Do you propose to have a side-channel
whereby the device informs the server of each of these
reboots, etc.?  Otherwise, how does the PXE server
know that this *second* PXE boot request is just as
legitimate as the *first*?

> Finally, you could have a honeypot node.  It could be
> running an app that would look like a node wanting to
> PXE boot, then validating the boot code it gets and
> sounding an alarm if it's suspect.  In fact, the PXE
> server should be able to run a process that detects
> other suspicious PXE servers.  You could even bait
> them with an inert boot and see who responds...

Again, that only applies to the portions of the network
that this server can *see*.  Unless you put the entire
thing in a Faraday box (which eliminates the need for
worrying about bogus servers), there is no way to
prevent "node X" from seeing a rogue PXE server that
the *real* PXE server would be aware of.

If everything followed the rules, there'd be no need
for worrying about the "CAN'T HAPPEN"s  :-/

On Fri, 02 Jul 2010 12:40:46 -0700, D Yuniskis
<not.going.to.be@seen.com> wrote:

>Hi Glyn,
>
>Glyn Davies wrote:
>> D Yuniskis <not.going.to.be@seen.com> wrote:
>>

<Snip>

>> Do a PXE boot and once up and running use a smartcard or other physical
>> dev to auth the connection to the net?
>
>Yeah, but anything that can intercept that PXE boot
>request and substitute some *other* executable image
>then has unhindered access to that physical/secure
>device (unless you rely on an organic memory module
>being nearby to participate in the process  :> )

Yes.. so your wired nodes are 'secure'
Your wireless nodes are not in a secure environment?
Is it possible to run to run them via a VPN onto the secure het, after
which normal PXE can happen?
I'm not sure you can PXE with wireless 802..etc - never heard of it,
but fine over a bridged network.

>
>> What is your secure network anyway?
>> Do you have 802.1x or anything?
>> 
>> It's a prototype anyway, so how extreme are the lengths you need to go
>> to?
>
>The prototype will be deployed in a production environment
>for alpha and beta testing.  Thereafter, it will be used
>to gather metrics on the various algorithms deployed within
>it.  As well as to see what sort of attacks are waged
>against it (including "red team" activities) and how it
>copes in those scenarios.
>
>I figure it will sit in place for the better part of a year
>before production hardware is committed.  Thereafter, it
>may survive another year or two as a "well instrumented
>test bed".

You didn't answer what kind of secure network you have - if you can
use commodity VPN to bridge these 'wireless' nodes, then you should be
sorted - and still able to PXE, no?

Glyn