MCU mimicking a SPI flash slave

Hi folks-

I'm hoping to tap into your various experiences to see if what I'm 
thinking is practical.

Our customer has a device into which a Datakey is plugged to extract 
data stored in the device.  The Datakey is a really just a trade name 
for a SPI flash with ergonomic features that resemble a real key (see 
datakey.com).  Our customer would like to replace the Datakey with a new 
design that will transmit the data wirelessly instead of storing it to 
the Datakey SPI flash.

We're proposing a design based on an MCU which will appear as a SPI 
slave flash device, store the data pumped to it from the customer's 
device, and forward it wirelessly.  I'm wondering if it's practical to 
accomplish it.  The firmware would have to be totally responsive 
temporally and in data content.  That sounds like a tough hill to climb 
that requires getting the interface just right, nearly perfect.  I see a 
lot of pitfalls that could make the effort a dead end.

Does anybody have any success or failure stories to relate that would 
help us gauge the feasibility of the proposed design?

Thanks - John Speth

On 6/14/2017 10:44 AM, John Speth wrote:
> Hi folks-
>
> I'm hoping to tap into your various experiences to see if what I'm thinking is
> practical.
>
> Our customer has a device into which a Datakey is plugged to extract data
> stored in the device.  The Datakey is a really just a trade name for a SPI
> flash with ergonomic features that resemble a real key (see datakey.com).

So, the "device" is the SPI master (the Datakey being the first downstream
slave)

> Our
> customer would like to replace the Datakey with a new design that will transmit
> the data wirelessly instead of storing it to the Datakey SPI flash.

But, the customer owns the IP for the "device"?  I.e., in the best case
(in terms of making YOUR job easier), they could redesign the device
(or its firmware) to be compatible with your new gizmo.  In the worst
case, they would want to make no changes to the "device" and expect
your gizmo to take up the challenge of behaving like a "real" Datakey.

Regardless, as they own(?) the IP for the device, it is possible to
understand EXACTLY how the device expects to interact with a genuine
Datakey and your "replacement" gizmo.  I.e., you don't really have to emulate
a Datakey but, rather, emulate a Datakey AS ACCESSED BY their device.

[This could give you considerable leeway in your implementation without
requiring a change to the "device".]

> We're proposing a design based on an MCU which will appear as a SPI slave flash
> device, store the data pumped to it from the customer's device, and forward it
> wirelessly.  I'm wondering if it's practical to accomplish it.  The firmware
> would have to be totally responsive temporally and in data content.  That
> sounds like a tough hill to climb that requires getting the interface just
> right, nearly perfect.  I see a lot of pitfalls that could make the effort a
> dead end.

Again, thinking worst case...

Your new device would have to be able to emulate the FLASH device (Datakey)
irrespective of the wireless link's functionality.  For example, if the
"device" pushes data into the Datakey at a particular burst rate, you have
to ensure you can accept AND RETAIN that data in the face of a dubious
wireless link.

What happens if the link *never* comes up?
Does the wireless link protocol include acknowledgements of received data?
What if the data (or some portion of it) is not acknowledged in a particular
timeframe (see below)?

Does the device do a read-back of the Datakey to verify the contents written?
Can you use (abuse) this to indicate a confirmation that the link is up and/or
the wireless transmission has completed successfully?

> Does anybody have any success or failure stories to relate that would help us
> gauge the feasibility of the proposed design?

The first hurdle is looking at the (peak) data rates (and frequencies) from
the "device" and characterize the "typical" and "worst case" traffic.  With
access to the sources (and schematics), you should be able to opine as to
their likely values.  Else, you're stuck with a protocol analyzer (or, a
'scope) and a hopeful attitude that you will actually *see* worst case
examples of these transactions.

 From that, you can determine if you can tackle it with a bit-banging interface.
Or, an SPI tricked to behave as a slave.  Or, as some amount of hardware
assist (external shift register?).

Until you can confidently claim that you can reliably behave as a *local*
SPI slave, the rest of the problem isn't worth evaluating.  Once that's
assured, you can start exploring ways of fitting some sort of acknowledgement
into the existing device's expectations of the Datakey.

(I.e., presumably, it has to deal with defective Datakeys *and* Datakeys
that are withdrawn while a communication transaction is in progress.
Can a Datakey be *bodged* by incorrect usage?  How does their "system"
accommodate this?  etc.)

[You've not mentioned costs, etc.]

John Speth wrote on 6/14/2017 1:44 PM:
> Hi folks-
>
> I'm hoping to tap into your various experiences to see if what I'm thinking
> is practical.
>
> Our customer has a device into which a Datakey is plugged to extract data
> stored in the device.  The Datakey is a really just a trade name for a SPI
> flash with ergonomic features that resemble a real key (see datakey.com).
> Our customer would like to replace the Datakey with a new design that will
> transmit the data wirelessly instead of storing it to the Datakey SPI flash.
>
> We're proposing a design based on an MCU which will appear as a SPI slave
> flash device, store the data pumped to it from the customer's device, and
> forward it wirelessly.  I'm wondering if it's practical to accomplish it.
> The firmware would have to be totally responsive temporally and in data
> content.  That sounds like a tough hill to climb that requires getting the
> interface just right, nearly perfect.  I see a lot of pitfalls that could
> make the effort a dead end.
>
> Does anybody have any success or failure stories to relate that would help
> us gauge the feasibility of the proposed design?

I have no experience in building such a product, but why not use an actual 
SPI data flash as the intermediate storage while the MCU monitors the action 
and forwards the data?  This would require periods when the MCU could read 
the data flash.  Is that possible or practical?

I had a design that needed a 30 MHz SPI like interface (not entirely 
compatible though) which was designed to read/write registers.  I considered 
using a fast CPU to emulate the interface in software, but it just couldn't 
meet the timing.  Even with a 700 MIPS processor I wasn't sure it could 
receive the command to read a register and then get the data onto the output 
fast enough.  I'm not sure of the timing of the SPI interface you have, but 
it will likely be hard to emulate the SPI in software.

-- 

Rick C

On Wed, 14 Jun 2017 10:44:00 -0700, John Speth wrote:

> Hi folks-
> 
> I'm hoping to tap into your various experiences to see if what I'm
> thinking is practical.
> 
> Our customer has a device into which a Datakey is plugged to extract
> data stored in the device.  The Datakey is a really just a trade name
> for a SPI flash with ergonomic features that resemble a real key (see
> datakey.com).  Our customer would like to replace the Datakey with a new
> design that will transmit the data wirelessly instead of storing it to
> the Datakey SPI flash.
> 
> We're proposing a design based on an MCU which will appear as a SPI
> slave flash device, store the data pumped to it from the customer's
> device, and forward it wirelessly.  I'm wondering if it's practical to
> accomplish it.  The firmware would have to be totally responsive
> temporally and in data content.  That sounds like a tough hill to climb
> that requires getting the interface just right, nearly perfect.  I see a
> lot of pitfalls that could make the effort a dead end.
> 
> Does anybody have any success or failure stories to relate that would
> help us gauge the feasibility of the proposed design?
> 
> Thanks - John Speth

Do you know the clock rate of the data, and how fast the turn-around 
needs to be?

Maybe a CPLD or a small FPGA, if you don't like Rick's (probably spot-on) 
suggestion of using an EEPROM as an intermediate device.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

On 15/06/17 02:54, rickman wrote:
> John Speth wrote on 6/14/2017 1:44 PM:
>> Hi folks-
>>
>> I'm hoping to tap into your various experiences to see if what I'm thinking
>> is practical.
>>
>> Our customer has a device into which a Datakey is plugged to extract data
>> stored in the device.  The Datakey is a really just a trade name for a SPI
>> flash with ergonomic features that resemble a real key (see datakey.com).
>> Our customer would like to replace the Datakey with a new design that will
>> transmit the data wirelessly instead of storing it to the Datakey SPI flash.
>>
>> We're proposing a design based on an MCU which will appear as a SPI slave
>> flash device, store the data pumped to it from the customer's device, and
>> forward it wirelessly.  I'm wondering if it's practical to accomplish it.
>> The firmware would have to be totally responsive temporally and in data
>> content.  That sounds like a tough hill to climb that requires getting the
>> interface just right, nearly perfect.  I see a lot of pitfalls that could
>> make the effort a dead end.
>>
>> Does anybody have any success or failure stories to relate that would help
>> us gauge the feasibility of the proposed design?
>
> I have no experience in building such a product, but why not use an actual SPI
> data flash as the intermediate storage while the MCU monitors the action and
> forwards the data?  This would require periods when the MCU could read the data
> flash.  Is that possible or practical?
>
> I had a design that needed a 30 MHz SPI like interface (not entirely compatible
> though) which was designed to read/write registers.  I considered using a fast
> CPU to emulate the interface in software, but it just couldn't meet the timing.
> Even with a 700 MIPS processor I wasn't sure it could receive the command to
> read a register and then get the data onto the output fast enough.  I'm not sure
> of the timing of the SPI interface you have, but it will likely be hard to
> emulate the SPI in software.

That sounds easy for the XMOS processors:
  - multicore (up to 32 100MIPS cores)
  - FPGA-like i/o, e.g. SERDES, buffers, programmable
    clocks to 250Mb/s, separate timer for each port
  - i/o timing changes can be specified/measured in
    software with 4ns resolution
  - guaranteed latencies <100ns
  - *excellent* programming model based on CSP
    (summary: of "Occam in C")
  - Eclipse-based dev environment

Currently I'm managing to count transitions in
software on two 50Mb/s inputs, plus do
simultaneous USB comms to a host computer.

Processors plus &pound;10 devkit available from Farnell
and DigiKey.

On 14/06/17 18:44, John Speth wrote:
> Hi folks-
>
> I'm hoping to tap into your various experiences to see if what I'm thinking is
> practical.
>
> Our customer has a device into which a Datakey is plugged to extract data stored
> in the device.  The Datakey is a really just a trade name for a SPI flash with
> ergonomic features that resemble a real key (see datakey.com).  Our customer
> would like to replace the Datakey with a new design that will transmit the data
> wirelessly instead of storing it to the Datakey SPI flash.
>
> We're proposing a design based on an MCU which will appear as a SPI slave flash
> device, store the data pumped to it from the customer's device, and forward it
> wirelessly.  I'm wondering if it's practical to accomplish it.  The firmware
> would have to be totally responsive temporally and in data content.  That sounds
> like a tough hill to climb that requires getting the interface just right,
> nearly perfect.  I see a lot of pitfalls that could make the effort a dead end.
>
> Does anybody have any success or failure stories to relate that would help us
> gauge the feasibility of the proposed design?

If you want guaranteed timing and FPGA-like io,
have a look at the XMOS processors. See my other
post.

On 14/06/17 19:44, John Speth wrote:
> Hi folks-
> 
> I'm hoping to tap into your various experiences to see if what I'm
> thinking is practical.
> 
> Our customer has a device into which a Datakey is plugged to extract
> data stored in the device.  The Datakey is a really just a trade name
> for a SPI flash with ergonomic features that resemble a real key (see
> datakey.com).  Our customer would like to replace the Datakey with a new
> design that will transmit the data wirelessly instead of storing it to
> the Datakey SPI flash.
> 
> We're proposing a design based on an MCU which will appear as a SPI
> slave flash device, store the data pumped to it from the customer's
> device, and forward it wirelessly.  I'm wondering if it's practical to
> accomplish it.  The firmware would have to be totally responsive
> temporally and in data content.  That sounds like a tough hill to climb
> that requires getting the interface just right, nearly perfect.  I see a
> lot of pitfalls that could make the effort a dead end.
> 
> Does anybody have any success or failure stories to relate that would
> help us gauge the feasibility of the proposed design?
> 
> Thanks - John Speth

If you make sure you have a MCU with good slave SPI support and DMA,
then the actual SPI transfers should not be an issue (assuming a sane
SPI clock speed).  The cpu would then not be involved in the
back-to-back SPI moves at all.  Pretty much any Cortex M3/4 device
should cope fine.

The key challenge is how to deal with the commands - the bits that need
some processing.  That will probably mean fast response to the first few
incoming SPI bytes via an interrupt function.  Your requirements here
will depend on the protocol, the clock speed, the processing time
required, etc.

Write-style SPI commands are no problem - just buffer up the SPI command
and data with DMA as it comes in.  It's the read-style ones that are the
killer.  For very fast SPI interfaces, it is not uncommon that with a
read command, the first byte returned is a dummy byte - that gives you
the time to fill your DMA buffer with the real data for the reply.  But
not all SPI protocols use that, and they might have some reads that need
immediate response.  If you can get that under control, the rest should
be (relatively) easy.

If you can't build this from a normal microcontroller because you need
too fast response, your options are FPGA (either as an assist to a
microcontroller, or with a processor in the FPGA) or perhaps an XMOS
microcontroller.  XMOS programming is a bit different from normal
microcontroller programming, but it will handle tasks like this easily.

On 15/06/17 08:14, David Brown wrote:
> On 14/06/17 19:44, John Speth wrote:
>> Hi folks-
>>
>> I'm hoping to tap into your various experiences to see if what I'm
>> thinking is practical.
>>
>> Our customer has a device into which a Datakey is plugged to extract
>> data stored in the device.  The Datakey is a really just a trade name
>> for a SPI flash with ergonomic features that resemble a real key (see
>> datakey.com).  Our customer would like to replace the Datakey with a new
>> design that will transmit the data wirelessly instead of storing it to
>> the Datakey SPI flash.
>>
>> We're proposing a design based on an MCU which will appear as a SPI
>> slave flash device, store the data pumped to it from the customer's
>> device, and forward it wirelessly.  I'm wondering if it's practical to
>> accomplish it.  The firmware would have to be totally responsive
>> temporally and in data content.  That sounds like a tough hill to climb
>> that requires getting the interface just right, nearly perfect.  I see a
>> lot of pitfalls that could make the effort a dead end.
>>
>> Does anybody have any success or failure stories to relate that would
>> help us gauge the feasibility of the proposed design?
>>
>> Thanks - John Speth
>
> If you make sure you have a MCU with good slave SPI support and DMA,
> then the actual SPI transfers should not be an issue (assuming a sane
> SPI clock speed).  The cpu would then not be involved in the
> back-to-back SPI moves at all.  Pretty much any Cortex M3/4 device
> should cope fine.
>
> The key challenge is how to deal with the commands - the bits that need
> some processing.  That will probably mean fast response to the first few
> incoming SPI bytes via an interrupt function.  Your requirements here
> will depend on the protocol, the clock speed, the processing time
> required, etc.
>
> Write-style SPI commands are no problem - just buffer up the SPI command
> and data with DMA as it comes in.  It's the read-style ones that are the
> killer.  For very fast SPI interfaces, it is not uncommon that with a
> read command, the first byte returned is a dummy byte - that gives you
> the time to fill your DMA buffer with the real data for the reply.  But
> not all SPI protocols use that, and they might have some reads that need
> immediate response.  If you can get that under control, the rest should
> be (relatively) easy.
>
> If you can't build this from a normal microcontroller because you need
> too fast response, your options are FPGA (either as an assist to a
> microcontroller, or with a processor in the FPGA) or perhaps an XMOS
> microcontroller.  XMOS programming is a bit different from normal
> microcontroller programming, but it will handle tasks like this easily.

I'll just say that I find XMOS programming to be *very*
easy compared to C+RTOS -- doubly so for a hardware engineer.

The "RTOS" is in hardware => very low API learning curve :)

The programming model (CSP) is *very* simple and also
comprehensive. It was invented in the 70s, and has stood
the test of time.

On 15/06/17 09:25, Tom Gardner wrote:
> On 15/06/17 08:14, David Brown wrote:
>> On 14/06/17 19:44, John Speth wrote:
>>> Hi folks-
>>>
>>> I'm hoping to tap into your various experiences to see if what I'm
>>> thinking is practical.
>>>
>>> Our customer has a device into which a Datakey is plugged to extract
>>> data stored in the device.  The Datakey is a really just a trade name
>>> for a SPI flash with ergonomic features that resemble a real key (see
>>> datakey.com).  Our customer would like to replace the Datakey with a new
>>> design that will transmit the data wirelessly instead of storing it to
>>> the Datakey SPI flash.
>>>
>>> We're proposing a design based on an MCU which will appear as a SPI
>>> slave flash device, store the data pumped to it from the customer's
>>> device, and forward it wirelessly.  I'm wondering if it's practical to
>>> accomplish it.  The firmware would have to be totally responsive
>>> temporally and in data content.  That sounds like a tough hill to climb
>>> that requires getting the interface just right, nearly perfect.  I see a
>>> lot of pitfalls that could make the effort a dead end.
>>>
>>> Does anybody have any success or failure stories to relate that would
>>> help us gauge the feasibility of the proposed design?
>>>
>>> Thanks - John Speth
>>
>> If you make sure you have a MCU with good slave SPI support and DMA,
>> then the actual SPI transfers should not be an issue (assuming a sane
>> SPI clock speed).  The cpu would then not be involved in the
>> back-to-back SPI moves at all.  Pretty much any Cortex M3/4 device
>> should cope fine.
>>
>> The key challenge is how to deal with the commands - the bits that need
>> some processing.  That will probably mean fast response to the first few
>> incoming SPI bytes via an interrupt function.  Your requirements here
>> will depend on the protocol, the clock speed, the processing time
>> required, etc.
>>
>> Write-style SPI commands are no problem - just buffer up the SPI command
>> and data with DMA as it comes in.  It's the read-style ones that are the
>> killer.  For very fast SPI interfaces, it is not uncommon that with a
>> read command, the first byte returned is a dummy byte - that gives you
>> the time to fill your DMA buffer with the real data for the reply.  But
>> not all SPI protocols use that, and they might have some reads that need
>> immediate response.  If you can get that under control, the rest should
>> be (relatively) easy.
>>
>> If you can't build this from a normal microcontroller because you need
>> too fast response, your options are FPGA (either as an assist to a
>> microcontroller, or with a processor in the FPGA) or perhaps an XMOS
>> microcontroller.  XMOS programming is a bit different from normal
>> microcontroller programming, but it will handle tasks like this easily.
> 
> I'll just say that I find XMOS programming to be *very*
> easy compared to C+RTOS -- doubly so for a hardware engineer.
> 
> The "RTOS" is in hardware => very low API learning curve :)
> 

XMOS programming is very easy for some things, but hard for other
things.  In particular, there is a limit to the scalability.  It starts
off easy splitting your work into lots of parallel tasks, putting them
each in threads, and everything is fine - until you run out of hardware
threads.  Then you end up destroying your structure to get a several
logically parallel tasks running in the same hardware thread, or running
a software RTOS in one hardware thread.  And when you have your multiple
high-speed interfaces working, you find you don't have enough ram left
for buffers.

XMOS devices can be a lot of fun, and I would enjoy working with them
again.  For some tasks, they are an ideal solution (like for this one),
once you have learned to use them.  But not everything is easy with
them, and you have to think in a somewhat different way.

> The programming model (CSP) is *very* simple and also
> comprehensive. It was invented in the 70s, and has stood
> the test of time.

Yes, CSP is a great tool.  I was taught it by some of the folks behind
it.  (Tony Hoare himself was head of the department, but I didn't have
him as a lecturer.)

AT Thursday 15 June 2017 16:02, David Brown wrote:

> On 15/06/17 09:25, Tom Gardner wrote:
>> On 15/06/17 08:14, David Brown wrote:
>>> On 14/06/17 19:44, John Speth wrote:
>>>> Hi folks-
>>>>
>>>> I'm hoping to tap into your various experiences to see if what I'm
>>>> thinking is practical.
>>>>
>>>> Our customer has a device into which a Datakey is plugged to extract
>>>> data stored in the device.  The Datakey is a really just a trade name
>>>> for a SPI flash with ergonomic features that resemble a real key (see
>>>> datakey.com).  Our customer would like to replace the Datakey with a
>>>> new design that will transmit the data wirelessly instead of storing it
>>>> to the Datakey SPI flash.
>>>>
>>>> We're proposing a design based on an MCU which will appear as a SPI
>>>> slave flash device, store the data pumped to it from the customer's
>>>> device, and forward it wirelessly.  I'm wondering if it's practical to
>>>> accomplish it.  The firmware would have to be totally responsive
>>>> temporally and in data content.  That sounds like a tough hill to climb
>>>> that requires getting the interface just right, nearly perfect.  I see
>>>> a lot of pitfalls that could make the effort a dead end.
>>>>
>>>> Does anybody have any success or failure stories to relate that would
>>>> help us gauge the feasibility of the proposed design?
>>>>
>>>> Thanks - John Speth
>>>
>>> If you make sure you have a MCU with good slave SPI support and DMA,
>>> then the actual SPI transfers should not be an issue (assuming a sane
>>> SPI clock speed).  The cpu would then not be involved in the
>>> back-to-back SPI moves at all.  Pretty much any Cortex M3/4 device
>>> should cope fine.
>>>
>>> The key challenge is how to deal with the commands - the bits that need
>>> some processing.  That will probably mean fast response to the first few
>>> incoming SPI bytes via an interrupt function.  Your requirements here
>>> will depend on the protocol, the clock speed, the processing time
>>> required, etc.
>>>
>>> Write-style SPI commands are no problem - just buffer up the SPI command
>>> and data with DMA as it comes in.  It's the read-style ones that are the
>>> killer.  For very fast SPI interfaces, it is not uncommon that with a
>>> read command, the first byte returned is a dummy byte - that gives you
>>> the time to fill your DMA buffer with the real data for the reply.  But
>>> not all SPI protocols use that, and they might have some reads that need
>>> immediate response.  If you can get that under control, the rest should
>>> be (relatively) easy.
>>>
>>> If you can't build this from a normal microcontroller because you need
>>> too fast response, your options are FPGA (either as an assist to a
>>> microcontroller, or with a processor in the FPGA) or perhaps an XMOS
>>> microcontroller.  XMOS programming is a bit different from normal
>>> microcontroller programming, but it will handle tasks like this easily.
>> 
>> I'll just say that I find XMOS programming to be *very*
>> easy compared to C+RTOS -- doubly so for a hardware engineer.
>> 
>> The "RTOS" is in hardware => very low API learning curve :)
>> 
> 
> XMOS programming is very easy for some things, but hard for other
> things.  In particular, there is a limit to the scalability.  It starts
> off easy splitting your work into lots of parallel tasks, putting them
> each in threads, and everything is fine - until you run out of hardware
> threads.  Then you end up destroying your structure to get a several
> logically parallel tasks running in the same hardware thread, or running
> a software RTOS in one hardware thread.  And when you have your multiple
> high-speed interfaces working, you find you don't have enough ram left
> for buffers.
> 
> XMOS devices can be a lot of fun, and I would enjoy working with them
> again.  For some tasks, they are an ideal solution (like for this one),
> once you have learned to use them.  But not everything is easy with
> them, and you have to think in a somewhat different way.
> 
>> The programming model (CSP) is *very* simple and also
>> comprehensive. It was invented in the 70s, and has stood
>> the test of time.
> 
> Yes, CSP is a great tool.  I was taught it by some of the folks behind
> it.  (Tony Hoare himself was head of the department, but I didn't have
> him as a lecturer.)

Just grabbing this thread.

XMOS sounds like something interesting to play with. 
How good is the development environment? 
Runs on Linux?

-- 
Reinhardt