Embedded Linux processors

I was idly looking to see what was out there in the low end Linux space -
something bigger than an ESP32 but more production friendly than a Raspberry
Pi.  I came across this excellent guide:

https://jaycarlson.net/embedded-linux/

He builds dev boards for 10 different chips from 7 vendors, just to see how
it all goes - both hardware and software.  The results are quite
interesting.

Any other recommendations for Linux-supporting SoCs that are nice for low
volume/hand production?

Theo

On 10/24/2022 7:20 AM, Theo wrote:
> I was idly looking to see what was out there in the low end Linux space -
> something bigger than an ESP32 but more production friendly than a Raspberry
> Pi.  I came across this excellent guide:
> 
> https://jaycarlson.net/embedded-linux/
> 
> He builds dev boards for 10 different chips from 7 vendors, just to see how
> it all goes - both hardware and software.  The results are quite
> interesting.
> 
> Any other recommendations for Linux-supporting SoCs that are nice for low
> volume/hand production?

As you've qualified the solution space with "Linux-supporting", I
assume you mean a Linux port is already available (for at least
the underlying architecture).

And, as you've discounted the rPi as "less production friendly", I
assume you're looking for *components*, not *assemblies*.

Looking for "low-cost linux boards" could give you an idea as to
the processors chosen for each.  But, they typically are "kitchen sink"
approaches to problems.

I'd, instead, look into the kernel and see if you can do away with
the PMMU (i.e., get it to work with all memory wired down and no
swap configured; then, remove the code associated with paging).

This may make some aspects of the implementation impractical.  E.g.,
my RTOS relies on a PMMU to share data across protection domains,
do zero copy ransfers, etc.  But, you may be able to live without
the things that rely on that mechanism.

[No idea as I've never looked inside the linux kernel]

Some of the older kernel versions (and ports) may give you an insight
into what can/can't be done.

This could expand the range of processors/SoCs that you could use
(though likely require some effort for a port).

On 26/10/2022 03:17, Don Y wrote:
> On 10/24/2022 7:20 AM, Theo wrote:
>> I was idly looking to see what was out there in the low end Linux space -
>> something bigger than an ESP32 but more production friendly than a 
>> Raspberry
>> Pi.  I came across this excellent guide:
>>
>> https://jaycarlson.net/embedded-linux/
>>
>> He builds dev boards for 10 different chips from 7 vendors, just to 
>> see how
>> it all goes - both hardware and software.  The results are quite
>> interesting.
>>
>> Any other recommendations for Linux-supporting SoCs that are nice for low
>> volume/hand production?
> 
> As you've qualified the solution space with "Linux-supporting", I
> assume you mean a Linux port is already available (for at least
> the underlying architecture).
> 
> And, as you've discounted the rPi as "less production friendly", I
> assume you're looking for *components*, not *assemblies*.

I wouldn't assume that (though the OP will have to clarify).  Pi's are 
fine for prototyping, but there are many reasons why they might not be a 
suitable choice for real products.  However, that does not at all 
suggest that it is a good idea to use chips directly rather than modules.

Unless your production runs are at least 10,000 a time, it is unlikely 
to be cost-effective to use anything other than pre-populated modules. 
Designing a board for large ball count BGAs, high speed memories, etc., 
is not quick or cheap, nor is their production.

> 
> Looking for "low-cost linux boards" could give you an idea as to
> the processors chosen for each.  But, they typically are "kitchen sink"
> approaches to problems.
> 
> I'd, instead, look into the kernel and see if you can do away with
> the PMMU (i.e., get it to work with all memory wired down and no
> swap configured; then, remove the code associated with paging).
> 

That could have been good advice - twenty years ago.

Now it is pointless to aim for such a minimal system.  The cheapest 
processors with MMU supported by Linux cost a few dollars.  The cheapest 
non-MMU microcontrollers that are capable of supporting Linux are at 
least ten dollars.  Swap has always been optional, but working without 
an MMU leads to a lot of complications and restrictions (such as no 
"fork" calls).  No one uses non-MMU Linux except for nerdy fun.  (And 
fun is /always/ a good reason for doing something.)


> This may make some aspects of the implementation impractical.  E.g.,
> my RTOS relies on a PMMU to share data across protection domains,
> do zero copy ransfers, etc.  But, you may be able to live without
> the things that rely on that mechanism.
> 
> [No idea as I've never looked inside the linux kernel]
> 
> Some of the older kernel versions (and ports) may give you an insight
> into what can/can't be done.
> 
> This could expand the range of processors/SoCs that you could use
> (though likely require some effort for a port).
> 

On 24/10/2022 16:20, Theo wrote:
> I was idly looking to see what was out there in the low end Linux space -
> something bigger than an ESP32 but more production friendly than a Raspberry
> Pi.  I came across this excellent guide:
> 
> https://jaycarlson.net/embedded-linux/
> 
> He builds dev boards for 10 different chips from 7 vendors, just to see how
> it all goes - both hardware and software.  The results are quite
> interesting.
> 
> Any other recommendations for Linux-supporting SoCs that are nice for low
> volume/hand production?
> 
> Theo

The key things to determine are what you consider "production friendly", 
and what you need.  You want a module, not a chip.  Some modules come 
with pins for connections, others with just solder pads, and some are 
made to fit SO-DIMM sockets or similar connectors.  Some modules have 
Ethernet, Wifi, Bluetooth, HDMI, USB, and other high-speed interfaces - 
others have much less.  Some have on-board eMMC or other NAND flash, 
others rely on external memory or SD-Cards.  Some have their power 
supply handling on board and need just a single supply at 3.3v or 5v, 
others need multiple supplies at different levels with careful bringup. 
  Some have long lifetimes and will be available for a decade, others 
are from companies that might be gone next month.  Some have excellent 
support from the supplier, some have excellent community support, and 
others have virtually no support at all.

We don't know anything about the product, its needs, or about what you 
can do yourself and what you need supplied.  All I can give you is 
general advice here regarding things to consider.  And be wary of trying 
to get minimal cost for the module - it can easily cost you more in the 
long run.  (Equally, high price of a module is no guarantee.)

There are many people making SoC's that can work well with Linux, mostly 
ARM Cortex-A but also some RISC-V now.  (There are also PPC, MIPS, and a 
few other cores, but those are in more specialised devices like network 
chips.)  There are no SoC's that are remotely suitable for hand production.

Another thing to consider, of course, is whether a Linux module is what 
you really want.  There are microcontrollers that are more powerful than 
ESP32 devices, such as NXP's i.mx RT line (with 500-1000 MHz Cortex-M7 
cores).  On the software side, there is Zephyr which sits somewhere 
between FreeRTOS and Linux and might be useful.  (I haven't tried Zephyr 
myself.)

David Brown <david.brown@hesbynett.no> wrote:
> > 
> > https://jaycarlson.net/embedded-linux/
> 
> The key things to determine are what you consider "production friendly", 
> and what you need.  You want a module, not a chip.  Some modules come 
> with pins for connections, others with just solder pads, and some are 
> made to fit SO-DIMM sockets or similar connectors.  Some modules have 
> Ethernet, Wifi, Bluetooth, HDMI, USB, and other high-speed interfaces - 
> others have much less.  Some have on-board eMMC or other NAND flash, 
> others rely on external memory or SD-Cards.  Some have their power 
> supply handling on board and need just a single supply at 3.3v or 5v, 
> others need multiple supplies at different levels with careful bringup. 
>   Some have long lifetimes and will be available for a decade, others 
> are from companies that might be gone next month.  Some have excellent 
> support from the supplier, some have excellent community support, and 
> others have virtually no support at all.

The above article covers all of those things in a nice way: some parts are
in 64 pin QFNs, some are in 0.8mm BGA which he reckons is doable to hand
solder (I haven't tried that...).  Some have abandonware software stacks,
others are in the mainline Linux tree.  etc etc

> We don't know anything about the product, its needs, or about what you 
> can do yourself and what you need supplied.  All I can give you is 
> general advice here regarding things to consider.  And be wary of trying 
> to get minimal cost for the module - it can easily cost you more in the 
> long run.  (Equally, high price of a module is no guarantee.)

I don't have a product :-)  But really just making a thought experiment
about what would happen if I did have a product - let's say an IoT thingy
(wifi, display, etc) in the <$100 sticker price, initial volumes let's say
hundreds.

The ESP32s are nice as they're a simple, cheap, wifi module.  If you wanted
to cut costs you could use the bare chip.  The Pis aren't: the Zero is a
nice form factor, but you can't buy it in volume.  The regular Pis can't
really be mounted on a custom PCB if you don't have a large enclosure.  The
Compute Modules are better, but still larger than an ESP32.  However you
can't really buy any of them at the moment, and if you could they would be
quite expensive.  The Pi2040 is an ok microcontroller but nothing special
(and wifi is an extra chip).  Also none of them have any protection from
someone changing or stealing your firmware.

It is interesting in the above article how much the complexity starts to
rise once you start going beyond a single chip solution: BGAs, DDR routing,
numerous power supplies and sequencing, etc.

> There are many people making SoC's that can work well with Linux, mostly 
> ARM Cortex-A but also some RISC-V now.  (There are also PPC, MIPS, and a 
> few other cores, but those are in more specialised devices like network 
> chips.)  There are no SoC's that are remotely suitable for hand production.

Some of the SIPs and BGAs in the article above are, allegedly.  However
'hand production' is really a proxy for production complexity.  If you can
build a 4 layer board and hand-mount it, you can build in low-ish volume on
a relatively cheap pick and place line.  If you need a 10 layer board and
package-on-package BGA mounting equipment, you can't do that without a much
greater volume to amortise the tooling costs.

Systems on module are a good solution to that but, if some of these SoCs are
niche, the modules are even more niche (hard to buy in small quantities,
produced by a tiny company, and so on).

> Another thing to consider, of course, is whether a Linux module is what 
> you really want.  There are microcontrollers that are more powerful than 
> ESP32 devices, such as NXP's i.mx RT line (with 500-1000 MHz Cortex-M7 
> cores).  On the software side, there is Zephyr which sits somewhere 
> between FreeRTOS and Linux and might be useful.  (I haven't tried Zephyr 
> myself.)

The iMX RT isn't one I've come across, thanks.  That's the kind of thing I'm
interested in.

The software side is one that's frequently neglected: one thing the
Raspberry Pi folks are really good at is maintaining their software stack. 
A lot of other (Chinese) Linux SoC vendors basically throw it all over the
wall and let the customers do the maintenance.  In some ways it's nice not
to play in that space.  OTOH once you get beyond a certain point it's nice
to be able to use 'grown up' tools (like a webserver that can easily do TLS,
not some stripped down microcontroller TLS stack that only does TLS 1.1 and
can't fit any more in RAM, or worse does no TLS at all).

I'm really mainly curious how this middle part of the market goes, and
wondering how others feel about it.

Theo

On 10/26/2022 1:06 AM, David Brown wrote:
> On 26/10/2022 03:17, Don Y wrote:
>> On 10/24/2022 7:20 AM, Theo wrote:
>>> I was idly looking to see what was out there in the low end Linux space -
>>> something bigger than an ESP32 but more production friendly than a Raspberry
>>> Pi.  I came across this excellent guide:
>>>
>>> https://jaycarlson.net/embedded-linux/
>>>
>>> He builds dev boards for 10 different chips from 7 vendors, just to see how
>>> it all goes - both hardware and software.  The results are quite
>>> interesting.
>>>
>>> Any other recommendations for Linux-supporting SoCs that are nice for low
>>> volume/hand production?
>>
>> As you've qualified the solution space with "Linux-supporting", I
>> assume you mean a Linux port is already available (for at least
>> the underlying architecture).
>>
>> And, as you've discounted the rPi as "less production friendly", I
>> assume you're looking for *components*, not *assemblies*.
> 
> I wouldn't assume that (though the OP will have to clarify).  Pi's are fine for 
> prototyping, but there are many reasons why they might not be a suitable choice 
> for real products.  However, that does not at all suggest that it is a good 
> idea to use chips directly rather than modules.
> 
> Unless your production runs are at least 10,000 a time, it is unlikely to be 
> cost-effective to use anything other than pre-populated modules. Designing a 
> board for large ball count BGAs, high speed memories, etc., is not quick or 
> cheap, nor is their production.

Did you *read* the article?

    "To this end, I designed a dev board from scratch for each application
    processor reviewed. Well, actually, many dev boards for each processor:
    roughly 25 different designs in total. This allowed me to try out different
    DDR layout and power management strategies — as well as fix some bugs
    along the way."

Perhaps you've no experience designing (and laying out and prototyping)
"modern" parts.  It's not rocket science.  The days of paying $2K for
a Leister are ancient history...  That was another point of the article.

>> Looking for "low-cost linux boards" could give you an idea as to
>> the processors chosen for each.  But, they typically are "kitchen sink"
>> approaches to problems.
>>
>> I'd, instead, look into the kernel and see if you can do away with
>> the PMMU (i.e., get it to work with all memory wired down and no
>> swap configured; then, remove the code associated with paging).
> 
> That could have been good advice - twenty years ago.
> 
> Now it is pointless to aim for such a minimal system.  The cheapest processors 
> with MMU supported by Linux cost a few dollars.

What do you do when your product *sells* for a few dollars?

> The cheapest non-MMU 
> microcontrollers that are capable of supporting Linux are at least ten 
> dollars.

How do you define "supporting Linux"?  I.e., "for which an existing build
exists?"

Most developers are only interested in the API and feature sets that
they have available to them.  If it "looks" like linux, in terms of
what they can expect it to do for them, they don't likely care about
the actual implementation.

>  Swap has always been optional, but working without an MMU leads to a 
> lot of complications and restrictions (such as no "fork" calls).

Fork needn't "create a copy of the parent process" -- if the
existing copy of the process can be used without duplication
(think XIP -- no gobs of RAM into which to copy the new process
image!).  All it need do is create a LOGICALLY new process container
(which needn't even have "protection" from other processes).

Fork is probably the *least* valuable use of a PMMU in a system.
An MMU that gives some (reasonable) control over accesses to
specific regions IN A UNIFIED ADDRESS SPACE would likely lead
to more robust code (in and of itself) than supporting a
classic fork().

>  No one uses 
> non-MMU Linux except for nerdy fun.  (And fun is /always/ a good reason for 
> doing something.)

<https://www.kernel.org/doc/html/latest/admin-guide/mm/nommu-mmap.html>
<https://www.techonline.com/tech-papers/supporting-linux-without-an-mmu/>

>> This may make some aspects of the implementation impractical.&nbsp; E.g.,
>> my RTOS relies on a PMMU to share data across protection domains,
>> do zero copy ransfers, etc.&nbsp; But, you may be able to live without
>> the things that rely on that mechanism.
>>
>> [No idea as I've never looked inside the linux kernel]
>>
>> Some of the older kernel versions (and ports) may give you an insight
>> into what can/can't be done.
>>
>> This could expand the range of processors/SoCs that you could use
>> (though likely require some effort for a port).

On 10/26/2022 1:06 AM, David Brown wrote:
> On 24/10/2022 16:20, Theo wrote:
>> I was idly looking to see what was out there in the low end Linux space -
>> something bigger than an ESP32 but more production friendly than a Raspberry
>> Pi.  I came across this excellent guide:
>>
>> https://jaycarlson.net/embedded-linux/
>>
>> He builds dev boards for 10 different chips from 7 vendors, just to see how
>> it all goes - both hardware and software.  The results are quite
>> interesting.
>>
>> Any other recommendations for Linux-supporting SoCs that are nice for low
>> volume/hand production?
>>
>> Theo
> 
> The key things to determine are what you consider "production friendly", and 
> what you need.  You want a module, not a chip.  Some modules come with pins for 
> connections, others with just solder pads, and some are made to fit SO-DIMM 
> sockets or similar connectors.  Some modules have Ethernet, Wifi, Bluetooth, 
> HDMI, USB, and other high-speed interfaces - others have much less.  Some have 
> on-board eMMC or other NAND flash, others rely on external memory or SD-Cards.  
> Some have their power supply handling on board and need just a single supply at 
> 3.3v or 5v, others need multiple supplies at different levels with careful 
> bringup.  Some have long lifetimes and will be available for a decade, others 
> are from companies that might be gone next month.  Some have excellent support 
> from the supplier, some have excellent community support, and others have 
> virtually no support at all.
> 
> We don't know anything about the product, its needs, or about what you can do 
> yourself and what you need supplied.  All I can give you is general advice here 
> regarding things to consider.  And be wary of trying to get minimal cost for 
> the module - it can easily cost you more in the long run.  (Equally, high price 
> of a module is no guarantee.)
> 
> There are many people making SoC's that can work well with Linux, mostly ARM 
> Cortex-A but also some RISC-V now.  (There are also PPC, MIPS, and a few other 
> cores, but those are in more specialised devices like network chips.)  There 
> are no SoC's that are remotely suitable for hand production.
> 
> Another thing to consider, of course, is whether a Linux module is what you 
> really want.  There are microcontrollers that are more powerful than ESP32 
> devices, such as NXP's i.mx RT line (with 500-1000 MHz Cortex-M7 cores).  On 
> the software side, there is Zephyr which sits somewhere between FreeRTOS and 
> Linux and might be useful.  (I haven't tried Zephyr myself.)

Low quantities isn't the only reason to avoid component level design.

When you buy a module, you are reliant on the supplier to continue
making said module available.  And, in many cases, WITHOUT CHANGES!
("Oh, the last batch we sent you has new firmware or replaces U11
with a functionally equivalent, but DIFFERENT, component.  It *shouldn't*
cause you any problems...")  You likely have no insight into their
DETAILED planning for a device -- especially if you are buying them
in small quantities!

[The same is true with components.  But, the module provider ALSO
is at risk to those component -- and service -- suppliers and you
likely aren't tracking what *his* current risks are!]

You have less information about the actual operating conditions
and limits of the "component (module!)".  And, can't easily do
anything to tailor those to your specific needs.  You can put in
place a SUPPLEMENTAL test/burn-in practice... ADDING to the
cost of the supplied module to assure yourself that it will
meet YOUR needs.

You have less control over form factors and packaging.  You
rely on their choice of connection mechanisms -- which may
not be suitable to your environment.  ("Can I SOLDER this DIMM
onto your module so it won't vibrate loose?")  Connectors and
caps are the bane of all designs.

[Most of the designs for my current project are implemented
on ~1"x2" double-sided boards that are *stacked* to make specific
"devices" -- because anything larger would have unfavorable
packaging.  I, obviously, want to ensure that I can leverage the
design such that I don't have very many *different* 1x2 boards
and don't want to burden the product cost with some hardware
feature that isn't useful on a board -- but that *might* be
useful on a board that targets hobbyists!]

The cost of designing, laying out and prototyping your own
boards -- regardless of production quantity -- is insignificant
compared to the cost of developing the software that runs on those
boards.

On 10/26/2022 2:09 AM, Theo wrote:
> I don't have a product :-)  But really just making a thought experiment
> about what would happen if I did have a product - let's say an IoT thingy
> (wifi, display, etc) in the <$100 sticker price, initial volumes let's say
> hundreds.

But, you're only looking at Linux (or, any "fleshy" OS) because you
think it will make WiFi, networking, display, etc. "more straight-forward".
You don't really care, functionally, if it is "Linux" (i.e., a particular
kernel) that makes that happen, do you?  As long as the API isn't
too bizarre...

The problem, I see, is ending up with lots of "features" that you
don't really need in a given product.

Do you *really* need a filesystem -- let alone support for a variety
of them (and a structure that facilitates supporting many even if you
only use *one*?).

Do you really need to be able to support multiple network interfaces
with a stack that is designed to allow "equivalent" interfaces to
their drivers to slide under it?

Once your app is up and running, will the page tables EVER change?

> The ESP32s are nice as they're a simple, cheap, wifi module.  If you wanted
> to cut costs you could use the bare chip.  The Pis aren't: the Zero is a
> nice form factor, but you can't buy it in volume.  The regular Pis can't
> really be mounted on a custom PCB if you don't have a large enclosure.  The
> Compute Modules are better, but still larger than an ESP32.  However you
> can't really buy any of them at the moment, and if you could they would be
> quite expensive.  The Pi2040 is an ok microcontroller but nothing special
> (and wifi is an extra chip).  Also none of them have any protection from
> someone changing or stealing your firmware.

That last isn't as easy to guard against as you might think...

> It is interesting in the above article how much the complexity starts to
> rise once you start going beyond a single chip solution: BGAs, DDR routing,
> numerous power supplies and sequencing, etc.

But there's no black magic, there.  This is all "common practice", now.
If you don't have the skills, you develop them (as the author suggests).
Layout tools do a lot of this for you.  And, if you are looking at
smallish "products", the hairy parts of the design are usually close
to the CPU and don't extend far into the field.

Eyesight gets to be a problem, as you get older.  Parts are moving
in the wrong direction (in terms of size!).  <frown>  But, a mantis
or stereo-microscope can be a win, there.  Or, subbing the fab
out to a (local!) group that can also handle some of your rework,
as may later be needed.

>> There are many people making SoC's that can work well with Linux, mostly
>> ARM Cortex-A but also some RISC-V now.  (There are also PPC, MIPS, and a
>> few other cores, but those are in more specialised devices like network
>> chips.)  There are no SoC's that are remotely suitable for hand production.
> 
> Some of the SIPs and BGAs in the article above are, allegedly.  However
> 'hand production' is really a proxy for production complexity.  If you can
> build a 4 layer board and hand-mount it, you can build in low-ish volume on
> a relatively cheap pick and place line.  If you need a 10 layer board and
> package-on-package BGA mounting equipment, you can't do that without a much
> greater volume to amortise the tooling costs.

Or, a firm that has already made that investment.

I'm always amused at the folks who design products around modules.
And, then have to design a daughter-card (which may, in fact, be a
*mother*!) to address the other issues necessary to their design
(few real world devices mate to pins on 0.1" centers!)

So, the effort they were trying to avoid becomes essential as
a consequence of their use of a "module"

> Systems on module are a good solution to that but, if some of these SoCs are
> niche, the modules are even more niche (hard to buy in small quantities,
> produced by a tiny company, and so on).

Exactly.  And, you have no say in HOW they are designed, fabricated,
packaged, etc.

>> Another thing to consider, of course, is whether a Linux module is what
>> you really want.  There are microcontrollers that are more powerful than
>> ESP32 devices, such as NXP's i.mx RT line (with 500-1000 MHz Cortex-M7
>> cores).  On the software side, there is Zephyr which sits somewhere
>> between FreeRTOS and Linux and might be useful.  (I haven't tried Zephyr
>> myself.)
> 
> The iMX RT isn't one I've come across, thanks.  That's the kind of thing I'm
> interested in.
> 
> The software side is one that's frequently neglected: one thing the
> Raspberry Pi folks are really good at is maintaining their software stack.
> A lot of other (Chinese) Linux SoC vendors basically throw it all over the
> wall and let the customers do the maintenance.  In some ways it's nice not
> to play in that space.  OTOH once you get beyond a certain point it's nice
> to be able to use 'grown up' tools (like a webserver that can easily do TLS,
> not some stripped down microcontroller TLS stack that only does TLS 1.1 and
> can't fit any more in RAM, or worse does no TLS at all).
> 
> I'm really mainly curious how this middle part of the market goes, and
> wondering how others feel about it.

If you want to be in a business (regardless of size), you have to invest
in the tools necessary to make that business work.  The tools can be
physical assets -- or, intellectual skillsets.

Only you can identify the likely direction your business (products)
will take.  So, only you can decide which "tools" are sensible
investments.

[I don't design the molds for my enclosures.  I do "CAD-sketches"
for a guy that refines them for me with the details of the stuff
that goes into -- and connects to -- each of them.  *He* is one
of my tools.]

Don Y <blockedofcourse@foo.invalid> wrote:
> On 10/26/2022 2:09 AM, Theo wrote:
> > I don't have a product :-)  But really just making a thought experiment
> > about what would happen if I did have a product - let's say an IoT thingy
> > (wifi, display, etc) in the <$100 sticker price, initial volumes let's say
> > hundreds.
> 
> But, you're only looking at Linux (or, any "fleshy" OS) because you
> think it will make WiFi, networking, display, etc. "more straight-forward".
> You don't really care, functionally, if it is "Linux" (i.e., a particular
> kernel) that makes that happen, do you?  As long as the API isn't
> too bizarre...

The reason people use Linux is for the software stacks.  It allows you to
write in a more friendly language, have better libraries for doing
complicated things, use existing tooling, not have to worry about boring
housekeeping things like the networking (does your thing support IPv6? 
Linux has for decades, does your homebrew embedded RTOS?  What about WPA3?). 
Can you interact securely with whatever cloud service your widget needs to
do its thing?  (especially if that service is not designed specifically for
talking to low-end widgets)

Essentially you trade off ease of software development for hardware
complexity.  If you're playing in the low volume game, development effort
and time to market is more important than saving cents on production costs.
If you're selling by the million the tradeoff is different.

> The problem, I see, is ending up with lots of "features" that you
> don't really need in a given product.
> 
> Do you *really* need a filesystem -- let alone support for a variety
> of them (and a structure that facilitates supporting many even if you
> only use *one*?).

If you want to run <tool> and that needs a filesystem, yes you do.  I'm sure
you could reimplement it to do without, but that takes effort.

> Do you really need to be able to support multiple network interfaces
> with a stack that is designed to allow "equivalent" interfaces to
> their drivers to slide under it?
> 
> Once your app is up and running, will the page tables EVER change?

That depends on the app.  The point here is to be able to use existing
software without having to re-engineer it.  Once you start re-engineering
things, that's where your time goes.

> > The ESP32s are nice as they're a simple, cheap, wifi module.  If you wanted
> > to cut costs you could use the bare chip.  The Pis aren't: the Zero is a
> > nice form factor, but you can't buy it in volume.  The regular Pis can't
> > really be mounted on a custom PCB if you don't have a large enclosure.  The
> > Compute Modules are better, but still larger than an ESP32.  However you
> > can't really buy any of them at the moment, and if you could they would be
> > quite expensive.  The Pi2040 is an ok microcontroller but nothing special
> > (and wifi is an extra chip).  Also none of them have any protection from
> > someone changing or stealing your firmware.
> 
> That last isn't as easy to guard against as you might think...

Indeed, which is why microcontrollers have various secure boot and encrypted
firmware support.

(which aren't perfect, but prevent somebody just pulling your flash chip and
reading it out)

> > It is interesting in the above article how much the complexity starts to
> > rise once you start going beyond a single chip solution: BGAs, DDR routing,
> > numerous power supplies and sequencing, etc.
> 
> But there's no black magic, there.  This is all "common practice", now.
> If you don't have the skills, you develop them (as the author suggests).
> Layout tools do a lot of this for you.  And, if you are looking at
> smallish "products", the hairy parts of the design are usually close
> to the CPU and don't extend far into the field.

Indeed, no black magic, just time and cost.  Don't do it if you don't need
it.

> If you want to be in a business (regardless of size), you have to invest
> in the tools necessary to make that business work.  The tools can be
> physical assets -- or, intellectual skillsets.
> 
> Only you can identify the likely direction your business (products)
> will take.  So, only you can decide which "tools" are sensible
> investments.

The thing here is choosing your battles.  Spend your time on the things that
add value to the product.  Don't make life needlessly harder when that's not
necessary.  Everything *can* be done, but some things shouldn't *need* to be
done.  If you're in the high-volume game, saving $1m using cheaper parts
makes sense.  If you're in the low-volume game, you might only save $1000
but spend $10K in time doing so.

Theo

On 26/10/2022 11:09, Theo wrote:
> David Brown <david.brown@hesbynett.no> wrote:
>>>
>>> https://jaycarlson.net/embedded-linux/
>>
>> The key things to determine are what you consider "production friendly",
>> and what you need.  You want a module, not a chip.  Some modules come
>> with pins for connections, others with just solder pads, and some are
>> made to fit SO-DIMM sockets or similar connectors.  Some modules have
>> Ethernet, Wifi, Bluetooth, HDMI, USB, and other high-speed interfaces -
>> others have much less.  Some have on-board eMMC or other NAND flash,
>> others rely on external memory or SD-Cards.  Some have their power
>> supply handling on board and need just a single supply at 3.3v or 5v,
>> others need multiple supplies at different levels with careful bringup.
>>    Some have long lifetimes and will be available for a decade, others
>> are from companies that might be gone next month.  Some have excellent
>> support from the supplier, some have excellent community support, and
>> others have virtually no support at all.
> 
> The above article covers all of those things in a nice way: some parts are
> in 64 pin QFNs, some are in 0.8mm BGA which he reckons is doable to hand
> solder (I haven't tried that...).  Some have abandonware software stacks,
> others are in the mainline Linux tree.  etc etc

If you are doing all this for fun and learning, where your own time is 
free and reliability is not an issue, then you can do some of this by 
hand.  If you are trying to make a product to sell to others and turn a 
profit, it's a completely different situation.

BGA's are okay to place by hand, but getting a good, even soldering 
result with kitchen-top tools is unlikely.  At best, you'll get 
something that works for a while - but put it to real use and the voids, 
half-contacts, partial short-circuits and other flaws will cause 
failures sooner or later as thermal stresses wear them out.  And then 
you have the 0.5 mm pitch QFN packages, the 0402 chicken feed 
components, and all the rest of it.

And if this is professional, don't forget the testing and certification 
you need, depending on where you are selling it - things like EMC 
testing and radio emission regulations.  If your home made device has 
Wifi or Bluetooth, and you want to sell it, the certification process 
will cost you hundreds of thousands of dollars (especially since you 
haven't a hope in hell of passing the tests when you do home production).

But it can certainly be fun as a hobby and to get a better understanding 
about how all this works.

> 
>> We don't know anything about the product, its needs, or about what you
>> can do yourself and what you need supplied.  All I can give you is
>> general advice here regarding things to consider.  And be wary of trying
>> to get minimal cost for the module - it can easily cost you more in the
>> long run.  (Equally, high price of a module is no guarantee.)
> 
> I don't have a product :-)  But really just making a thought experiment
> about what would happen if I did have a product - let's say an IoT thingy
> (wifi, display, etc) in the <$100 sticker price, initial volumes let's say
> hundreds.
> 
> The ESP32s are nice as they're a simple, cheap, wifi module.  

Yes - they are often a first choice for when you want Wifi and/or Bluetooth.

> If you wanted
> to cut costs you could use the bare chip.  

No, you can't.  You can't design a working Wifi module for the price of 
100 ESP32 modules, assuming you value the hours spent appropriately for 
an electronics engineer.  You can't produce a working Wifi module in 
your kitchen or garage, because the required components are too small to 
handle by hand.  And that's before you try and certify the thing so that 
it is legal to sell.

At my company, we have experienced electronics designers with top-class 
design software.  We have production facilities for high-speed automated 
production of low and mid volume production runs, capable of placing and 
soldering parts that are barely visible to the naked eye, with optical 
and x-ray inspection systems.  We would not consider making a product 
with Wifi using bare chips - we would use ready-made modules.  If we 
can't do it, /you/ can't do it.


> The Pis aren't: the Zero is a
> nice form factor, but you can't buy it in volume.

Of course you can order them in volume.  More appropriate, perhaps, are 
the Pi compute modules - which you can also order in volume.  You are 
asking for hundreds, while distributors will happily take orders in tens 
of thousands for these.

However, like almost everything else in the electronics industry these 
days, you'll be hard pushed to find much stock of Pis, or any other 
Linux module, Linux-capable SoC, or the other components involved.  So 
if you need something in the short term, take whatever you can find in 
stock.

Hopefully the current component shortage situation won't last forever, 
and then you'll be able to order Pi Zeros and Pi Compute Modules in 
whatever quantity suits.

>  The regular Pis can't
> really be mounted on a custom PCB if you don't have a large enclosure.  The
> Compute Modules are better, but still larger than an ESP32.  However you
> can't really buy any of them at the moment, and if you could they would be
> quite expensive.  The Pi2040 is an ok microcontroller but nothing special
> (and wifi is an extra chip).  Also none of them have any protection from
> someone changing or stealing your firmware.
> 
> It is interesting in the above article how much the complexity starts to
> rise once you start going beyond a single chip solution: BGAs, DDR routing,
> numerous power supplies and sequencing, etc.

Linux systems are /never/ a single chip solution.  And yes, it is can 
often be the other chips that are the biggest challenges - or their 
supporting small components.

> 
>> There are many people making SoC's that can work well with Linux, mostly
>> ARM Cortex-A but also some RISC-V now.  (There are also PPC, MIPS, and a
>> few other cores, but those are in more specialised devices like network
>> chips.)  There are no SoC's that are remotely suitable for hand production.
> 
> Some of the SIPs and BGAs in the article above are, allegedly.  However
> 'hand production' is really a proxy for production complexity.  If you can
> build a 4 layer board and hand-mount it, you can build in low-ish volume on
> a relatively cheap pick and place line.  If you need a 10 layer board and
> package-on-package BGA mounting equipment, you can't do that without a much
> greater volume to amortise the tooling costs.
> 

That is partly correct, partly misunderstanding.

The board layer count affects the cost of the pcb itself, and the effort 
(and tools) required for the design.  It doesn't affect the board 
manufacturing (you don't make the pcb yourself), although it can limit 
the suppliers that can make it for you.

If you want to make professional quality boards and sell them, then you 
do not do it with hand mounting - even if some guy on the internet says 
it's possible.  If you don't have the volumes involved to have the 
production tools needed for automated pick and place, optical 
inspection, proper solder ovens, etc., outsource the board production. 
There is no shortage of companies who will do this even for runs of 
hundreds of boards - you can choose between more local suppliers that 
will have well-trained staff that will work with you to improve the 
design, all the way to anonymous far-eastern companies that will work 
cheaply and give you exactly what you ask for, mistakes and all.

There are some kinds of boards that are fine for small scale 
manufacturing with simple machines - Linux boards are not one of them. 
A base board for mounting a Linux module, might be a lot more practical 
for your own production.

> Systems on module are a good solution to that but, if some of these SoCs are
> niche, the modules are even more niche (hard to buy in small quantities,
> produced by a tiny company, and so on).
> 

The niche SoCs are not normally on modules.  The people who buy a SoC 
with MIPS or PPC cores do so because they are making massive network 
switches, car engine controllers, and the like.

>> Another thing to consider, of course, is whether a Linux module is what
>> you really want.  There are microcontrollers that are more powerful than
>> ESP32 devices, such as NXP's i.mx RT line (with 500-1000 MHz Cortex-M7
>> cores).  On the software side, there is Zephyr which sits somewhere
>> between FreeRTOS and Linux and might be useful.  (I haven't tried Zephyr
>> myself.)
> 
> The iMX RT isn't one I've come across, thanks.  That's the kind of thing I'm
> interested in.

The more "fun" parts in the family are fair sized BGA's.  They are a 
nice group of parts.

> 
> The software side is one that's frequently neglected: one thing the
> Raspberry Pi folks are really good at is maintaining their software stack.
> A lot of other (Chinese) Linux SoC vendors basically throw it all over the
> wall and let the customers do the maintenance.  In some ways it's nice not
> to play in that space.  OTOH once you get beyond a certain point it's nice
> to be able to use 'grown up' tools (like a webserver that can easily do TLS,
> not some stripped down microcontroller TLS stack that only does TLS 1.1 and
> can't fit any more in RAM, or worse does no TLS at all).
> 

IMHO the "encrypt everything" movement is a silly idea and a massive 
waste of effort and resources.  Sure, you want your bank website traffic 
to use SSL, but it is completely unnecessary for the great majority of 
web traffic.

But I agree that sometimes it is nice to have plenty of resources in 
your embedded system, whatever you use them for.

> I'm really mainly curious how this middle part of the market goes, and
> wondering how others feel about it.
> 
> Theo