Embedded Scripting -- Tcl? Lua? Thoughts? Suggestions?

On 14/03/15 12:45, Don Y wrote:
> Are you sure you're not thinking of tk?

tcl is the language, tk the toolkit - they're very close friends.

> Likewise, I thought the ability to perform multiple assignments
> (with a *single* assignment operator) would cause problems.
> how is that *read* different from x=1  y=3)

The only real use for multiple assignment is swapping values without 
using an explicit temporary:
x, y = y, x

> Likewise, the potential for confusion when arguments are omitted
> from function invocations.  I.e., it doesn't do much to help
> protect the programmer from "little (painful) mistakes":
>     "Oh, crap!  Foo() takes *3* parameters!)

Calling a 3-argument function with 2 arguments should return a function 
that takes the remaining argument :) (I.e. currying)

> I've been surgically removing/rewriting features of Limbo (my
> scripting choice)

Where can I read more about Limbo? Is this the language from Inferno?
Hmm, that looks a lot like what I described in my other message :)

David Brown <david.brown@hesbynett.no> writes:
> A disadvantage [of Lua] is that there is only one number type, which
> is floating point by default, but you can change that to integer 

They finally added integers to Lua, as of version 5.3, released earlier
this year.

Tim Wescott <seemywebsite@myfooter.really> writes:
> I'm planning on having a chunk of serial flash (or an SD card)... my
> understanding is that I can expect Lua to use about 100k of flash,
> which is a bit much for me but bearable. ...  We briefly considered
> Java, but it just looks entirely too big -- it would basically drive
> us to use Linux or some other "real" OS ... I'm looking for opinions,
> so fire away!

OK, embedded enterpreters were one of my favorite subjects for a while,
so here goes.

1. The smallest SD cards available these days are in the gigabytes, so
100k is nothing.  The interpreter's RAM footprint is probably of more
concern on a small embedded system.  How much ram are you willing to
use?  I also have to ask whether you're going to be the one writing the
customizations at the user's request, or if the customer him/herself
wants to write customizations.  You should probably stay away from weird
or difficult languages if the customer is doing the customization.

2. TCL and Lua are both quite easy to embed but Lua is a more modern and
generally better language, so I'd pick it over Tcl.  Lua has more of a
programmer community these days (lots of mobile games and apps embed it)
and there's a high performance compiler (LuaJIT) available if you want
that.

3. However, from my language snob perspective, Lua and Tcl are both
somewhat distasteful.  Also, you'll probably get much better user uptake
from Javascript (another distasteful language) since so many people know
it.  Javascript will need a lot more machine resources and I don't know
what the embedding side is like, but you might want to consider it if
you have the CPU and RAM for it.

4. Other alternatives include Lisp dialects--I used to be into these.
You might look at GNU Guile (Scheme), Hedgehog Lisp (sort of a
functional subset of Scheme), and Lysp
(http://piumarta.com/software/lysp/).  I guess each of these have
drawbacks.

5. You can use Javacard or J2ME in an embedded (non-Linux) environment
but I'm not sure why you'd want to.  It requires external compilation
tools and a somewhat bureaucratic approach to development.  I mention it
because you brought it up.  Main thing it tries to supply (with some
success but not total) is security against untrusted, potentially
hostile scripts trying to mess with the surrounding application or with
other scripts in the same container.  I don't know if that's an issue
for what you're doing.

6. You can also consider Python though its embedding challenges are
harder.  I think it can be used without Linux but I've never done it.
If you go this route it's probably better to write your outer
application in it, calling C modules from Python as necessary, rather
than the other way around (writing the outer app in C and loading Python
into it).  

7. If you're -really- resource constrained or your scripts need realtime
guarantees, there is Forth, but I'd stay away from it if you have enough
cpu and memory to run something like Lua.

Of all the above, based on the limited info you've given, the quick
answer is probably "Lua unless you can afford Javascript or you're a
language geek and like Lisp".  If you can say more about your
application and your hardware constraints, that might help narrow things
down.

Hi Clifford,

On 3/13/2015 7:14 PM, Clifford Heath wrote:
> On 14/03/15 12:45, Don Y wrote:
>> Are you sure you're not thinking of tk?
>
> tcl is the language, tk the toolkit - they're very close friends.
>
>> Likewise, I thought the ability to perform multiple assignments
>> (with a *single* assignment operator) would cause problems.
>> how is that *read* different from x=1  y=3)
>
> The only real use for multiple assignment is swapping values without using an
> explicit temporary:
> x, y = y, x

Yes -- saving a temporary.  But, it's a feature that can easily be
abused (no direct consequences of doing so -- for the *abuser*).

Limbo suffers from this same problem:
     x, y := 3;
sets *both* x and y to '3'.  x, y = 2, 3 is an error.

To it's *credit*, the lua style of multiple assignment can be
accomplished using tuples:
    (x, y) := (2, 3);
though tuples are far more useful for other things!

>> Likewise, the potential for confusion when arguments are omitted
>> from function invocations.  I.e., it doesn't do much to help
>> protect the programmer from "little (painful) mistakes":
>>     "Oh, crap!  Foo() takes *3* parameters!)
>
> Calling a 3-argument function with 2 arguments should return a function that
> takes the remaining argument :) (I.e. currying)
>
>> I've been surgically removing/rewriting features of Limbo (my
>> scripting choice)
>
> Where can I read more about Limbo? Is this the language from Inferno?
> Hmm, that looks a lot like what I described in my other message :)

The vita nuova web site has has the two seminal papers on Limbo
(A Descent into Limbo; The Limbo Programming Language).  I htink
there are two (?) texts that also address the language/"system"
(as Limbo is intimately tied up in Inferno... where the language
ends and the system begins can be debated).

But, it's more fleshy than a scripting language.  E.g., contains
communication primitives *in* the language, seamlessly supports
RPC, multitasking, demand loadable modules, exceptions, GC, etc.

So, it's a good language to develop *real* applications in addition
to just hacking together simple little "basic" scripts.

The biggest problem is all the syntax that can easily intimidate
a casual user.

E.g.,
    i = 1;
    i := 1;
behave similarly but subtly different.  A casual user might be
stressed to discover his problem boiled down to a missing/extra
colon!  Or, "<-" vs. "->" vs. "=>".  Niggly details.  I, for
example, found it frustrating that you can't declare const lists.
Or even *initialize* a list in it's entirety!

(the current compiler leaves a lot to be desired in terms of
localizing errors)

FWIW, I think the current license terms are much more generous than
the *paid* license I acquired (change in their business model?)
So, you can download the environment -- BSD, Windows, OSX, Linux,
Slowaris, Plan 9 (of course) -- and play with it.  At one point,
you could run the VM in a browser (not sure if that is still supported).

On Fri, 13 Mar 2015 20:16:32 -0700, Paul Rubin wrote:

> Tim Wescott <seemywebsite@myfooter.really> writes:
>> I'm planning on having a chunk of serial flash (or an SD card)... my
>> understanding is that I can expect Lua to use about 100k of flash,
>> which is a bit much for me but bearable. ...  We briefly considered
>> Java, but it just looks entirely too big -- it would basically drive us
>> to use Linux or some other "real" OS ... I'm looking for opinions,
>> so fire away!
> 
> OK, embedded enterpreters were one of my favorite subjects for a while,
> so here goes.
> 
> 1. The smallest SD cards available these days are in the gigabytes, so
> 100k is nothing.  The interpreter's RAM footprint is probably of more
> concern on a small embedded system.  How much ram are you willing to
> use?  I also have to ask whether you're going to be the one writing the
> customizations at the user's request, or if the customer him/herself
> wants to write customizations.  You should probably stay away from weird
> or difficult languages if the customer is doing the customization.
> 
> 2. TCL and Lua are both quite easy to embed but Lua is a more modern and
> generally better language, so I'd pick it over Tcl.  Lua has more of a
> programmer community these days (lots of mobile games and apps embed it)
> and there's a high performance compiler (LuaJIT) available if you want
> that.
> 
> 3. However, from my language snob perspective, Lua and Tcl are both
> somewhat distasteful.  Also, you'll probably get much better user uptake
> from Javascript (another distasteful language) since so many people know
> it.  Javascript will need a lot more machine resources and I don't know
> what the embedding side is like, but you might want to consider it if
> you have the CPU and RAM for it.
> 
> 4. Other alternatives include Lisp dialects--I used to be into these.
> You might look at GNU Guile (Scheme), Hedgehog Lisp (sort of a
> functional subset of Scheme), and Lysp
> (http://piumarta.com/software/lysp/).  I guess each of these have
> drawbacks.
> 
> 5. You can use Javacard or J2ME in an embedded (non-Linux) environment
> but I'm not sure why you'd want to.  It requires external compilation
> tools and a somewhat bureaucratic approach to development.  I mention it
> because you brought it up.  Main thing it tries to supply (with some
> success but not total) is security against untrusted, potentially
> hostile scripts trying to mess with the surrounding application or with
> other scripts in the same container.  I don't know if that's an issue
> for what you're doing.
> 
> 6. You can also consider Python though its embedding challenges are
> harder.  I think it can be used without Linux but I've never done it. If
> you go this route it's probably better to write your outer application
> in it, calling C modules from Python as necessary, rather than the other
> way around (writing the outer app in C and loading Python into it).
> 
> 7. If you're -really- resource constrained or your scripts need realtime
> guarantees, there is Forth, but I'd stay away from it if you have enough
> cpu and memory to run something like Lua.
> 
> Of all the above, based on the limited info you've given, the quick
> answer is probably "Lua unless you can afford Javascript or you're a
> language geek and like Lisp".  If you can say more about your
> application and your hardware constraints, that might help narrow things
> down.

I'm trying to stick with an ARM M4 cored processor that has 256kB to 
512kB of flash, and maybe 64kB of RAM, and a 72MHz CPU speed.  Everything 
can run at "human time", so delays of no less than 50ms are OK.  There is 
an issue that a really hot operator shouldn't be able to overrun the 
machine, but the buttons are actuated by a wand (it's complicated -- they 
just are), which should slow things down.

Even if I loved Lisp (or Forth) more than anything else in the world, I'd 
still use Lua if it meant better user acceptance.  I'll consider 
Javascript for that very reason, particularly if I can find an open-
source interpreter that seems serious and well-supported.

-- 
www.wescottdesign.com

Tim Wescott <tim@seemywebsite.com> writes:
> I'm trying to stick with an ARM M4 cored processor that has 256kB to 
> 512kB of flash, and maybe 64kB of RAM, and a 72MHz CPU speed. 

Hmm, ok, I think Javascript is not practical in something that small.
Lua can work but will be a little bit cramped.  How much of that flash
and ram will actually be available to Lua (i.e. not needed by the
non-Lua code)?  How complicated do you expect the scripts to be?  Are
you ok with cross-compiling the scripts?  That means loading a script by
compiling it to bytecode with a PC application, then loading the
bytecode to your embedded board, instead of including the compiler in
the board so you can talk to it with a terminal.

http://wiki.eluaproject.net/FAQ#minimumrequirements

has some info.  TCL (especially old versions) is probably smaller.  I
know there are some bigger Cortex M4 parts, like ST has one with 2MB of
flash and 256k of ram -- do they cost too much or something?

> I'll consider Javascript for that very reason, particularly if I can
> find an open- source interpreter that seems serious and
> well-supported.

You'd have to use a bigger processor, but it still may be worth it.  I
worked on something a couple years ago that could have been done in C on
an 8 bitter, but we used an embedded ARM7/Linux system with 64 meg of
ram for development convenience, adding maybe $5 to the hardware BOM
which was acceptable for what we were doing.  I think it was well
justified by the flexibility it gave us.  We were able to ship a better
product faster.

I don't know anything about JS embeddability.  I've only programmed it
at the application level.  I know that various applications other than
web browsers do embed JS, but it may be a lot more complicated than
using Lua.

Will your scripts have to process much non-numeric data like strings and
complicated structures?  If not, maybe there's some BASIC dialect that
your users would find comfortable, especially if they're hardware types
who have been around for a while.

Paul Rubin <no.email@nospam.invalid> writes:
> I think Javascript is not practical in something that small.

Hmm, this may be of interest: http://www.espruino.com/

It claims to run JS in a 64k ram part like yours.

I just came across it and don't know any more.  I don't know what
dialect of JS it uses.  It could be limited in a way that frustrates
your users who are used to programming browser JS.  I'm sure that 
fancy modern JS implementations like V8 are way larger.

Don Y <this@is.not.me.com> wrote:
> On 3/13/2015 5:28 PM, mroberds@att.net wrote:
>
>> For a while, sometimes Tcl was used as sort of the Linux equivalent
>> of Visual Basic - for when you wanted a simple GUI but didn't want to
>> spend weeks screwing around with X/(GTK|Qt)/C++.
>
> Are you sure you're not thinking of tk?

Now that I think about it, all the GUI stuff seemed to be a
combination: Tcl/tk.  I guess you can use Tcl on its own without tk.

>> The thing I found odd about the choice of Lua for this application
>> was that *it didn't have any bitwise operators*.
>
> I didn't like the fact that you could mix and match types without
> anything alerting you to the fact that you're trying to do something
> "wrong".  E.g., if (5 == "5")...

I don't remember ever crashing Wireshark by screwing up my Lua scripts.
Things just wouldn't decode / display on the Wireshark GUI correctly.
If that happened, I could always save the captured packets to a file,
fix my Lua script, reload the file in Wireshark, and have the script
run again.

The Lua interpreter also supported a simple printf-like function that
would just display on the Wireshark console and not in the GUI.
Wireshark would open a separate 80x25 text window when you used the
console.  I found this useful for answering "am I pulling the right
bytes out of the packet or not", and *then* I could work on pushing
those bytes to the GUI in a useful way.

Matt Roberds

On Fri, 13 Mar 2015 16:27:12 -0500, Tim Wescott
<seemywebsite@myfooter.really> wrote:

>The title says most of it.
>
>I'm working on a project for which the customer wants a lot of flexibility 
>to customize a product.  The existing "macro language" is ancient (ca 
>1985), cryptic (it looks like machine language), limited, clunky, etc.  
>The customer is willing to change.
>
>I'm planning on having a chunk of serial flash (or an SD card) with a file 
>system on it (maybe FAT, maybe something home-rolled and primitive).  
>Scripts will be stored as text files, and read in as appropriate.
>
>Rather than just use some clunky cryptic and limited language that's 
>unique to the problem and to me (and charging my customer up the wazoo for 
>it), I'd like to use something that's open-source and available.  
>Something that uses modest resources is to be preferred -- my 
>understanding is that I can expect Lua to use about 100k of flash, which 
>is a bit much for me but bearable.  I don't know how Tcl compares -- I 
>just know that it's an alternative.
>
>So -- anyone have any direct experience with either of these, or with 
>something else?  We briefly considered Java, but it just looks entirely 
>too big -- it would basically drive us to use Linux or some other "real" 
>OS (even the little linuxes are too big for my tastes) -- I'd really 
>rather just have a scripting engine that gets called whenever a script 
>needs to run, and otherwise doesn't need much hand-holding from the 
>application code.
>
>I'm looking for opinions, so fire away!


My limited experience is that Lua is a more "normal" language, not
that Tcl is hard, it's just going to look weirder to most users, and
Lua (in its smaller forms) has a smaller footprint, and is a bit
faster than Tcl (whether either of those is an issue is obviously
application dependent, but Tcl's "everything is a string" metaphor
hurts performance some).

Both interface fairly easily to your application, and allow you to add
your own commands/functions easily.  Both are actively supported,
developed, and have widespread use.

If you eventually move to Java, Tcl does have a JVM version, which
might be useful for migration purposes.

I really don't think you'd go far wrong with either though.

On 13.3.15 23:27, Tim Wescott wrote:
> The title says most of it.
>
> I'm working on a project for which the customer wants a lot of flexibility
> to customize a product.  The existing "macro language" is ancient (ca
> 1985), cryptic (it looks like machine language), limited, clunky, etc.
> The customer is willing to change.
>
> I'm planning on having a chunk of serial flash (or an SD card) with a file
> system on it (maybe FAT, maybe something home-rolled and primitive).
> Scripts will be stored as text files, and read in as appropriate.
>
> Rather than just use some clunky cryptic and limited language that's
> unique to the problem and to me (and charging my customer up the wazoo for
> it), I'd like to use something that's open-source and available.
> Something that uses modest resources is to be preferred -- my
> understanding is that I can expect Lua to use about 100k of flash, which
> is a bit much for me but bearable.  I don't know how Tcl compares -- I
> just know that it's an alternative.
>
> So -- anyone have any direct experience with either of these, or with
> something else?  We briefly considered Java, but it just looks entirely
> too big -- it would basically drive us to use Linux or some other "real"
> OS (even the little linuxes are too big for my tastes) -- I'd really
> rather just have a scripting engine that gets called whenever a script
> needs to run, and otherwise doesn't need much hand-holding from the
> application code.
>
> I'm looking for opinions, so fire away!
>

My vote to Jim-TCL or LUA.

-- 

Tauno Voipio