Embedded Scripting -- Tcl? Lua? Thoughts? Suggestions?

The title says most of it.

I'm working on a project for which the customer wants a lot of flexibility 
to customize a product.  The existing "macro language" is ancient (ca 
1985), cryptic (it looks like machine language), limited, clunky, etc.  
The customer is willing to change.

I'm planning on having a chunk of serial flash (or an SD card) with a file 
system on it (maybe FAT, maybe something home-rolled and primitive).  
Scripts will be stored as text files, and read in as appropriate.

Rather than just use some clunky cryptic and limited language that's 
unique to the problem and to me (and charging my customer up the wazoo for 
it), I'd like to use something that's open-source and available.  
Something that uses modest resources is to be preferred -- my 
understanding is that I can expect Lua to use about 100k of flash, which 
is a bit much for me but bearable.  I don't know how Tcl compares -- I 
just know that it's an alternative.

So -- anyone have any direct experience with either of these, or with 
something else?  We briefly considered Java, but it just looks entirely 
too big -- it would basically drive us to use Linux or some other "real" 
OS (even the little linuxes are too big for my tastes) -- I'd really 
rather just have a scripting engine that gets called whenever a script 
needs to run, and otherwise doesn't need much hand-holding from the 
application code.

I'm looking for opinions, so fire away!

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Hi Tim,

On 3/13/2015 2:27 PM, Tim Wescott wrote:
> The title says most of it.
>
> I'm working on a project for which the customer wants a lot of flexibility
> to customize a product.  The existing "macro language" is ancient (ca
> 1985), cryptic (it looks like machine language), limited, clunky, etc.
> The customer is willing to change.
>
> I'm planning on having a chunk of serial flash (or an SD card) with a file
> system on it (maybe FAT, maybe something home-rolled and primitive).
> Scripts will be stored as text files, and read in as appropriate.
>
> Rather than just use some clunky cryptic and limited language that's
> unique to the problem and to me (and charging my customer up the wazoo for
> it), I'd like to use something that's open-source and available.
> Something that uses modest resources is to be preferred -- my
> understanding is that I can expect Lua to use about 100k of flash, which
> is a bit much for me but bearable.  I don't know how Tcl compares -- I
> just know that it's an alternative.
>
> So -- anyone have any direct experience with either of these, or with
> something else?  We briefly considered Java, but it just looks entirely
> too big -- it would basically drive us to use Linux or some other "real"
> OS (even the little linuxes are too big for my tastes) -- I'd really
> rather just have a scripting engine that gets called whenever a script
> needs to run, and otherwise doesn't need much hand-holding from the
> application code.
>
> I'm looking for opinions, so fire away!

I think Blue is better than Red.

Bagels beat donuts hands down!  (unless you're *not* in NYC; then
damn near ANYTHING beats a bagel!)

Chicago Pizza beats New York Pizza.

42

Connery over all others (Moore, Brosnan, Lazenby, Dalton, Craig)

Anything with butter.

As to *your* question...

What sorts of things does the customer expect to *do* with those scripts?
And, (besides space) what other constraints are you needing to place
on those actions?  E.g., any timeliness constraints?  Performance?
Numerical precision?  etc.

Who is the intended user:  *just* the client?  Or, will these things be
exposed to users?

What sorts of things do the scripts interact with (displays, etc.)?
E.g., do you need to be able to manipulate a GUI without constraint?
Or, are you just serving up text and/or canned dialogs?

What sort of user input device (keyboard, keypad, etc.)?  Other I/O's?

I opted for a (bastardized) Limbo as my scripting language.  But, I
use it to generate "real" applications and not just automate some
preexisting actions.  However, it won't fit your memory requirements.

Can you tolerate something as "crude" as a bastardized BASIC?  I.e.,
add "product specific" hooks to it to address some of the special needs
(displays, etc.) at a "native" level.  A BASIC interpreter/compiler
can be really small.

The same can be said of C interpreters, mini-compilers (JIT).  But,
you'd be targeting a different class of "developer" (user).

--don

On 13/03/15 21:27, Tim Wescott wrote:
> The title says most of it.
>
> I'm working on a project for which the customer wants a lot of flexibility
> to customize a product.  The existing "macro language" is ancient (ca
> 1985), cryptic (it looks like machine language), limited, clunky, etc.
> The customer is willing to change.
>
> I'm planning on having a chunk of serial flash (or an SD card) with a file
> system on it (maybe FAT, maybe something home-rolled and primitive).
> Scripts will be stored as text files, and read in as appropriate.
>
> Rather than just use some clunky cryptic and limited language that's
> unique to the problem and to me (and charging my customer up the wazoo for
> it), I'd like to use something that's open-source and available.
> Something that uses modest resources is to be preferred -- my
> understanding is that I can expect Lua to use about 100k of flash, which
> is a bit much for me but bearable.  I don't know how Tcl compares -- I
> just know that it's an alternative.
>
> So -- anyone have any direct experience with either of these, or with
> something else?  We briefly considered Java, but it just looks entirely
> too big -- it would basically drive us to use Linux or some other "real"
> OS (even the little linuxes are too big for my tastes) -- I'd really
> rather just have a scripting engine that gets called whenever a script
> needs to run, and otherwise doesn't need much hand-holding from the
> application code.
>
> I'm looking for opinions, so fire away!

I wish everyone was as mature and rational! I'm personally sick
and tired of scrotty little scripting languages that grey like
Topsy until nobody, including their originators^H^H^H^H^H
perpetrators, understood them anymore.

I haven't used it myself, but Lua looks like it might be a good
starting point, particularly since it is embeddable in/with C.

ISTR there being an embeddable JavaScript (not Java!) interpreter,
which might have the benefit of more familiar syntax.

On 13/03/15 22:27, Tim Wescott wrote:
> The title says most of it.
>
> I'm working on a project for which the customer wants a lot of flexibility
> to customize a product.  The existing "macro language" is ancient (ca
> 1985), cryptic (it looks like machine language), limited, clunky, etc.
> The customer is willing to change.
>
> I'm planning on having a chunk of serial flash (or an SD card) with a file
> system on it (maybe FAT, maybe something home-rolled and primitive).
> Scripts will be stored as text files, and read in as appropriate.
>
> Rather than just use some clunky cryptic and limited language that's
> unique to the problem and to me (and charging my customer up the wazoo for
> it), I'd like to use something that's open-source and available.
> Something that uses modest resources is to be preferred -- my
> understanding is that I can expect Lua to use about 100k of flash, which
> is a bit much for me but bearable.  I don't know how Tcl compares -- I
> just know that it's an alternative.
>
> So -- anyone have any direct experience with either of these, or with
> something else?  We briefly considered Java, but it just looks entirely
> too big -- it would basically drive us to use Linux or some other "real"
> OS (even the little linuxes are too big for my tastes) -- I'd really
> rather just have a scripting engine that gets called whenever a script
> needs to run, and otherwise doesn't need much hand-holding from the
> application code.
>
> I'm looking for opinions, so fire away!
>

I used Lua in a project a number of years ago, and would use it again if 
I needed scripting.  It has a few key advantages over TCL.  It is a 
nicer language, it is quite easy to integrate in a C project (it 
consists of just a few C files), and it's easy to make your own 
functions callable from Lua.  A disadvantage is that there is only one 
number type, which is floating point by default, but you can change that 
to integer if it suits you better (and it's certainly better than 
everything being a string, as it is in TCL).

On Fri, 13 Mar 2015 15:03:09 -0700, Don Y wrote:

> Hi Tim,
> 
> On 3/13/2015 2:27 PM, Tim Wescott wrote:
>> The title says most of it.
>>
>> I'm working on a project for which the customer wants a lot of
>> flexibility to customize a product.  The existing "macro language" is
>> ancient (ca 1985), cryptic (it looks like machine language), limited,
>> clunky, etc. The customer is willing to change.
>>
>> I'm planning on having a chunk of serial flash (or an SD card) with a
>> file system on it (maybe FAT, maybe something home-rolled and
>> primitive). Scripts will be stored as text files, and read in as
>> appropriate.
>>
>> Rather than just use some clunky cryptic and limited language that's
>> unique to the problem and to me (and charging my customer up the wazoo
>> for it), I'd like to use something that's open-source and available.
>> Something that uses modest resources is to be preferred -- my
>> understanding is that I can expect Lua to use about 100k of flash,
>> which is a bit much for me but bearable.  I don't know how Tcl compares
>> -- I just know that it's an alternative.
>>
>> So -- anyone have any direct experience with either of these, or with
>> something else?  We briefly considered Java, but it just looks entirely
>> too big -- it would basically drive us to use Linux or some other
>> "real" OS (even the little linuxes are too big for my tastes) -- I'd
>> really rather just have a scripting engine that gets called whenever a
>> script needs to run, and otherwise doesn't need much hand-holding from
>> the application code.
>>
>> I'm looking for opinions, so fire away!
> 
 << smart-assery snipped >>
> 
> As to *your* question...
> 
> What sorts of things does the customer expect to *do* with those
> scripts?
> And, (besides space) what other constraints are you needing to place on
> those actions?  E.g., any timeliness constraints?  Performance?
> Numerical precision?  etc.
> 
> Who is the intended user:  *just* the client?  Or, will these things be
> exposed to users?
> 
> What sorts of things do the scripts interact with (displays, etc.)?
> E.g., do you need to be able to manipulate a GUI without constraint?
> Or, are you just serving up text and/or canned dialogs?
> 
> What sort of user input device (keyboard, keypad, etc.)?  Other I/O's?
> 
> I opted for a (bastardized) Limbo as my scripting language.  But, I use
> it to generate "real" applications and not just automate some
> preexisting actions.  However, it won't fit your memory requirements.
> 
> Can you tolerate something as "crude" as a bastardized BASIC?  I.e.,
> add "product specific" hooks to it to address some of the special needs
> (displays, etc.) at a "native" level.  A BASIC interpreter/compiler can
> be really small.
> 
> The same can be said of C interpreters, mini-compilers (JIT).  But,
> you'd be targeting a different class of "developer" (user).

The product will have a character display, and it'll store text files for 
later retrieval.  It has a measurement thingie (I don't want to get too 
specific -- but it makes measurements) that'll cough up a number when it's 
actuated, and it has a sea of (well, around 100) buttons that the user can 
press.

My intention at the moment is to make the box this big dumb thing with 
some system code that knows to call a function for each measurement and 
button press (i.e., when the user presses button 1, the function named 
"button_1" will get called if it exists).  Then I'll encode the actual 
functionality as a bunch of functions in the language-o-choice.

The nice thing about this is that if a user comes up with some desired 
customization (which we expect, as we're replacing an existing product for 
which this was common), we can either tell them to start coding and have 
fun, or we can charge them $$ to code the thing up to their specifications.

The real driver for wanting to do this as a scripting language is to make 
it easy for the user to customize.  I'm imagining that some users won't 
want to touch the customization with a 10 foot pole, some will be happy to 
customize a button or two, some will get their feet wet and then call us, 
and some will go whole hog with the thing, and do entire systems without 
us even knowing it's going on.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

On Fri, 13 Mar 2015 15:03:09 -0700, Don Y wrote:

> Hi Tim,
> 
> On 3/13/2015 2:27 PM, Tim Wescott wrote:
>> The title says most of it.
>>
>> I'm working on a project for which the customer wants a lot of
>> flexibility to customize a product.  The existing "macro language" is
>> ancient (ca 1985), cryptic (it looks like machine language), limited,
>> clunky, etc. The customer is willing to change.
>>
>> I'm planning on having a chunk of serial flash (or an SD card) with a
>> file system on it (maybe FAT, maybe something home-rolled and
>> primitive). Scripts will be stored as text files, and read in as
>> appropriate.
>>
>> Rather than just use some clunky cryptic and limited language that's
>> unique to the problem and to me (and charging my customer up the wazoo
>> for it), I'd like to use something that's open-source and available.
>> Something that uses modest resources is to be preferred -- my
>> understanding is that I can expect Lua to use about 100k of flash,
>> which is a bit much for me but bearable.  I don't know how Tcl compares
>> -- I just know that it's an alternative.
>>
>> So -- anyone have any direct experience with either of these, or with
>> something else?  We briefly considered Java, but it just looks entirely
>> too big -- it would basically drive us to use Linux or some other
>> "real" OS (even the little linuxes are too big for my tastes) -- I'd
>> really rather just have a scripting engine that gets called whenever a
>> script needs to run, and otherwise doesn't need much hand-holding from
>> the application code.
>>
>> I'm looking for opinions, so fire away!

<< snip >>

> Can you tolerate something as "crude" as a bastardized BASIC?  I.e.,
> add "product specific" hooks to it to address some of the special needs
> (displays, etc.) at a "native" level.  A BASIC interpreter/compiler can
> be really small.

Oh, and -- we've booted the idea of BASIC around, too.  As far as I'm 
concerned, the first letter or two of BASIC stands for BArf, and I'll have 
to write scripts for the thing.  So it's not high on our list of preferred 
options.

-- 

Tim Wescott
Wescott Design Services
http://www.wescottdesign.com

Tim Wescott <seemywebsite@myfooter.really> wrote:
> So -- anyone have any direct experience with either of these, or with
> something else?

I did a tiny amount of Tcl 10+ years ago, but that was on a Linux
desktop with GUI.  For a while, sometimes Tcl was used as sort of the
Linux equivalent of Visual Basic - for when you wanted a simple GUI but
didn't want to spend weeks screwing around with X/(GTK|Qt)/C++.  I've
never tried embedding Tcl in anything.

I *have* used embedded Lua more recently.  Wireshark (a packet sniffer,
formerly called Ethereal) understands lots and lots of packet formats,
but if you've come up with your own weird thing, it just gives you a hex
dump of the payload.  You can write your own packet parser in Lua and
Wireshark will run it on the payload for you.  The basic application is
to make the packet more human-readable.  You can also use it along with
Wireshark's native ability to trigger on or sort by certain data fields
to only log packets of interest.

The thing I found odd about the choice of Lua for this application was
that *it didn't have any bitwise operators*.  You couldn't do stuff like
"if(packet[3] & 0x04) { /*...*/ }".  I did a lot of multiply and divide
by 2, and tests like "if(packet[5] > 127)" to see if the high bit was
set.  Also, IIRC, Lua's default numeric type was float.

This was a couple of years ago; maybe "full" Lua could do bitwise back
then and Wireshark was only implementing a subset, or maybe Lua couldn't
do it back then but can do it now.

Wireshark also had the option of writing a packet dissector in C, and
then compiling it into a library (.so or .dll depending on platform);
this was said to give you faster decoding.  Most of the protocols I was
looking at sent packets at no more than a few dozen Hz; I think one of
them topped out around 1 KHz.  The Lua scripts didn't seem to have any
problem doing "live" decodes on a new-in-2007 PC.

I know I've seen the option to embed a Lua interpreter in something more
"embedded", possibly U-Boot.  Lua's Wikipedia page lists a lot of places
where it has been embedded.

I will give my standard advice: ship a few example scripts with the
product.  You can assume that people know how to use arithmetic
operators and things like that, but the details of how to get data out
of and back into your hardware will be unique to your implementation.
I think Wireshark had a limited example in the manual, but since it's
popular, I was able to find additional examples online.  Most
applications won't be able to count on that, though.

Matt Roberds

On 3/13/2015 5:06 PM, Tim Wescott wrote:

>> Can you tolerate something as "crude" as a bastardized BASIC?  I.e.,
>> add "product specific" hooks to it to address some of the special needs
>> (displays, etc.) at a "native" level.  A BASIC interpreter/compiler can
>> be really small.
>
> Oh, and -- we've booted the idea of BASIC around, too.  As far as I'm
> concerned, the first letter or two of BASIC stands for BArf, and I'll have
> to write scripts for the thing.  So it's not high on our list of preferred
> options.

Presumably, your goal is to make life pleasant for the *customer*, not
*yourself*!  :>

I offered BASIC as an option as you can write one in a few KB and is
easily understood by "non-programmers".  I've worked on instruments
where (my) customer (e.g., chemist) was charged with scripting behaviors
that *his* customers would blindly invoke (e.g., to determine if the
hospital patient has Hep-C, HIV, etc.).  So, lots riding on (my)
customer "getting it right" as I'm not looking over his shoulder to
verify he's considered every exception, boundary case, etc.  (All I
have to do is ensure the language is well-behaved and has no "gotchas"
that are likely to trip someone up two-levels removed from me!

Hi Matt,

On 3/13/2015 5:28 PM, mroberds@att.net wrote:
> Tim Wescott <seemywebsite@myfooter.really> wrote:
>> So -- anyone have any direct experience with either of these, or with
>> something else?
>
> I did a tiny amount of Tcl 10+ years ago, but that was on a Linux
> desktop with GUI.  For a while, sometimes Tcl was used as sort of the
> Linux equivalent of Visual Basic - for when you wanted a simple GUI but
> didn't want to spend weeks screwing around with X/(GTK|Qt)/C++.  I've
> never tried embedding Tcl in anything.

Are you sure you're not thinking of tk?

I looked at a bunch of OTS languages when I had to settle on a
scripting (i.e., "application") language.  Many are far too cryptic
for casual users (I wanted to target "real programmers", hobbyists
and "homeowners" -- lots of competing criteria!).

Tcl has some features that are troublesome (e.g., upvar, uplevel) even
for "conventional" programmers.  Imagine trying to explain them to
*casual* programmers/users.

> I *have* used embedded Lua more recently.  Wireshark (a packet sniffer,
> formerly called Ethereal) understands lots and lots of packet formats,
> but if you've come up with your own weird thing, it just gives you a hex
> dump of the payload.  You can write your own packet parser in Lua and
> Wireshark will run it on the payload for you.  The basic application is
> to make the packet more human-readable.  You can also use it along with
> Wireshark's native ability to trigger on or sort by certain data fields
> to only log packets of interest.
>
> The thing I found odd about the choice of Lua for this application was
> that *it didn't have any bitwise operators*.  You couldn't do stuff like
> "if(packet[3] & 0x04) { /*...*/ }".  I did a lot of multiply and divide
> by 2, and tests like "if(packet[5] > 127)" to see if the high bit was
> set.  Also, IIRC, Lua's default numeric type was float.

I didn't like the fact that you could mix and match types without
anything alerting you to the fact that you're trying to do something
"wrong".  E.g., if (5 == "5")...

Likewise, I thought the ability to perform multiple assignments
(with a *single* assignment operator) would cause problems.  E.g.,
x, y = 1, 3 -- how would you "read that over the phone"?
"x and y get 1 and 3, respectively"?  "x gets 1, y gets 3" (then
how is that *read* different from x=1  y=3)

Likewise, the potential for confusion when arguments are omitted
from function invocations.  I.e., it doesn't do much to help
protect the programmer from "little (painful) mistakes":
    "Oh, crap!  Foo() takes *3* parameters!)

I've been surgically removing/rewriting features of Limbo (my
scripting choice) to try to ensure users can "get it right"
without scratching their heads to figure out what's wrong
with their code.  Too many multiple character operators, etc.

> This was a couple of years ago; maybe "full" Lua could do bitwise back
> then and Wireshark was only implementing a subset, or maybe Lua couldn't
> do it back then but can do it now.
>
> Wireshark also had the option of writing a packet dissector in C, and
> then compiling it into a library (.so or .dll depending on platform);
> this was said to give you faster decoding.  Most of the protocols I was
> looking at sent packets at no more than a few dozen Hz; I think one of
> them topped out around 1 KHz.  The Lua scripts didn't seem to have any
> problem doing "live" decodes on a new-in-2007 PC.
>
> I know I've seen the option to embed a Lua interpreter in something more
> "embedded", possibly U-Boot.  Lua's Wikipedia page lists a lot of places
> where it has been embedded.
>
> I will give my standard advice: ship a few example scripts with the
> product.  You can assume that people know how to use arithmetic
> operators and things like that, but the details of how to get data out
> of and back into your hardware will be unique to your implementation.
> I think Wireshark had a limited example in the manual, but since it's
> popular, I was able to find additional examples online.  Most
> applications won't be able to count on that, though.
>
> Matt Roberds
>

On 14/03/15 08:27, Tim Wescott wrote:
> I'd like to use something that's open-source and available.
> Something that uses modest resources is to be preferred
> I'm looking for opinions, so fire away!

I've been around scripting languages since I built my first one 35 years 
ago, and I don't think there's a really good solution for your 
application yet... based on my definition of "good". The problem is 
two-fold:

* Languages whose parser is very small tend to be hard to read & write 
(Lisp, Forth) and tend not to have any static type checking, and

* Scripting languages tend to be garbage-collected, which adds to the 
memory requirement and introduces unpredictable delays.

What I think would work best for embedded application :

* has a static data model (just global and local variables, but no 
dynamic allocation),

* has a highly humanised syntax that pre-compiles into a byte code 
format on a host computer, where it can be statically type-checked, but 
is still easy to execute on the target.

I've been thinking about (and even started) writing such a beast since, 
oh, about 1995 I think. I actually set out to write an event-driven 
(actor) language like this, for high-school robotics programs, back when 
my kids were starting high school. I really don't think it's a very 
difficult thing to do (just a weekend project for me), but I'm not aware 
that it's been done. I'd like to hear if anyone knows of an existing 
candidate that meets these design criteria.

Clifford Heath.