When a Mongoose met a MicroPython

• Tutorials
• Communications
• C Programming

Mongooses and very small pythons can coexist, and even be friends.

This is more a framework than an actual application, with it you can integrate MicroPython and Cesanta's Mongoose. Mongoose runs when called by MicroPython and is able to run Python functions as callbacks for the events you decide in your event handler. The code is completely written in C, except for the example Python callback functions, of course. To try it, you can just build this example on a Linux machine, and, with just a small tweak, you can also run it on any ESP32 board.

How to build

Building for Linux does not have any special requirements beyond a C compiler and make, it will clone the MicroPython repository and build the code on-the-fly. If you then want to build for an ESP32, that requires Docker, as we've seen on a previous post: Bellegram.

Clone this Github repository and change to the "pythongoose" directory, then call make:

$ git clone https://github.com/scaprile/mongoose_apps.git
$ cd pythongoose
$ make

This also starts the server, so you should see no prompt back...

LINK build-standard/micropython
   text       data        bss        dec        hex    filename
 666745       2504       7304     676553      a52c9    build-standard/micropython
make[1]: Leaving directory '/home/scaprile/work/cesanta/mongoose/examples/pythongoose/micropython/ports/unix'
micropython/ports/unix/build-standard/micropython main.py

See it in action

Now connect with a browser to your machine at port 8000: http://localhost:8000/, you should see OK displayed.

$ curl -s http://localhost:8000/
<H1>OK</H1>

As this is just a framework, there are no actual API functions nor web pages, but this is extremely easy to add following Mongoose tutorials; though we might add some in the future, who knows, what do you think ?

Anyway, this framework defines one endpoint in order to test its functionality, so if we GET or POST at http://localhost:8000/api/epname we will see the Python script taking over and doing its stuff. It is easy then to replicate this and change epname to any apropriate endpoint name.

It is not good practice to leave sensible API calls open to anyone, so this framework incorporates simple Basic Authorization (you probably already went ahead and are questioning why I didn't tell you so before...), so as soon as we enter admin as our username and istrator as its password, another Python script will be called to validate user credentials. Why? Because some of us think that handling, storing and retrieving user databases is easier to do in Python than in C.

$ curl -s http://localhost:8000/api/epname
Auth required
$ curl -su admin:istrator http://localhost:8000/api/epname
Hi
$ curl -su admin:istrator http://localhost:8000/api/epname --data-binary "r u there ?"
Hi

The output on the terminal running the example will be:

micropython/ports/unix/build-standard/micropython main.py
f1 
f1 r u there ?

How does it work ?

The code is based on Mongoose HTTP server and MicroPython integration examples.

Mongoose is an event-driven framework that decouples us from the underlying networking stack and possible operating system, calling an event handler callback whenever there is an event of interest; for this to work, we need to periodically call its event manager. This is usually done in an infinite loop, though this time that loop will be Python code handling all needs of the Python application, and calling a function our module will export. There is also an initialization function that Python initialization code will call, passing the proper callback functions for API event and authorization. So, something like the following, we'll get to the parameters later:

this:

calls this:

When the HTTP server receives a request for any /api/ URI, our event handler will call Mongoose functions to get any authorization information present in the request, and pass it to the Python function, that will check credentials and return a boolean telling us to reject the request or proceed further

The Python function is rather simple, we just check and match user and password in a dictionary:

If authorization is denied, which will happen first time, the browser will get the proper status code and indications to open a request, so the user can fill proper credentials. From then, all requests will include those credentials.

After the requesting user has been validated, the endpoint is identified and its Python handler function is called:

For the sake of simplicity, this Python function does nothing but print its name and calling arguments, returning with a simple text... I'm sure you'll come up with smarter and definitely more useful things to do, the point is that Mongoose will take that response and properly send it to the browser as a response to its request, as seen above.

And that's it, from here to a full functioning RESTful application requires imagination and replicating this framework as needed.

You may be wondering how on earth does MicroPython know how to call Mongoose, or even worse, how does Mongoose know how to call a MicroPython function and how can it deal with Python objects... Well, if you care to dive a bit more, that is the task of the code that turns those weirdly named Mongoose functions into a Python module.

The art of MicroPython module crafting

Micropythons's C API is documented, though its best document is file obj.h itself and its siblings. So, what we need to do is to write C code that will interact with that API in order to obtain strings and integers from esoteric things like Python object, and also to be able to craft those ethereal entities from our mundane bytes and nul-terminated strings.

So, starting from the beginning, the init function call will extract the callback addresses and call our init function, yeah, that one we've seen above.

You are probably wondering what are those shouting names, what is a ROOT, what root, why, and etc. Well, first, we know we need to put those function addresses somewhere so we can call them back. The catch is, that at some point, a nice garbage collector will traverse memory searching for objects with no reference to them, hungry for memory, and eager to delete those that are not in use, that is, no one is referencing them. So, we need to keep a reference to them in the Python world (not in our current C world, invisible to Python). This may probably change in the future, which is probably good and bad, as this is an internal API and can be changed at any moment so keeping up with future releases will also be an art, but this time on our fault. The price to pay for such a deep integration.

Beyond that, and though it is not needed at this point, we introduced an additional level of complexity. At the tail of the function there is a macro that tells MicroPython builder parser how many arguments this function expects. There are simpler options for one, two, and three arguments, but beyond that you need to specify a variable number of arguments, with the possibility of having optionals (not present), so between a minimum and a maximum number of arguments. For expansion's sake, we chose that path.

Take the poll function, for example, that is an easy one as it only has one argument:

So, we've introduced MP-callable functions that turn arguments into C-world stuff and return MP objects. Now let's meet a C-callable function that will turn its arguments into MP-objects, call an MP function, and turn its returned object back into C-consumable data:

A string object is created from the passed data, a one argument Python function is called with that argument, and returns a string object. We take that, turn it into a C string, and pass the data back to the C world.

Finally, the authorization function is a more involved version of the former, as it passes two arguments and checks the returned object to see if it has to return success or failure.

As you've probably noticed, there is an implicit and tacit agreement on the number of objects passed and returned.

Let's go deeper

"A dream within a dream ?"

Mongoose uses a Berkeley Sockets interface to interact with the underlying networking stack. It can also provide its own TCP/IP stack, but we won't use it this time. Mongoose then interacts with Linux using Berkeley Sockets.

When we call mg_http_listen(), Mongoose parses the URL and obtains the address to listen on; in this case it will listen on all interfaces. It then opens a socket and stays listening there. We do that, at initialization time, when main.py calls our init function.

When we open a browser and point to this address and port, it will send a TCP connection request, the TCP/IP stack in our machine will accept it, then the browser will request the URI we asked it to request.

At this point, Mongoose will start to receive TCP data on a port it knows it is expecting HTTP protocol data, so it will buffer that data and process its headers, then wait to have a full request. Some of the deeper level internals are similar to those explained on a previous post: Bellegram. Beyond differences in OSs, let's note that here Mongoose will run every time MicroPython lets it run, when main.py calls our poll function, as seen above.

Once a full request has been received, the event manager calls our event handler callback function, passing an MG_EV_HTTP_MSG event as argument. In that function, we do all the processing described above, receive a response, and call mg_http_reply() to send it. This function call will place data in a Mongoose buffer, and the event manager will later call the socket send function as the TCP/IP stack signals there is room for more data to be sent. As the content we are sending is pretty small, this will happen right away.

Sign in