Simple filesystem for embedded platform

Started by pozz February 1, 2018
I'm using lwip that runs httpd over LPC1769 MCU.  Now the filesystem for 
httpd can't fit the internal Flash memory anymore.  So I'm going to use 
an external SPI Flash memory.

I need some suggestions about a filesystem to use.  I need:

- a tool that creates a binary (or Intel HEX) image from a folder on my
   Windows development machine (the image will be downloaded to Flash);
- file management functions (open, close, read);
- simple integration with lwip httpd;

I don't need to change the filesystem, so a read-only filesystem is enough.
On 01/02/18 09:39, pozz wrote:
> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for > httpd can't fit the internal Flash memory anymore. So I'm going to use > an external SPI Flash memory. > > I need some suggestions about a filesystem to use. I need: > > - a tool that creates a binary (or Intel HEX) image from a folder on my > Windows development machine (the image will be downloaded to Flash); > - file management functions (open, close, read); > - simple integration with lwip httpd; > > I don't need to change the filesystem, so a read-only filesystem is enough.
Do you really need open, close and read functions? All you actually need is to copy a block of SPI memory into ram. That might simplify things a bit. Does your filesystem need to be independent of the main program? Your filesystem tool could read through the files in a directory, concatenate them together into a single binary file, and generate a C header with something like: enum { start_index_html = 0, length_index_html = 4040, start_favicon_ico = 4040, length_favicon_ico = 840, start_node_js = 4880, length_node_js = 10391, ... }; If that is enough, then your "tool" could be done as a bash script if you wanted (I'd use Python myself). It will be a little more work if you want to track filenames as well, rather than having them fixed in the code, but you could generate a table of indexes, lengths, and names. It is easier if that part can be linked directly in the C code, but it would be possible to have it at the start of the binary blob for the SPI.
Il 01/02/2018 10:00, David Brown ha scritto:
> On 01/02/18 09:39, pozz wrote: >> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for >> httpd can't fit the internal Flash memory anymore. So I'm going to use >> an external SPI Flash memory. >> >> I need some suggestions about a filesystem to use. I need: >> >> - a tool that creates a binary (or Intel HEX) image from a folder on my >> Windows development machine (the image will be downloaded to Flash); >> - file management functions (open, close, read); >> - simple integration with lwip httpd; >> >> I don't need to change the filesystem, so a read-only filesystem is enough. > > Do you really need open, close and read functions? All you actually > need is to copy a block of SPI memory into ram. That might simplify > things a bit.
Yes, you're right. open/close here can be avoided. read is a must (your copy from SPI to memory RAM is a *read*).
> Does your filesystem need to be independent of the main program? Your > filesystem tool could read through the files in a directory, concatenate > them together into a single binary file, and generate a C header with > something like: > > enum { > start_index_html = 0, length_index_html = 4040, > start_favicon_ico = 4040, length_favicon_ico = 840, > start_node_js = 4880, length_node_js = 10391, > ... > }; > > If that is enough, then your "tool" could be done as a bash script if > you wanted (I'd use Python myself).
I can't understand. Suppose you need to open the URI "/index.html". How to search for this filename in your enum list?
> It will be a little more work if you want to track filenames as well, > rather than having them fixed in the code, but you could generate a > table of indexes, lengths, and names. It is easier if that part can be > linked directly in the C code, but it would be possible to have it at > the start of the binary blob for the SPI.
I know I can write something from zero, but I'm sure something already written that can be reused.
On 01/02/18 12:04, pozz wrote:
> Il 01/02/2018 10:00, David Brown ha scritto: >> On 01/02/18 09:39, pozz wrote: >>> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for >>> httpd can't fit the internal Flash memory anymore. So I'm going to use >>> an external SPI Flash memory. >>> >>> I need some suggestions about a filesystem to use. I need: >>> >>> - a tool that creates a binary (or Intel HEX) image from a folder on my >>> Windows development machine (the image will be downloaded to Flash); >>> - file management functions (open, close, read); >>> - simple integration with lwip httpd; >>> >>> I don't need to change the filesystem, so a read-only filesystem is >>> enough. >> >> Do you really need open, close and read functions? All you actually >> need is to copy a block of SPI memory into ram. That might simplify >> things a bit. > > Yes, you're right. open/close here can be avoided. read is a must (your > copy from SPI to memory RAM is a *read*). > > >> Does your filesystem need to be independent of the main program? Your >> filesystem tool could read through the files in a directory, concatenate >> them together into a single binary file, and generate a C header with >> something like: >> >> enum { >> start_index_html = 0, length_index_html = 4040, >> start_favicon_ico = 4040, length_favicon_ico = 840, >> start_node_js = 4880, length_node_js = 10391, >> ... >> }; >> >> If that is enough, then your "tool" could be done as a bash script if >> you wanted (I'd use Python myself). > > I can't understand. Suppose you need to open the URI "/index.html". How > to search for this filename in your enum list?
First ask yourself, /do/ you need to search for this URI in a list? An embedded web server will often have just one web page - it might not even have any other files or URI's at all. Then you don't need to parse the incoming URI in any way - you return the same "file" each time. For the web server in an application I have, I have 4 files - index.html, favicon.ico, 404.html, and a jquery.js file. The searching can be done be a set of "if (strcmp....)" checks that are fixed in the code. (My server is a little bit beyond that, but not much.) You are not trying to re-create Apache here - there is no need to make things more complex than they have to be for the task in hand. I don't know what features you need here, and whether such a method for "searching" is enough - but it is possible.
> > >> It will be a little more work if you want to track filenames as well, >> rather than having them fixed in the code, but you could generate a >> table of indexes, lengths, and names. It is easier if that part can be >> linked directly in the C code, but it would be possible to have it at >> the start of the binary blob for the SPI. > > I know I can write something from zero, but I'm sure something already > written that can be reused.
My script for reading a bunch of files and putting them in a static data structures linked directly with my C code (I guess my microcontroller has more flash than yours!) is about 130 lines of Python. Half of that is static text copied directly to the generated header and C file. I wrote that myself, along with the httpd daemon in C, because I found it easier to get the features I wanted than to use the demo server and code from the LWIP httpd server. The "filesystem" bit was easy. Perhaps writing it from zero is not as hard as you think it might be - while finding and adapting existing solutions may be more effort. (Of course I cannot give you an answer here, I am just giving ideas for consideration.)
pozz <pozzugno@gmail.com> wrote:

> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for > httpd can't fit the internal Flash memory anymore. So I'm going to use > an external SPI Flash memory. > > I need some suggestions about a filesystem to use. I need: > > - a tool that creates a binary (or Intel HEX) image from a folder on my > Windows development machine (the image will be downloaded to Flash); > - file management functions (open, close, read); > - simple integration with lwip httpd; > > I don't need to change the filesystem, so a read-only filesystem is enough.
http://elm-chan.org/fsw/ff/00index_p.html Bye Jack -- Yoda of Borg am I! Assimilated shall you be! Futile resistance is, hmm?
On Thu, 01 Feb 2018 09:39:40 +0100, pozz wrote:

> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for > httpd can't fit the internal Flash memory anymore. So I'm going to use > an external SPI Flash memory. > > I need some suggestions about a filesystem to use. I need: > > - a tool that creates a binary (or Intel HEX) image from a folder on my > Windows development machine (the image will be downloaded to Flash); > - file management functions (open, close, read); > - simple integration with lwip httpd; > > I don't need to change the filesystem, so a read-only filesystem is enough.
If the file open frequency is not that often: reserve the first (or few) sectors on the SPI flash for some sort of file name list - file name, start address in SPI flash, length In your open code read through the file list, find the file and set an internal file hanle to the address on the SPI flash. On the read side, read from SPI, increment address, etc. If opens happen frequenly, read the file name map from SPI on boot and save in some type of memory table. You can scan that quick and then set the file address in your file handle structure For writing have some tool that creates an image and upload it. Or maybe even upload a tar or zip image and have the web server take care of it. If your file list is not that long just pick a sector to start the files, unpack the files from the stream coming in to your web server, write to SPI and keep track of starting address and length. At the end go write the file list sector(s). You bulk earse the SPI flash at the beginning so everything is ready. Updating a single file is a lot tricker and if that is the case you really need some type of flash file system. -- Chisolm Republic of Texas
On Thursday, February 1, 2018 at 3:58:40 PM UTC-5, Jack wrote:
> http://elm-chan.org/fsw/ff/00index_p.html
Chan's file system is great stuff - used it many times. But Pozz needs WAY less than even Chan's petit version. Pozz just needs a simple small set of read-only files...
Il 01/02/2018 12:21, David Brown ha scritto:
> On 01/02/18 12:04, pozz wrote: >> Il 01/02/2018 10:00, David Brown ha scritto: >>> On 01/02/18 09:39, pozz wrote: >>>> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for >>>> httpd can't fit the internal Flash memory anymore. So I'm going to use >>>> an external SPI Flash memory. >>>> >>>> I need some suggestions about a filesystem to use. I need: >>>> >>>> - a tool that creates a binary (or Intel HEX) image from a folder on my >>>> Windows development machine (the image will be downloaded to Flash); >>>> - file management functions (open, close, read); >>>> - simple integration with lwip httpd; >>>> >>>> I don't need to change the filesystem, so a read-only filesystem is >>>> enough. >>> >>> Do you really need open, close and read functions? All you actually >>> need is to copy a block of SPI memory into ram. That might simplify >>> things a bit. >> >> Yes, you're right. open/close here can be avoided. read is a must (your >> copy from SPI to memory RAM is a *read*). >> >> >>> Does your filesystem need to be independent of the main program? Your >>> filesystem tool could read through the files in a directory, concatenate >>> them together into a single binary file, and generate a C header with >>> something like: >>> >>> enum { >>> start_index_html = 0, length_index_html = 4040, >>> start_favicon_ico = 4040, length_favicon_ico = 840, >>> start_node_js = 4880, length_node_js = 10391, >>> ... >>> }; >>> >>> If that is enough, then your "tool" could be done as a bash script if >>> you wanted (I'd use Python myself). >> >> I can't understand. Suppose you need to open the URI "/index.html". How >> to search for this filename in your enum list? > > First ask yourself, /do/ you need to search for this URI in a list? An > embedded web server will often have just one web page - it might not > even have any other files or URI's at all. Then you don't need to parse > the incoming URI in any way - you return the same "file" each time. For > the web server in an application I have, I have 4 files - index.html, > favicon.ico, 404.html, and a jquery.js file. The searching can be done > be a set of "if (strcmp....)" checks that are fixed in the code. (My > server is a little bit beyond that, but not much.) > > You are not trying to re-create Apache here - there is no need to make > things more complex than they have to be for the task in hand. I don't > know what features you need here, and whether such a method for > "searching" is enough - but it is possible.
My static web files will be around 20. However I got your point: auto-generate a hardcoded file-list made by a sequence of !strcmp(). I think it would be ok.
>>> It will be a little more work if you want to track filenames as well, >>> rather than having them fixed in the code, but you could generate a >>> table of indexes, lengths, and names. It is easier if that part can be >>> linked directly in the C code, but it would be possible to have it at >>> the start of the binary blob for the SPI. >> >> I know I can write something from zero, but I'm sure something already >> written that can be reused. > > My script for reading a bunch of files and putting them in a static data > structures linked directly with my C code (I guess my microcontroller > has more flash than yours!) is about 130 lines of Python. Half of that > is static text copied directly to the generated header and C file. I > wrote that myself, along with the httpd daemon in C, because I found it > easier to get the features I wanted than to use the demo server and code > from the LWIP httpd server. The "filesystem" bit was easy. Perhaps > writing it from zero is not as hard as you think it might be - while > finding and adapting existing solutions may be more effort. (Of course > I cannot give you an answer here, I am just giving ideas for consideration.)
On 02/02/18 13:22, pozz wrote:
> Il 01/02/2018 12:21, David Brown ha scritto:
>> First ask yourself, /do/ you need to search for this URI in a list? An >> embedded web server will often have just one web page - it might not >> even have any other files or URI's at all. Then you don't need to parse >> the incoming URI in any way - you return the same "file" each time. For >> the web server in an application I have, I have 4 files - index.html, >> favicon.ico, 404.html, and a jquery.js file. The searching can be done >> be a set of "if (strcmp....)" checks that are fixed in the code. (My >> server is a little bit beyond that, but not much.) >> >> You are not trying to re-create Apache here - there is no need to make >> things more complex than they have to be for the task in hand. I don't >> know what features you need here, and whether such a method for >> "searching" is enough - but it is possible. > > My static web files will be around 20. However I got your point: > auto-generate a hardcoded file-list made by a sequence of !strcmp(). I > think it would be ok. >
That's it, yes. An alternative is to generate a hash from the strings and use that in a switch statement. This all depends somewhat on whether your C code in internal flash and your "files" in external flash need to be independent or if they can be tightly coupled. The idea is to avoid making something more complicated than necessary, while still being flexible enough to handle changes or new files.
Am 02.02.2018 um 13:22 schrieb pozz:
> Il 01/02/2018 12:21, David Brown ha scritto: >> First ask yourself, /do/ you need to search for this URI in a list? An >> embedded web server will often have just one web page - it might not >> even have any other files or URI's at all. Then you don't need to parse >> the incoming URI in any way - you return the same "file" each time. For >> the web server in an application I have, I have 4 files - index.html, >> favicon.ico, 404.html, and a jquery.js file. The searching can be done >> be a set of "if (strcmp....)" checks that are fixed in the code. (My >> server is a little bit beyond that, but not much.) >> >> You are not trying to re-create Apache here - there is no need to make >> things more complex than they have to be for the task in hand. I don't >> know what features you need here, and whether such a method for >> "searching" is enough - but it is possible. > > My static web files will be around 20. However I got your point: > auto-generate a hardcoded file-list made by a sequence of !strcmp(). I > think it would be ok.
You can even let your generator generate the strcmp sequence. I would probably go for some sort of struct { uint32_t start; uint32_t size; char name[24]; }; and let my generator build an array of that. Last time I did that, I stored the structs in the flash as well, to get independence between the program and data images. Stefan