Reply by Stefan Reuther●February 2, 20182018-02-02
Am 02.02.2018 um 13:22 schrieb pozz:
> Il 01/02/2018 12:21, David Brown ha scritto:
>> First ask yourself, /do/ you need to search for this URI in a list? An
>> embedded web server will often have just one web page - it might not
>> even have any other files or URI's at all. Then you don't need to parse
>> the incoming URI in any way - you return the same "file" each time. For
>> the web server in an application I have, I have 4 files - index.html,
>> favicon.ico, 404.html, and a jquery.js file. The searching can be done
>> be a set of "if (strcmp....)" checks that are fixed in the code. (My
>> server is a little bit beyond that, but not much.)
>>
>> You are not trying to re-create Apache here - there is no need to make
>> things more complex than they have to be for the task in hand. I don't
>> know what features you need here, and whether such a method for
>> "searching" is enough - but it is possible.
>
> My static web files will be around 20. However I got your point:
> auto-generate a hardcoded file-list made by a sequence of !strcmp(). I
> think it would be ok.
You can even let your generator generate the strcmp sequence.
I would probably go for some sort of
struct {
uint32_t start;
uint32_t size;
char name[24];
};
and let my generator build an array of that. Last time I did that, I
stored the structs in the flash as well, to get independence between the
program and data images.
Stefan
Reply by David Brown●February 2, 20182018-02-02
On 02/02/18 13:22, pozz wrote:
> Il 01/02/2018 12:21, David Brown ha scritto:
>> First ask yourself, /do/ you need to search for this URI in a list? An
>> embedded web server will often have just one web page - it might not
>> even have any other files or URI's at all. Then you don't need to parse
>> the incoming URI in any way - you return the same "file" each time. For
>> the web server in an application I have, I have 4 files - index.html,
>> favicon.ico, 404.html, and a jquery.js file. The searching can be done
>> be a set of "if (strcmp....)" checks that are fixed in the code. (My
>> server is a little bit beyond that, but not much.)
>>
>> You are not trying to re-create Apache here - there is no need to make
>> things more complex than they have to be for the task in hand. I don't
>> know what features you need here, and whether such a method for
>> "searching" is enough - but it is possible.
>
> My static web files will be around 20. However I got your point:
> auto-generate a hardcoded file-list made by a sequence of !strcmp(). I
> think it would be ok.
>
That's it, yes. An alternative is to generate a hash from the strings
and use that in a switch statement.
This all depends somewhat on whether your C code in internal flash and
your "files" in external flash need to be independent or if they can be
tightly coupled. The idea is to avoid making something more complicated
than necessary, while still being flexible enough to handle changes or
new files.
Reply by pozz●February 2, 20182018-02-02
Il 01/02/2018 12:21, David Brown ha scritto:
> On 01/02/18 12:04, pozz wrote:
>> Il 01/02/2018 10:00, David Brown ha scritto:
>>> On 01/02/18 09:39, pozz wrote:
>>>> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for
>>>> httpd can't fit the internal Flash memory anymore. So I'm going to use
>>>> an external SPI Flash memory.
>>>>
>>>> I need some suggestions about a filesystem to use. I need:
>>>>
>>>> - a tool that creates a binary (or Intel HEX) image from a folder on my
>>>> Windows development machine (the image will be downloaded to Flash);
>>>> - file management functions (open, close, read);
>>>> - simple integration with lwip httpd;
>>>>
>>>> I don't need to change the filesystem, so a read-only filesystem is
>>>> enough.
>>>
>>> Do you really need open, close and read functions? All you actually
>>> need is to copy a block of SPI memory into ram. That might simplify
>>> things a bit.
>>
>> Yes, you're right. open/close here can be avoided. read is a must (your
>> copy from SPI to memory RAM is a *read*).
>>
>>
>>> Does your filesystem need to be independent of the main program? Your
>>> filesystem tool could read through the files in a directory, concatenate
>>> them together into a single binary file, and generate a C header with
>>> something like:
>>>
>>> enum {
>>> start_index_html = 0, length_index_html = 4040,
>>> start_favicon_ico = 4040, length_favicon_ico = 840,
>>> start_node_js = 4880, length_node_js = 10391,
>>> ...
>>> };
>>>
>>> If that is enough, then your "tool" could be done as a bash script if
>>> you wanted (I'd use Python myself).
>>
>> I can't understand. Suppose you need to open the URI "/index.html". How
>> to search for this filename in your enum list?
>
> First ask yourself, /do/ you need to search for this URI in a list? An
> embedded web server will often have just one web page - it might not
> even have any other files or URI's at all. Then you don't need to parse
> the incoming URI in any way - you return the same "file" each time. For
> the web server in an application I have, I have 4 files - index.html,
> favicon.ico, 404.html, and a jquery.js file. The searching can be done
> be a set of "if (strcmp....)" checks that are fixed in the code. (My
> server is a little bit beyond that, but not much.)
>
> You are not trying to re-create Apache here - there is no need to make
> things more complex than they have to be for the task in hand. I don't
> know what features you need here, and whether such a method for
> "searching" is enough - but it is possible.
My static web files will be around 20. However I got your point:
auto-generate a hardcoded file-list made by a sequence of !strcmp(). I
think it would be ok.
>>> It will be a little more work if you want to track filenames as well,
>>> rather than having them fixed in the code, but you could generate a
>>> table of indexes, lengths, and names. It is easier if that part can be
>>> linked directly in the C code, but it would be possible to have it at
>>> the start of the binary blob for the SPI.
>>
>> I know I can write something from zero, but I'm sure something already
>> written that can be reused.
>
> My script for reading a bunch of files and putting them in a static data
> structures linked directly with my C code (I guess my microcontroller
> has more flash than yours!) is about 130 lines of Python. Half of that
> is static text copied directly to the generated header and C file. I
> wrote that myself, along with the httpd daemon in C, because I found it
> easier to get the features I wanted than to use the demo server and code
> from the LWIP httpd server. The "filesystem" bit was easy. Perhaps
> writing it from zero is not as hard as you think it might be - while
> finding and adapting existing solutions may be more effort. (Of course
> I cannot give you an answer here, I am just giving ideas for consideration.)
Reply by Dave Nadler●February 1, 20182018-02-01
On Thursday, February 1, 2018 at 3:58:40 PM UTC-5, Jack wrote:
Chan's file system is great stuff - used it many times.
But Pozz needs WAY less than even Chan's petit version.
Pozz just needs a simple small set of read-only files...
Reply by Joe Chisolm●February 1, 20182018-02-01
On Thu, 01 Feb 2018 09:39:40 +0100, pozz wrote:
> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for
> httpd can't fit the internal Flash memory anymore. So I'm going to use
> an external SPI Flash memory.
>
> I need some suggestions about a filesystem to use. I need:
>
> - a tool that creates a binary (or Intel HEX) image from a folder on my
> Windows development machine (the image will be downloaded to Flash);
> - file management functions (open, close, read);
> - simple integration with lwip httpd;
>
> I don't need to change the filesystem, so a read-only filesystem is enough.
If the file open frequency is not that often:
reserve the first (or few) sectors on the SPI flash for some sort of file
name list - file name, start address in SPI flash, length
In your open code read through the file list, find the file and set an
internal file hanle to the address on the SPI flash. On the read side,
read from SPI, increment address, etc.
If opens happen frequenly, read the file name map from SPI on boot and save
in some type of memory table. You can scan that quick and then set the
file address in your file handle structure
For writing have some tool that creates an image and upload it. Or maybe
even upload a tar or zip image and have the web server take care of it.
If your file list is not that long just pick a sector to start the files,
unpack the files from the stream coming in to your web server, write to SPI
and keep track of starting address and length. At the end go write the
file list sector(s). You bulk earse the SPI flash at the beginning so
everything is ready.
Updating a single file is a lot tricker and if that is the case you really
need some type of flash file system.
--
Chisolm
Republic of Texas
Reply by Jack●February 1, 20182018-02-01
pozz <pozzugno@gmail.com> wrote:
> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for
> httpd can't fit the internal Flash memory anymore. So I'm going to use
> an external SPI Flash memory.
>
> I need some suggestions about a filesystem to use. I need:
>
> - a tool that creates a binary (or Intel HEX) image from a folder on my
> Windows development machine (the image will be downloaded to Flash);
> - file management functions (open, close, read);
> - simple integration with lwip httpd;
>
> I don't need to change the filesystem, so a read-only filesystem is enough.
> Il 01/02/2018 10:00, David Brown ha scritto:
>> On 01/02/18 09:39, pozz wrote:
>>> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for
>>> httpd can't fit the internal Flash memory anymore. So I'm going to use
>>> an external SPI Flash memory.
>>>
>>> I need some suggestions about a filesystem to use. I need:
>>>
>>> - a tool that creates a binary (or Intel HEX) image from a folder on my
>>> Windows development machine (the image will be downloaded to Flash);
>>> - file management functions (open, close, read);
>>> - simple integration with lwip httpd;
>>>
>>> I don't need to change the filesystem, so a read-only filesystem is
>>> enough.
>>
>> Do you really need open, close and read functions? All you actually
>> need is to copy a block of SPI memory into ram. That might simplify
>> things a bit.
>
> Yes, you're right. open/close here can be avoided. read is a must (your
> copy from SPI to memory RAM is a *read*).
>
>
>> Does your filesystem need to be independent of the main program? Your
>> filesystem tool could read through the files in a directory, concatenate
>> them together into a single binary file, and generate a C header with
>> something like:
>>
>> enum {
>> start_index_html = 0, length_index_html = 4040,
>> start_favicon_ico = 4040, length_favicon_ico = 840,
>> start_node_js = 4880, length_node_js = 10391,
>> ...
>> };
>>
>> If that is enough, then your "tool" could be done as a bash script if
>> you wanted (I'd use Python myself).
>
> I can't understand. Suppose you need to open the URI "/index.html". How
> to search for this filename in your enum list?
First ask yourself, /do/ you need to search for this URI in a list? An
embedded web server will often have just one web page - it might not
even have any other files or URI's at all. Then you don't need to parse
the incoming URI in any way - you return the same "file" each time. For
the web server in an application I have, I have 4 files - index.html,
favicon.ico, 404.html, and a jquery.js file. The searching can be done
be a set of "if (strcmp....)" checks that are fixed in the code. (My
server is a little bit beyond that, but not much.)
You are not trying to re-create Apache here - there is no need to make
things more complex than they have to be for the task in hand. I don't
know what features you need here, and whether such a method for
"searching" is enough - but it is possible.
>
>
>> It will be a little more work if you want to track filenames as well,
>> rather than having them fixed in the code, but you could generate a
>> table of indexes, lengths, and names. It is easier if that part can be
>> linked directly in the C code, but it would be possible to have it at
>> the start of the binary blob for the SPI.
>
> I know I can write something from zero, but I'm sure something already
> written that can be reused.
My script for reading a bunch of files and putting them in a static data
structures linked directly with my C code (I guess my microcontroller
has more flash than yours!) is about 130 lines of Python. Half of that
is static text copied directly to the generated header and C file. I
wrote that myself, along with the httpd daemon in C, because I found it
easier to get the features I wanted than to use the demo server and code
from the LWIP httpd server. The "filesystem" bit was easy. Perhaps
writing it from zero is not as hard as you think it might be - while
finding and adapting existing solutions may be more effort. (Of course
I cannot give you an answer here, I am just giving ideas for consideration.)
Reply by pozz●February 1, 20182018-02-01
Il 01/02/2018 10:00, David Brown ha scritto:
> On 01/02/18 09:39, pozz wrote:
>> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for
>> httpd can't fit the internal Flash memory anymore. So I'm going to use
>> an external SPI Flash memory.
>>
>> I need some suggestions about a filesystem to use. I need:
>>
>> - a tool that creates a binary (or Intel HEX) image from a folder on my
>> Windows development machine (the image will be downloaded to Flash);
>> - file management functions (open, close, read);
>> - simple integration with lwip httpd;
>>
>> I don't need to change the filesystem, so a read-only filesystem is enough.
>
> Do you really need open, close and read functions? All you actually
> need is to copy a block of SPI memory into ram. That might simplify
> things a bit.
Yes, you're right. open/close here can be avoided. read is a must (your
copy from SPI to memory RAM is a *read*).
> Does your filesystem need to be independent of the main program? Your
> filesystem tool could read through the files in a directory, concatenate
> them together into a single binary file, and generate a C header with
> something like:
>
> enum {
> start_index_html = 0, length_index_html = 4040,
> start_favicon_ico = 4040, length_favicon_ico = 840,
> start_node_js = 4880, length_node_js = 10391,
> ...
> };
>
> If that is enough, then your "tool" could be done as a bash script if
> you wanted (I'd use Python myself).
I can't understand. Suppose you need to open the URI "/index.html". How
to search for this filename in your enum list?
> It will be a little more work if you want to track filenames as well,
> rather than having them fixed in the code, but you could generate a
> table of indexes, lengths, and names. It is easier if that part can be
> linked directly in the C code, but it would be possible to have it at
> the start of the binary blob for the SPI.
I know I can write something from zero, but I'm sure something already
written that can be reused.
Reply by David Brown●February 1, 20182018-02-01
On 01/02/18 09:39, pozz wrote:
> I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for
> httpd can't fit the internal Flash memory anymore. So I'm going to use
> an external SPI Flash memory.
>
> I need some suggestions about a filesystem to use. I need:
>
> - a tool that creates a binary (or Intel HEX) image from a folder on my
> Windows development machine (the image will be downloaded to Flash);
> - file management functions (open, close, read);
> - simple integration with lwip httpd;
>
> I don't need to change the filesystem, so a read-only filesystem is enough.
Do you really need open, close and read functions? All you actually
need is to copy a block of SPI memory into ram. That might simplify
things a bit.
Does your filesystem need to be independent of the main program? Your
filesystem tool could read through the files in a directory, concatenate
them together into a single binary file, and generate a C header with
something like:
enum {
start_index_html = 0, length_index_html = 4040,
start_favicon_ico = 4040, length_favicon_ico = 840,
start_node_js = 4880, length_node_js = 10391,
...
};
If that is enough, then your "tool" could be done as a bash script if
you wanted (I'd use Python myself).
It will be a little more work if you want to track filenames as well,
rather than having them fixed in the code, but you could generate a
table of indexes, lengths, and names. It is easier if that part can be
linked directly in the C code, but it would be possible to have it at
the start of the binary blob for the SPI.
Reply by pozz●February 1, 20182018-02-01
I'm using lwip that runs httpd over LPC1769 MCU. Now the filesystem for
httpd can't fit the internal Flash memory anymore. So I'm going to use
an external SPI Flash memory.
I need some suggestions about a filesystem to use. I need:
- a tool that creates a binary (or Intel HEX) image from a folder on my
Windows development machine (the image will be downloaded to Flash);
- file management functions (open, close, read);
- simple integration with lwip httpd;
I don't need to change the filesystem, so a read-only filesystem is enough.