Hi lpc2000 group!

For a data logger I want to store incoming data on a SD-flash memory
card as quickly as possible. I am using LPC2103 runnung at 4x16MHz, and
I have connected a 1GByte SD-Flash to the SSP (SPI1) port; no FAT-
filesystem (for speed reasons), just pure "data dump" onto the flash.

I have never used SPI nor SD Flash cards before, so this might be
impossible or stupid, but I have no clue how to improve performance. It
seems I have to write data in blocks of 512 bytes (one sector);
transferring the block needs about 400us (which is slower than expected
but might be fine) - however, the SD card needs about 1000us (1ms) to
store the received 512 bytes before it accepts a new block. So in total
I need ~1.5ms to store a block of 512bytes, resulting in 340KByte/sec...

This is significantly slower than my PC can store data on the same card
(I measured about 3.5MBytes/sec); so I am wondering if I make some
stupid mistake... Can I already send data to the card while it is
storing the previous chunk? Can I send multiple blocks and it will only
store after receiving n blocks? Can I increase the block size while it
will still stick to 1ms to store larger blocks?

Just to be very clear: I am not using FAT; I am following the "direct"
SPI protocol to talk to SD cards.
Any suggestions?

Thank a lot in advance!!

Jorg
PS: Here is a dump of my card's CSD (CardSpecificData) register, maybe
it helps.
mmc_csd_structure: 0
mmc_prot: 0
mmc_taac: 2D (5 5)
mmc_nsac: 0 (0)
mmc_tran_speed: 32 (6 2) (50)
mmc_ccc: 1B5
mmc_read_bl_len: 9 (512)
mmc_read_bl_partial: 1
mmc_write_blk_misalign: 0
mmc_read_blk_misalign: 0
mmc_dsr_imp: 0
mmc_c_size: 3929
mmc_vdd_r_curr_min: 7
mmc_vdd_r_curr_max: 6
mmc_vdd_w_curr_min: 7
mmc_vdd_w_curr_max: 6
mmc_c_size_mult: 7 (128)
mmc_erase_sector_size: 31 (32)
mmc_erase_grp_size: 28 (29)
mmc_wp_grp_size: 0 (1)
mmc_wp_grp_enable: 0
mmc_default_ecc: 0
mmc_r2w_factor: 5 (32)
mmc_write_bl_len: 9 (512)
mmc_write_bl_partial: 0
mmc_copy: 0
mmc_perm_write_protect: 0
mmc_tmp_write_protect: 0
mmc_eec: 0

mmc_size_bytes: 1030225920
mmc_size_sector: 512
mmc_size_sectors: 2012160
mmc_max_transfer_speed: 25000000bps
mmc_max_read_timeout: 2000us (200us typical)
mmc_max_write_timeout: 64000us (6400us typical) Start a New Thread

>
> I have never used SPI nor SD Flash cards before, so this might be
> impossible or stupid, but I have no clue how to improve performance. It
> seems I have to write data in blocks of 512 bytes (one sector);
> transferring the block needs about 400us (which is slower than expected
> but might be fine) - however, the SD card needs about 1000us (1ms) to
> store the received 512 bytes before it accepts a new block. So in total
> I need ~1.5ms to store a block of 512bytes, resulting in 340KByte/sec...
Looking at one of the SanDisk specs, a block write can vary between
0.5 mS (typical) and 240 mS (maximum). That's all over the map!
>
> This is significantly slower than my PC can store data on the same card
> (I measured about 3.5MBytes/sec); so I am wondering if I make some
> stupid mistake... Can I already send data to the card while it is
> storing the previous chunk? Can I send multiple blocks and it will only
> store after receiving n blocks? Can I increase the block size while it
> will still stick to 1ms to store larger blocks?

Read the specs for the MMC card. You probably can't afford the specs
for SD. These cards are SLOW; VERY, VERY SLOW. Specs available at
SanDisk.

I would be very worried about your PC measurement. Are you sure the
write was complete or just that it was queued for eventual output?

Maybe CF would be a better approach.

Richard Start a New Thread

joconradt wrote:
> Hi lpc2000 group!
>
> For a data logger I want to store incoming data on a SD-flash memory
> card as quickly as possible. I am using LPC2103 runnung at 4x16MHz, and
> I have connected a 1GByte SD-Flash to the SSP (SPI1) port; no FAT-
> filesystem (for speed reasons), just pure "data dump" onto the flash.
>
> I have never used SPI nor SD Flash cards before, so this might be
> impossible or stupid, but I have no clue how to improve performance. It
> seems I have to write data in blocks of 512 bytes (one sector);
> transferring the block needs about 400us (which is slower than expected
> but might be fine) - however, the SD card needs about 1000us (1ms) to
> store the received 512 bytes before it accepts a new block. So in total
> I need ~1.5ms to store a block of 512bytes, resulting in 340KByte/sec...
>
> This is significantly slower than my PC can store data on the same card
> (I measured about 3.5MBytes/sec); so I am wondering if I make some
> stupid mistake... Can I already send data to the card while it is
> storing the previous chunk? Can I send multiple blocks and it will only
> store after receiving n blocks? Can I increase the block size while it
> will still stick to 1ms to store larger blocks?
>
> Just to be very clear: I am not using FAT; I am following the "direct"
> SPI protocol to talk to SD cards.

There is indeed a multiple block write command (cmd 25) in SPI mode and
I'm using it successfully in a work-in-progress project.
It looks definitely faster, but I haven't measured the throughput vs.
single block write mode yet. There is a small delay after writing
each x blocks (x or 32 in my case with a specific card, can't
remember exactly), I think that indicates the internal flash
block size of the sd card, but this may vary from card to card.

I think a PC based card reader can be faster because it knows
the SD protocol, which I think is more efficient than the SPI
protocol (4 bits/clock instead of 1).

Actually I am using a FAT filesystem. I added two functions to the
elm-chan TinyFatfs. One to find the "free block" area, i.e. the
cluster after the last used cluster, until the last cluster.
I then record data in raw SPI multiple block write mode to the
free block area. Finally, another function chains the clusters
together and assigns the recorded data to an actual file. This
combines the speed of raw writing with the convenience of a file
system. File size has a granularity of 1 cluster though.

Just out of curiosity, what kind of data are you storing?
I'm recording an ultrasonic signal. One of the LPC2148 timers
generates a clock to trigger the built-in ADC which in turn
triggers the FIQ interrupt. The FIQ does very simple preprocessing
(removing DC offset) and stores the sampled data into a 16 kB circular
buffer. This allows sampling without missing data in case of minor write
latency. The main loops simply polls the buffer and writes a block
when at least 512 bytes are available in the buffer.

Kind regards,
Bertrik Start a New Thread

Bertrik,

Is this the bat detector? If so, I saw your page on that, and
that's very cool. The one thing I was curious about is that you call it
a 'bat detector', but it appears to be more of a 'bat recorder'. Does
the software process the signal and provide an indication that a bat
squeak was heard?

I have bats in my belfry... Well, OK, I have 2 bats that live in the
gable vent at the peak of my house. They're cute little brown bats, and
based on the amount of crap they leave in the driveway, they must be
killing scores of insects every night.

I was thinking your project, combined with a little bit of Ethernet
or RF to provide a remote indicator to my PC would be cool. Then the
next logical step is to place a camera that focuses on the vent, and can
image them coming and going. Do you think a 6000 watt-second flash
would stun them too badly? :)

--jc

Bertrik Sikken wrote:
>
> joconradt wrote:
> > Hi lpc2000 group!
> >
> > For a data logger I want to store incoming data on a SD-flash memory
> > card as quickly as possible. I am using LPC2103 runnung at 4x16MHz, and
> > I have connected a 1GByte SD-Flash to the SSP (SPI1) port; no FAT-
> > filesystem (for speed reasons), just pure "data dump" onto the flash.
> >
> > I have never used SPI nor SD Flash cards before, so this might be
> > impossible or stupid, but I have no clue how to improve performance. It
> > seems I have to write data in blocks of 512 bytes (one sector);
> > transferring the block needs about 400us (which is slower than expected
> > but might be fine) - however, the SD card needs about 1000us (1ms) to
> > store the received 512 bytes before it accepts a new block. So in total
> > I need ~1.5ms to store a block of 512bytes, resulting in
> 340KByte/sec. ..
> >
> > This is significantly slower than my PC can store data on the same card
> > (I measured about 3.5MBytes/sec) ; so I am wondering if I make some
> > stupid mistake... Can I already send data to the card while it is
> > storing the previous chunk? Can I send multiple blocks and it will only
> > store after receiving n blocks? Can I increase the block size while it
> > will still stick to 1ms to store larger blocks?
> >
> > Just to be very clear: I am not using FAT; I am following the "direct"
> > SPI protocol to talk to SD cards.
>
> There is indeed a multiple block write command (cmd 25) in SPI mode and
> I'm using it successfully in a work-in-progress project.
> It looks definitely faster, but I haven't measured the throughput vs.
> single block write mode yet. There is a small delay after writing
> each x blocks (x or 32 in my case with a specific card, can't
> remember exactly), I think that indicates the internal flash
> block size of the sd card, but this may vary from card to card.
>
> I think a PC based card reader can be faster because it knows
> the SD protocol, which I think is more efficient than the SPI
> protocol (4 bits/clock instead of 1).
>
> Actually I am using a FAT filesystem. I added two functions to the
> elm-chan TinyFatfs. One to find the "free block" area, i.e. the
> cluster after the last used cluster, until the last cluster.
> I then record data in raw SPI multiple block write mode to the
> free block area. Finally, another function chains the clusters
> together and assigns the recorded data to an actual file. This
> combines the speed of raw writing with the convenience of a file
> system. File size has a granularity of 1 cluster though.
>
> Just out of curiosity, what kind of data are you storing?
> I'm recording an ultrasonic signal. One of the LPC2148 timers
> generates a clock to trigger the built-in ADC which in turn
> triggers the FIQ interrupt. The FIQ does very simple preprocessing
> (removing DC offset) and stores the sampled data into a 16 kB circular
> buffer. This allows sampling without missing data in case of minor write
> latency. The main loops simply polls the buffer and writes a block
> when at least 512 bytes are available in the buffer.
>
> Kind regards,
> Bertrik Start a New Thread

Where their are two bats their probably are many.

Norm
----- Original Message -----
From: J.C. Wren
To: l...
Sent: Wednesday, September 19, 2007 9:55 PM
Subject: Re: [lpc2000] super fast data dump to SD memory card possible?
Bertrik,

Is this the bat detector? If so, I saw your page on that, and
that's very cool. The one thing I was curious about is that you call it
a 'bat detector', but it appears to be more of a 'bat recorder'. Does
the software process the signal and provide an indication that a bat
squeak was heard?

I have bats in my belfry... Well, OK, I have 2 bats that live in the
gable vent at the peak of my house. They're cute little brown bats, and
based on the amount of crap they leave in the driveway, they must be
killing scores of insects every night.

I was thinking your project, combined with a little bit of Ethernet
or RF to provide a remote indicator to my PC would be cool. Then the
next logical step is to place a camera that focuses on the vent, and can
image them coming and going. Do you think a 6000 watt-second flash
would stun them too badly? :)

--jc

Bertrik Sikken wrote:
>
> joconradt wrote:
> > Hi lpc2000 group!
> >
> > For a data logger I want to store incoming data on a SD-flash memory
> > card as quickly as possible. I am using LPC2103 runnung at 4x16MHz, and
> > I have connected a 1GByte SD-Flash to the SSP (SPI1) port; no FAT-
> > filesystem (for speed reasons), just pure "data dump" onto the flash.
> >
> > I have never used SPI nor SD Flash cards before, so this might be
> > impossible or stupid, but I have no clue how to improve performance. It
> > seems I have to write data in blocks of 512 bytes (one sector);
> > transferring the block needs about 400us (which is slower than expected
> > but might be fine) - however, the SD card needs about 1000us (1ms) to
> > store the received 512 bytes before it accepts a new block. So in total
> > I need ~1.5ms to store a block of 512bytes, resulting in
> 340KByte/sec. ..
> >
> > This is significantly slower than my PC can store data on the same card
> > (I measured about 3.5MBytes/sec) ; so I am wondering if I make some
> > stupid mistake... Can I already send data to the card while it is
> > storing the previous chunk? Can I send multiple blocks and it will only
> > store after receiving n blocks? Can I increase the block size while it
> > will still stick to 1ms to store larger blocks?
> >
> > Just to be very clear: I am not using FAT; I am following the "direct"
> > SPI protocol to talk to SD cards.
>
> There is indeed a multiple block write command (cmd 25) in SPI mode and
> I'm using it successfully in a work-in-progress project.
> It looks definitely faster, but I haven't measured the throughput vs.
> single block write mode yet. There is a small delay after writing
> each x blocks (x or 32 in my case with a specific card, can't
> remember exactly), I think that indicates the internal flash
> block size of the sd card, but this may vary from card to card.
>
> I think a PC based card reader can be faster because it knows
> the SD protocol, which I think is more efficient than the SPI
> protocol (4 bits/clock instead of 1).
>
> Actually I am using a FAT filesystem. I added two functions to the
> elm-chan TinyFatfs. One to find the "free block" area, i.e. the
> cluster after the last used cluster, until the last cluster.
> I then record data in raw SPI multiple block write mode to the
> free block area. Finally, another function chains the clusters
> together and assigns the recorded data to an actual file. This
> combines the speed of raw writing with the convenience of a file
> system. File size has a granularity of 1 cluster though.
>
> Just out of curiosity, what kind of data are you storing?
> I'm recording an ultrasonic signal. One of the LPC2148 timers
> generates a clock to trigger the built-in ADC which in turn
> triggers the FIQ interrupt. The FIQ does very simple preprocessing
> (removing DC offset) and stores the sampled data into a 16 kB circular
> buffer. This allows sampling without missing data in case of minor write
> latency. The main loops simply polls the buffer and writes a block
> when at least 512 bytes are available in the buffer.
>
> Kind regards,
> Bertrik
>
> Start a New Thread

J.C. Wren wrote:
> Bertrik,
>
> Is this the bat detector? If so, I saw your page on that, and
> that's very cool. The one thing I was curious about is that you call it
> a 'bat detector', but it appears to be more of a 'bat recorder'. Does
> the software process the signal and provide an indication that a bat
> squeak was heard?

Indeed, this is for my bat detector and you're right it's more of a
bat recorder right now. Initially I tried to make the software do
everything: record at high speed and provide a down-converted audible
signal as well, but (although possible) it is near the limit of the
capabilities of a LPC2148, at least without using hand optimised code.
The main goal was to do at least ultrasonic recording, because this
is something that is something that current hobby bat detectors lack.

> I have bats in my belfry... Well, OK, I have 2 bats that live in the
> gable vent at the peak of my house. They're cute little brown bats, and
> based on the amount of crap they leave in the driveway, they must be
> killing scores of insects every night.

Nice to hear. If the amount of droppings becomes noticeable, you may
have more than 2 bats. The best time to count them is just after sunset,
although this time of year they may have already left the house as
their young are now raised and they're preparing for hibernation.
(at least that's what the bats in the Netherlands do).

> I was thinking your project, combined with a little bit of Ethernet
> or RF to provide a remote indicator to my PC would be cool. Then the
> next logical step is to place a camera that focuses on the vent, and can
> image them coming and going. Do you think a 6000 watt-second flash
> would stun them too badly? :)

No idea how bright that is, but I think they would be bothered by it.
I would go with infrared instead. Nowadays you can get relatively cheap
wireless cameras (still need power though) that can see in infrared.
As far as I know, current detection systems rely on bats crossing
infrared beams, but it's not a perfect system. A bat colony in my town
is instrumented with RFID reader systems (and many bats in that colony
are tagged with tiny subdermal RFID tags). Even radar could be possible.

Kind regards,
Bertrik

>
> --jc
>
> Bertrik Sikken wrote:
>> joconradt wrote:
>>> Hi lpc2000 group!
>>>
>>> For a data logger I want to store incoming data on a SD-flash memory
>>> card as quickly as possible. I am using LPC2103 runnung at 4x16MHz, and
>>> I have connected a 1GByte SD-Flash to the SSP (SPI1) port; no FAT-
>>> filesystem (for speed reasons), just pure "data dump" onto the flash.
>>>
>>> I have never used SPI nor SD Flash cards before, so this might be
>>> impossible or stupid, but I have no clue how to improve performance. It
>>> seems I have to write data in blocks of 512 bytes (one sector);
>>> transferring the block needs about 400us (which is slower than expected
>>> but might be fine) - however, the SD card needs about 1000us (1ms) to
>>> store the received 512 bytes before it accepts a new block. So in total
>>> I need ~1.5ms to store a block of 512bytes, resulting in
>> 340KByte/sec. ..
>>> This is significantly slower than my PC can store data on the same card
>>> (I measured about 3.5MBytes/sec) ; so I am wondering if I make some
>>> stupid mistake... Can I already send data to the card while it is
>>> storing the previous chunk? Can I send multiple blocks and it will only
>>> store after receiving n blocks? Can I increase the block size while it
>>> will still stick to 1ms to store larger blocks?
>>>
>>> Just to be very clear: I am not using FAT; I am following the "direct"
>>> SPI protocol to talk to SD cards.
>> There is indeed a multiple block write command (cmd 25) in SPI mode and
>> I'm using it successfully in a work-in-progress project.
>> It looks definitely faster, but I haven't measured the throughput vs.
>> single block write mode yet. There is a small delay after writing
>> each x blocks (x or 32 in my case with a specific card, can't
>> remember exactly), I think that indicates the internal flash
>> block size of the sd card, but this may vary from card to card.
>>
>> I think a PC based card reader can be faster because it knows
>> the SD protocol, which I think is more efficient than the SPI
>> protocol (4 bits/clock instead of 1).
>>
>> Actually I am using a FAT filesystem. I added two functions to the
>> elm-chan TinyFatfs. One to find the "free block" area, i.e. the
>> cluster after the last used cluster, until the last cluster.
>> I then record data in raw SPI multiple block write mode to the
>> free block area. Finally, another function chains the clusters
>> together and assigns the recorded data to an actual file. This
>> combines the speed of raw writing with the convenience of a file
>> system. File size has a granularity of 1 cluster though.
>>
>> Just out of curiosity, what kind of data are you storing?
>> I'm recording an ultrasonic signal. One of the LPC2148 timers
>> generates a clock to trigger the built-in ADC which in turn
>> triggers the FIQ interrupt. The FIQ does very simple preprocessing
>> (removing DC offset) and stores the sampled data into a 16 kB circular
>> buffer. This allows sampling without missing data in case of minor write
>> latency. The main loops simply polls the buffer and writes a block
>> when at least 512 bytes are available in the buffer. Start a New Thread