DMA issue while recording data on SATA HD

Hi there,
Before I post my problem, let me present the overall picture.
We have developed a realtime embedded data acquisition system based on
A/D, FPGA, SATA controller, SATA HDD, microcontroller and Ethernet.
The purpose of the system is to record seismic data from the sensors,
digitize them and write them on the SATA HDD. Later, the recorded data
can be retrieved through Ethernet.
We have used Spartan 3 FPGA, Silicon Image SATA controller (Sil3512)
and SATA HDD. There is no OS in our system, we have developed our own
simple FAT, the FPGA acts as the host device which talks to the SATA
controller through a dedicated PCI interface. There is no PCI slot on
our system, we have used opencores PCI bridge on the FPGA which talks
to the SATA controller through its PCI interface. There are two ping
pong buffers in the FPGA, each 512 Kbytes. The digitized data is
stored in these buffers in a ping pong fashion at 36 Mbytes/s. The
FPGA programs the SATA controller DMA. The data is read out from the
buffers at 66 MHz and transported on the PCI bus at 33 MHz towards
SATA controller when the SATA controller initiates the read DMA. The
system works perfect when the incoming data rate is 24 Mbytes/sec. But
as we increase the data rate to 36 Mbytes/s (which is what we
require), we see loss of data occurring. Here is the algorithm for
data recording:

1.	Continue filling buffer 1 and buffer 2 with digitized data in a
round robin fashion.
2.	As soon as buffer 1 is full, program the DMA for SATA controller so
that it comes and reads out the filled buffer while buffer 2 is being
filled.
3.	Wait for the DMA done from the SATA controller from the first DMA.
4.	On finding DMA done, wait for buffer2 to be filled.
5.	As soon as buffer 2 is filled, program the DMA for SATA controller
so that it comes and reads out the filled buffer 2 while buffer 1 is
being filled.
6.	Wait for the DMA done from the SATA controller from the second DMA.
7.	On finding DMA done, wait for buffer1 to be filled.
8.	Repeat steps 2-7 till the data acquisition is stop.


The data loss occurs due to delay in step 3 and step 6 where the FPGA
waits for the DMA done signal from the SATA controller. At times, the
SATA controller takes longer than anticipated and due to this, data
buffers overflow. 36Mbytes/s should not be an aggressive data rate for
the SATA controller + SATA HDD specially when the PCI link is
dedicated only between the host and the SATA controller.. We have
tried out different numbers with PCI bridge configuration and SATA
controller PCI configuration but things have not improved. We are
using UDMA 6 mode and 80 GB 7400 RPM SATA HD. Anyone out there to
guide us how to handle this problem? Let me know if more information
is required. Thanks in advance.

Best Regards

Op Tue, 27 May 2008 11:54:29 +0200 schreef maverick  
<sheikh.m.farhan@gmail.com>:
> [...]
>
> The data loss occurs due to delay in step 3 and step 6 where the FPGA
> waits for the DMA done signal from the SATA controller. At times, the
> SATA controller takes longer than anticipated and due to this, data
> buffers overflow. 36Mbytes/s should not be an aggressive data rate for
> the SATA controller + SATA HDD specially when the PCI link is
> dedicated only between the host and the SATA controller. [...]

Indeed also because most modern disks have an internal buffer sized in the  
order of megabytes.  Maybe you (or the DMA controller) are waiting for the  
physical disk write to complete, before continuing?



-- 
Gemaakt met Opera's revolutionaire e-mailprogramma:  
http://www.opera.com/mail/

On May 27, 4:38=A0pm, "Boudewijn Dijkstra" <boudew...@indes.com> wrote:
> Op Tue, 27 May 2008 11:54:29 +0200 schreef maverick =A0
> <sheikh.m.far...@gmail.com>:
>
> > [...]
>
> > The data loss occurs due to delay in step 3 and step 6 where the FPGA
> > waits for the DMA done signal from the SATA controller. At times, the
> > SATA controller takes longer than anticipated and due to this, data
> > buffers overflow. 36Mbytes/s should not be an aggressive data rate for
> > the SATA controller + SATA HDD specially when the PCI link is
> > dedicated only between the host and the SATA controller. [...]
>
> Indeed also because most modern disks have an internal buffer sized in the=
 =A0
> order of megabytes. =A0Maybe you (or the DMA controller) are waiting for t=
he =A0
> physical disk write to complete, before continuing?
>
> --
> Gemaakt met Opera's revolutionaire e-mailprogramma: =A0http://www.opera.co=
m/mail/

But these hard disks guarantee around 60-65 MBytes/s sustained
throughput. That should include all these timings !

maverick wrote:
> On May 27, 4:38=EF=BF=BDpm, "Boudewijn Dijkstra" <boudew...@indes.com> wro=
te:
> > .....
> >
> > Indeed also because most modern disks have an internal buffer sized in t=
he =EF=BF=BD
> > order of megabytes. =EF=BF=BDMaybe you (or the DMA controller) are waiti=
ng for the =EF=BF=BD
> > physical disk write to complete, before continuing?
> > ....
>
> But these hard disks guarantee around 60-65 MBytes/s sustained
> throughput. That should include all these timings !

Can you merge a number of successive writes which you now do as
separate ATA write commands in a single one? That is, the drive will
think you write say 20 buffers worth of length, you will only
reprogram
the DMAC on the fly. If the problem persists, the issue is likely
something other than what Boudewijn suggessts. If this solves the
problem, he is right (my guess is he is right, throughput and latency
can differ dramatically).

Didi

------------------------------------------------------
Dimiter Popoff               Transgalactic Instruments

http://www.tgi-sci.com
------------------------------------------------------
http://www.flickr.com/photos/didi_tgi/sets/72157600228621276/

Original message: http://groups.google.com/group/comp.arch.embedded/msg/0ce3=
32f221d72b7a?dmode=3Dsource

In article <bb237122-e433-487e-ba0b-b8b0db02d769@e39g2000hsf.googlegroups.com>, maverick <sheikh.m.farhan@gmail.com> wrote:
>The data loss occurs due to delay in step 3 and step 6 where the FPGA
>waits for the DMA done signal from the SATA controller. At times, the
>SATA controller takes longer than anticipated and due to this, data
>buffers overflow. 36Mbytes/s should not be an aggressive data rate for
>the SATA controller + SATA HDD specially when the PCI link is
>dedicated only between the host and the SATA controller.. We have
>tried out different numbers with PCI bridge configuration and SATA
>controller PCI configuration but things have not improved. We are
>using UDMA 6 mode and 80 GB 7400 RPM SATA HD. 

Are you writing to the inner or outer sectors on the HDD? This makes a large 
difference. For example the HDD that's in my PC right now will do 57 MB/s on 
the outer sectors but only 30 MB/s on the inner sectors. 

If you can temporarily remove the HDD from your device and plug it into a 
PC, try benchmarking it with a program like "HD Tune" to get an idea of its 
actual performance. 

If you need a sustained 36 MB/s on the entire surface of the disk, there are 
10000 rpm SATA drives. You might also consider going larger than 80GB even if 
you don't need that much room since (all things being equal) a 500GB 3.5" 
drive at 7200RPM will usually be faster than an 80GB 3.5" drive at 7200RPM.

If that's still not good enough, just use two hard drives (if you have enough 
room in your enclosure) and alternate the writes. 
--Tom.

On May 27, 11:54=A0am, maverick <sheikh.m.far...@gmail.com> wrote:
> Hi there,
> Before I post my problem, let me present the overall picture.
> We have developed a realtime embedded data acquisition system based on
> A/D, FPGA, SATA controller, SATA HDD, microcontroller and Ethernet.
> The purpose of the system is to record seismic data from the sensors,
> digitize them and write them on the SATA HDD. Later, the recorded data
> can be retrieved through Ethernet.
> We have used Spartan 3 FPGA, Silicon Image SATA controller (Sil3512)
> and SATA HDD. There is no OS in our system, we have developed our own
> simple FAT, the FPGA acts as the host device which talks to the SATA
> controller through a dedicated PCI interface. There is no PCI slot on
> our system, we have used opencores PCI bridge on the FPGA which talks
> to the SATA controller through its PCI interface. There are two ping
> pong buffers in the FPGA, each 512 Kbytes. The digitized data is
> stored in these buffers in a ping pong fashion at 36 Mbytes/s. The
> FPGA programs the SATA controller DMA. The data is read out from the
> buffers at 66 MHz and transported on the PCI bus at 33 MHz towards
> SATA controller when the SATA controller initiates the read DMA. The
> system works perfect when the incoming data rate is 24 Mbytes/sec. But
> as we increase the data rate to 36 Mbytes/s (which is what we
> require), we see loss of data occurring. Here is the algorithm for
> data recording:
>
> 1. =A0 =A0 =A0Continue filling buffer 1 and buffer 2 with digitized data i=
n a
> round robin fashion.
> 2. =A0 =A0 =A0As soon as buffer 1 is full, program the DMA for SATA contro=
ller so
> that it comes and reads out the filled buffer while buffer 2 is being
> filled.
> 3. =A0 =A0 =A0Wait for the DMA done from the SATA controller from the firs=
t DMA.
> 4. =A0 =A0 =A0On finding DMA done, wait for buffer2 to be filled.
> 5. =A0 =A0 =A0As soon as buffer 2 is filled, program the DMA for SATA cont=
roller
> so that it comes and reads out the filled buffer 2 while buffer 1 is
> being filled.
> 6. =A0 =A0 =A0Wait for the DMA done from the SATA controller from the seco=
nd DMA.
> 7. =A0 =A0 =A0On finding DMA done, wait for buffer1 to be filled.
> 8. =A0 =A0 =A0Repeat steps 2-7 till the data acquisition is stop.
>
> The data loss occurs due to delay in step 3 and step 6 where the FPGA
> waits for the DMA done signal from the SATA controller. At times, the
> SATA controller takes longer than anticipated and due to this, data
> buffers overflow. 36Mbytes/s should not be an aggressive data rate for
> the SATA controller + SATA HDD specially when the PCI link is
> dedicated only between the host and the SATA controller.. We have
> tried out different numbers with PCI bridge configuration and SATA
> controller PCI configuration but things have not improved. We are
> using UDMA 6 mode and 80 GB 7400 RPM SATA HD. Anyone out there to
> guide us how to handle this problem? Let me know if more information
> is required. Thanks in advance.
>
> Best Regards

As I seen the problem occurs with increased data rate then it might be
also problem of the internal SATA controller design. I know from past
it might be a problem of internal architecture. Some SoC vendors are
pushing to save a cost for die. Usual way to write/read data to/from
SATA controller (DMA<->SATA transfer) is through internal FIFO in SATA
controller. DMA writes/reads to/from FIFO of the SATA controller. If
FIFO is too small (and I already saw such designs) then it easily
causes stop of the data transfer while writing because FIFO gets
easily empty if DMA is not capable to refil FIFO fast enough.I saw
e.g. design of FIFO from CEVA in CPU from Freescale with only 64bytes
- compare with size of one physical sector of 512 bytes. So it's a
problem of the arbitration between CPU and DMA while accessing SDR/
DDR. Size of FIFO just boosts problem. Every such stopped transfer (I
think mechanism called HOLD, HLDA) requires retransmittion for few
bytes. This definitely results in significant performance penalty. The
same may happen while reading data, just a problem of full FIFO. Also
make sure that filesystem adds some overhead, specially journaling
filesystem, but basically any filesystem, because data is not
sequential. More security/safety results in more overhead. Last but
not least: do not forget about fragmentation. Fragmented HDD even with
very simple filesystem gives you another significant penalty. You
should do regular defragmentation. To do perfect measurement you would
need SATA analyzer.

On May 27, 11:54=A0am, maverick <sheikh.m.far...@gmail.com> wrote:
> Hi there,
> Before I post my problem, let me present the overall picture.
> We have developed a realtime embedded data acquisition system based on
> A/D, FPGA, SATA controller, SATA HDD, microcontroller and Ethernet.
> The purpose of the system is to record seismic data from the sensors,
> digitize them and write them on the SATA HDD. Later, the recorded data
> can be retrieved through Ethernet.
> We have used Spartan 3 FPGA, Silicon Image SATA controller (Sil3512)
> and SATA HDD. There is no OS in our system, we have developed our own
> simple FAT, the FPGA acts as the host device which talks to the SATA
> controller through a dedicated PCI interface. There is no PCI slot on
> our system, we have used opencores PCI bridge on the FPGA which talks
> to the SATA controller through its PCI interface. There are two ping
> pong buffers in the FPGA, each 512 Kbytes. The digitized data is
> stored in these buffers in a ping pong fashion at 36 Mbytes/s. The
> FPGA programs the SATA controller DMA. The data is read out from the
> buffers at 66 MHz and transported on the PCI bus at 33 MHz towards
> SATA controller when the SATA controller initiates the read DMA. The
> system works perfect when the incoming data rate is 24 Mbytes/sec. But
> as we increase the data rate to 36 Mbytes/s (which is what we
> require), we see loss of data occurring. Here is the algorithm for
> data recording:
>
> 1. =A0 =A0 =A0Continue filling buffer 1 and buffer 2 with digitized data i=
n a
> round robin fashion.
> 2. =A0 =A0 =A0As soon as buffer 1 is full, program the DMA for SATA contro=
ller so
> that it comes and reads out the filled buffer while buffer 2 is being
> filled.
> 3. =A0 =A0 =A0Wait for the DMA done from the SATA controller from the firs=
t DMA.
> 4. =A0 =A0 =A0On finding DMA done, wait for buffer2 to be filled.
> 5. =A0 =A0 =A0As soon as buffer 2 is filled, program the DMA for SATA cont=
roller
> so that it comes and reads out the filled buffer 2 while buffer 1 is
> being filled.
> 6. =A0 =A0 =A0Wait for the DMA done from the SATA controller from the seco=
nd DMA.
> 7. =A0 =A0 =A0On finding DMA done, wait for buffer1 to be filled.
> 8. =A0 =A0 =A0Repeat steps 2-7 till the data acquisition is stop.
>
> The data loss occurs due to delay in step 3 and step 6 where the FPGA
> waits for the DMA done signal from the SATA controller. At times, the
> SATA controller takes longer than anticipated and due to this, data
> buffers overflow. 36Mbytes/s should not be an aggressive data rate for
> the SATA controller + SATA HDD specially when the PCI link is
> dedicated only between the host and the SATA controller.. We have
> tried out different numbers with PCI bridge configuration and SATA
> controller PCI configuration but things have not improved. We are
> using UDMA 6 mode and 80 GB 7400 RPM SATA HD. Anyone out there to
> guide us how to handle this problem? Let me know if more information
> is required. Thanks in advance.
>
> Best Regards

Maverick, I forgot to write: In the case of free space in the cache of
the HDD (memory which physicaly resides on HDD, usualy 8 or 16MB) you
should forget about speed of 30-60MBytes/sec (sustained data rate),
but you have to take into account speed of SATA interface -
150/300MBytes/sec, depending which SATA you have. In that moment you
write into the cache at the full speed of SATA interface. Then wrong
setup of arbitration between CPU and DMA while accessing SDR/DDR
together with too small SATA controller FIFO may show up.