Linux and dmaengine - problem with checking the number of bytes transferred in a segment

Hi,

I'm writing a driver for the device, that transferrs data via AXI Stream interface in packets with variable length (however the maximum length of the packet is known).
The data should be transferred to the buffers allocated in the application and
accessed by the kernel via get_user_pages.
The application allocates a buffer for each packet and submits it to the driver via ioctl call. The driver prepares it for dma using the:
1) get_user_pages_fast
2) sg_alloc_table_from_user_pages
3) dma_map_sg

When the transfer is started, each buffer is
1) synced with dma_sync_sg_for_device
2) converted to dma_async_tx_descriptor with dmaengine_prep_slave_sg
3) configured with proper callback and parameters
4) submitted via descriptor's tx_submit method.

When all buffers are submitted, the transfer is started with dma_async_issue_pending

After each packet is transmitted and tlast is asserted, the callback is executed. And here is the problem:

There is no way to find how many bytes has been transferred.
Of course I can work it around by zeroing the whole buffer before the transfer and requesting, that the device sends the "end of packet" magic word at the end, but this is definitely a suboptimal solution.

OK. I can avoid zeroing the buffer if I require sending the header with the packet's sequential number at the begining, and magic EOP with the same sequential number at the end (So I'll avoid a situation, when the packet is interrupted in the middle, but EOP from the previous one is still in the buffer - of course I'm going to reuse the buffers). 

I'm absolutely surprised, that dmaengine does not offer such fundamental functionality as finding the length of the completed transfer.
The problem was noticed in 2013 for USB devices: http://www.spinics.net/lists/linux-usb/msg89942.html however it seems that the solution proposed at the end, based on using the dmaengine_tx_status does not work for completed transfers.

Have I overlooked something?

TIA & Regards,
Wojtek

PS. I have prepared a simple code, which demonstrates the problem for a single packet: https://github.com/wzab/Z-turn-examples/archive/v0.3.tar.gz ,
the "axi_dma_prj1" subdirectory.
PS2. The problem was also reported to the Xilinx forum: https://forums.xilinx.com/t5/Embedded-Linux/AXI-DMA-checking-the-number-of-really-transferred-bytes-for/m-p/715636/highlight/true#M16801
and to the stackoverflow: http://stackoverflow.com/questions/38982140/the-dmaengine-in-linux-how-to-check-the-number-of-actually-transferred-bytes-i

wzab01@gmail.com wrote:
> Hi,
>
> I'm writing a driver for the device, that transferrs data via AXI
> Stream interface in packets with variable length (however the maximum
> length of the packet is known). The data should be transferred to the
> buffers allocated in the application and accessed by the kernel via
> get_user_pages. The application allocates a buffer for each packet
> and submits it to the driver via ioctl call. The driver prepares it
> for dma using the: 1) get_user_pages_fast 2)
> sg_alloc_table_from_user_pages 3) dma_map_sg
>
> When the transfer is started, each buffer is 1) synced with
> dma_sync_sg_for_device 2) converted to dma_async_tx_descriptor with
> dmaengine_prep_slave_sg 3) configured with proper callback and
> parameters 4) submitted via descriptor's tx_submit method.
>
> When all buffers are submitted, the transfer is started with
> dma_async_issue_pending
>
> After each packet is transmitted and tlast is asserted, the callback
> is executed. And here is the problem:
>
> There is no way to find how many bytes has been transferred.

Well, you get a callback when it's "done". "Actually transferred" is
not a hard and fast concept anyway.

> Of course I can work it around by zeroing the whole buffer before the
> transfer and requesting, that the device sends the "end of packet"
> magic word at the end, but this is definitely a suboptimal solution.
>


I think you have to measure the impact of zeroing buffers on
performance  to actually say that. Actually measuring that is not
completely trivial.

FWIW, I zero buffers until I decide I can 1) afford not to and
2) measure a justifiable impact on performance.

Might be worth making "zero/don't zero" settable via ioctl() control
for the driver. That should not be too onerous.

> OK. I can avoid zeroing the buffer if I require sending the header
> with the packet's sequential number at the begining, and magic EOP
> with the same sequential number at the end (So I'll avoid a
> situation, when the packet is interrupted in the middle, but EOP from
> the previous one is still in the buffer - of course I'm going to
> reuse the buffers).
>
> I'm absolutely surprised, that dmaengine does not offer such
> fundamental functionality as finding the length of the completed
> transfer.

Well, DMA is like that. The information is not always even available. "I
shot an arrow in the air; it fell to earth I know not where..."

> The problem was noticed in 2013 for USB devices:
> http://www.spinics.net/lists/linux-usb/msg89942.html however it seems
> that the solution proposed at the end, based on using the
> dmaengine_tx_status does not work for completed transfers.
>
> Have I overlooked something?
>
> TIA & Regards, Wojtek
>
> PS. I have prepared a simple code, which demonstrates the problem for
> a single packet:
> https://github.com/wzab/Z-turn-examples/archive/v0.3.tar.gz , the
> "axi_dma_prj1" subdirectory. PS2. The problem was also reported to
> the Xilinx forum:
> https://forums.xilinx.com/t5/Embedded-Linux/AXI-DMA-checking-the-number-of-really-transferred-bytes-for/m-p/715636/highlight/true#M16801
>
>
and to the stackoverflow: 
http://stackoverflow.com/questions/38982140/the-dmaengine-in-linux-how-to-check-the-number-of-actually-transferred-bytes-i
>

-- 
Les Cargill