Forums

pci and caching

Started by John Larkin December 3, 2005
Consider a Pentium PC with main memory and a PCI bus. One can plug
memory-type devices into the PCI bus, things like video or ADC
buffers, CPCI cards, and including, I suppose, more program-space
memory.

A couple of questions:

Is there (I guess there must be) a mechanism for a Windows program to
directly map a chunk of PCI-bus memory into its virtual address space?
Anybody know how this works?

Does anybody know how the BIOS decides what should be cached? There's
nothing in a device's PCI config registers that says "don't cache me"
as far as I can tell. 

I know the guy who wrote the book "PCI Bus Demystified" so I asked
him; he hadn't a clue about any of this.

Thanks,

John



John Larkin wrote:
> Consider a Pentium PC with main memory and a PCI bus. One can plug > memory-type devices into the PCI bus, things like video or ADC > buffers, CPCI cards, and including, I suppose, more program-space > memory. > > A couple of questions: > > Is there (I guess there must be) a mechanism for a Windows program to > directly map a chunk of PCI-bus memory into its virtual address space? > Anybody know how this works? >
Don't know about windows, but on linux, you'd use mmap() to map a physical memory range such as a PCI VGA card to a process's virtual memory. Check out linux device drivers for more info: http://www.xml.com/ldd/chapter/book/ . Specifically, check out chapter 13. I guess on windows its the same. Look for documentation of mmap() or memmap() etc for windows.
> Does anybody know how the BIOS decides what should be cached? There's > nothing in a device's PCI config registers that says "don't cache me" > as far as I can tell. >
Don't know, but mmap() seems to handle this. CPUs that implement caching typically have ways to prevent certain memory I/O from being cached. Usually this can only be done in supervisory mode so you really need to ask your OS to set it up for you.
On Sat, 03 Dec 2005 09:31:04 -0800, John Larkin wrote:

> Consider a Pentium PC with main memory and a PCI bus. One can plug > memory-type devices into the PCI bus, things like video or ADC > buffers, CPCI cards, and including, I suppose, more program-space > memory. > > A couple of questions: > > Is there (I guess there must be) a mechanism for a Windows program to > directly map a chunk of PCI-bus memory into its virtual address space? > Anybody know how this works? >
Not really. Once upon a time I wrote some DOS programs that did this (using djgpp) but I've never done it in windows. You may have to write device driver code to do this sort of thing. Normal application code may not have enough privilege to perform this type of mapping.
> Does anybody know how the BIOS decides what should be cached? There's > nothing in a device's PCI config registers that says "don't cache me" > as far as I can tell. >
I'm not 100% sure what you mean by cached, in this instance. Do you mean cached by the CPU itself (L2 cache)? If so, I have no idea. I am pretty sure that when control initially passes to the BIOS the cache is disabled, though. At some point, the BIOS turns on the cache, but this may be very late in the boot process. But if you are talking about accesses being cached by the North Bridge, then I would guess that PCI accesses are never cached. I think the way this words is that memory access from the CPU goes over the host bus to the North Bridge, and the North Bridge routes the access to PCI space or SDRAM as appropriate. If it is a PCI access, the North Bridge would not ever cache, I assume.
> I know the guy who wrote the book "PCI Bus Demystified" so I asked > him; he hadn't a clue about any of this. > > Thanks, > > John
What are you trying to do? As far as I know (which may not be very far!), you don't need to worry about this issue unless you are doing something really obscure. I have written DOS code (again, with djgpp) which accessed PCI config space (and memory mapped areas) for reading and writing and there were no problems with caching. HTH! --Mac
On Sun, 04 Dec 2005 05:00:14 GMT, Mac <foo@bar.net> wrote:

> >What are you trying to do? As far as I know (which may not be very far!), >you don't need to worry about this issue unless you are doing something >really obscure.
I'm thinking about doing a CPCI board that would look like a smallish block of stuff in memory space (as opposed to i/o space) on the PCI bus. I was wondering if a Win application could get at it directly, without doing a driver call for every i/o access, and how, in general, a PC decides what's cachable and what's not. My board would deliver realtime data in the register block ('volatile' in C-speak, I think) so it must not be cached at any level. I'm guessing everything beyond the contiguous ram space is not cached, and/or maybe anything above the 2Gig line doesn't get cached. Funny how little seems to be known about this.
> >I have written DOS code (again, with djgpp) which accessed PCI config >space (and memory mapped areas) for reading and writing and there were no >problems with caching.
Yeah, we have a couple of PowerBasic programs that run under DOS or 9x, that search PCI config space for a device and then drag it down into a hole in real space, between 640K and 1M. Of course, first you have to locate an unused, uncached hole to plop it into, and that seems to be different from bios to bios. We've found systems that have unused, cached holes! John
>I'm thinking about doing a CPCI board that would look like a smallish >block of stuff in memory space (as opposed to i/o space) on the PCI >bus. I was wondering if a Win application could get at it directly, >without doing a driver call for every i/o access, and how, in general, >a PC decides what's cachable and what's not. My board would deliver >realtime data in the register block ('volatile' in C-speak, I think) >so it must not be cached at any level.
I'm rusty on this stuff, so be suspicious. I/O space is a kludge for ISA. Best to avoid it, but I don't think it's really any different. (other than not many address bits) I think there is a cachable bit in the config space options. If you have appropriate driver support, you can map some of an applications virtual address space to "memory" on your device, and it's easy to make that uncached. With appropriate driver support, I've written diagnostic/hacks that ran in user space. Interrupts are a bit tricky. I forget the details. I think there was a magic location to read that turned off the interrupt. We probably had some API to tell the driver the address of that location, it would read it and save the answer, and more API to get that data (probably that last one) and a count of how many interrupts had happened since the last time you asked.
>Yeah, we have a couple of PowerBasic programs that run under DOS or >9x, that search PCI config space for a device and then drag it down >into a hole in real space, between 640K and 1M. Of course, first you >have to locate an unused, uncached hole to plop it into, and that >seems to be different from bios to bios. We've found systems that have >unused, cached holes!
The deal with PCI is that the BIOS allocates PCI addreses. If a device wants X bits of address, they must be aligned on an X bit address boundry. Unless you are very lucky, that will result in holes. (Lucky means you filled in all the holes with chunks from other devices.) -- The suespammers.org mail server is located in California. So are all my other mailboxes. Please do not send unsolicited bulk e-mail or unsolicited commercial e-mail to my suespammers.org address or any of my other addresses. These are my opinions, not necessarily my employer's. I hate spam.
On Sat, 03 Dec 2005 09:31:04 -0800, John Larkin wrote:
 
> Does anybody know how the BIOS decides what should be cached? There's > nothing in a device's PCI config registers that says "don't cache me" > as far as I can tell.
Not PCI config registers. Memory configuration registers. MTRRs (Memory Type Range Registers) in the processor and chipset control such things.
> I know the guy who wrote the book "PCI Bus Demystified" so I asked him; > he hadn't a clue about any of this.
Memory on the PCI bus can no longer cached (deprecated in V2.2, IIRC). Cacheable memory on the PCI bus is a mess so most systems, even before it was yanked out of the spec, didn't support it. I *highly* recommend "PCI System Architecture" by Shanley and Anderson as a introduction/reference. MindShare has a very good series of books on busses and processors. They do an *excellent* series of courses too, if a tad expensive. Their books are available in dead tree form or as e-books: http://www.mindshare.com -- Keith
On Sat, 03 Dec 2005 22:08:35 -0800, John Larkin wrote:

> On Sun, 04 Dec 2005 05:00:14 GMT, Mac <foo@bar.net> wrote: > >> >>What are you trying to do? As far as I know (which may not be very far!), >>you don't need to worry about this issue unless you are doing something >>really obscure. > > I'm thinking about doing a CPCI board that would look like a smallish > block of stuff in memory space (as opposed to i/o space) on the PCI > bus. I was wondering if a Win application could get at it directly, > without doing a driver call for every i/o access, and how, in general, > a PC decides what's cachable and what's not. My board would deliver > realtime data in the register block ('volatile' in C-speak, I think) > so it must not be cached at any level. >
Hmm. It seems like other people must have done this (or something like it) before. I am guessing that this will work fine without any special precautions.
> I'm guessing everything beyond the contiguous ram space is not cached, > and/or maybe anything above the 2Gig line doesn't get cached. Funny how > little seems to be known about this. >
Well, there are people out there who know this stuff, but they might not be reading here. There is a lot of information about Intel Architecture that is hard to ferret out. Most of it doesn't seem to be in any kind of real specification, either. Anyway, I am quite sure that accesses to memory mapped areas on the PCI bus are totally distinct from true memory accesses in the sense that the North Bridge is well aware which area it is accessing. So I don't think you have to worry about cacheing there. As for the L2 cache, I'm not sure how that is managed. Maybe it only caches instructions. But it obviously doesn't interfere with normal device driver operation, so I don't think you have to worry about it, either. As you can tell, I'm just guessing, but it seems like there are a lot of things that wouldn't work right if these types of accesses were cached by hardware.
>> >>I have written DOS code (again, with djgpp) which accessed PCI config >>space (and memory mapped areas) for reading and writing and there were no >>problems with caching. > > Yeah, we have a couple of PowerBasic programs that run under DOS or > 9x, that search PCI config space for a device and then drag it down > into a hole in real space, between 640K and 1M. Of course, first you > have to locate an unused, uncached hole to plop it into, and that > seems to be different from bios to bios. We've found systems that have > unused, cached holes! > > John
Oh, wow. That sounds kind of hard. With djgpp and a protected mode stub, you can write real 32-bit programs for DOS. No worries about dragging stuff down below 1M. You don't get any kind of GUI, but for some tasks that is not a problem. The Intel Architecture and associated DOS baggage is unbelievable arcane. It sure would be nice to dump it all and start over. ;-) --Mac
On Sun, 04 Dec 2005 17:41:56 GMT, Mac <foo@bar.net> wrote:


>> Yeah, we have a couple of PowerBasic programs that run under DOS or >> 9x, that search PCI config space for a device and then drag it down >> into a hole in real space, between 640K and 1M. Of course, first you >> have to locate an unused, uncached hole to plop it into, and that >> seems to be different from bios to bios. We've found systems that have >> unused, cached holes! >> >> John > >Oh, wow. That sounds kind of hard. With djgpp and a protected mode stub, >you can write real 32-bit programs for DOS. No worries about dragging >stuff down below 1M. You don't get any kind of GUI, but for some tasks >that is not a problem.
It wasn't bad... just had to figure out a couple of bios calls. It's available for anybody who's interested. We also wrote a couple of cute utilities that scan the 0..1M address space; one shows the data contents, one graphs access time vs address. Between the two you can pretty much figure out where the BIOS has put things (like shadows of itself!) and what's cached. We like to use DOS for our dedicated VME/PCI test programs (rackmount PC or VME card embedded Pentium) because we can own the CPU and run pretty much realtime, even turning off or dancing around the 18 Hz clock tick in extreme cases.
> >The Intel Architecture and associated DOS baggage is unbelievable arcane. >It sure would be nice to dump it all and start over. ;-)
Well, I'm typing on, essentially, a heavily kluged 8008 CPU, which was a ghastly bad architecture to start on. How we wound up with Intel and Microsoft as the dominant computer architecture is one of the tragedies of our time. Think of how things would be if IBM had gone with the 68K and anybody but Bill. John
Keith wrote:

> Memory on the PCI bus can no longer cached (deprecated in V2.2, > IIRC). Cacheable memory on the PCI bus is a mess so most systems, > even before it was yanked out of the spec, didn't support it.
I'm not sure where you got this info, but it's news to me! :O To answer the OP: As for mapping PCI memory from windows, there's a toolkit called TVICPCI <http://entechtaiwan.net/dev/pci/index.shtm> which grants user space access to PCI resources, which of course includes memory. They have a demo version for evaluation. You won't be able to tell Windows to use this as generic memory space (ie. Windows won't be executing out of it), but you will be able to use it for your own data. There's a bit in the PCI BAR that specifies whether or not the region is pre-fetchable. If set, a PCI master will know that it is allowed to use MRL (memory read line) or MRM (memory read multiple) on that address space. That doesn't necessarily mean it will. Regards, Mark
John, the NT DDK provides an example program that does just this.
Please see this link from MS:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;Q189327

The down side is that you are opening up protected hardware resources
to user apps, which isn't 100% kosher.

I have seen this applied to Win2k and XP. Beats me what would be
required for 9x.