pci and caching

Consider a Pentium PC with main memory and a PCI bus. One can plug
memory-type devices into the PCI bus, things like video or ADC
buffers, CPCI cards, and including, I suppose, more program-space
memory.

A couple of questions:

Is there (I guess there must be) a mechanism for a Windows program to
directly map a chunk of PCI-bus memory into its virtual address space?
Anybody know how this works?

Does anybody know how the BIOS decides what should be cached? There's
nothing in a device's PCI config registers that says "don't cache me"
as far as I can tell. 

I know the guy who wrote the book "PCI Bus Demystified" so I asked
him; he hadn't a clue about any of this.

Thanks,

John

John Larkin wrote:
> Consider a Pentium PC with main memory and a PCI bus. One can plug
> memory-type devices into the PCI bus, things like video or ADC
> buffers, CPCI cards, and including, I suppose, more program-space
> memory.
>
> A couple of questions:
>
> Is there (I guess there must be) a mechanism for a Windows program to
> directly map a chunk of PCI-bus memory into its virtual address space?
> Anybody know how this works?
>

Don't know about windows, but on linux, you'd use mmap() to map a
physical memory range such as a PCI VGA card to a process's virtual
memory. Check out linux device drivers for more info:
http://www.xml.com/ldd/chapter/book/ . Specifically, check out chapter
13.

I guess on windows its the same. Look for documentation of mmap() or
memmap() etc for windows.


> Does anybody know how the BIOS decides what should be cached? There's
> nothing in a device's PCI config registers that says "don't cache me"
> as far as I can tell.
>

Don't know, but mmap() seems to handle this. CPUs that implement
caching typically have ways to prevent certain memory I/O from being
cached. Usually this can only be done in supervisory mode so you really
need to ask your OS to set it up for you.

On Sat, 03 Dec 2005 09:31:04 -0800, John Larkin wrote:

> Consider a Pentium PC with main memory and a PCI bus. One can plug
> memory-type devices into the PCI bus, things like video or ADC
> buffers, CPCI cards, and including, I suppose, more program-space
> memory.
> 
> A couple of questions:
> 
> Is there (I guess there must be) a mechanism for a Windows program to
> directly map a chunk of PCI-bus memory into its virtual address space?
> Anybody know how this works?
> 

Not really. Once upon a time I wrote some DOS programs that did this
(using djgpp) but I've never done it in windows. You may have to write
device driver code to do this sort of thing. Normal application code may
not have enough privilege to perform this type of mapping.

> Does anybody know how the BIOS decides what should be cached? There's
> nothing in a device's PCI config registers that says "don't cache me"
> as far as I can tell. 
> 

I'm not 100% sure what you mean by cached, in this instance. Do you mean
cached by the CPU itself (L2 cache)? If so, I have no idea. I am pretty
sure that when control initially passes to the BIOS the cache is disabled,
though. At some point, the BIOS turns on the cache, but this may be very
late in the boot process.

But if you are talking about accesses being cached by the North Bridge,
then I would guess that PCI accesses are never cached.

I think the way this words is that memory access from the CPU goes over
the host bus to the North Bridge, and the North Bridge routes the access
to PCI space or SDRAM as appropriate. If it is a PCI access, the North
Bridge would not ever cache, I assume.

> I know the guy who wrote the book "PCI Bus Demystified" so I asked
> him; he hadn't a clue about any of this.
> 
> Thanks,
> 
> John

What are you trying to do? As far as I know (which may not be very far!),
you don't need to worry about this issue unless you are doing something
really obscure.

I have written DOS code (again, with djgpp) which accessed PCI config
space (and memory mapped areas) for reading and writing and there were no
problems with caching.

HTH!

--Mac

On Sun, 04 Dec 2005 05:00:14 GMT, Mac <foo@bar.net> wrote:

>
>What are you trying to do? As far as I know (which may not be very far!),
>you don't need to worry about this issue unless you are doing something
>really obscure.

I'm thinking about doing a CPCI board that would look like a smallish
block of stuff in memory space (as opposed to i/o space) on the PCI
bus. I was wondering if a Win application could get at it directly,
without doing a driver call for every i/o access, and how, in general,
a PC decides what's cachable and what's not. My board would deliver
realtime data in the register block ('volatile' in C-speak, I think)
so it must not be cached at any level.

I'm guessing everything beyond the contiguous ram space is not cached,
and/or maybe anything above the 2Gig line doesn't get cached. Funny
how little seems to be known about this.

>
>I have written DOS code (again, with djgpp) which accessed PCI config
>space (and memory mapped areas) for reading and writing and there were no
>problems with caching.

Yeah, we have a couple of PowerBasic programs that run under DOS or
9x, that search PCI config space for a device and then drag it down
into a hole in real space, between 640K and 1M. Of course, first you
have to locate an unused, uncached hole to plop it into, and that
seems to be different from bios to bios. We've found systems that have
unused, cached holes!

John

>I'm thinking about doing a CPCI board that would look like a smallish
>block of stuff in memory space (as opposed to i/o space) on the PCI
>bus. I was wondering if a Win application could get at it directly,
>without doing a driver call for every i/o access, and how, in general,
>a PC decides what's cachable and what's not. My board would deliver
>realtime data in the register block ('volatile' in C-speak, I think)
>so it must not be cached at any level.

I'm rusty on this stuff, so be suspicious.

I/O space is a kludge for ISA.  Best to avoid it, but I don't
think it's really any different. (other than not many address bits)

I think there is a cachable bit in the config space options.

If you have appropriate driver support, you can map some of
an applications virtual address space to "memory" on your
device, and it's easy to make that uncached.

With appropriate driver support, I've written diagnostic/hacks
that ran in user space.  Interrupts are a bit tricky.  I forget
the details.  I think there was a magic location to read that
turned off the interrupt.  We probably had some API to tell the
driver the address of that location, it would read it and save
the answer, and more API to get that data (probably that last one)
and a count of how many interrupts had happened since the last
time you asked.



>Yeah, we have a couple of PowerBasic programs that run under DOS or
>9x, that search PCI config space for a device and then drag it down
>into a hole in real space, between 640K and 1M. Of course, first you
>have to locate an unused, uncached hole to plop it into, and that
>seems to be different from bios to bios. We've found systems that have
>unused, cached holes!

The deal with PCI is that the BIOS allocates PCI addreses.  If a
device wants X bits of address, they must be aligned on an X bit
address boundry.  Unless you are very lucky, that will result in
holes.  (Lucky means you filled in all the holes with chunks
from other devices.)

-- 
The suespammers.org mail server is located in California.  So are all my
other mailboxes.  Please do not send unsolicited bulk e-mail or unsolicited
commercial e-mail to my suespammers.org address or any of my other addresses.
These are my opinions, not necessarily my employer's.  I hate spam.

On Sat, 03 Dec 2005 09:31:04 -0800, John Larkin wrote:
 
> Does anybody know how the BIOS decides what should be cached? There's
> nothing in a device's PCI config registers that says "don't cache me"
> as far as I can tell. 

Not PCI config registers.  Memory configuration registers. MTRRs (Memory
Type Range Registers) in the processor and chipset control such things.

> I know the guy who wrote the book "PCI Bus Demystified" so I asked him;
> he hadn't a clue about any of this.

Memory on the PCI bus can no longer cached (deprecated in V2.2, IIRC).
Cacheable memory on the PCI bus is a mess so most systems, even before it
was yanked out of the spec, didn't support it.

I *highly* recommend "PCI System Architecture" by Shanley and Anderson as
a introduction/reference.  MindShare has a very good series of books on
busses and processors.  They do an *excellent* series of courses too, if a
tad expensive. Their books are available in dead tree form or as e-books:
http://www.mindshare.com  

-- 
  Keith

On Sat, 03 Dec 2005 22:08:35 -0800, John Larkin wrote:

> On Sun, 04 Dec 2005 05:00:14 GMT, Mac <foo@bar.net> wrote:
> 
>>
>>What are you trying to do? As far as I know (which may not be very far!),
>>you don't need to worry about this issue unless you are doing something
>>really obscure.
> 
> I'm thinking about doing a CPCI board that would look like a smallish
> block of stuff in memory space (as opposed to i/o space) on the PCI
> bus. I was wondering if a Win application could get at it directly,
> without doing a driver call for every i/o access, and how, in general,
> a PC decides what's cachable and what's not. My board would deliver
> realtime data in the register block ('volatile' in C-speak, I think)
> so it must not be cached at any level.
> 

Hmm. It seems like other people must have done this (or something
like it) before. I am guessing that this will work fine without any
special precautions.

> I'm guessing everything beyond the contiguous ram space is not cached,
> and/or maybe anything above the 2Gig line doesn't get cached. Funny how
> little seems to be known about this.
> 

Well, there are people out there who know this stuff, but they might not
be reading here. There is a lot of information about Intel Architecture
that is hard to ferret out. Most of it doesn't seem to be in any kind of
real specification, either.

Anyway, I am quite sure that accesses to memory mapped areas on the PCI
bus are totally distinct from true memory accesses in the sense that the
North Bridge is well aware which area it is accessing. So I don't think
you have to worry about cacheing there.

As for the L2 cache, I'm not sure how that is managed. Maybe it only
caches instructions. But it obviously doesn't interfere with normal device
driver operation, so I don't think you have to worry about it, either.
As you can tell, I'm just guessing, but it seems like there are a lot of
things that wouldn't work right if these types of accesses were cached by
hardware.

>>
>>I have written DOS code (again, with djgpp) which accessed PCI config
>>space (and memory mapped areas) for reading and writing and there were no
>>problems with caching.
> 
> Yeah, we have a couple of PowerBasic programs that run under DOS or
> 9x, that search PCI config space for a device and then drag it down
> into a hole in real space, between 640K and 1M. Of course, first you
> have to locate an unused, uncached hole to plop it into, and that
> seems to be different from bios to bios. We've found systems that have
> unused, cached holes!
> 
> John

Oh, wow. That sounds kind of hard. With djgpp and a protected mode stub,
you can write real 32-bit programs for DOS. No worries about dragging
stuff down below 1M. You don't get any kind of GUI, but for some tasks
that is not a problem.

The Intel Architecture and associated DOS baggage is unbelievable arcane.
It sure would be nice to dump it all and start over.  ;-)

--Mac

On Sun, 04 Dec 2005 17:41:56 GMT, Mac <foo@bar.net> wrote:


>> Yeah, we have a couple of PowerBasic programs that run under DOS or
>> 9x, that search PCI config space for a device and then drag it down
>> into a hole in real space, between 640K and 1M. Of course, first you
>> have to locate an unused, uncached hole to plop it into, and that
>> seems to be different from bios to bios. We've found systems that have
>> unused, cached holes!
>> 
>> John
>
>Oh, wow. That sounds kind of hard. With djgpp and a protected mode stub,
>you can write real 32-bit programs for DOS. No worries about dragging
>stuff down below 1M. You don't get any kind of GUI, but for some tasks
>that is not a problem.

It wasn't bad... just had to figure out a couple of bios calls. It's
available for anybody who's interested. We also wrote a couple of cute
utilities that scan the 0..1M address space; one shows the data
contents, one graphs access time vs address. Between the two you can
pretty much figure out where the BIOS has put things (like shadows of
itself!) and what's cached.

We like to use DOS for our dedicated VME/PCI test programs (rackmount
PC or VME card embedded Pentium) because we can own the CPU and run
pretty much realtime, even turning off or dancing around the 18 Hz
clock tick in extreme cases.

>
>The Intel Architecture and associated DOS baggage is unbelievable arcane.
>It sure would be nice to dump it all and start over.  ;-)

Well, I'm typing on, essentially, a heavily kluged 8008 CPU, which was
a ghastly bad architecture to start on. How we wound up with Intel and
Microsoft as the dominant computer architecture is one of the
tragedies of our time. Think of how things would be if IBM had gone
with the 68K and anybody but Bill.

John

Keith wrote:

> Memory on the PCI bus can no longer cached (deprecated in V2.2,
> IIRC). Cacheable memory on the PCI bus is a mess so most systems,
> even before it was yanked out of the spec, didn't support it.

I'm not sure where you got this info, but it's news to me! :O

To answer the OP:

As for mapping PCI memory from windows, there's a toolkit called TVICPCI 
<http://entechtaiwan.net/dev/pci/index.shtm> which grants user space 
access to PCI resources, which of course includes memory. They have a 
demo version for evaluation. You won't be able to tell Windows to use 
this as generic memory space (ie. Windows won't be executing out of it), 
but you will be able to use it for your own data.

There's a bit in the PCI BAR that specifies whether or not the region is 
pre-fetchable. If set, a PCI master will know that it is allowed to use 
MRL (memory read line) or MRM (memory read multiple) on that address 
space. That doesn't necessarily mean it will.

Regards,
Mark

John, the NT DDK provides an example program that does just this.
Please see this link from MS:

http://support.microsoft.com/default.aspx?scid=kb;EN-US;Q189327

The down side is that you are opening up protected hardware resources
to user apps, which isn't 100% kosher.

I have seen this applied to Win2k and XP. Beats me what would be
required for 9x.