x86 real mode

Hi George,

On 10/17/2014 12:29 PM, George Neuner wrote:
> On Fri, 17 Oct 2014 07:23:40 -0700, Don Y <this@is.not.me.com> wrote:
>
>> What are the most conservative, *practical* expectations I can
>> make living in x86 real mode:  TEXT of 64K and DATA of (disjoint)
>> 64K?  BSS in it's own segment?  Or, shared with DATA?
>
> If you use the tiny (64K code+data) or small (64K code, 64K data)
> model, .bss and stack share space with your data.  If you choose a
> model with multiple data segments, .bss and stack can be separate (but
> tool dependent you may have to ask for it explicitly).

It seems the "safest" (most conservative) assumption is to figure
a single 64KB address space.  If, instead, the x86 ALWAYS prepared
a separate data space (etc), then I could assume a larger model.

Or, if the tiny model was IMPRACTICAL for any use (and was just
included as an homage to the 8085).

Or, if ints were always 32b, etc.

I.e., it seems safe to assume the tiny model was intended to be
*usable* and not just "an engineering/marketing exercise".

>> And, how seemlessly will the compiler let me *implicitly* move
>> those segments around as well as between them?  (e.g., practical
>> limitations on code/data sizes).  "PC" handled 640K so should
>> I expect that to be the size of my playground?
>
> The linker determines where your segments get placed (assuming the OS
> permits the requested placement).

OK.  Point of my question being the oddities of the x86 architecture
aren't (necessarily) exposed in the software.  I need to come up
with a solution that is readily portable to non-x86 architectures.

>> [Presumably, any "object" is constrained to fit within a single
>> segment]
>
> In the "huge" model arrays and structs can straddle segments. However,
> you should take care that no individual element of an array or struct
> lies across a segment boundary ... I have found that many compilers
> don't renormalize intermediate pointers when indexing from a huge
> pointer, so you can get into trouble when the offset is close to or
> exceeds 64K.  I've been bitten by that more times than I care to
> admit.
> [The solution is to deliberately construct a new huge pointer.  When
> you store the new pointer it will be normalized and thus safe to use.]

Understood.  I recall the 4KB object limitations that the Z180 presented.
This shouldn't be hard to accommodate (if considered AHEAD OF TIME and
not shoehorned into the design after-the-fact).

(actually, I think I can probably leverage some of my more ancient codebases
if I can locate them  :-/  I hadn't considered that possibility!)

Thanks!
--don

Tauno Voipio <tauno.voipio@notused.fi.invalid> wrote:

(snip)
> Actually, Intel did not intend to use the 8086 real mode
> as such, only as bootstrap mode for 80286. The success
> of 8088 and original PC came as a surprise, and it influenced
> the design of 80386 with the virtual 8086 mode.

Well, the 8086 was an upgrade from the 8080 or 8085. 
They knew by then that people were running out of address space.
In 1976, I used an 8080 system with 64K unmarked (factory samples,
or something like that) DRAM. 

Systems with (externally) bank switched RAM weren't all that
unusual, so maybe putting something like bank switch inside
the chip wouldn't have been so strange. 

But yes, I am pretty sure that Intel didn't expect the 80286
to mostly be used in systems only running real mode.
 
> There were no PC's at the design time of 8086 and 80286,
> and the aim was for well protected real-time multitasking
> applications. So it was natural that 80286 does not voluntarily
> return back to real mode from protected mode. This led to
> a massive kludge in PC/AT, where the keyboard controller
> can reset the main processor and the RTC CMOS RAM will
> contain a code explaining why the reset was this time.

I believe that kludge is still there today.  Also, the A20 
address line switch.

Also, the original IBM/PC used a 14.31818MHz crystal, convenient
for generating NTSC video with the CGA. That signal went on the
bus for that reason.  The processor then ran on that frequency
divided by three (that is the way the clock generator works).

Now, all ISA based PCs need the 14.31818 MHz crystal to supply that
line, just in case someone uses a CGA.  A few times I put a CGA
in my AT clone to generate test patterns for TV alignment.

-- glen

On 10/17/2014 2:02 PM, Grant Edwards wrote:
> On 2014-10-17, Don Y <this@is.not.me.com> wrote:
>> On 10/17/2014 8:36 AM, Arlet Ottens wrote:
>>> On 10/17/2014 05:28 PM, Don Y wrote:
>>>
>>>> Not a question of toolchain but, rather, what the *environment*
>>>> already "expects" (provides).  Hence the "*practical*" qualifier in
>>>> my original post.
>>>
>>> The environment is provided by the toolchain startup code, so it is a
>>> question of toolchain.
>>
>> The environment is defined by the rest of the code which I have to
>> co-operate.  It is poorly documented -- hence my query as to what I
>> could likely expect to encounter.
>
> So you wanted us to explain the memory model used by code that we
> haven't seen and wasn't even mentioned until now?

*I* haven't seen the code, either.  Please note the wording of my initial post:
     "What are the most conservative, *practical* expectations I can
     make living in x86 real mode:"

expectation:  a strong belief that something will happen or be the
               case in the future

Had I a *choice* in defining the implementation, I would have asked what
POSSIBILITIES were available to me in making that choice.  *THAT* is a
question to which I can easily get an answer without relying on folks'
"experiences".

George Neuner <gneuner2@comcast.net> wrote:

(snip)

> In the "huge" model arrays and structs can straddle segments. However,
> you should take care that no individual element of an array or struct
> lies across a segment boundary ... I have found that many compilers
> don't renormalize intermediate pointers when indexing from a huge
> pointer, so you can get into trouble when the offset is close to or
> exceeds 64K.  I've been bitten by that more times than I care to
> admit.

Seems like a compiler bug to me.  If you assign to a far pointer,
then it is your fault.

> [The solution is to deliberately construct a new huge pointer.  When
> you store the new pointer it will be normalized and thus safe to use.]
 
> Arithmetic on "far" pointers affects only the 16-bit offset and wraps
> at the segment boundary.  This means that two far pointers having
> different segments can't reasonably be compared.  If you are using
> multiple data segments and you need to compare pointers, you have to
> use normalized "huge" pointers.  Do be aware that huge pointer
> arithmetic can be quite a bit slower due to (re)normalization of the
> results.

You don't say what language you use, but C only allows comparison
of pointers to different objects as equal or not equal. With large
model, objects (arrays and structs) can't get larger than 64K.
(For C, they can't even equal 64K.)  A C compiler should generate
properly comparable (for equal or not equal) pointers in large
model.
 
> You really only need to choose the huge model if you expect a single
> array or struct to exceed 64K.  If you need huge pointers purely for
> comparison, they can be mixed with far pointers in any of the multiple
> data segment models.  Because far pointer arithmetic is faster, they
> should be preferred wherever you can live with their limitations.

In many cases, I have used 2D arrays, actual C arrays of pointers
to arrays.  As long as the array of pointers and pointed-to arrays
fit in 64K, you can get plenty big enough. 

I believe in the OS/2 1.x days, I had 16M on my system before
converting to OS/2 2.x and appropriate compilers.

I could generate arrays to at least 1 million elements of double,
so about 1000x1000, which easily fits. 

>>I imagine this will all be accomplished in the linkage editor
>>(not visible to the source code).
 
> Depends on the tool - some compilers use pragmas to control placement
> of objects into particular segments.  Placement of the segments
> themselves is controlled by the linker (and/or OS).
 
>>[What a screwed up architecture!]
 
> It isn't bad until you grow beyond 64K data and are forced into
> explicit use of far or huge pointers.  

As long as you stay within small or large, it isn't all that hard.
It is mixed models (using a few far pointers in small model) that
gets confusing. Go completely to large, never use the far keyword,
and everything works fine.

> There's a model that allows
>>64K code and <=64K data where the compiler transparently handles code
> pointers so you don't have to worry about them.

There is medium and compact, and I forget which is which.

I usually generate libraries for small and large, and only used those.
(When you install the MS C compiler, you select which models you
plan to use, and it generates the appropriate libraries.)

-- glen

On 10/17/2014 2:48 PM, glen herrmannsfeldt wrote:
> Tauno Voipio <tauno.voipio@notused.fi.invalid> wrote:
>
> (snip)
>> Actually, Intel did not intend to use the 8086 real mode
>> as such, only as bootstrap mode for 80286. The success
>> of 8088 and original PC came as a surprise, and it influenced
>> the design of 80386 with the virtual 8086 mode.
>
> Well, the 8086 was an upgrade from the 8080 or 8085.

The 8085 was essentially an 8080 with an internal clock driver
(and a couple of other little fixups -- RIM/SIM, INT5.5/6.5/7.5,
etc.)

> They knew by then that people were running out of address space.

Yes, an Our Hero, Bill figured 640K was more than anyone would
EVER need!  :-/

[How much nicer things would have been had the 68K won that design-in]

> In 1976, I used an 8080 system with 64K unmarked (factory samples,
> or something like that) DRAM.

I can recall getting 16Kx1's on an 8085 and DROOLING over "all the space"!

> Systems with (externally) bank switched RAM weren't all that
> unusual, so maybe putting something like bank switch inside
> the chip wouldn't have been so strange.

This is, in fact, what the Z180/64180 devices did for the Z80
(though in a really kludgy fashion).

TI (?) also had some tonka toy parts that did this (for those
of us who didn't want to live with 74189's serving that function).

> But yes, I am pretty sure that Intel didn't expect the 80286
> to mostly be used in systems only running real mode.

But, the 286 powered *up* in real mode.  Which, no doubt, is
why real mode persisted (persists?).

>> There were no PC's at the design time of 8086 and 80286,
>> and the aim was for well protected real-time multitasking
>> applications. So it was natural that 80286 does not voluntarily
>> return back to real mode from protected mode. This led to
>> a massive kludge in PC/AT, where the keyboard controller
>> can reset the main processor and the RTC CMOS RAM will
>> contain a code explaining why the reset was this time.
>
> I believe that kludge is still there today.  Also, the A20
> address line switch.
>
> Also, the original IBM/PC used a 14.31818MHz crystal, convenient
> for generating NTSC video with the CGA. That signal went on the
> bus for that reason.  The processor then ran on that frequency
> divided by three (that is the way the clock generator works).
>
> Now, all ISA based PCs need the 14.31818 MHz crystal to supply that
> line, just in case someone uses a CGA.  A few times I put a CGA
> in my AT clone to generate test patterns for TV alignment.

IIRC, the original "ISA" bus wasn't even formally characterized
until after the fact.

Am 17.10.2014 um 23:32 schrieb Don Y:

> It seems the "safest" (most conservative) assumption is to figure
> a single 64KB address space.

Huh?  How do you figure that the possibility that causes your own code 
to make _the_most_ assumptions about the behaviour of unknown, 
surrounding code, to be the _safest_?  When did it become safe to make 
unwarranted assumptions?

> If, instead, the x86 ALWAYS prepared
> a separate data space (etc), then I could assume a larger model.

Well, yes, it essentially always does just that.  All code addressing is 
relative to CS, all data addressing is relative to DS or SS, at least by 
default.  And since all those are allowed to be different, the only 
a-priori _safe_ assumption obviously is that they will be.

> Or, if the tiny model was IMPRACTICAL for any use

It's practical if the program is small enough to allow it, and the 
program loader being used supports it.

The usual one at the time did: tiny model is what MS-DOS *.com format 
programs use.  They're copied verbatim from disk to RAM (beginning 0x100 
bytes into a 64 KB page) , then all three primary segment registers are 
set to the start of that RAM region (i.e. CS=DS=SS=constant) and the 
loader just jumps to CS::0x0100.  So code and pre-initialized data are 
in one big block of RAM.  The remainder of those 64K are assumed to be 
used for .bss and stack.

The advantage was that the program loader doesn't have to make any code 
modifications to handle relocation of the code ... all addresses in the 
code are just offsets relative to the load base, which itself doesn't 
ever appear in the code, because it's kept in the segment registers.

The disadvantage was that the loader _didn't_ perform relocations, so 
there was no way to load bigger programs.  So along came the EXE program 
file format, which is a whole different kettle of fish.

> I.e., it seems safe to assume the tiny model was intended to be
> *usable* and not just "an engineering/marketing exercise".

It stopped being truly usable when programs outgrew 64 KB for code and 
all(!) data combined.

Don Y <this@is.not.me.com> wrote:

(snip, I wrote)

>> Well, the 8086 was an upgrade from the 8080 or 8085.
 
> The 8085 was essentially an 8080 with an internal clock driver
> (and a couple of other little fixups -- RIM/SIM, INT5.5/6.5/7.5,
> etc.)

And a 5V only power supply. As well as I know it, the 8085 and Z80
are but successors to the 8080, but different groups from intel.
(One group left, one stayed.)  Being 5V only made it a lot easier
to use in small systems.
 
>> They knew by then that people were running out of address space.
 
> Yes, an Our Hero, Bill figured 640K was more than anyone would
> EVER need!  :-/
 
> [How much nicer things would have been had the 68K won that design-in]

Might have been a 68008, though. They seem to like the 8 bit bus.
 
>> In 1976, I used an 8080 system with 64K unmarked (factory samples,
>> or something like that) DRAM.
 
> I can recall getting 16Kx1's on an 8085 and DROOLING 
> over "all the space"!
 
>> Systems with (externally) bank switched RAM weren't all that
>> unusual, so maybe putting something like bank switch inside
>> the chip wouldn't have been so strange.
 
> This is, in fact, what the Z180/64180 devices did for the Z80
> (though in a really kludgy fashion).
 
> TI (?) also had some tonka toy parts that did this (for those
> of us who didn't want to live with 74189's serving that function).
 
>> But yes, I am pretty sure that Intel didn't expect the 80286
>> to mostly be used in systems only running real mode.
 
> But, the 286 powered *up* in real mode.  Which, no doubt, is
> why real mode persisted (persists?).

The IBM/370 powers up with DAT off. You have to have some way
to initialize the tables you need before you start using them.
 
>>> There were no PC's at the design time of 8086 and 80286,
>>> and the aim was for well protected real-time multitasking
>>> applications. So it was natural that 80286 does not voluntarily
>>> return back to real mode from protected mode. This led to
>>> a massive kludge in PC/AT, where the keyboard controller
>>> can reset the main processor and the RTC CMOS RAM will
>>> contain a code explaining why the reset was this time.

I was recently reading "The Intel Trinity" which probably says
when they started working on the design of the 286. 

The kludge was needed for OS/2 1.x to run DOS box code. 
IBM figured that out in time to get it into the PC/AT.

>> I believe that kludge is still there today.  Also, the A20
>> address line switch.

>> Also, the original IBM/PC used a 14.31818MHz crystal, convenient
>> for generating NTSC video with the CGA. That signal went on the
>> bus for that reason.  The processor then ran on that frequency
>> divided by three (that is the way the clock generator works).

>> Now, all ISA based PCs need the 14.31818 MHz crystal to supply that
>> line, just in case someone uses a CGA.  A few times I put a CGA
>> in my AT clone to generate test patterns for TV alignment.
 
> IIRC, the original "ISA" bus wasn't even formally characterized
> until after the fact.

IBM always was, and still is, good at documented what they did,
though not always why they did it. There are manuals that give lots
of detail on every line of the bus, and BIOS listings with comments.

But then there were mistakes along the way. Edge triggered 
interrupts make it hard to share INT lines. 

-- glen

Am 18.10.2014 um 00:00 schrieb glen herrmannsfeldt:
> Seems like a compiler bug to me.  If you assign to a far pointer,
> then it is your fault.

Since there was no formal definition other than the individual 
compiler's documentation of what a "far" pointer did and did not to in 
various situations, the only way it could be a bug would have been if it 
didn't match its own documentation.

> You don't say what language you use, but C only allows comparison
> of pointers to different objects as equal or not equal.

Since neither "far" nor "huge" pointers even exist in the language 
standard, none of its rules apply to them unless the compiler authors 
decide to say so.

They decided not to.  I actually had to grow some data structures' sizes 
to the next power-of-two so a huge pointer could correctly walk a >64KiB 
array of it.  The horror!

Hans-Bernhard Br&#4294967295;ker <HBBroeker@t-online.de> wrote:
> Am 17.10.2014 um 23:32 schrieb Don Y:
 
(snip)
>> If, instead, the x86 ALWAYS prepared
>> a separate data space (etc), then I could assume a larger model.
 
> Well, yes, it essentially always does just that.  All code addressing is 
> relative to CS, all data addressing is relative to DS or SS, at least by 
> default.  And since all those are allowed to be different, the only 
> a-priori _safe_ assumption obviously is that they will be.

Not so unusual to keep DS and SS together.

Also, even for the 8080 you could have separate address space
for stack and data. There was a way to decode which data references
were to the stack.  I don't know anyone ever did that, though.
 
>> Or, if the tiny model was IMPRACTICAL for any use
 
> It's practical if the program is small enough to allow it, and the 
> program loader being used supports it.

Well, pretty much they did it the same way as CP/M, which allows
for easy porting.
 
> The usual one at the time did: tiny model is what MS-DOS *.com format 
> programs use.  They're copied verbatim from disk to RAM (beginning 0x100 
> bytes into a 64 KB page) , then all three primary segment registers are 
> set to the start of that RAM region (i.e. CS=DS=SS=constant) and the 
> loader just jumps to CS::0x0100.  So code and pre-initialized data are 
> in one big block of RAM.  The remainder of those 64K are assumed to be 
> used for .bss and stack.

> The advantage was that the program loader doesn't have to make any code 
> modifications to handle relocation of the code ... all addresses in the 
> code are just offsets relative to the load base, which itself doesn't 
> ever appear in the code, because it's kept in the segment registers.
 
> The disadvantage was that the loader _didn't_ perform relocations, so 
> there was no way to load bigger programs.  So along came the EXE program 
> file format, which is a whole different kettle of fish.
 
The descriptions I remember for COM didn't say that you were stuck
with 64K, but that it was the program's responsibility to keep
track of everything. You had all the rest of memory to use,
COMMAND.COM would be reloaded from disk if you wrote over it.
 
>> I.e., it seems safe to assume the tiny model was intended to be
>> *usable* and not just "an engineering/marketing exercise".
 
> It stopped being truly usable when programs outgrew 64 KB 
> for code and all(!) data combined.

-- glen

Hans-Bernhard Br&#4294967295;ker <HBBroeker@t-online.de> wrote:
> Am 18.10.2014 um 00:00 schrieb glen herrmannsfeldt:
>> Seems like a compiler bug to me.  If you assign to a far pointer,
>> then it is your fault.
 
> Since there was no formal definition other than the individual 
> compiler's documentation of what a "far" pointer did and did not to in 
> various situations, the only way it could be a bug would have been 
> if it didn't match its own documentation.

Well, it has to match the standard that the compiler claims.

If you aren't using mixed models, the compiler has to consistently
support the model in use. (That is, no near or far keyword.)

But most probably didn't want to go all the way to huge, as it
would slow down (and increase code size) for all pointers. 

Mixed model, it is your responsibility to get it right.

>> You don't say what language you use, but C only allows comparison
>> of pointers to different objects as equal or not equal.
 
> Since neither "far" nor "huge" pointers even exist in the language 
> standard, none of its rules apply to them unless the compiler authors 
> decide to say so.

The C standard allows for a variety of pointer models, including
those of large and huge model. It doesn't allow for the "far"
and "huge" keyword.
 
> They decided not to.  I actually had to grow some data structures' sizes 
> to the next power-of-two so a huge pointer could correctly walk a >64KiB 
> array of it.  The horror!

-- glen