0. Abstract
With the release of MPE/iX (now called MPE/iX) on
HPPA (Hewlett Packard Precision Architecture), many new features have
arrived for the programmer. These include mapped files and a very large
address space. One new feature overlooked by many is the RISC
architecture. Although RISC means "reduced complexity", optimizing
performance on RISC is paradoxically more complex than on the classic
HP3000. This paper asks: "what can we do to maximize performance?" Some
answers are presented, and particular attention is given to the
characteristics of mapped files, the file system, and Native Mode versus
Compatibility Mode.
1. Mapped Files
This section will introduce mapped files and discuss their performance
characteristics.
1.1 Mapped File Introduction
From a programmer's viewpoint, MPE/iX has two basic
types of files: the ordinary, record-oriented files that have existed
since the birth of MPE, and mapped files.
A mapped file is an MPE ordinary file that is going
to be accessed via virtual memory loads and stores instead of (or in
addition to) via file system intrinsics. Instead of calling FOPEN, a
programmer can call the new HPFOPEN intrinsic, and specify that a file
is to be opened for "mapped" access. This will result in two pieces of
information being returned to the program: a file number (like FOPEN
would have returned), and a virtual memory address. The virtual memory
address returned is the address of the first byte of data in the file.
If the address is stored in a pointer, as shown in the following
example, and the pointer is then "de-referenced", the first byte from
the file is brought into memory.
HP Pascal/XL SPLash!
var filedata : ^char; virtual byte pointer filedata;
firstbyte : char; byte firstbyte;
double filedata'spaceid = filedata;
... ...
hpfopen (..., filedata, ...); hpfopen (..., filedata'spaceid, ...);
firstbyte := filedata^; firstbyte := filedata;
Note: the above example was done with HP Pascal/XL, but
most of the rest of the examples in this paper will be done in SPLash!, a
native mode version of SPL/V, which allows easy manipulation of 32 bit
and 64 bit virtual addresses. Mapped file access is also available in HP
C/XL. (The SPLash! example shows the need to get the pointer passed by
reference...the intrinsic declaration of HPFOPEN has no way of telling a
compiler that that parameter expects to be a pointer-by-reference, so
SPLash! would treat "filedata" as a request to pass the address that
filedata *points to*, instead of the address of the filedata pointer itself.)
With the above fragment of code, let's look at fetching the first two 80-byte
records.
byte array
rec0' (0 : 79),
rec1' (0 : 79);
move rec0' := filedata, (80); ! get first 80 bytes
move rec1' := filedata (80), (80); ! get second 80 bytes
If the file system had been used to access the first two records, as in:
fread (fid, rec0', -80);
fread (fid, rec1', -80);
then the total CPU utilized by the FREADs would be much greater than the CPU
used by the two "move" statements.
1.2 How are Mapped Files Implemented?
In MPE/iX, all files are stored on disc as an array of bytes. A
file is called a "mapped file" if it happens to have been opened by a
user who requested its virtual address be returned as a result of the
HPFOPEN intrinsic. At the lowest level of MPE/iX, ALL disc files are
always opened as mapped files. Usually, we call a file a "mapped file"
if we intend to access its data via virtual memory along with (or
instead of via) the file system intrinsics.
Two aspects of disc files have changed from MPE V to MPE/iX:
- The file label is not stored as part of the file.
- There is no wasted space between records or between blocks.
The first change is a decade overdue. The second change is a direct result of
the virtual memory system of HPPA.
When any disc file is opened in MPE/iX, a module called the
"Virtual Space Manager" allocates a range of virtual addresses
sufficient to cover the entire file. The process is called "mapping", as
in: mapping the file into virtual memory. "Mapping" provides a
one-to-one correspondence between a virtual memory address and a byte of
disc data for every byte in the file.
If a program tries to use a virtual address that has been mapped
onto a file to fetch a byte of data, the following is done by hardware:
- Extract the upper 53 bits of the 64 bit virtual address, calling it the
VPN (Virtual Page Number).
- Is the virtual page "in" memory. (I.e.: is there a physical page of 2,048
bytes that has been assigned to that VPN?)
- If yes, then using the bottom 11 bits (the page "offset") of the original
64 bit virtual address, index into the physical page, fetch the byte, and
return.
- If no, interrupt and ask the software to bring our page into physical
memory.
- When our page arrives in memory, our process will be restarted at step #1
above.
- The above process can be phrased in a simpler manner:
- If the virtual address is in real memory, fetch the data;
otherwise do a "page fault" and swap the page into memory and then fetch
the byte.
Note: this description of virtual memory is simplified, and omits features such
as the Translation Lookaside Buffer (TLB).
Thus, to fetch the first byte of the 100th record of an 80-byte
record file, we can simply take the virtual address of the first byte of
the file, add 8000 to it, and then fetch a byte from that address.
Sooner, or later, the byte will appear in the register that we asked it
to be loaded into.
- The detailed workings of virtual memory are quite complex, and beyond the
scope of this paper. For now, let's just remember:
- When bytes of a file are accessed via a virtual address, the data is
brought into memory as needed by the operating system via "page faults".
Once a page is in memory, its data can be accessed at main-memory speeds.
On a typical MPE/iX machine, many millions of bytes of mapped files could
be in memory all at the same time.
If anything is stored into the virtual address, the physical page is
marked dirty. Dirty pages are eventually written out to disc, but this
process might not occur for quite some time.
When we talk about a "page" in reference to the CPU hardware, we
generally mean a "physical page" of 2,048 bytes. At most other times,
"page" refers to a "logical page" (sometimes incorrectly called a
"virtual page") of 4,096 bytes. When a logical page is brought into
memory, it will occupy two consecutive physical pages.
1.3 Prefetch
"Prefetching" is the act of bringing more data from disc into
memory than was immediately requested by a user, in an attempt to
prevent a second disc read shortly after the first.
The disc caching code on MPE V had two "dials" the system manager
could twist to control the amount of data prefetched. One dial to
control the size of cache domains created for sequential disc reads, and
another to control the size of domains created for random disc reads.
On MPE/iX, the system manager has no such controls. Instead, the
prefetch size is determined (at present) by one primary factor: what
subsystem is asking for the data to be read from disc. If the request to
read data from disc is from the memory manager (due to a page fault),
one logical page is read. If the request is from the file system,
several logical pages are read.
Clearly, this has enormous performance implications. Consider a
program accessing a file of 256 byte records in a sequential manner.
Assuming the file has about 90,000 records, and assuming that the file
system requests 4 logical pages at a time, then the memory mapped access
will have 5,625 page faults versus 1,406 for the file system accessor.
(Remember: a logical page is 4,096 bytes, and a physical page is 2,048
bytes. Unless dealing with the lowest levels of MPE/iX, we normally
refer to logical pages.)
As a test of the above, a program was run that did a simple
sequential read of the file SL.PUB.SYS (89,867 records of 256 bytes).
This file takes about 22 megabytes of disc space. The following table
show the CPU and Elapsed times required to read the file. In between
each run, a separate 16 megabyte file was read in an attempt to flush as
much of the SL.PUB.SYS file data from memory as possible (see the
section: Measurement Problems).
The following table shows the time the test program needed to read SL.PUB.SYS.
The test program was running in Native Mode.
SL.PUB.SYS sequential read (times in milliseconds)
CPU Elapsed Delta Access Method
----- ------- ------ -------------
19686 146298 126612 Memory Mapped
35398 44361 8963 FreadDir
36590 44957 8367 Fread
39465 46802 7337 FreadDir & FreadSeek
48650 51949 3299 Fread & FreadSeek
The "Delta" column shows the amount of time the program was presumably
waiting for the data to come from disc.
The "FreadDir" access method consisted of using the FREADDIR
intrinsic with ascending record numbers, which results in reading
exactly the same records as the FREAD intrinsic. The last two rows added
a call to the FREADSEEK intrinsic in an attempt to have MPE/iX prefetch
data before it was read. For those two tests, FREADSEEK was called once
every 4 reads, with a request to prefetch the fourth record following
the current.
The implications:
- Use sequential FREADDIR to sequentially read a file that is not already
in memory (see note below);
- Don't use FREADSEEK. At least in these tests, it never seems to help,
and only costs extra CPU time.
Taking the first delta figure, 126,612, and guessing that we can do a
disc read in 22.5 milliseconds, we get an estimate of 5,627 disc reads,
which matches our prediction.
If we take the delta for the FREAD test, 8,367, and using the
same estimate of 22.5 milliseconds per disc read, we see 372 disc reads.
This implies that FREAD is prefetching in chunks of 15 or 16 logical
pages, not the 4 originally assumed.
Note that with the FREAD & FREADSEEK test the delta was cut about in half,
at the cost of greatly increased CPU time.
A second large file was tested, NL.PUB.SYS (64,275 records of 256 bytes each,
16 megabytes):
CPU Elapsed Delta Access Method NL.PUB.SYS
----- ------- ------ ------------- (sequential)
11507 74920 63413 Memory Mapped
22109 26240 4131 FreadDir
23857 27364 3507 Fread
25735 28124 2389 FreadDir & FreadSeek
28887 31151 2264 Fread & FreadSeek
These results mirror those for reading SL.PUB.SYS.
1.4 Memory Resident Data
The previous section examined the performance of mapped files
versus the file system for data that was out on disc. Frequently, the
data for a file will happen to be resident in memory. This is the case
when a file is accessed multiple times in a relatively short period.
This section examines the performance of accessing file data that is
already in memory. Using the same Native Mode program (an SPL/V program
compiled with SPLash!), the file CATALOG.PUB.SYS was sequentially read.
This file has 7040 records of 80 bytes each for a total of 0.5
megabytes.
CPU Elapsed Access Method
---- ------- -------------
181 182 Mapped File
1660 1677 FreadDir
1678 1680 Fread
1959 1976 FreadDir & FreadSeek
1977 1994 Fread & FreadSeek
The file CATALOG was read once to bring it into memory. The time to do this
is not reflected in the above table.
Note that the elapsed time is just slightly more than the CPU time. This is
because the process is never paused to wait for disc I/O.
The implications:
- If the file's data is likely to be in memory, use mapped file access!
- FREADSEEK should not be used for files where the data is in memory already.
1.5 NM vs CM vs OCT
MPE/iX can execute in any of three modes: Native Mode (executing
RISC instructions), Compatibility Mode (emulating classic HP3000 CISC
instructions), and a blend of the two produced by the Object Code
Translator (OCT). Briefly, a Compatibility Mode (CM) program can be run
through the OCT to produce a hybrid program file that contains the
original CISC instructions as well as their translation into RISC
instructions. OCT'ed programs must obey ALL the same restrictions as CM
programs (e.g.: 16-bit wide stack of 65,535 bytes). (For more
information on OCT, CM, and NM, the reader is directed to the book
"Beyond RISC" from Software Research Northwest.)
The data in the preceding tests was obtained from a Native Mode
program. This section examines the performance of the file system when
called from the three types of program code: NM, OCT, and CM. As a
reminder of what can be accomplished by what my partner, Steve Cooper,
calls the "second migration", mapped file access is also shown in the
table. The "second migration" is the process of adapting programs to
take advantage of the new features in MPE/iX. The "first migration" is
the one HP talks about: porting a program to Native Mode (which usually
means minimal changes).
The file CATALOG.PUB.SYS was sequentially read in the same
manners as before, with the IDENTICAL program compiled in SPL/V (CM),
run through the Object Code Translator (OCT), and compiled by SPLash!
(NM). The following table shows the results:
CATALOG.PUB.SYS (times in milliseconds)
CPU Elapsed Mode Access Method
---- ------- ---- ---------
181 182 NM Mapped (requires NM)
1660 1677 NM FreadDir
1678 1680 NM Fread
1959 1976 NM FreadDir & FreadSeek
1977 1994 NM Fread & FreadSeek
3326 3343 OCT FreadDir
3838 3854 OCT Fread
4196 4214 CM FreadDir
4850 4881 CM Fread
5196 5216 OCT FreadDir & FreadSeek
5670 5690 OCT Fread & FreadSeek
6471 6493 CM FreadDir & FreadSeek
7473 7493 CM Fread & FreadSeek
The implications:
- NM is far faster than CM or OCT.
- Calling FREADSEEK from CM or OCT programs is even more of a penalty than
calling it from NM programs.
- FREADDIR is still slightly faster than FREAD.
The test program was produced from the source file "READER" with the following commands:
CM: spl reader, $newpass, $null
prep $oldpass, reader.cm
OCT: octcomp reader.cm, readero.cm, , noovf
NM: splasm reader
Note that the "noovf" option on the "octcomp" command tells the OCT
that the program does not expect to generate arithmetic overflows and to
optimize its translation with that in mind. This results in slightly
faster OCT'ed programs.
The basic reason that the CM and OCT programs are so much slower
is that simple disc files are handled by Native Mode portions of MPE/iX.
Some types of disc files are still handled by Compatibility Mode
portions of MPE/iX, ported from MPE V/E. These include message files,
RIO files, Circular files, and KSAM files.
When a CM or OCT program calls the FREAD intrinsic to read a
record from an ordinary disc file, the FREAD intrinsic must "switch" to
Native Mode and call the Native Mode FREAD intrinsic. This switch is not
inexpensive. OCT programs pay the same switch overhead as CM programs
because they are still emulating the Classic instruction set, albeit
faster than the emulator. NM programs (e.g.: HP Pascal/XL and SPLash!)
are already in Native Mode when they call FREAD, so no switch is necessary.
The next test shows the results of serially reading a KSAM file
of 1,000 80 byte records from NM, OCT, and CM programs. As in the
CATALOG test, the file was brought into memory before the start of the test.
CPU Elapsed Mode Access Method
---- ------- ---- -------------
2475 2494 OCT Fread
2677 2696 CM Fread
3239 3257 NM Fread
Note that the FREAD intrinsic returns the records in key order, not the
chronological order in which they were written.
Note that the FreadDir test was dropped. The FREADDIR intrinsic cannot be
used on KSAM files.
The mapped file test was dropped because it reads the data in chronological
order, not key order.
The implications:
If KSAM is being used heavily, don't migrate the programs into NM
until a native mode version of KSAM is available (from HP or another
vendor).
[Update 96/03/07: KSAM/iX is in Native Mode]
2. Memory & Disc Utilization
In MPE V, stacks were limited to a maximum of 65,535 bytes. In
MPE/iX, the limitation is 1 gigabyte (1,073,741,824 bytes). (This limit
includes the CM stack & heap, the NM stack, the NM heap, and the XRT.)
In MPE V, if any part of the stack was in memory, then the entire
stack was in memory. In MPE/iX, only the logical pages recently
referenced are likely to be in memory at any time. Additionally, only
those pages that have EVER been referenced are allocated disc storage.
As more and more stack/heap pages are touched, more and more pages are
allocated on disc. This means that having an array of 1,000,000 bytes in
SPLash! (or Pascal/XL, or any NM language) is not expensive...until you
use it. A megabyte array will have 1 million bytes of virtual address
assigned to it, but the disc storage will range from 0 to 256 logical pages!
Disc files are allocated storage exactly like the stack/heap:
only those pages ever touched are allocated disc sectors. (Since extents
may be allocated several logical pages at a time, some rounding-up does
occur.) This means that it is feasible to have "sparse" files. For
example, a file with 1 byte for every possible Social Security number
would have a limit of 999,999,999 bytes. If a single write is done to
record 2345, then a single extent will be allocated. A test done on MPE
XL 1.1 resulted in an extent of 2,048 sectors being allocated. This does
not mean that all future extents will be of equal size. Unfortunately,
the programmer has no control over the extent size.
3. Data Alignment
On the Classic HP3000, the natural data alignment was 16 bits.
With rare exceptions, 32-bit and 64-bit data could be placed at any
16-bit boundary with impunity and no performance ramifications.
On the HPPA HP3000s, the natural data alignment is 32 bits for
32-bit data, and (sometimes) 64-bits for 64-bit data. (The 64- bit
alignment applies primarily to IEEE 64 bit floating point numbers.)
As a result, if code is ported from a CM language to its NM
equivalent, one of two problems can result: program aborts (or other
errors) due to misaligned data; or performance slowdowns.
Most NM compilers provide a means of specifying that certain
variables are only 16-bit aligned. When this is done, then the compilers
will typically emit 3 instructions to load a 32 bit variable instead of
the 1 that would have been required if the variable was aligned on a
32-bit boundary. This is necessary because the RISC hardware does not
allow the LDW (Load 32-bit Word) instruction to be given an address that
is not a multiple of 4 bytes (32 bits). Instead, 2 LDH (Load 16-bits)
instructions and one DEP (deposit) instruction must be used to build the
32 bit value in a register.
No performance data is shown here because the implications are clear from the
instruction count: 1 versus 3.
4. SORT vs HPSORT
Compatibility Mode programs that call the SORT intrinsics still get the old
sort package, running in OCT.
Native Mode programs have a choice of two intrinsics to do
sorting: SORT and HPSORT. These two intrinsics are interfaces to a new
sort package which runs in Native Mode. The native mode sort package
lacks some of the features of the CM sort facility (e.g.: the ability to
pass procedures to do the comparison), and has one additional wrinkle:
sometimes it calls the CM sort to do the sort!
In its present incarnation, NM Sort will call CM Sort when it
gets a "difficult" sort. This includes sorts that specify an alternate
collating sequence.
Additionally, when NM Sort does stay in NM, it does NOT open a
temporary file called SORTSCR. Instead, it uses two temporary files that
are either nameless or have a name like HPSORT1 and HPSORT2 (?),
depending on the release of MPE/iX. This means that if a fairly simple
sort is requested from a NM program, the programmer cannot point the
sort scratch file to a disc drive he/she knows is separate from the
input and output data.
In short, NM Sort is still evolving. Test runs should be made before converting
to NM simply to call NM sort.
5. System Performance
The overall system performance can still be affected by proper tuning of the
C, D, and E subqueues via the TUNE command.
The choice of disc drives for a file can also be controlled in
the usual manner (e.g.: BUILD FOO;DEV=3). However, the number of extents
cannot be easily controlled any more. The basic choice is one extent or
many extents.
Main memory is vital to the performance of the system. Unlike MPE V, which
tended to degrade slowly, MPE/iX will suffer a very sharp drop in performance
when not enough memory is available. Economize on everything else ... and buy
memory.
A 950 (and 955) will support up to 256 megabytes (128 per memory
controller). Three vendors offer memory for the machine: HP, Kelly
Computer Systems (the first to put 256 megabytes in a user's computer),
and EMC. Sites with Classic HP3000s may be interested in Kelly's RAMDISC
for the 3000, which can be traded in on HPPA memory when needed.
6. NM vs. CM : Intrinsics
In an earlier section, we determined that some types of files are still
implemented with Compatibility Mode code.
File system intrinsics are not the only ones that might actually
be implemented in CM. The ASCII, BINARY, DASCII and DBINARY Native Mode
intrinsics currently switch to CM to do their work. Although this may
change in the future, the performance implications are still interesting today.
Porting a program into Native Mode may reveal other intrinsics that are still
implemented in Compatibility Mode.
The following table shows the result of calling the ASCII
intrinsic a large number of times from programs written in NM, OCT, and CM:
CPU Elapsed Mode
---- ------- ----
9051 9084 NM
11688 11728 OCT
12211 12252 CM
Although the Native Mode program was the fastest, it is by a very narrow margin.
The ASCII/BINARY/etc. intrinsics have always been a performance
bottleneck on MPE V. They haven't changed in MPE/iX. The following table
shows the results of calling the ASCII intrinsic versus calling a
"clone" of the intrinsic:
CPU Elapsed Mode Procedure
---- ------- ---- ---------
457 471 NM ASCII clone
9009 9040 NM ASCII intrinsic
Similar savings can be obtained for BINARY, DASCII, DBINARY, and
CTRANSLATE. Contact the author for NMOBJ files that can be used as
replacement intrinsics.
7. Measurement Problems
Measuring performance on MPE/iX is extremely difficult. Unlike
MPE V, MPE/iX provides no control over what disc data is (or is not) in
memory. As a result, tests must be run multiple times with best-case (or
average-case) times used.
(Note: the program
FLUSH can be used to flush pages of
closed files out of memory.)
The difficulty of measuring performance is at its worst when
looking at disc I/O. The following is a partial list of features that
would aid this type of analysis:
- An intrinsic that will make free all pages of memory that are not marked
memory-resident or locked.
- An intrinsic that will force all pages that are dirty to disc.
- An intrinsic that would return for a virtual address information like:
size of object and number of logical pages currently in memory.
The first feature would allow the system to be returned to a known
"blank slate" state, allowing repeatable performance testing.
Note: an intrinsic allows the system manager/performance tuner/
software developer the ability to exercise the above functions
programmatically. This is clearly superior to simply having a command
for two reasons:
- A command can be written by the user which simply calls the intrinsic.
The opposite is not inexpensively true.
- Intrinsics are not as easy to abuse by the casual user.
[Update 96/03/07: the author has a utility program, FFLUSH, which
flushes the pages for all closed files out of memory. This allows for
relatively easy and repeatable testing for many of the timing questions
considered in this paper.]
One valuable tool used in this paper is DEBUG. Given a virtual
address associated with a mapped file, the debugger can be used to
determine the number of logical pages that are currently in memory.
Assuming the file starts at virtual address $123.0, then the debugger
command:
= vainfo ($123.0, "pages_in_mem"), #
will report (in decimal) the number of 4,096 byte logical pages that are
currently in memory.
8. Conclusions
Obtaining optimum performance with MPE/iX is more difficult than
on MPE V ... there are more things to tune, with much less knowledge.
Things to remember:
- The amount of memory on the machine is critical;
- Migration to Native Mode is important, but should not be done blindly. If
an application is a heavy KSAM or message file user, do some timing tests
first.
- The "second migration" is more important ... it means taking advantage of
the new features.
Perhaps, when MPE/iX begins to stabilize, and third-party performance
tools are developed and marketed, the folklore on how to maximize
performance will begin to grow as it did under MPE V. In the meantime,
keep the faith!
NOTE: All timings in this paper were obtained running under MPE
XL 1.1. Initial testing on MPE XL 1.2 shows no major differences.
WAIT!
Postscript!
FREADSEEK has been given a bad name in this article. Well, like
the "goto", it has its uses. Further testing (and a lot of thought)
resulted in a modification to the test program that was reading
SL.PUB.SYS with the results:
SL.PUB.SYS sequential read (times in milliseconds)
CPU Elapsed Delta Access Method
----- ------- ----- -------------
15273 49525 34252 Memory Mapped & FreadSeek <---
19686 146298 126612 Memory Mapped
35398 44361 8963 FreadDir
35903 65936 30033 FreadDir & FreadSeek
36590 44957 8367 Fread
39769 64529 24760 Fread & FreadSeek
Notice the incredible change in the "Memory Mapped & FreadSeek"
numbers (first line). The crucial difference here is in the timing and
quantity of calls to the FREADSEEK intrinsic. Earlier testing showed
that the best case "throughput" for reading data with a mapped file
(where the data was already memory resident) was about 3,111 bytes per
millisecond. (Obtained from the memory-resident speed of reading
CATALOG.PUB.SYS (80 * 7040 bytes) in 181 milliseconds.) Clearly, any
prefetch should be done far enough ahead of time that the data is in
memory by the time it is needed. The above calculation showed that if we
assume it takes 30 milliseconds to read data from disc, then it must be
requested 30 * 3,111 bytes before it is needed.
The test program was adjusted to prefetch 128 records ahead
(instead of 4 records ahead). The next round of timings showed a gain,
but not as much as hoped for. Then, we realized that the prefetch was
reading 8 logical pages. So, after processing 24 logical pages (100,000
bytes) the test program was prefetching 8 logical pages instead of 24!
The program was modified again, to fetch 24 logical pages at a time,
resulting in the times shown above.
Moral: prefetching via FREADSEEK is worth the time, but ONLY
after careful analysis. Failure to prefetch at the right time, or not
enough data, is worse than not prefetching at all.
![[3khat16.ico]](../../../img/3khat16.ico)
HP3000![[archive16.gif]](../../../img/archive16.gif "Wayback Archive")