Large Size Datasets in TurboIMAGE/iX
MPE documents | |
 Complete PDF | |
| Table of Contents | |
| Index |
Commercial Systems Division-Databases
Introduction
This enhancement in TurboIMAGE/iX removes the current 80 GB limit on the size of a dataset so as to allow users to have really large datasets. Now the only limiting factor is the number of entries in a dataset, which can be up to 2**31 -1.As the solution involves changes in the internal data structures of IMAGE, a new utility DBBIGSET provides an easy way for existing databases to migrate to the new structures. Once migrated, the structural details will be transparent to the users. User interface changes have been made to DBSCHEMA, DBUTIL and DBINFO. Old and new format databases can co-exist in the same system.
Dataset Size Limitations
The IMAGE database management system has been around for more than two decades.When HP first introduced the product, the maximum size of a dataset was limited to the MPE file size. However, when users felt the need for much larger dataset sizes, the 'JUMBO' feature was introduced. With this feature, a user can create a dataset with a size greater than the MPE file size of 4GB. Even though the dataset physically spans across multiple MPE files called chunk control files and chunkset files, the JUMBO feature allows them to be seen as a single logical file. Even with this feature, users could only define a dataset up to 40 GB in size, because of the limit of block numbers, whereas the JUMBO feature per se doesn't have this restriction. Earlier, IMAGE used to transfer data in blocks, which necessitated addressing of a record through a block number and an offset within the block. The maximum number of blocks allowed to be addressed in IMAGE was 2**23 -1, and the maximum block size was 5120 bytes. The total file size with these, turns out to be 42,949,667,840 bytes (approximately 40 GB). A quick solution has been provided in IMAGE version C.07.10 released with MPE/iX version 6.0, wherein the limit of 40 GB has been extended to 80 GB. This was done by increasing the maximum number of blocks to 2**24 -1. The current enhancement lifts the 80 GB restriction by changing the way that records are addressed internally.
Entryname vs Entrynumber formats
To understand the solution in detail, a brief discussion of the terms entryname and entrynumber is worthwhile. As discussed above, in IMAGE, records (entries) were referenced through a 24 bit block number and an 8 bit offset (slot number) within the block. This format is referred to as 'entryname' format. For example, the 101st record in a dataset, with a blocking factor of 8, will be referenced as hex c05, represented as the 24-bit blocknumber of 12 (hex c) and a slot within the block of 5. Now we can use a full 32-bit number to refer a record number. We call this 'entrynumber' format. So, the 101st record in a dataset will be represented as just 101, irrespective of the blocking factor used.All existing databases prior to this release use entryname format. With this solution, a database can be created either in the entrynumber format or the entryname format. However, if large datasets greater than 80 GB are to be created, it can be done only if the database is created in entrynumber format.
Changes in User Interfaces
DBSCHEMA
To protect existing database environments and scripts, the new DBSCHEMA (version C.09.00 of IMAGE) will create databases in entryname format by default. However, if a database needs to be created in the entrynumber format, there is a new $CONTROL LARGESET directive in DBSCHEMA. Once DBSCHEMA encounters this directive, it will create the root file to support large datasets, and set the 'root version' to 'C6'. It may be noted that a database, as a whole, will need to support large size datasets (in other words, one can not define a particular dataset alone to be in entrynumber format).DBUTIL
DBUTIL changes are nearly transparent to the user, except that show dbname all, will have the following message, if entrynumber format is used: This database supports dataset size > 80 GB Also, if the migration process failed, a subsequent show dbname all will have the following message: Migration to large dataset failed. Please restore db from backup All other commands in DBUTIL will behave as today.DBINFO
DBINFO has been modified to return information on various features used by the database. Now, DBINFO Mode 406, will return in the buffer element #17 the various features used in the database. The word is interpreted as below:| Bit # | Value/ Description |
|---|---|
| 15 | 1: Database contains at least one expandable detail dataset0: Database contains no expandable detail dataset |
| 14 | 1: Database has at least one jumbo dataset 0: Database contains no jumbo sets |
| 13 | 1: At least one master dataset in the database has a B-Tree0: No datasets in the database have B-trees |
| 12 | 1: Database has at least one expandable master dataset 0: Database contains no expandable master datasets |
| 11 | Unused |
| 10 | 1: Database is in entrynumber format (i.e support for large size dataset available)0: Database in entryname format (Dataset size can only be upto 80 GB) |
| 09 | 1: Database uses new limits (# of sets = 240,# of items =
1200, # paths per masters = 64) 0: Database uses old limits (# of sets = 199, #of items = 1024, # paths per master = 16) |
| 08-01 | Unused |
Migration Utility
Existing databases can be moved to support large datasets through the migration utility provided with the solution. The migration utility is DBBIGSET.PUB.SYS, which has PM capability and has to be executed from the group where the database exists. Migration is a one-time activity, and it requires exclusive access to the database. Since the internal structures are being changed, a full backup of the database must be taken before starting the migration. Alternatively, if sufficient resources are available one could make a copy of the database, do the migration on the copy, verify the integrity and copy the migrated database back to the original. No data integrity checking of the database is performed during this process. During migration, the RL'CONDITION word in root file label #0 will be set to 'LS'. If for any reason the migration can't be completed, appropriate error messages will be displayed, and the user must revert back to the old database environment from the previous backup. The syntax for running migration utility is RUN DBBIGSET.PUB.SYS;PARM=mig-option;INFO='dbname' Where mig-option =
1: migrate from entryname format to entrynumber format
-1: migrate from entrynumber format to entryname format dbname = unqualified database name (without group or account)
 |
NOTE: It should be kept in mind that, migration involves changes to internal structures of the database, and a backup must be taken before starting the migration. |
Entrynumber format vs. Jumbos vs. MPE Large Files
It has to be noted that this enhancement per se will not make use of the large file feature of MPE release 6.5. Creating a database in entrynumber format or migrating a database to entrynumber format just allows the datasets to grow beyond 80 GB (which is the current limit). There is no harm in keeping the database in entrynumber or entryname format, if the dataset size is less than 80 GB. Also, there is no impact if some databases are kept in entrynumber format and others in entryname format in the same system. However, as of version C.09.02 of IMAGE, the only mechanism to have a dataset size greater than 4 GB, is to use 'Jumbo' feature. This will be the case, till IMAGE makes use of the large file feature of MPE.
Conclusion
In summary, this release
Enhances TurboIMAGE/iX to support large dataset sizes beyond 80 GB (currently up to 10 TB, if the maximum entrysize of 2378 halfwords is used, and the dataset contains the maximum number of records 2**31-1).
Provides a way of migrating existing databases, if large size dataset support is required.
Ensures that existing databases and applications continue to work as today.
|
Business BASIC and the TurboIMAGE/iX Limits Increase |
Chapter 5 Product Release History |