AIX V5.3 software maintenance best practices
This web page is meant to supplement, not replace, other documentation which has been published:
The contents of this web page solely reflect the personal views of the authors and do not necessarily represent the views, positions, strategies or opinions of IBM or IBM management. Please use the Add Comment link at the bottom of the page to provide feedback. Note: Until you sign up and log in (using links in the upper right corner of this web page), you will not see the Add Comment link and you can not add a comment.
The structure and contents of this web page are loosely based on the AIX updates Version 3: How to work the puzzle web page published by Shiv Dutta and Brad Cobb, but they are not responsible for this web page nor its contents.
Speaking the IBM Support Center's language
The terminology associated with AIX fixes is confusing. Once the terms are understood, maintaining an AIX system becomes significantly easier:
- Fileset
A set of files that are installed together as a single unit. All licensed program products (including AIX itself) are packaged and delivered as one or more filesets. A fileset has a name (eg, bos.diag.com) and a VRMF level (eg, 5.1.0.10). The term "fileset level" is usually used instead of "VRMF level".
(VRMF is an acronym standing for version, release, modification, and fix. For example, PEX_PHIGS.graPHIGS.rte.base VRMF or fileset level 5.1.0.10 is version 5, release 1, modification level 0, fix level 10 of the PEX_PHIGS.graPHIGS.rte.base fileset. With AIX V5.3 below TL7, the first digit of a fileset's fix level seems to be the Technology Level and the second digit seems to be the Service Pack in which the update was first delivered. Starting with TL7, a fileset's modification level will be the Technology Level in which the fileset update was first delivered, and a fileset's fix level will be incremented from 0 as each new update is delivered within the TL, according to the IBM AIX 5L Operating System Service Strategy Details and Best Practices document downloadable from the Service and support best practices for UNIX servers web page.)
- PTF - Program Temporary Fix
A program temporary fix (PTF) provides a fix for one or more software product defect(s). A PTF is temporary only in the sense that a fix is no longer required once the defect is addressed in the next release of the product. A single PTF often fixes multiple defects. A PTF applies to a single fileset. If more than one fileset must be updated to deliver new function, an Invisible Packaging PTF (IPP) is defined. An IPP does not update any filesets. Instead, it points to a set of PTFs that collectively deliver the new function.
For example, PTF U476294 updates fileset PEX_PHIGS.graPHIGS.rte.base. Specifically, U476294 updates the fileset to the 5.1.0.10 VRMF level. PTF U476294 fixes four defects (identified by APARs, defined below).
A PTF which has been applied to an AIX system (but not committed) can be rejected (backed out).
- PMR - Problem Management Record
A Problem Management Record (PMR) is created when a customer opens a software service request with the IBM Support Center to ask a question or report a problem.
- APAR - Authorized Program Analysis Report
An authorized program analysis report (APAR) identifies and describes a software product defect. An APAR number can be used to obtain a PTF for the defect, if a PTF is available. When documenting required fixes, it's best to specify APARs for which fixes are required, rather than PTF or PMR numbers. (It is always possible to determine if the PTF (fix) for an AIX APAR is installed on a system using the command 'instfix -iavk APAR_NUMBER', assuming the APAR is not fixed by an Invisible Packaging PTF. There is no other way to query for the presence of a PTF on an AIX image.)
An APAR can specify other APARs as requisites, in which case fixes for those other APARs will (for pre-reqs or co-reqs) or might (for if-reqs) be required to install the APAR's fix.
Continuing with our example, PTF U476294 fixes APARs IY18782, IY18936, IY19534, and IY19765.
According to the IBM AIX 5L Operating System Service Strategy Details and Best Practices document downloadable from the Service and support best practices for UNIX servers web page, if a defect occurs in more than one release (ie, AIX V5.2 & V5.3), then an APAR is now created for each release. But starting with TL7, an APAR will be created for each Technology Level (see below) for which a fix must be generated.
- TL - Technology Level
An AIX Technology Level (TL) is an update to AIX containing fixes and new hardware or software features. (AIX software updates are now bundled in Technology Levels rather than Maintenance Levels.) A new AIX Technology Level is released roughly every six months. For more information regarding the AIX Technology Level concept (and a comparison of Technology Level concept with the Maintenance Level concept), see the IBM AIX Operating System Release and Service Strategy document downloadable from the IBM AIX Operating System Release and Service Strategy web page.
- SP - Service Pack
An AIX Service Pack (SP) is a bundle of PTFs released between Technology Levels. The PTFs/fixes address highly pervasive, critical, or security-related issues. Service Packs are provided for AIX N and N-1 releases (for example, V5.3 and V5.2) on the latest Technology Level for each release (for example, 5300-04 and 5200-08). For more information regarding AIX Service Packs, see the IBM AIX 5L Operating System Service Strategy Details and Best Practices document downloadable from the Service and support best practices for UNIX servers web page.
- Interim Fix
An AIX Interim Fix is a fix custom-built by the IBM Support Center to fix a defect. An AIX Interim Fix addresses a defect in a single fileset and at a single VRMF level of that fileset.
Unlike a PTF, an AIX Interim Fix is tested only to confirm that it fixes the defect it is intended to address. Since an interim fix can therefore be built and delivered more rapidly than a PTF, security fixes are often delivered as interim fixes. The IBM Support Center is able to build interim fixes which address non-security related issues only on supported Technology Levels. Beginning with AIX 5300-06, Technology Levels are supported for approximately two years.
For more information regarding Interim Fixes, see the Interim fix management article.
- alt_disk_install
The AIX alt_disk_install command allows a root sysadmin to create an alternate rootvg on another set of disk drives. The alternate rootvg can be configured by restoring a mksysb image to it while AIX continues to run from the primary rootvg, or the primary rootvg can be "cloned" to the alternate rootvg and updates and fixes can then be installed on the alternate rootvg while AIX continues to run. (Upgrading AIX to a new release - eg, V5.2 to V5.3 - on alternate disks requires a Network Installation Management (NIM) server.) When the sysadmin is ready, AIX can be rebooted from the alternate rootvg disks. Changes can be backed out by rebooting AIX from the original primary rootvg.
In AIX V5.3, alt_disk_install has been replaced by alt_disk_copy , alt_disk_mksysb , and alt_rootvg_op . The alt_disk_install module will continue to ship as a wrapper to the new commands, but it will not support any new functions, flags, or features.
- multibos
The multibos utility (available with AIX V5.3 Maintenance Level 3 and above) allows a root sysadmin to create and maintain two bootable instances of AIX in the same root volume group (rootvg), so root can install AIX updates and fixes which require a reboot without disrupting the running AIX instance. When the sysadmin is ready, the AIX instance can be rebooted to implement the updates and fixes which have been installed on the standby instance. Changes can be backed out by rebooting the original running instance. For more information regarding the multibos utility, see the Using the multibos utility article.
- SMIT
The AIX system management interface tool (SMIT) is a menu-driven interface to many AIX systems functions, including those functions which install, update, and otherwise manage AIX and other software components. Most AIX sysadmins use SMIT to manage software, rather than learning the rather complex command line interface.
Hint: When you request that SMIT do something, SMIT invokes an AIX command to do the work. As noted in the SMIT article, when you press F6, SMIT displays the command it is about to invoke. So you can use SMIT to learn the AIX command interface for system administration, while getting the work done with SMIT until you become proficient with the command interface.
Hint: The smitty command starts SMIT in ASCII (nongraphical) mode. The smit commmand starts SMIT in AIXwindows (graphical) mode. When you start SMIT, you can provide an optional FastPath name, which causes SMIT to start on the menu page specified by the FastPath. As noted in the SMIT article, when you press F8, SMIT displays the FastPath name which will cause SMIT to start on the menu page currently being displayed.
Maintaining AIX software
If a Production environment is completely stable - no growth in workload, no new hardware, and no new application, middleware, or system management software, then there is no need to introduce new AIX updates. Please note: Very few Production environments meet all these criteria. And, in the long run, change is inevitable. For example, it was necessary to introduce new AIX updates to accommodate the change to Daylight Savings Time which was implemented on March 11, 2007.
And even if nothing else changes, there is motivation to keep System p firmware fairly current. System p firmware is constantly enhanced to do a better job identifying failing hardware components. So if/when a server fails, the more current the firmware on the server, the more likely IBM is to identify and replace a failing component on the first try, thereby avoiding a recurrence of a hardware problem before it is fixed. System p firmware is constantly enhanced to do a better job recovering from hardware failures without taking a server outage. So the more current the firmware, the less likely a server is to crash when a hardware failure occurs. The Fix Level Recommendation Tool (FLRT) can be used to determine minimum recommended levels for AIX, p5 system firmware, Hardware Management Console machine code, Virtual I/O Server software, and several other optional system components. Use FLRT to determine if an AIX Technology Level is compatible with other system component levels.
If the cost or impact of an unscheduled Production outage is high, AIX software maintenance best practices require a Development environment, a Quality Assurance/User Acceptance Test environment, and a Production environment for each solution. The QA/UAT should have identical software and hardware to Production. An infrastructure must exist to drive a peak production workload into the QA/UAT system. The number of CPUs, disk drives, etc are sometimes reduced in the QA/UAT environment to reduce cost. Those who decide to implement a scaled down QA/UAT infrastructure must acknowledge that there are certain classes of problems (related to scalability) that will only be encountered in Production.
Software maintenance best practices require HACMP clusters in the Development and QA/UAT environments if there is an HACMP cluster in Production.
Before a change is introduced in Production, a recent AIX Technology Level and Service Pack should first be introduced. The change should be rolled through Development -> QA/UAT -> Production and tested at each step, including a performance stress-test in QA/UAT. In some cases, it will be necessary to update application, middleware, or system management software to accommodate the new AIX Technology Level, when those components would not otherwise need to be changed.
If old components (eg, application software) aren't supported or won't run on the new AIX Technology Level, it will be necessary to change AIX and other component levels at the same time. But if possible, it is desirable to introduce and stress-test the AIX software updates, then introduce and stress-test other new components one at a time. Changes should be introduced into Production in the same two phases - first AIX software updates, then the other new component perhaps a week later.
(Introducing changes in phases improves one's ability to isolate the trigger for any new problem to a change in a particular component. But please note that the software defect causing a problem does not necessarily lie in the last component changed. That is, defects in system management software (for example) may be benign when used with a given version of middleware, yet cause problems when used with a higher level of that middleware. So even if a problem with system management software appears immediately after a middleware change is introduced, the defect may lie in the system management software rather than in the middleware.)
Please note: In some cases, it may be necessary to update server firmware before updating AIX to a new Technology Level. The AIX 5L Version 5.3 Release Notes for TL05 contain a table showing the minimum firmware levels required for AIX V5.3 on various servers given various usage scenarios. Release notes for Technology Levels other than TL05 can be found by following the appropriate link from the Release Notes Index for AIX 5.3 and Expansion Pack web page .
If frequent changes (to hardware, middleware, system management, and/or application software) are made to a Production environment, it can make sense to update AIX on a regular schedule rather than tying AIX updates to other changes. A new AIX Technology Level is released every six months. Service Packs are released more frequently than that. If updating AIX once a year is an appropriate target in a given environment, in most such environments it will work best to schedule that update in the summer (late July or early August) and to install the Technology Level which was released a few months earlier, along with a recent Service Pack. The second Technology Level released in a given year contains functional enhancements. Early adopters of new functional enhancements might instead want to schedule the yearly AIX update late in the year (or early the following year) and to install the second Technology Level released in a given year.
When contemplating updating AIX to a new Technology Level, select the Technology Level carefully.
Please note: In response to a request for the latest AIX maintenance, the IBM Support Center will sometimes send a customer every last PTF that has been dropped at the time the request is made. It seems prudent to avoid installing such latest maintenance, since if a problem is introduced by the latest maintenance, one must either (1) wait for a new fix to be developed or (2) somehow figure out which PTF is at fault and back it off. Customers should make sure they request and receive a CD containing an AIX Technology Level and Service Pack.
 | Important note
Software updates for optional software products (the C and C++ compilers, HACMP, etc) are not included in a Technology Level, nor are updates for Java. (PTFs for optional software products and Java will occasionally be found in a Technology Level or a Service Pack, but only when such PTFs are requisites for AIX PTFs in the Technology Level or Service Pack.)
- IBM C/C++ compiler
software maintenance
The C Set ++ Runtime is shipped on AIX installation media and installed along with base AIX. If the C Set ++ Runtime environment is used (by, for example, Oracle), current C Set ++ Runtime software updates should be installed when a new AIX Technology Level is installed.
There is no need for a compiler license to install and use a compiler's run-time environment. The compiler itself (required to compile source code) is a separately orderable licensed program product. An AIX license does not include a compiler license, so there is extra cost for a compiler license. *If a compiler is installed, current compiler software updates should be installed when a new AIX Technology Level is installed.
See the Latest updates for supported IBM C and C++ compilers web page to download the latest software updates for the C Set ++ Runtime environment and for supported IBM XL C Enterprise Edition for AIX and the IBM XL C/C++ Enterprise Edition for AIX compilers. For the latest C Set ++ Runtime environment software updates, follow the link from there to the IBM C+ Runtime Environment Components for AIX+. And for software updates for C/C++ compiler versions which are no longer supported, follow the link to Unsupported C and C+ compilers updates+. Installation instructions and prerequisite information are available on the web page from which the update is downloaded.
Note: The C Set ++ Runtime environment is delivered in fileset xlC.aix50.rte. AIX V5.3 installation media at the TL04 level delivers the xlC.aix50.rte fileset at the 6.0.0.13 level, for which IBM no longer provides support. As of 9/29/2007, the oldest supported level of the xlC.aix50.rte fileset which can be downloaded is 8.0.0.10.
If an IBM compiler is being used in conjuction with non-IBM software, consult the software vendor for recommended IBM compiler fileset levels. If, for example, the compiler will be used with Oracle 10gR2, consult Oracle MetaLink note 43208.1 for information about Oracle's current certifications of 10gR2 with IBM XL compilers.
A major component of the C Set ++ Runtime environment is the /usr/lpp/xlC/lib/libC.a shared library delivered by the xlC.aix50.rte fileset and linked to from /usr/lib/libC.a by the xlC.rte fileset. In case you are wondering, the /usr/lib/libc.a shared library is delivered by the bos.rte.libc fileset. AIX Technology Levels do include updates to the bos.rte.libc fileset.
- IBM's Java technology
software maintenance
If Java is used, current software updates for the Java version(s) should be installed when a new AIX Technology Level is installed. If Java is being used in conjuction with other software, consult the vendor of that software for recommended Java levels.
The Java version(s) installed on AIX can be identified with the commands lslpp -l | grep -i java and the default Java version can be identified with the java -fullversion command.
APAR numbers identifying currently available software update packages for a Java version can be obtained by pointing a browser at IBM's Java AIX Download and service information web page and clicking on the Fix Info link for the appropriate Java version.
Once an APAR number has been obtained, the software update package for that APAR can be downloaded as described below.
- Non-IBM multipath Fibre Channel device drivers
|
We've got a problem...
Now, let's assume you encounter what seems to be an AIX defect. The following questions are worth asking. If you call the IBM Support Center for help, the questions are likely to come up anyway.
- Is there a more recent AIX Technology Level available?
Point a browser at the Fix packs for AIX 5.3 operating system web page, select All 5.3 in the Select a Technology Level select box, and click for a list of all available Technology Levels and the date each was released.
- How do I install a more recent AIX Service Pack or Technology Level?
Point a browser at the Fix packs for AIX 5.3 operating system web page, select All 5.3 in the Select a Technology Level select box, and click , then follow the link to the desired Technology Level and Service Pack. Every Technology Level/Service Pack web page has an Installation instructions link (on the Package information tab) and a Download tab.  | Important note The IBM AIX 5L Operating System Service Strategy Details and Best Practices document (downloadable from the Service and support best practices for UNIX servers web page) says:
"Technology Levels Must be Applied as a Unit The rule has been changed that previously allowed applying individual updates/PTFs from a TL. The rule now says that installing a Technology Level is an "all or nothing" operation. Requisites are now added so the whole Technology Level is installed. Before applying a TL, you should always create a backup and plan on restoring that backup if you need to rollback to your previous level."
There are several ways to rollback to a previous AIX level.
When installing a Technology Level, use smitty update_all and leave the COMMIT software updates field set to yes and the SAVE replaced files field set to no (the default settings). And before installing a Technology Level, on the smitty update_all menu set the PREVIEW only? (install operation will NOT occur) field to yes to confirm that the entire installation will succeed. If the preview indicates that any part of the TL install will fail, resolve the issues before proceeding with the actual install. Among other things, the preview will list the filesets which will be affected by the TL (or, for that matter, any install or update operation you want to preview).
The PTFs in a Service Pack can be rejected, if necessary. If you want the option of later rejecting Service Pack updates when applying a Service Pack in SMIT, set the COMMIT software updates? field set to no and the SAVE replaced files? field set to yes when installing the Service Pack. (To be able to reject Service Pack updates, you must first install and commit a base Technology Level and then apply the desired Service Pack. The base Technology Level must, of course, be the TL within which the desired SP falls.) Before applying a Service Pack, all previous updates should be committed (fast path: smitty commit), so that if any PTFs must be rejected, the only updates rejected will be those from the Service Pack. To preview the commit operation (obtain a list of updates which will be committed without actually committing anything), set the PREVIEW only? (commit operation will NOT occur) field to yes. |
- How do I determine which filesets and PTFs (APAR fixes) are included in an AIX Service Pack or Technology Level?
Every Service Pack and Technology Level web page has an Complete list of updates link (on the Package information tab) which provides that information.
- I could swear I was running Technology Level 6 Service Pack 1 on my server/LPAR, but oslevel -s now shows 5300-04. What the heck is going on?
You probably installed a new fileset (one that had never been previously installed) from AIX V5.3 media at the TL04 level. Please note that if additional filesets or bundles are installed after a Technology Level has been installed, it is important to reinstall the Technology Level.
Hint: Use the command oslevel -sl 5300-06-01 to display a list of filesets which are below the level required by AIX TL06 SP01, for example.
Reinstall TL06 SP01 to bring the filesets up to the level required. If only a few filesets will be affected when a TL/SP combination is reinstalled, it is probably okay to apply the TL/SP without committing it, so that the new fileset updates can be rejected later if desired, contrary to the advice above not to reject a TL, which is applicable when a TL is installed for the first time.
- How can I view the description of an APAR (which sometimes includes a list of fileset levels exposed to the defect)?
- Point a browser at the Technical help database for AIX
web site
- enter the APAR's number (eg, IY89429) in the "Search AIX operating systems:" field
- select the APARs button in the "Limit search to selected documents:" section
- click

- click on the item for the desired APAR in the search results list (the results list will be short)
- How can I get a list showing the level(s) of the fileset(s) delivered by the fix for a particular APAR?
- View the description of the APAR (as described above) and see the PTF to Fileset Mapping section of the APAR description
- How can I download the fix for a particular APAR?
- View the description of the APAR (as described above) and click the Obtain fix for this APAR link on the APAR page which appears
 | Hint Before downloading the fix for an APAR, confirm that the APAR describes a defect in AIX V5.3, not some other version. The Reported release field in the APAR information section of the APAR text specifies the version to which an APAR applies. The Reported release field is 530 for AIX V5.3 APARs. |
 | Note If a PTF, Service Pack, or Technology Level is not yet available to fix an APAR, the Obtain fix for this APAR link will not appear, in which case it will be possible to subscribe to the APAR and receive a notification when a fix becomes available. Even if a fix is not yet available in a PTF, Service Pack, or Technology Level, the IBM Support Center might be able to provide an interim fix. |
 | Hint To download an APAR's PTF without any other fileset updates, uncheck the following boxes on the Packaging options web page (the page linked to by Obtain fix for this APAR):
- Include requisites
- Include fixes that correct regressions.
- Replace superseded fixes with the latest.
Note: These boxes will not appear when the APAR fix is delivered as part of a Service Pack or Technology Level. They will only appear when the APAR fix is a PTF.
Note: Unchecking these boxes may eliminate from the download list some fileset updates which you may later realize you need. You are least likely to need to replace superseded fixes, so it is generally fairly safe to uncheck that box.
Please note the Understanding terms link, which displays a page containing useful definitions:
- Requisites - Describes fixes which must be installed before the given fix can be installed.
- Regressions - Describes Fixes which have been found to have problems, sometimes referred to as PEs (PTFs in Error).
- Superseded - Describes fixes for which there is a newer level fix for the same fileset. PTFs are categorized as superseded to encourage users to update to a later level.
|
 | Hint To build a custom fix package, which eliminates from the download list some (but not all) fixes which won't be required given the set of filesets (and the fileset levels) installed on your AIX system, provide detailed system fix information (output of the lslpp -Lc command) on the Packaging options page. |
 | Fix Central changed in November 2007 The Fix Central changes supporting AIX Service Strategy document published in November 2007 says:
One of the first changes you will notice is that IBM is now promoting the full distribution of fix packs. Downloading packages for specific problems is being discontinued. A fix to a specific problem will be available in one or more fix packs. A fix pack can be either a Service Pack or a Technology Level package.
While it is not obvious how to download the fix for a particular APAR, it can be done if a PTF has been made available for the APAR. Going forward, fixes for many APARs will only be released as part of a Fix Pack. Individual PTFs will not be available for those APARs.
Note: A Fix search option has been added to the Fix type select box (one of the last selections which appears on the Fix Central main page). An APAR fix can be found as described above using the Fix search search engine, so it is possible to download the fix for a particular APAR from the Fix Central web site rather than the Technical help data base for AIX. |
- How can I obtain fixes without downloading them?
If you have an active software maintenance agreement (SWMA) for AIX , you can call the IBM Support Center (800-IBM-SERV, Option 2, then 3) and open a service request to ask that AIX fixes be shipped to you.
- How can I download an update for a particular AIX fileset?
Point a browser at the AIX FTP site at service.boulder.ibm.com. ( The site contents are best viewed by selecting Internet Explorer View -> Details.)
 | Note It appears that all fileset updates are not published on the AIX FTP site web site. For example, as of 11/19/2007, bos.perf.perfstat at the 5.3.0.61 level is not on the site. To confirm the existence of a particular fileset level, the Technical help database for AIX web site can be searched using the fileset name and level as search arguments, but when doing so, one may have to crawl through a bunch of false hits. |
Speaking the IBM Support Center's language, Part II
Here are some additional AIX fix terms which might be encountered when downloading maintenance:
- Critical
Describes fixes (PTFs) which are deemed very important.
- HIPER
Describes APARs (defects) which are high impact and pervasive - that is, have significant adverse consequences when encountered and are seen in many different environments.
- Pre-req/prereq
Pre-requisite or requisite. Describes a fix (PTF) which must be installed before a particular PTF can be installed. For example, U476294 is a pre-req of U480015, so U476294 must be installed before U480015 can be installed.
- Co-req/coreq
Co-requisite. Describes a fix (PTF) which must be installed along with a particular PTF. There is little difference between a co-requisite and a pre-requisite, except that pre-requisites must be installed first, whereas co-requisite order is not specified. For example, PTFs U408465 and U408470 are co-reqs of one another, so neither one can be installed without the other.
- If-req/ifreq
If-requisite. Describes a fix (PTF) which changes an optional fileset in some way required by the change to another fileset. For example, PTF U482686 (which updates fileset bos.diag.com) is an if-req of PTF U496235 (which updates fileset devices.scsi.disk.diag.com). So if optional fileset bos.diag.com is installed, then U482686 must be installed before U496235 can be installed.
- Regressions
Describes fixes which have been found to have problems, sometimes referred to as PEs (PTFs in Error).
- Superseded
Describes fixes for which there is a newer level fix for the same fileset. PTFs are categorized as superseded to encourage users to update to a later level.
- lslpp
(list licensed program products)
An AIX command which reports the status of filesets installed on the AIX system on which it is run. It reports the status of all software which is installed using the AIX installp command, including AIX filesets, filesets for IBM licensed program products (eg, HACMP and DB2), and non-IBM software, if the vendor has chosen to package and deliver the software in installp format.
Hint: The command rpm -qa lists Linux binaries which have been installed on AIX . Please note that the smitty install_software fast path can be used to install on AIX Linux binaries which are complied for System p and packaged in RPM format.
|
|