Administrative Domains on IBMi – PowerHA

I would like to share my experience from implementing and configuring the feature of PowerHA – The Administrative Domains. This functionality allows synchronization of specific objects stored on the SYSBAS. It should be used in conjunction with an iASP based solution. Thus, if you don’t use the iASP, you will not find the Administrative Domain useful.

An iASP can bring a lot of benefits, especially for high availability solutions, but it also bring some pain. Very painful is the implementation. But once it done, and all application setup is adjusted, it is usually very comfortable and only small overhead is needed for administration. Nevertheless, not all objects can be stored on the iASP. Exactly for these ‘non iASP’ objects the Administrative domain is designed.

These are requirements for the Admin domain:

• PowerHA license (5770HAS )
• already configured the cluster
• QRESVRSEC system value must be set to 1. It is good idea to set the value few months before the implementation.

The following objects can be handled by Admin Domain starting with OS V7R1:

• NWS configurations (*NWSCFG)
• NWSH device descriptions (*NWSHDEV)
• NWS storage spaces (*NWSSTG)
• Tape device descriptions (*TAPDEV)
• Optical device descriptions (*OPTDEV)
• Ethernet line descriptions (*ETHLIN)
• Token-ring line descriptions (*TRNLIN)
• Printer devices
• Authorization lists
• User profiles (*USRPRF)
• Class (*CLS)
• Job description (*JOBD)
• ASP device description (*ASPDEV)
• System values (*SYSVAL)
• Network attributes (*NETA)
• Environment variables (*ENVVAR)
• TCP/IP attributes (*TCPA)
• Subsystem descriptions (*SBSD)
• Network server descriptions (*NWSD) of types *WINDOWSNT, *IXSVR, and *ISCSI.

As you can see, only part of the objects can be ‘handled’ by the Admin Domain. I use word handled rather then replicated, because the Admin Domain does not fully replicate the objects (I don’t know how IBM does it exactly). In my opinion, the Admin Domain retrieves an object on one system, and create it on another node which is part of the cluster, and the Admin Domain configuration. The logic is different then software replication solutions which could replicate the entire object either via Save/Restore command or read/analyze journals entries and apply process on the other node. (I write about consequences from this approach below)

Anyway, only objects listed above can be handled by the Admin Domain, so IBM assumed that all other objects will be stored and ‘replicated’ within the iASP.

Every object which supposed to be replicated and synchronized must be added to the ‘monitored resources’ via command ADDCADMRE.

This is Peer to Peer replication – there is no source and target system. Any change on the monitored object will be propagated among the nodes regardless where it was initiated. For instance, if an administrator add his user profile with *ALL attributes to be monitored, the password will be always synchronized between the systems regardless where it was changed.

It is possible to chose what changes should be synchronized (CHGCAD) between the nodes. Either (*LASTCHG) the last change that was made before the node joined the cluster, or (*ACTDMN) only changes made on active nodes in an active cluster. The second can be used, if we don’t want to propagate the changes made on the DR during maintenance.

In my opinion every implementation of the Admin Domain must handle user profiles. It is extremely expensive to run 3rd party replication software on the iASP system to synchronize mainly user profiles. That is when the Admin Domain kicks in. But there are challenges with handling user profiles via Admin Domain.

As I mentioned before, every object must be entered to the monitored resources. If you create a new user profile via CRTUSRPRF immediately after you need to enter this profile to the Admin Domain. So, you need to initiate another command. The best is to use have a CL program and attach it into the exit program QIBM_QSY_CRT_PROFILE.

The bigger problem is deletion of user profiles/objects. Because the Admin Domain does not delete objects! If you delete an object on system A, it will remain on system B, and the Administrative Domain reports it as a failed object. Therefore, it is a little tricky to handle delete operations. Leaving an obsolete *JOBD on the DR node (for some time) is not so harm, as leaving an *USRPRF. One of the ideas for handling user profiles might look following:  add a CL to the exit program for profile deletion. Whenever the profile is deleted on a system, the program will change the profile status to “disabled” and text field to “To be deleted” on the other nodes in the cluster. It would prevent from logging in to the DR node. Furthermore, there must be some automatic process (once a day) which will delete all profiles with status “To be deleted”.

With the Admin Domain implementation you have to add also all users to the monitored resources. The tool set does not offer any command to add generic users or *ALL. So, again the best is to develop some CL and add users in batch. BUT REMEMBER – you probably do not want to add all users to the Admin Domain. Some users must stay as system specific, and they can’t have the same parameters against all systems. Most likely user profiles used by some tools must be system unique.

IBM Lab Services knows about these issues, and ‘solution gaps’. Therefore, they developed sets of programs within QZRDPWRHA library (extra charged toolkit). Within this toolkit, you can find multiple commands which would help with implementation and maintenance of the Admin Domain. For instance, there is a command (DLYPFRCLU) which handles deleting user profiles on all nodes via a cluster. IMO, the tools are very handy. And probably without this tool set, my decision about implementation of the Admin Domain would be more difficult.

Do not forget about monitoring of “monitored resource entry (MRE) “- objects handled by the Admin Domain. It is a good practice to be notified if an object gets the status different then “Consistent”. You can do it via command PRTCADMRE.

Another important hint – if your HA solution is build from more then two nodes. For instance, the production node, the DR, and the Backup Node (Flashcopy for taking offline backups) the Admin domain can be created only between Prod and DR. You don’t need to get sysbase objects synchronized on the backup node. In terms of profiles, you do not need them either. Since V6R1 if a profile is missing and it owns an object on the iASP. The Vary on process will create the profile with status *DISABLED, and following text in the description field “CREATED BY AUTO-CONFIGURATION”.

At the begging of this post I wrote about some consequences coming from the fact that the object are retrieved and created rather then save and restore. And this is the biggest disadvantage of the Admin Domain, and I haven’t found easy solution for it. The Admin Domain does not handle/synchronize authorities on the objects!

While you can still deal with Authorization Lists to handle the authorities on JOBD or SBSD, I didn’t find the way how to handle authorities on the user profiles.

The picture illustrates an adjusted authority on a user profile.

User BARTOL has *USE authorities for profile BART. Such configuration will be NEVER synchronized across nodes by the Admin Domain. And this can not be handled by the AUTL.

This limitation is not mentioned in any IBM materials which I was working with. And It may be a serious limitation  for some implementations.

Update September 22 2017.

We used QZRDPWRHA/ADDCADUSR program to load all user profiles to the Admin Domain. The program did not finish and stuck in DLTOVR status. In the job log MSGID CPF2499 was listed. Actually this is related with a bug in command ADDCADMRE and an user profile where password was set with QRETSVRSEC 0. If you try to add such profile interactively, a error message will be displayed. If you try to add the user profile in a batch process, it goes to DLTOVR status due to wrong message handling.

I’ve been told that IBM is aware about the problem and will fix it in next release. For you it means that you can’t try to add an user profile with QRETSVRSEC 0 because the entire process fails.

In summary. The Administrative Domain is a basic solution to handle objects stored on the SYSBAS. It has very basic functionality, and it requires dozen of different CLs in order to automate it ( I highly recommend QZRDPWRHA toolset). From the other hand, it is for free (Power HA license), and certainly saves money needed for any other 3rd party replication product.

‘