November 27, 2010

Expire inventory hangs on TSM 6.1.3.2.

TSM Server Version:6.1.3.2
OS Plateform Version:AIX 6.1 64 bit

I recently faced that expire inventory runs very slow on tsm 6.1.3.2 and hangs on some particular node's data expiration. I noticed hanging on some nodes was random in nature. I was expecting windows 2008 system state the culprit and responsible for this mess but I found that it was hanging on some AIX nodes as well. I openned a PMR and was advised by IBM support to upgrade server to the maintenance level 6.1.4.2 as TSM 6.1 expire inventory hangs while processing some grouped objects like windows system state.
After applying this fix expiration inventory hang problem has been eliminated but expiration process was still running very slow which was creating locks on some threads restarting and slowing down some client backup sessions. Further from IBM support I have been given a cumulative fix for this problem which I am due to apply.
During trobleshooting and analysing the problem I ran instrumentation on tsm server and found that "DB2 MFtch Exec" is eating most of the time of a client backup session.
It has been identified as a db2 bug.
Once I apply this fix I'll update the results on this post.

 Applying thix efix has resolved expiration problem completely.

Related APARS:
APAR IC69729 - V6.1 AND V6.2 SERVER CAN EXPERIENCE SLOW EXPIRATION
PROCESSING OF GROUPED OBJECTS 6.1.5 fixhttp://www-01.ibm.com/support/docview.wss?uid=swg1IC69729

APAR IC69156 - V6.1 SERVER CAN EXPERIENCE SLOW EXPIRATION PROCESSING
ON GROUPED OBJECTS fix at 6.1.4
http://www-01.ibm.com/support/docview.wss?uid=swg1IC69156

APAR IC70269 - PERFORMANCE PROBLEMS ON LEVEL 6.1.4.0 OF THE TIVOLI
STORAGE MANAGER SERVER fix 6.1.4.1
http://www-01.ibm.com/support/docview.wss?uid=swg1IC70269

Applying efix and scheduling undocumented expiration command listed below has resolved slow expiration problem completely.Before applying efix I was advised to upgrade tsm server on to the level 6.1.4.3 since fix was built on this code.I also splited TSM DB on 4 separate disks to eliminate disk bottleneck as earlier TSMDB was configured on only one disk and whenever any high I/O intensive process was running it was making this disk 100% utilized.Spliting DB on 4 disks allowed higher I/O operations balacing them on all four disks.
Do not forget to configre tsm db disks into RAID 5 ( recommended raid level for tsm db) as we can't mirror tsb db any more starting with version 6.1.

Expiration command: expi inv dur=600 sk=y restart=no

Information on how to move TSM DB to alternate location can be found in article "Moving TSM V6 database to new directories".


No comments:

Post a Comment