July 29, 2011

ANR2971E Database backup/restore/rollforward terminated -DB2 sqlcode -2071 error

OS Platform: AIX 6.1
TSM server Version:6.2.2.30

ANR2971E Database backup/restore/rollforward terminated -DB2 sqlcode -2071 error. This error comes when libApiTSM64 library file could not be loaded which resides in /usr/tivoli/tsm/client/api/bin64 directory. Reinstall the latest version of ba client and problem should be resolved. In my case I installed 6.2.2.3 which is the latest client version available on ftp site and problem got resolved. It could be because of incorrect installation or corrupt library file.

When reviewing db2diag.log file for diagnostic,  message "An error occurred while accessing the shared library" can confuse with tsm library sharing in case of library manager/client setup. SQL message here refers to the library file libApiTSM64.a in client api directory.

July 23, 2011

stale NFS handle ( GPFS filesystem backup using TSM )

GPFS mmbackup uses dsm.opt at default location i.e. /usr/tivoli/tsm/client/ba/bin in case of AIX if file is inside bin64 then we need to create symbolic link for dsm.opt to point at default location.
Or this problem could be because of a filesystem earlier mounted as NFS made local.




July 21, 2011

LTO3-LTO4 compatibility

I came across a situation where I was asked whether LTO3 media written on LTO4 drive is readable in LTO3 drive.
Answer is yes because LTO does not allow writing a different format on one media type.
I have seen this in one of my customer's environment where library is using a mix of LTO3 and LTO4 drives with all LTO3 media and there has never been any issue while restoring data using LTO3 drives ,doesn't matter whether data was written in LTO4 drive. On LTO3 media , LTO4 would be writing in LTO3 format only.
All LTO generations are backward compatible with one generation back for both read/write and two generation back for read only operations.

July 19, 2011

IBM Device driver Link

ftp://ftp.software.ibm.com/storage/devdrvr

tapeutil no more supported

Tapeutil utility has been replaced by ITDT( IBM Tape Diagnostic Tool) which is a separate utility and can be downloaded from IBM fix central or ftp site. IBM tape device driver (i.e. Atape.driver) comes with AIX filesets.

July 15, 2011

Empty primary storage pool tapes not converting into scratch tapes

Oops!! I found that the primary storage pool volumes which have been ejected from library using move media command are not returning into scratch status, these are showing as empty volumes in q vol command.

To resolve the problem I ran  move media on one of the volume and that deleted the volume from storage pool and now this can be used as scratch.
Command used for moving media:
move media <vol_name> stg=<stgpool_name> wherestate=mountablenotinlib

July 14, 2011

Moving TSM V6 database to new directories

TSM Server Version: 6.1
OS Platform: AIX 6.1

You can move TSM DB from one location to alternate location using following method:
  • Create new directories where you want to move DB.
  • Make sure DB instance user is the owner of these directories.
  • List these directories inside a file. List each directory on a single line.For  example below are the contents of file dbvol.txt
/tsmdb/db001
/tsmdb/db002
/tsmdb/db003
/tsmdb/db004
  • Halt TSM server
  • Remove tsm db using following command:
dsmserv  removedb TSMDB1 -i /home/tsminst1/tsminst1
  • Restore tsm db using following command:
dsmserv -i /home/tsminst1/tsminst1 restore db todate=today on=/home/tsminst1/dbvol.txt

where todate option can be used to restore database to any point in date and dbvol.txt is the file which contains db direcories location.Version 6 onwards require volume history file to restore tsm db.
Commands need to run by db instance owner id.

At last when restore db finishes following message comes.This means db restore is completed.
ANR0369I Stopping the database manager because of a server shutdown.

You may receive waning messages like below during db restore.These can be ignored.

ANR0498W Session < session no.>  refused for <node name> because restore DB is in progress.

July 10, 2011

Upgrading TSM 6.2.1.0 to 6.2.2.30

Procedure same as explained in article "Upgrading tsm 6.1 to the next maintenance level"

Upgrading from TSM 6.1 to TSM 6.2

Existing TSM version: 6.1.4.3
Target TSM version: 6.2.2.30
OS Platform: AIX 6.1 64 bit

Prerequisites:
  • DB2 requires IOCP to be enabled on AIX platforms. Enable this manually
    using "smitty iocp". If the problem persists contact a technical service
    representative.
  • DB2 instance user id and password should be ready before installation
  • /opt/tivoli/tsm must be having 2000 MB allocated
  • For installation root user id is required
  • remote user login should be enabled on system i.e ssh, test if yo can login using instance user id and password using ssh at default port.
  • Check if previous installation has been done using default values otherwise /etc/tivoli/tsm/instance.info may be required to be renamed. 
Procedure:
  • Download licensed version of TSM 6.2.1 from passport advantage site for customer.
  • Copy .bin file on to TSM server.
  • Extract contents by executing .bin file.
  • Issue ./install.bin -i console command inside the installation direcory.

December 3, 2010

Exchange VSS backups failing after cancelling a previous backup

TDP for exchange version:6.1.2.1
BA client version:6.2.1.0
TSM server version:6.1.4.2
Exchange OS Plateform Version:2008 R2

To resolve the problem, run a VSS backup on the storage group on which the backup was stopped. If the successive VSS backup fail with a backup-in-progress error, stop and restart the following services from the Services snap-in. This will clear the backup-in-progress state for the storage groups that are affected by this failure: Microsoft® Exchange Information Store service.
Microsoft Exchange Replication Service

November 28, 2010

Few usefull 3494 library commands


1. Query Library Data:
mtlib -l /dev/lmcp0/ -qL | more

2.Query Volume Count:
mtlib -l /dev/lmcp0/ -qK

3.Query Statistical Data:
mtlib -l /dev/lmcp0/ -qS

4.Query Volume Inventory:
mtlib -l /dev/lmcp0/ -qI

5.Query Expanded Device Information:
mtlib -l /dev/lmcp0/ -qD -E

6.Query Mounted volumes:
mtlib -l /dev/lmpc0/ -qM

7.Query Reserverd volumes:
mtlib -l /dev/lmcp0/ -qR

Upgrading tsm 6.1 to the next maintenance level

Initial TSM server version:6.1.3.2
Target TSM server version:6.1.4.2
OS Plateform version:AIX 6.1.4.0 64 bit

  • Download tsm server code from IBM ftp site:
  • Halt tsm server. Halting TSM server would automatically stop DB2 server.
  • If after stopping TSM server db2 still keep on running ( process db2sync ) then it can be stopped  using db2stop and If db2stop not working db2stop force can be tried or simply kill db2sync process.
  • I experienced for some reason in some cases halting tsm server does not kill dsmserv process immediately so need to kill dsmserv and then stop tsm db2 server as stated above.
  • Save the downloaded bin file inside a dir and extract code by executing ./6.1.4.2-TIV-TSMALL-AIX.bin.This will extract code inside the same directory.
  • Using root execute following command:
./install.bin -i console

Applying fixes would look similar to fresh tivoli storage manager installation except it would not include license component.

Installed component would include following:

TSM Server
DB2 9.5
TSM Devices
TSM Storage Agent.

November 27, 2010

TSM ba client 6.1.3.2 experiences slow backup performance

TSM ba client version:6.1.3.2
OS plateform version:Windows 2008 R2

I noticed that clients which are at level 6.1.3.2 they backup slow than those which are at 6.2.0.1.
I upgraded clients to version 6.2.0.1 and found improvements in backup performance.
I noticed that clients which are running on windows 2008 R2 with exchange 2010 configured and running are specially backing up slow.

I am planning to upgrade them to the latest fix available at the time which is 6.2.1.3 as taking them upto 6.2.1.0 has not really helped in their filesystem backup performance.

Expire inventory hangs on TSM 6.1.3.2.

TSM Server Version:6.1.3.2
OS Plateform Version:AIX 6.1 64 bit

I recently faced that expire inventory runs very slow on tsm 6.1.3.2 and hangs on some particular node's data expiration. I noticed hanging on some nodes was random in nature. I was expecting windows 2008 system state the culprit and responsible for this mess but I found that it was hanging on some AIX nodes as well. I openned a PMR and was advised by IBM support to upgrade server to the maintenance level 6.1.4.2 as TSM 6.1 expire inventory hangs while processing some grouped objects like windows system state.
After applying this fix expiration inventory hang problem has been eliminated but expiration process was still running very slow which was creating locks on some threads restarting and slowing down some client backup sessions. Further from IBM support I have been given a cumulative fix for this problem which I am due to apply.
During trobleshooting and analysing the problem I ran instrumentation on tsm server and found that "DB2 MFtch Exec" is eating most of the time of a client backup session.
It has been identified as a db2 bug.
Once I apply this fix I'll update the results on this post.

 Applying thix efix has resolved expiration problem completely.

Related APARS:
APAR IC69729 - V6.1 AND V6.2 SERVER CAN EXPERIENCE SLOW EXPIRATION
PROCESSING OF GROUPED OBJECTS 6.1.5 fixhttp://www-01.ibm.com/support/docview.wss?uid=swg1IC69729

APAR IC69156 - V6.1 SERVER CAN EXPERIENCE SLOW EXPIRATION PROCESSING
ON GROUPED OBJECTS fix at 6.1.4
http://www-01.ibm.com/support/docview.wss?uid=swg1IC69156

APAR IC70269 - PERFORMANCE PROBLEMS ON LEVEL 6.1.4.0 OF THE TIVOLI
STORAGE MANAGER SERVER fix 6.1.4.1
http://www-01.ibm.com/support/docview.wss?uid=swg1IC70269

Applying efix and scheduling undocumented expiration command listed below has resolved slow expiration problem completely.Before applying efix I was advised to upgrade tsm server on to the level 6.1.4.3 since fix was built on this code.I also splited TSM DB on 4 separate disks to eliminate disk bottleneck as earlier TSMDB was configured on only one disk and whenever any high I/O intensive process was running it was making this disk 100% utilized.Spliting DB on 4 disks allowed higher I/O operations balacing them on all four disks.
Do not forget to configre tsm db disks into RAID 5 ( recommended raid level for tsm db) as we can't mirror tsb db any more starting with version 6.1.

Expiration command: expi inv dur=600 sk=y restart=no

Information on how to move TSM DB to alternate location can be found in article "Moving TSM V6 database to new directories".


November 23, 2010

Configuring TDP for Exchange 2010 Full VSS Backups

Exchange Server OS Plateform: Windows 2008 R2
TDP for Exchange version: 6.1.2.1
TSM ba client version: 6.2.1.0

TDP for exchange 6.1.2.1 needs ba client version 6.2.1.0 to be installed.
For detailed system requirements for TDP for exchange 6.1 click on below link.

Detailed Documentation available on following link:

Additionnal Software Requirement:
Basic VSS operations require one of the following applications:
–IBM Tivoli Storage Manager Microsoft Exchange VSS Integration Module Version 5.5.2 or later
– IBM Tivoli Storage FlashCopy Manager Version 2.1 or later
If backups are stored directly on Tivoli Storage Manager Server, these applications are not required.
Written procedure is tested on windows 2008 r2 exchange 2010 stand alone environment with
tdp 6.1.2.1 and ba client version 6.2.1.0 backing up directly to TSM server. It assumes default client installation location.

Procedure:
  • Download Windows 64 bit TSM ba client 6.2.1.0 from IBM ftp site and install it. 
  • Register ba client node on TSM server and configure it's backup.
  • Configure CAD and scheduler services for ba client running below commands:
dsmcutil install cad /name:"TSM Client Acceptor" /password:xxxxxxx /cadschedname:"TSM Scheduler Service" /httpport:1581 /autostart:yes /startnow:yes

dsmcutil install /name:"TSM Scheduler Service" /password:tsmadmin /autostart:no /startnow:no
  • Configure WEB Client services using below command:
dsmcutil install remoteagent /name:"TSM Remote Client Agent" /node:xxxxxxx /password:xxxxxx /optfile:dsm.opt" /partnername:"TSM Client Acceptor"
  • Download TDP for Exchange 6.1 from passport advantage and install it. Download TDP for exchange patch 6.1.2.1 from IBM ftp site and apply it on installed base software.
  • Resister Exchange TDP node in TSM server specifically useid=none not configured.
  • Grant proxy node authority to ba client running following command :
grant proxynode target=<exchange tdp node name> agent=<exchange ba client node name>
  • Configure dsm.opt for exchange with following parameters:
NODename          <Exchange node name>
CLUSTERnode       no
COMPRESSIon       Off
PASSWORDAccess    Generate
COMMMethod        TCPip
TCPPort                   1500
TCPServeraddress  xx.xx.xx.xx
TCPWindowsize     63
TCPBuffSize       32
SCHEDMODE             Polling
SCHEDLOGRetention     15,d
ERRORLOGRretention    15,d 
SCHEDLOGNAME        C:\Program Files\Tivoli\TSM\TDPExchange\tdpsched.log
ERRORLOGNAME        C:\Program Files\Tivoli\TSM\TDPExchange\tdperror.log 
  • Configure tdpexc.cfg for exchange with following parameters:
logprun 14
vsspolicy * * full tsm <mgmt_class_name>
localdsmagentnode <ba client  node name>
storemailboxinfo no
STOREMAILBOXINFO NO option would enable fast backup since exchange 2010 takes a lot of time in saving mailbox history file.If this option is selected then at the time of restoration of moved mail box user needs to provide it's location.
  • Edit excfull.smp sample file as per your environment and save it as excfull.cmd.
  • Define a client schedule on tsm server using below sample command:
def sched <domain name>  <sched name> action=command obj="c:\progra~1\tivoli\tsm\tdpexchange\tdpexc.cmd" startt=hh:mm:ss priority=2 duration=2 duru=h period=1 perunit=day dayofweek=any
  • Associate TDP node with this schedule
def assoc <domain name> <sched name> <node_name1>
  • Configure TSM TDP Scheduler Service running following command:
    dsmcutil install /name:"TSM TDP Scheduler Service" /node:xxxxxxx /password:xxxxxxx /optfile:"c:\program files\tivoli\tsm\tdpexchange\dsm.opt" /autostart:yes /startnow:yes

    User running TSM TDP Scheduler services should have following rights:

    1) Member of AD's Organization Management group.
    2) Local admin

    November 22, 2010

    TSM ba clients miss scheduled backups ( servers behind firewall)

    TSM ba client version=6.2.1.0
    Client OS version=Windows 2008 R2

    Recently I faced this problem of tsm clients behind firewall miss their scheduled backups.I deleted  CAD and configured tsm scheduler directly to poll tsm server.This resolved the problem.

    Probable reason could be that when TSM server contacts tsm client behind firewall it fails to contact tsm client and hence don't get any reply.
    I also noticed that client was not listening at default port 1501 while it was listening on server comm port 1500.
    If any one struggling with this kind of issue they can try this workaround.

    ANS0106E Message index not found for message 14085

    TSM Cleint Version=6.2.1.0
    OS version=Windows 2008 R2

    ANS0106E Message index not found for message 14085

    This error comes because dscenu.txt file gets corrupted.This could be because of the upgradation of client while scheduler service running.Uninstalling and reinstalling ba client can resolve the problem or the very simple fix of this problem is to copy the file dscenu.txt from other client running on same level.This file resides inside ba client dir ( C:\Program Files\Tivoli\TSM\Baclient).