Data Protector Migration Unix to other OS

Are there many good reasons to change from Data Protector on Unix to a different operating system? I do not know many… However, there are customers required to do this step and to migrate to Windows or Linux; often combined with a upgrade to the current version of Data Protector. And here the story starts: there is no path for migration available. The reason is simple, the DCBF folders are not compatible between Unix and Windows nor is there a chance to do an export and import of the internal database. The DCBF folders on HP-UX are stored in big-endian format and on Linux in little-endian format and there is no tool for translation available. What the difference is can be search at your favorite search engine.

The article is somehow larger than usual, you’ll need 30 minutes to work through, the article is addressed to experts. At the end of this article you’ll find the link to the export and import scripts.

Because of the problems described above the migration from HP-UX Cell to windows is limited to copy the folder /etc/opt/omni (backup jobs, schedules, …). To allow a easy migration for the rest of your Data Protector environment I show the steps I did for migration from HP-UX 11.11 – DP 6.11 to Windows 2008 R2 – DP 6.2. The most important part during the migration is to import your old media into the new cell. I have customers with 500 and up to 2000 media to be imported into the new environment. With the import of your media only you recreate your history (restore chain), as during the import the DCBF folders will fill up and the filename versions are imported. Many customers do eschew to import media as they do not know that a command is available since Data Protector 6.11 was introduced. This command allows you to export the catalog to a file.

In this article the migration is shown for a migration from HP-UX 11.23 to Windows 2008 R2;  the steps enable you to do a successful migration. The documented steps are one of many ways to do a migration. Please refer to the requirements at the end of this article otherwise your migration might fail.

During the migration following major steps were done:

1 Installation of the new cell server on Windows 2008 R2

2 Library / Drive:

2.1 physical move to the new cell server (SCSI Library)

2.2 Zoning changes for the new cell server (FC Library)

3 Configuration of the new cell server:

3.1 global,

3.2 omnirc,

3.3 Additional DCBF folders, i.e.:

omnidbutil -add_dcdir "D:\ProgramData\OmniBack\db40\dcbf1" -seq 1

omnidbutil -add_dcdir "D:\ProgramData\OmniBack\db40\dcbf2" -seq 2

3.4 Additional Filenames Extension Files, i.e.: (repeating this step adds more extensions)

omnidbutil -extendfnames "D:\ProgramData\OmniBack\db40\datafiles\cdb" -maxsize 2047

3.5 additional optimizations as needed (i.e.: Archiving – see article with obrindex.dat)

4 Deactivate the schedule on the old cell server using the command /opt/omni/sbin/omnitrig -stop (this will remove the trigger from crontab)

5 Export of the clients off the old cell using the command omnicc -export_host <Hostname>, use the same name as found in cell_info file. To automate this step you could "cat" the cell_info file and print the hostname using awk, the output can be used within a shell script. Important: do not forget to save the output as you need the host names for the import process. Example for the export:

MYHOSTS=$(cat /etc/opt/omni/config/server/cell/cell_info | awk '{print $2}')

for EXPORTHOST in ${MYHOSTS}

do

/opt/omni/bin/omnicc -export_host ${EXPORTHOST}

done

6 If you own additional installation servers and you need the servers in the new cell, export these installation servers with omnicc -export_is <hostname> (save the output for the import process).

7 Import the clients into the new cell, the host names are still known from the export. The used command is omnicc -import_host <Hostname>, for the import the same rules apply as during the export.

8 Import the additional installation servers into the new cell with the command omnicc -import_is <hostname>. If you need to upgrade your installation server, the installation is done manually and locally on the system. Important: Do not forget to apply the patches if there are any. If you need a new installation server in the new cell for distribution of Linux/Unix clients, a RedHat or SLES (64 Bit) system is installed very fast in VMware, the installation of Data Protector is done within minutes.

9 Upgrade your clients to the current Data Protector version using the GUI distribution mechanism (Important: If you still use Windows 2000, do not try to upgrade these clients and keep the old version as DP 6.2 does not support Windows 2000 clients any longer; however staying with the old version is supported for all the functions from the old version. You might need to have a DP 6.11 GUI for the import of your Windows 2000 clients into the new cell.). Please keep in mind that it might be necessary to create a SSH Key on the installation server and to distribute the key to your Linux/Unix clients.

10 Creation of pools und devices on the new cell server:

10.1 In small enviroments the libraries, drives and pools can be created by hand. In this case you could eliminate dozens of “historical” pools, in midsize enviroments 10 pools are enough and may fit all your needs. The same is true for drives, you might want to change from blocksize 64k to the higher value 256k or 512k (depending on the used controller and drives).

10.2 Delete all unneeded pools per script, in this case all pools except "Default File" and "Default LTO-Ultrium" are deleted:

omnimm -remove_pool "Default AIT"

omnimm -remove_pool "Default DDS"

omnimm -remove_pool "Default DLT"

omnimm -remove_pool "Default DTF"

omnimm -remove_pool "Default Exabyte"

omnimm -remove_pool "Default Optical"

omnimm -remove_pool "Default QIC"

omnimm -remove_pool "Default SAIT"

omnimm -remove_pool "Default SD-3"

omnimm -remove_pool "Default SuperDLT"

omnimm -remove_pool "Default T10000"

omnimm -remove_pool "Default T3480/T4890/T9490"

omnimm -remove_pool "Default T3590"

omnimm -remove_pool "Default T3592"

omnimm -remove_pool "Default T9840"

omnimm -remove_pool "Default T9940"

omnimm -remove_pool "Default Tape"

11 Create new pools per command, i.e.:

omnimm -create_free_pool "FreePool" "LTO-Ultrium" 60 250

omnimm -create_pool "Filesystem" "LTO-Ultrium" -free_pool "FreePool"

omnimm -create_pool "Database" "LTO-Ultrium" -free_pool "FreePool"

omnimm -create_pool "DataProtectorInternalDatabase" "LTO-Ultrium" -no_free_pool -no_move_free_media

12 Create libraries and drives:

12.1 Use the “Auto configure” function within Data Protector.

12.2 Use the tool sanconf, documentation can be found in the Client Reference Guide.

12.3 Download your libraries and drives using the command omnidownload on the old cell server, modify the files (change the cell servername), copy the files to the new cell server, create the librares and drives using the command omniupload on the new cell server.

12.4 Use the tool savedevices (search for it at www.data-protector.org), current version can be used for migrations from Windows to Windows only, for Unix/Linux it must be modified.

12.5 As a last way export the mmdb on the old cell server and import it on the new cell server. At the end delete all media (as you import it with the scripts) and change the cell name, as the device will be created with the old cell server name. You might use the following commands:

omnidbutil -writedb -mmdb /<Export folder> – transfer the flder to the new cell server, i.e. to c:\temp.

omnidbutil -readdb -mmdb c:\temp

omnidbutil -change_cell_name [<old cell server>]

13 Copy the stuff you need from /etc/opt/omni to the new cell server, if possible apply the unix2dos tool on the files to change the files from unix format to dos format. Common folders could be:

13.1 barlists

13.2 barschedules

13.3 datalists

13.4 schedules

13.5 copylists

13.6 consolidationlists

13.7 integ

From userlist migrate your own users only. The first 3-4 lines on the new cell server were created during the installation, overwriting this file might lead to “denied access” messages on the new cell server or even worse the services might not start anymore.

So, the preparing work has been done and the new cell server is ready, next step is to get the catalog information from the old media and the import of these media into the new cell server. At the top of this article I already said that no media needs to be moved physically as with Data Protector 6.11 a command was introduced for that. The commands I used in my scripts, the exportcatalog.pl and the importcatalog.pl Perl script; on demand modify it according your requirements. To get these scripts working, I again refer to the requirements at the end of this article. During the export of the catalog on the Unix system the Data Protector daemons are restarted after each export. The reason is the high memory consumption of the RDS daemon, which might crash the RDS daemon. If you do not need that simple comment it out. In addition in file .omnirc I added the parameter _M_ARENA with value 1:32 and in file rdmserver.ini I limited the threads to 32. If you need additional information on these settings drop a comment or send me a mail.

The scripts:

Export:

1 The script is started using the command perl exportcatalog.pl, you do not need to install perl, the perl modules from Data Protector should be sufficient.

2 The script was written to be used on Unix using perl and system calls. On request (donation?) it will be written as shell script.

3 exportcatalog.pl:

# define the pools on source cell server
# all the pools must exist before starting the script
# it is recommended to move 100-200 pools to the transfer pool
# once all media are exported, continue with next 100-200 media

# in this pool we search for our media
my $pool="_TransferPool";


# on error we move the media to failed pool
my $errorpool="_TransferPoolError"; 


# once the media is exported it will be moved to a new pool
my $successpool="Z_MediaExported"; 


# define the paths we need during the export when no search path is set
my $omni_bin="/opt/omni/bin/";
my $omni_sbin="/opt/omni/sbin/";


# the directory to store the mcf files, a separate mount point was created for the migration
my $output="/exportcatalog"; 


# other stuff we need
my $command;
my @media;
my $media;
my $mediumid;
my $result;


# this will return the media ID - [XX1234L2] for all media in the transfer pool
$command=$omni_bin."omnimm -list_pool ".$pool." \| awk \'\{print \$2\}\'";
@media=`$command`;


# nextlines will do the export
foreach $media (@media)
{
  # next 3 lines to restart DP in case we have a memory problem with RDS (this makes the export faster)
  $command=$omni_sbin."omnisv -stop"; `$command`;
  $command=$omni_sbin."omnisv -start"; `$command`;
  $command=$omni_sbin."omnitrig -stop"; `$command`;

  chomp $media;
  # remove the brackets from the media ID - [XX1234L2] --> XX1234L2
  $media=~ s/\[//ig;
  $media=~ s/\]//ig;
  
  # we only run if we have a media
  if (($media ne "Medium") && ($media ne ""))
  {
    print "working on media: ".$media."\n";
    $command=$omni_bin."omnimm -copy_to_mcf ".$media." -output_directory ".$output." > ".$output."/".$media.".txt";
    `$command`;
    $command="cat ".$output."/".$media.".txt \| grep \'1 out of 1\'";
    $result=`$command`;
    if (!($result =~ /1 out of 1 catalog\(s\) successfully copied to file\(s\)./))
    {
      print "ERROR: export of media failed - moving to error pool\n\n";
      $command=$omni_bin."omnimm -move_medium ".$media." ".$errorpool;
      `$command`;
    }
    else
    {
      print "SUCCESS: media exported - moving to success pool\n";
      $command=$omni_bin."omnimm -move_medium ".$media." ".$successpool;
      `$command`;
      $command="cat ".$output."/".$media.".txt \| grep \' ('";
      $result=`$command`;
      ($dummy, $mediumid)=split(/\(/,$result,2);
      $mediumid=~ s/\)//ig;
      $mediumid=~ s/\.//ig;
      $mediumid=~ s/\:/_/ig;
      chomp $mediumid;
      print "renaming mcf file ".$mediumid.".mcf to done_".$mediumid.".mcf\n\n";
      $command="mv ".$output."/".$mediumid.".mcf ".$output."/done_".$mediumid.".mcf";
      `$command`;
      ### you may want to ftp the files afterwards to a different server when not picked up from the UNIX server
      $command="chmod 777 ".$output."/done_".$mediumid.".mcf";
      `$command`;
    }
  }
}

Import:

1 The script is started using the command perl importcatalog.pl, you do not need to install perl, the perl modules from Data Protector should be sufficient.

2 To have a ongong import of exported media you could add a scheduled task on the new cell server till the migration has been finished.

3 All other stuff is documented in the script.

4 importcatalog.pl:

# the same pool we used during the export, all media will be imported here
my $pool="_TransferPool";


# on the UNIX server a samba shared was used to export the catalog
# so we use the share to pickup the mcf files
# when samba cannot be configured you could enhance the exportcatalog.pl to upload
# the mcf files to another server using i.e. FTP
# my $source="Z:\\Migration\\exportedmedia\\";
# aternate source (line above) should be used when Samba is not used
my $source="\\\\unixserver\\exportcatalog\\";


# our working directory and the directory when the import is done
my $destination1="Z:\\Migration\\working\\";
my $destination2="Z:\\Migration\\done";


# other stuff we need
my $command;
my @media;
my $media;
my $mediumid;
my $result;


# we copy (no movement before import was successful) the mcf file to the working directory 
$command="copy ".$source."done*.mcf ".$destination1; `$command`;

# we build the array of files to be imported
$command="dir ".$destination1."done*.mcf /o/b"; @media=`$command`;


# next lines will do the work, on error the mcf file stays in the working directory until manually moved
foreach $media (@media)
{
  chomp $media;
  print "working on media: ".$media."\n";
  $command = "omnimm -import_from_mcf ".$destination1."".$media." -no_pool_prefix -orig_pool -import_as_original";
  $result=`$command`;
  if (!($result =~ /1 out of 1 media successfully imported./))
  {
    print "ERROR: import of media failed - renaming media\n";
    $command="ren ".$destination1."".$media." ERROR_".$media;
    `$command`;
  }
  else
  {
    print "SUCCESS: media imported - moving media to another location\n";
    $command="move ".$destination1."".$media." ".$destination2; `$command`;
    $command="del /q ".$source."".$media; `$command`;
  }
}

The export / import of the media is done automatically using the scripts, no media changed necessary, the migration will run besides your daily work with minimal todo. In mixed environments with 500 media to be imported this part of the migration will take up to 1 week. The scripts can be downloaded here:

[wpdm_file id=8]

Requirements / Prerequisites:

1 The old cell server stays alive during the migration and till all media are imported.

2 The cell server name and the IP address are kept on the old cell server. It would be possible to change that on the old cell server, however in this case you’ll need emergency licenses or EVAL licenses – please ask yor HP partner, HP Partner Account Manager or HP Sales/Presales.

3 The licenses are migrated to the new IP address using HP webware site.

4 The licenses will be imported on the new cell server when the export of the media on the old cell server has been finished – till this time the new cell server stays with the 60 days, so called Instant On Licenses.

5 Once the licenses are migrated to the new cell server the old cell server must be deactivated otherwise you violate the license agreement.

6 On the old cell server current patches are installed, otherwise the export of the media might fail. Information: withhin the patches from may 2011 a problem with option “Log Files, Log Directories, Log Folder, Log All” was fixed when exporting and importing media.

7 In file global the variable EnableMCFCompression=1 is activated. The variable will be used for the copy catalog session, the generated catalog file will be compressed.

8 There is sufficient disk space available for the migration.

9 The media are labled with barcode lable, if not using barcode you need to modify the script to get the right medium ID.

10 On the old and the new cell server perl must be installed, a simple check with perl -v shows the installed version.

This great document can be found at www.data-protector.org only, so to all consultants and customers: if this article was helpful for you and you are able to profit in your projects, I would be glad when you honor this work with a donation (donation button on the upper right site).