The need, to backup the data from your servers to media and thus to maintain data security does not stop when backed up to tape, there is often a request to protect the tapes from further appending or overwriting or to move the media off the tape library and to store it on a safe place (safe, backup computer center, …). Each Data Protector admin knows this specific situation, finding out which media was used in the last full backup for vaulting. The experienced admin will configure Data Protector to send a report or use many different pools to keep the daily backup, weekly backup and monthly backup apart. Problem: The media sets often should then be moved to a own vaulting pool before they are taken off the library.
Solution: Just for this problem I wrote a small Perl script. As always, I write my scripts in Perl, as this language is very flexible for such simple tasks. The script does not need a full Perl installation, the stripped down version, which is included within the Data Protector installation on the Cell Server is sufficient. Functional way, in short: It searches for sessions in a specific period, analyzes the media and then the media will be moved to a vaulting pool.
How it works: The script searches for sessions of the time frames specified in the script. For the sessions the used media are detected. The list of media is now cleared of any existing duplicates (one media may have to be shared by multiple backup jobs) and for each media the pool is detected. If the the pool where the media belongs to should be excluded (configured inside the script), then the respective media is skipped, otherwise, the media is moved into the configured destination pool. If you need a bit more complex version it is possible more than one copy of the script with different values for the variables set and run. This also applies to different types of media (LTO and what else is there on the market…). At the end of the script an email with the information which tapes were moved to the destination pool is sent. Who does not like this feature can comment out the call to the binary… If you like it you need to get the tool “Blat” and to configure it – Link: http://www.blat.net. For simplicity, I have everything (the script, the batch file to call the script and the Blat binaries) in the bin directory of Data Protector, but even that can be changed. It could also have an extension where the media are moved into the mail slots (5 more lines of code). This of course makes sense only if the library has configured mail slots and the number of media to be moved offsite is less than or equal to the amount of mail slots in place.
Use case: You have configured the pools FilesystemWeekend, FilesystemDaily (for the file backup), DatabasesWeekend, DatabasesDaily (for database backup), DPDatabase, Vaulting und FreePool. On each monday morning the media which were used from friday evening until monday mornning must be moved into pool “Vaulting”. The daily pools and the DPDatabase pool (The backup of the internal database I always do to a separate pool!) should not be included for vaulting in this case. The required settings for the script can be find below.
Settíngs: What do you need to change for our use case?? Please note the with a backslash quoted @ for the email addresses and that yo have configure “blat” beforehand (only 1 command to execute).
$exclude_pools="FilesystemDaily,DatabasesDaily,DPDatabase,Vaulting";
$vault_pool="Vaulting";
$timeframe_start="60";
$timeframe_duration="60";
$mailrecipient="servicedesk\@data-protector.org";
$mailserver="127.0.0.1";
$mailsender="dataprotector\@data-protector.org";
$mailsubject="Media vaulting";
[wpdm_file id=4]
UPDATE Download: There was a minor bug when counting the media – new download below.
[wpdm_file id=5]
Use it for…: The script is best suited for small to mid-size companies. For larger environments I recommend to evaluate HP Media Operations. As an alternative I sell a script for large environments, it can be enhanced on your needs and on your request. To get more information and a offer, please send a mail with the subject “Off-Site Tape Vaulting” to daniel-braun@data-protector.org
Donate: If you like this article and if the script helps you in your daily Data Protector business, I would be glad to get a donation from you (5 € are sufficient) via PayPal. The link to donate you’ll find in the upper right corner of this site…
Reference: If you are behind a proxy and unable to download the file… below you will find the complete script. If you like it smaller, please feel free to modify, the script does not need to have the “prints” and can be cutted to the half size…
#!perl -I../lib/perl -X -CA
### needed modules
use Omniback;
# Get DP folders
my $OMNIHOME = Omniback::CmnQuotePath($Omniback::PAN{'BASE'});
my $OMNIHOME_STR = $OMNIHOME;
$OMNIHOME_STR =~ s/\"//g;
my $OMNIBIN = "\"${OMNIHOME_STR}/bin\"";
# define the exclude pools, pool for vaulting, ...
$exclude_pools="Pool1,Pool2,PoolX";
# the pool mus exist in DP
$vault_pool="Vaulting";
# the time frame to retriev the session information
# assuming the script should run monday 8am, the timeframe of
# 60 month covers backups starting friday evening
$timeframe_start="60";
$timeframe_duration="60";
# the stuff needed to send a mail at the end of the script
# you need to download blat (mail program) search for blat at google
$mailrecipient="firstname.lastname\@domain.com"; # please remind the quoted (at)
$mailserver="192.168.2.10"; # the ip address of the smtp server
$mailsender="dataprotector\@domain.com"; # please remind the quoted (at)
$mailsubject="Media vaulting"; # Subject for the mail
print "Generating Report\n";
print "- skipping pools: $exclude_pools\n";
print "- target pool is: $vault_pool\n";
print "- start time: $timeframe_start\n";
print "- duration: $timeframe_duration\n";
print "\n\nNote: Ignore messages regarding \"No medium found\"\n\n";
@exec=`${OMNIBIN}/omnirpt -report list_sessions -timeframe $timeframe_start $timeframe_duration -tab`;
foreach (@exec)
{
chomp $_;
next if /^#/;
(@sessionid)=split(/\t/,$_,23);
push(@sessions,$sessionid[22]);
}
foreach (@sessions)
{
@exec=`${OMNIBIN}/omnidb.exe -session \"$_\" -media`;
$media = $exec[2];
chomp $media;
($media_short,$dummy)=split(/ /,$media,2);
if ($media_short ne "")
{
push(@media,$media_short);
}
}
@newmedia=&del_double(@media);
print "Summary\n";
@movedmedia=();
foreach (sort(@newmedia))
{
($media1,$dummy)=split(/ /,$_,2);
$media1 =~ s/\[//ig;
$media1 =~ s/\]//ig;
@exec=`${OMNIBIN}/omnimm -media_info \"$media1\" -detail`;
$pool = $exec[2];
chomp $pool;
chop $pool;
$pool =~ s/Pool name : //ig;
if ($exclude_pools =~ /$pool/)
{
print "skipping media: $media1 \n";
}
else
{
print "moving media: $media1 \n";
@exec=`${OMNIBIN}/omnimm -move_medium \"$media1\" $vault_pool`;
push(@moved_media,$media1);
}
}
$anzahl=@moved_media;
if ($anzahl gt 0)
{
$message="";
$message="Vaulting media:\n============================\n\n";
$message=$message."Following media were moved to pool: $vault_pool for vaulting.\n";
$message=$message."Media:\n";
foreach (@moved_media)
{
$message=$message.$_."\n";
}
$message=$message."\n\nMail generated by robot\n\n";
`${OMNIBIN}/blat -to $mailrecipient -f $mailsender -subject \"$mailsubject\" -body \"$message\"`;
}
exit;
sub del_double{
my %all;
grep {$all{$_}=0} @_;
return (keys %all);
}
Nice little trick – but why would you need to move it to a vaulting pool? Moving does not change any properties on the physical media, label nor the data/catalog retention enforcement so how does this make it easier for the backup admin compared to a used media report in his mailbox that kicks off at the end of the job?
I also disagree with the need for that many media pools, admins generally prefer to have homogenous retention times for as much data in their environment as possible (where it makes sense ofcourse). Firstly then homogenous retention times makes defining or revising a backup SLA a much less complicated task. Secondly there would be less media pools for the backup admin to supervise so administration is easier, and thirdly you have less unused space left on media in your vault locations. In your example above then at the typical customer, the full backup sets for both file, db and idb data would be vaulted for the same time but would be residing on minimum 3 media – none of which would be full – thats a lot of wasted capacity and money out the window if eg. using a simple gfs-rotation scheme for a year.
/Morten
Hi Morten, Yes you are right this will not change anything on the media. At least I set the vaulting pool to non appendable. Regarding your question, I personally do not care about the pools too; however, this is what I learned from customers to have such a vaulting pool and to have media with full backups moved to a special pool. In addition many customers don’t like to mix incremental and full backups on the same media. Thats why I created the script. In this case the customer don’t need to search different pools manually, he just have to have a look in one pool. When taking media off-site the customer could go to slots, sort for the pools, mark te media and select eject. Of course for an expert like you and me this is not a big deal and we would not need such a script, but not all DP admins are experts 🙂
This worked great. Do you have a sample of a script to bring the Vaulted tapes back into the silo and DP? We are having trouble with that.
Hi, no, but its quite easy, please contact me directly.
I cant seem to find your email address. How do I contact you directly?
Austin, it’s just daniel-braun(at)data-protector.org or daniel-braun(at)db-productions.de