Re: [SAGE] Linux Backup Software -- HELP needed in finding a reasonable one

[Dan Foster <sage@evilphb.org>]

Hot Diggety! M. D. Parker was rumored to have written:
> So I was wondering what backup system (either free or commercial)
> people are having success with out there before trying to
> reinevnt the wheel or getting a package that is ugly.

We use IBM Tivoli Storage Manager (TSM or ITSM, depending on the name du
jour...) to back up all of our systems that we manage. Works great and is
enterprise class with appropriate management features and is scalable, and
just simply works.

Backup failures that we have are tracked daily via a special failure report
and usually empty; if something fails, usually because the machine crashed
or the filesystem was dirty, or something silly like that.

TSM takes a few months to completely become comfortable with it from an
administrative and deployment standpoint, but the nice thing is that the
commands are almost identical (except for the few OS-specific options such
as references to device names or directories, for instance) on any TSM
server or client OS platform. Just as importantly, all you need to go from
know-nothing to seasoned veteran is essentially a single manual (also
available as a PDF file from the web site).

The TSM *server* can be on a number of different OS platforms -- Solaris,
AIX, Linux, Windows, HP/UX, OS/400, and MVS.

The TSM client platforms supported: VMS (3rd party software called ABC),
but all the rest are native to TSM -- Linux, Solaris, Windows, AIX, HP/UX,
MVS, OS/400, MacOS (9 and X), and Netware.

Setting up a new TSM client is as simple as a script (that I wrote in sh)
to run on the server to define the client (name, frequency, time of day),
and another sh script that I wrote to push it out to the client from a
management workstation and do the handful of one-time client-setup steps.

Works great.

(And yes, that's my equivalent of a five star rating ;) Ask an engineer
what he/she thinks of something... if he/she says 'works great', there's
just simply no higher praise!)

I also wrote a bunch of other scripts (primarily called via cron once a
day) to manage the server side -- reporting of success/failures, how many
scratch tapes are available, what is the TSM DB backup tape volser name
(for easy restores of TSM DB data itself if it ever gets wiped), and so
forth. That stuff (in its newest incarnation) has been running in full
production for a year this month and it just... works, day after day,
without needing admin intervention other than for the weekly on/off-site
tape swaps (which is also a simple script to run that selects tapes, prints
a nice postscript printout, sends email with the data, checks out the tape
from library, moves them to the I/O station door for removal, etc).

There's also an excellent ADSM-L mailing list with about 1700-ish TSM sites
or developers from all over the world that participates with helpful tips
or warnings or informal problem solving. Likewise for the other major
backup server software -- Doug Hughes runs various veritas-* mailing lists
including one for NetBackup, and I'm pretty sure there's lists hosted
elsewhere for Amanda, Legato, etc.

A senior manager recently saw a review of several major backup server
software at:

http://www.networkcomputing.com/showitem.jhtml?articleID=18200314

(It doesn't cover free stuff, but does an ok summary/write-up on the major
commercial tape backup stuff.)

Not that much "meat" to it but it's quick to read through (just need 5 mins
to read the whole thing, including all the reviews) and certainly probably
a reasonable high level overview. I would also generally agree with the
comments and weighting it made about ITSM 5.2.2.

It reviews the various major backup server software -- Legato Networker,
Veritas NetBackup, etc.

That group of 7 software seems to be pretty closed aligned on the whole;
nothing had an higher overall rating of B-, and nothing had lower than a
C+. They just somewhat vary in their individual strengths and weaknesses.

(I used Legato at last job which literally saved my job two days after its
installation, when a sh script goof by someone else didn't check $HOMEDIR
or $USER so a mistake caused that to be null... so rm -rf $HOMEDIR/$USER
expanded to rm -rf / and you see why the beet-red CEO immediately demanded
I restore all the lost data on the spot or I was going home without a job
right there and then! Fastest way to impress the importance of backups on a
relatively new system administrator. :-) )

>From my understanding, Veritas NetBackup is pretty darned good, and to a
somewhat lesser extent (but still pretty good), Legato Networker. These are
the two primary payware alternatives to TSM that I would have had
considered given my particular needs here had I not inherited an existing
setup and had 'tabula rasa' to work with. Not a slam on the other products
per se, either, since I'm sure they're usable and probably just as good if
your needs are somewhat less varied in scope or size.

The only real flaw with ITSM is that its pricing scheme keeps changing with
every version -- and I don't mean just the prices themselves, but *what*
counts (number of cpus? type of system? number of systems? number of users?
number of defined backup client systems? type of tape lib?) changes to the
point where it's real hard to get a straight answer out of a sales rep
that's valid for longer than about three months... and the pricing itself
could certainly be a little more cost-competitive.

Whomever keeps changing the pricing around in a major way every few months
really needs a spanking. :P It makes it hard for TSM sales reps to explain
to potential or existing customers in a sane way how much their desired
configuration is going to cost, which makes it harder for the customers to
explain to *their* own bean counters... which, in turn, makes it harder to
_buy_ TSM or licenses...

On the bright side, we usually find TSM to be a buy it once, forget about
it kind of deal unless you need to eventually buy additional client
licenses or want to buy more libraries (add'l library licenses) or
something.

Don't even *need* to keep upgrading it to every major version, either. Once
we found a stable version that we were happy with on both client and server
end, we stuck with it and it's just... worked, day in, day out, including
when we needed it for real.

Also, to do bare-metal-restores (BMR) stuff, usually requires purchasing a
third party add-on product or a script to disable a TSM-managed tape drive,
then do an OS-specific backup directly to it, then re-enable TSM but tell
it to never touch that tape for handling TSM data. Nothing major, perhaps
about 10 minutes of shell scripting and some backup / BMR restore tests
until you're 100% confident.
 
> I'm basically looking for software that will drive 
> the tape drives concurrently, manage the catalog,
> not write things in proprietary format, autochange tapes when
> needed to continue a backup, do restores.  Basically
> the standard stuff.

'Enterprise' doesn't have to mean "frightfully expensive and applicable
only to 5000+ employee companies". There are lower tiered/priced versions
of enterprise backup software, and often the fees are dependent on what
features (and number of clients/tapes/etc) that you actually use, so you
don't end up paying for junk that you never use.

What this stuff gives you is manageability, support, performance,
scalability (if your needs ever significantly grow, you don't have to
switch hardware or software... just plug more stuff in and run a command or
two...), and a single software running on a wide variety of machines.

Furthermore, enterprise backup software stuff also often has special hooks
for more complex setups such as integrating with SANs, NDMP stuff, server
to server backups/restores and disaster recovery stuff, and a whole bunch
of other things.

There's also commercial backup software targeted towards the small / medium
sized places for far less cost, too, and they sound like reasonable
solutions from what I've heard of their users and needs, too.

All this stuff is usually part of the 'scalability' angle as well as
investment protection if your needs ever grow significantly bigger in the
future.

It certainly has, here! Over 8 years, we went from a single DDS-2 DAT drive
to a 7 tape DAT autoloader to a 20 tape Magstar (3570) library to a 324
tape/4 drives single 3570 library to 610 tape/12 LTO-1 drives library
replicated in multiple sites (with even more tape drives and new generation
stuff possibly on the way...)

It's been a major godsend for us to use a single software to manage all
that hardware, growth, and platforms, and set up the stuff just once ever.
(scripts, server config, standardized client configs, etc)

> My company would rather go with a open source solution but
> I am considering commercial packages as well.

Yeah, it can be done via open source stuff. Just that you get to hit the
ground running much sooner with existing software that has all the required
features ready to be used, although that's certainly not a slam on open
source stuff. Amanda's supposed to be good, even with its 'difficult to
initially configure in an hurry' reputation, and there are probably other
open source software.

In defense of Amanda's learning curve, that also applies to a number of
other backup server software products as well since backup software are
often amongst the most critical software you'll ever run... so it has to be
able to correctly deal with a wide range of situations and platforms, which
usually implies an healthy number of options, potential configuration, and
so forth... which leads to a slight learning curve.

No big deal if you're an intermediate or advanced system administrator but
often frightfully intimidating for the junior system administrator when
confronted with the reality to need to read through docs to see options,
write scripts to manage stuff, think up devious scenarios that can cause
failures for either backups or restores, defend/test against them, etc.

I like open source stuff in general and use a good deal of them; I just
haven't used them for backups with anything bigger than single-tape dump
over ssh and that was a long time ago, so I'll have to defer to someone
else to better point out useful open source backup software possibilities
should you feel that the commercial stuff is out of your price range. (This
is often a reality at a tiny start-up, mom-and-pop shop, or academic
environment... been there, done that, know what that's like.)

-Dan