[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [SAGE] SUMMARY: circular log trimming tool (long)
On Thu, 1 May 2003, Quentin Fennessy wrote:
> I'm wandering slightly off the original topic -- but John Sellens
> term "cascading rename of log files" reminded me of something.
>
> I don't like "cascading rename of log files" and here is why:
>
> If you have an incident reported in a log file, and you
> reference that incident by filename then that reference
> will inevitably become stale after one cycle of log file
> renaming.
>
> ie: the file server melted down this morning,
> the evidence is in /net/m5/etc/messages today.
>
> tomorrow it may be in messages.0
> next week in messages.1
>
> Suppose that each logfile name is unique--by adding date information
> in the filename:
>
> the file server melted down, etc
> the evidence is in /net/m5/etc/messages-m5-2003-ww12,
> also available as /net/m5/etc/messages (via symlink).
>
> with this technique the logfile name will never change
> until the file is deleted.
> you can also copy the logfiles from several sources into
> the same directory without collision:
>
> messages-m5-2003-ww12
> messages-wopr-2003-ww12
> messages-deepthought-2003-ww12
> messages-escher-2003-ww12
>
> Your log rotations has to be a little smarter, but this is
> really a SMOP.
This provides a good segway into a YALR (yet another log rotator) tool
that supports naming and rotating (in various ways) log files with a
very flexible array of date-based extensions. For example, names
like:
syslog.31-May-03
syslog.01-June-03
would be created and handled in the correct order. A printf-like "%"
conversion naming facility is supported.
The script is at:
ftp://ftp.cs.duke.edu/pub/des/scripts/rotlog
and here's the help (-h option) message (WARNING: LONG!):
Usage: rotlog [-F nfmt] [-S sep] [-M max] [-mpslvr] file (numeric fmt)
rotlog [-F dfmt] [-S sep] [-M max] [-mpslvc] [-P interval] \
[-D "d[ m] y]]"] [--lc] [--uc] file (date-based fmt)
rotlog -h (for help)
`rotlog' rotates a log file or other cumulative data file. These
options are supported:
-F the format of the suffix, default is "N", see below;
-S separator character(s) to use between filename and suffix,
default is ".";
-M max number of backup copies, max >= 1;
-P previous, set date values for a previous date, see below;
-D date, for testing;
-m use `mv' instead of copying, this might be a problem for
some daemons;
-p preserve, try to preserve owner, group, modes;
-r rotate, renumber other files as needed, and move saved log
to low spot in sequence;
-c clobber, overwrite any file with the same name;
--lc use all lowercase for any text date suffix components;
--uc use all uppercase for any text date suffix components;
-l list only, just list the files that match the suffix pattern;
-s suffix only, just print out the generated suffix;
-d debug, don't really move or copy files, provide more output;
-v verbose, print commands;
-h print help and exit;
file [path/]file to rotate
Suffix format (-F) can be specified by the following, which are
replaced by the appropriate substitutions:
Exactly one of the following numeric sequence patterns:
N, N0 next higher decimal number, not padded (default);
N[1-9] next higher decimal number, using zero padding to
specified number of digits;
Note: with a numeric pattern, new files are added at the upper
end of the sequence, unless -r is specified, in which case
existing files are renumbered, and the new file becomes the
lowest numbered backup copy.
Or any combination of the following date-based patterns:
%yl year long [0000-9999];
%ys year short [00-99];
%ya year absolute [0-99];
%ml month name long (full);
%ms month name short (3 chars.);
%mp month number padded [01-12];
%ma month number absolute [1-12];
%dp day of month padded [01-31];
%da day of month absolute [1-31];
%kl week day name long (full);
%ks week day name short (3 chars.);
%ka week day number absolute [1-7];
%rp day of year padded [001-366];
%ra day of year absolute [1-366];
%wp week number padded [00-53];
%wa week number absolute [0-53];
%% one "%";
? [any other character] that character;
For example, "%yl-%ms-%dp" would yeild something like
"2001-Feb-07"
Note: white space characters should not be used in formats.
Previous interval (-P): format is "x[N]", where "x" is an interval
type, and "N" is an integer; interval type can be "d" for days, "w"
for weeks, "m" for months, or "y" for years; for example, "-P d3"
means to set the date elements to three days ago; omitting the
number is equivalent to N = 1; these options can be combined, for
example, "-P d -P m".
A separator of zero or more characters between the filename and
the suffix can be specified with -S; the default is ".".
For no separator between the file basename and the suffix, use
"-S ''"; ie, an explicit null argument.
When using -M, files that match the basename are also scanned for
matches against the suffix pattern, and the ones determined to be
the oldest that exceed the maximum are removed. With a numeric
suffix series, this will be the lower numbered files, unless -r is
specified, in which case upper numbered files are selected. With
date-based suffixes, those determined to be the oldest based on
the suffix pattern are selected (see Cautionary Note below).
Generally, when backing up the log file, `rotlog' will copy the
file to the new name, then truncate the original; this helps to
preserve the file status, and is less likely to confuse processes
that are using the file. Use -m to move the file to the new name,
and then create an empty file. In either case, -p can be used to
attempt to preserve file status (owner, group, permissions) for
both files.
Normally, `rotlog' will not overwrite a file; use -c to override
this behavior. This should only happen in certain circumstances,
such as when a repeating date pattern is used for the file suffix.
If -s is specified, the new file suffix is printed, and no other
action is taken. With -l (el), files that match the suffix
pattern are listed, and no other action is taken.
The default for pattern matches and substitutions for text date
suffix components is for the first letter to be uppercase and the
remaining letters lowercase; for example, April. With --lc,
all lowercase is used. With --uc, all uppercase is used.
Normally, date-based calculations are based on the current date;
for testing and perhaps other situations, -D can be used to select
an alternative base date. Additionally, for testing, -d can be
used to display instead of execute commands, and -v can be used
for greater verbosity.
Cautionary note: Certain combinations of options and date
suffixes do not make sense, and will not work right. For example,
"rotlog -F %kl -M 5 /var/adm/xlog" will work, but "rotlog -F %ys%kl
-M 5 /var/adm/xlog" will not. If in doubt, use some of the
debugging options, and test first on test data, being sure to try
boundary cases such as beginning/end of month or year.
--
Daniel E. Singer, System Administrator
Dept. of Computer Science, Duke University, Durham NC 27708 USA
"Non cognosco. In hoc tantum laboro."