Re: [SAGE] Log collection and analysis solutions

[Brad Knowles <brad@shub-internet.org>]

On 11/16/07, Sean Kelly wrote:

>  What have other people used and had success with for doing this sort of
>  thing? Further, what good tools exist out there for doing real-time
>  monitoring of these logs to monitor for ongoing events such as attacks,
>  system problems, and so forth.

In terms of doing real-time processing of log data, one interesting 
thing that we've heard about here at the LISA conference is SEC -- 
Simple Event Correlator, see <http://kodu.neti.ee/~risto/sec/>.

Of course, there's always tools like logwatch and swatch, but I think 
they target a slightly different type of problemspace.


I would say that I think you want to distinguish between the live 
processing of real-time log data from the historical processing of 
log data (perhaps including performance information), either for 
trending, growth projections, or maybe just tracking down what 
happened to which e-mail message at what time, etc....

The kinds of tools, either open-source (like Lire from LogReport.org 
or webalizer), or commercial (like Splunk) are going to be very 
different depending on which kind of problem they're trying to solve.


See also <http://en.wikipedia.org/wiki/Web_log_analysis_software> and 
related pages.

-- 
Brad Knowles <brad@shub-internet.org>
LinkedIn Profile: <http://tinyurl.com/y8kpxu>