ARIN Justified...

[Bill Van Emburg]

Simon wrote:
> 
> We have servers with over 5-10 million hits and parse logs daily at night. It takes about 2 hours to parse the logs per
> machine. Mostly due to resolving IPs. To get just the bandwidth, 10 million hits log file can be parsed in matter of
> minutes. So, you just need better tools ;-) As for other traffic such as FTP, there is a log file which can be parsed, too.
> We actually do this for anonymous FTP. I don't know who charges for POP/SMTP traffic, but same method can be
> implied here to calculate the bandwidth, too. It's matter of having right tools for the job. They are out there or you can
> have a programmer write custom set for your needs. Keep in mind, I'm referring to virtual web hosting, not dedicated.
> 

Attempting to parse all those different log files and consolidate the
info is certainly not elegant, nor a particularly great use of CPU, and
again, it does not tell you the actual bandwidth usage, merely the
application-level data.  It gets worse, when you consider that each of
our shared hosting customers has their own, separate web server, ftp
server, etc. running.  Even in shared hosting, each of our customers has
their own distinct server processes.  This very quickly becomes a
logistical nightmare, as well as a larger problem to parse.  Finally,
we're talking about more than double the hits you are describing.  It is
distinctly possible that the tool problems we're having are still
related to sheer volume.

Something I didn't mention before: we also have to measure streaming
media bandwidth consumption.  Correct me if I'm wrong, but I'm not aware
of a way to do that from log files, for any existing streaming server.
-- 

				     -- Bill Van Emburg
				     	Quadrix Solutions, Inc.
Phone: 732-235-2335, x206		(mailto:bve at quadrix.com)
Fax:   732-235-2336			(http://quadrix.com)
		The eBusiness Solutions Company