CookbookLogrep

Logrep is a very flexible query language for ad-hoc analysis of webserver logs. It's not as powerful as SQL but it sits somewhere between SQL and bespoke code you might write in the moment. Check out HelpLogrep for full explanations of all of the options.

Average response time and deviation by URL class:

$ logrep -o 'count(*),avg(msec),dev(msec),class' -s '30:1,2:desc'

91420	79.78	9.90	img
28308	1301.81	7.38	js
24912	99.94	10.75	css
9966	353.81	7.66	search
9916	4001.31	3.54	news
7273	206.61	4.98	home
1665	23.36	7.29	xml
...

Top 404 Not Found and 500 Error URLs:

$ logrep -f 'status>399' -o 'count(*),status,url' -s '10:1'

7	404	/about.hmtl
4	404	/javascripts/all.js
4	404	/__utm.gif
3	500	/Paris-hliton
2	404	/MSOffice/cltreq.asp?UL=1&ACT=4&BUILD=2614&STRMVER=4&CAPREQ=0
2	404	/_vti_bin/owssvr.dll?UL=1&ACT=4&BUILD=2614&STRMVER=4&CAPREQ=0
...

Look at the last 10,000 lines for requests from a particular IP address:

$ logrep -l 10000 -f 'ip~192.168.23' -o ts,url

1212742764	/Alicyn-YpKY1Ip
1212742765	/Miles-Jupp
1212742765	/Ed-Redlich
1212742765	/Robert-Shaye
1212742766	/q/%22likes-jt-ross-music%22
1212742767	/q/%22North-Korean-defector%22
1212742768	/Ingrid-Mattson/pictures
1212742769	/Josip-Broz-Tito/websites
...

Quick spot-check of requests by status code:

$ logrep -o 'status,count(*),avg(msec),min(msec),max(msec)'

304	1542	4.28	0	694
302	10	48.20	10	190
200	8446	399.34	0	134763
404	1	9.00	9	9
500	1	13.00	13	13

How are the crawlers hitting your site, broken down by day?

$ logrep --robots-only --output 'botname,year,month,day,count(*),avg(msec),dev(msec)' --sort '30:2,3,4:asc' 

Google	2008	6	6	7	888.86	0.70
msnbot	2008	6	6	1	919.00	0.00
-	2008	6	6	1	598.00	0.00
cuill	2008	6	6	1	1232.00	0.00
Slurp	2008	6	6	19	1095.79	1.64
...

Slowest countries?

$ logrep -o 'avg(msec),country' -s '10:1'

2231.00	Pakistan
2126.00	India
367.00	Hong Kong
1347.00	Thailand
1253.00	Brasil
1207.54	United Kingdom
1187.00	Venezuela
1133.00	Argentina
1085.00	Australia
919.00	Germany

Top referring domains that are not from 'example.com':

$ logrep -o 'refdom,count(*)' -f 'refdom!~example.com' -s '10:2'

-	852
offers.perfspot.com	156
friendster-layouts.com	71
www.xangaspy.net	54
xangaspy.net	27
search-mussinesses.com	24
pagead2.googlesyndication.com	23
search-bovenland.com	14
ads.adbrite.com	14
malayalampadam.com	13

Keep a watch out for slow requests as they happen:

$ logrep -m tail -f 'msec>1000' -o 'ts,ip,msec,class,url'

1214147493	127.0.0.1	13280	foo	/foo/
1214148491	127.0.0.1	27922	foo	/foo/
1214192358	127.0.0.1	1315	home	/
1215529978	::1	24097	xml	/crossdomain.xml
...