Skip to content
LumenaryAcademy
Professional 70 min

Testing, tools and a log analyser

Test::More and prove, the tools Perl professionals use every day, and a complete, tested command-line log analyser.

Code tested · 2AI-draftedAwaiting review by a computer scientist
MotivationH

Code you can trust next year

Perl has a strong testing culture, and it is older than most. Since the late 1980s, Perl's own source has come with a test suite, and the convention that every CPAN module ships with tests, run automatically when you install it, is a big part of why CPAN works. The format those tests print, TAP (the Test Anything Protocol), began in Perl and is now used by test tools in many languages.

That culture matters especially for the kind of code Perl is used for: scripts that run unattended at 3 a.m., parse formats nobody fully documented, and live for decades. This last lesson has three parts. First, how to test Perl code with Test::More and prove. Second, the tools working Perl programmers use every day. Third, a complete project that brings the whole course together: a command-line analyser for web server logs, with regular expressions, nested data, a class, error handling, options and tests.

Worked examplesH

Test::More

Definition— Test::More and TAP

Test::More is a core module of assertion functions. Each assertion prints one line of TAP: ok N - description if it passed, not ok N - description and a diagnostic if it failed. At the end, done_testing() prints the plan, 1..N, so a harness can tell that the script did not stop early. A test script exits with status 0 only if every test passed.

The program below tests a small function that turns durations like 1h30m into seconds. In a project, the function would live in a module under lib/ and the tests in a file under t/ ending in .t; here both are in one file.

perl
use strict;
use warnings;
use Test::More;

# The code under test: normally in lib/, here inline.
sub parse_duration {
    my ($text) = @_;
    die "empty duration\n" unless defined $text && length $text;
    die "cannot parse '$text'\n" unless $text =~ /\A(?:\d+[hms])+\z/;
    my %unit = (h => 3600, m => 60, s => 1);
    my $seconds = 0;
    while ($text =~ /(\d+)([hms])/g) {
        $seconds += $1 * $unit{$2};
    }
    return $seconds;
}

is(parse_duration('90s'), 90, 'seconds only');
is(parse_duration('1h30m'), 5400, 'hours and minutes');
is(parse_duration('2m5s'), 125, 'minutes and seconds');
cmp_ok(parse_duration('1h'), '>', parse_duration('59m'), 'an hour is longer than 59 minutes');

subtest 'bad input dies with a clear message' => sub {
    for my $bad ('', 'soon', '5x', '10m later') {
        my $ok = eval { parse_duration($bad); 1 };
        ok(!$ok, "'$bad' is rejected");
        like($@, qr/\A(?:empty duration|cannot parse)/, '... with a useful error');
    }
};

is_deeply(
    [ map { parse_duration($_) } qw(1s 1m 1h) ],
    [ 1, 60, 3600 ],
    'units scale as expected',
);

done_testing();

It printed:

text
ok 1 - seconds only
ok 2 - hours and minutes
ok 3 - minutes and seconds
ok 4 - an hour is longer than 59 minutes
# Subtest: bad input dies with a clear message
    ok 1 - '' is rejected
    ok 2 - ... with a useful error
    ok 3 - 'soon' is rejected
    ok 4 - ... with a useful error
    ok 5 - '5x' is rejected
    ok 6 - ... with a useful error
    ok 7 - '10m later' is rejected
    ok 8 - ... with a useful error
    1..8
ok 5 - bad input dies with a clear message
ok 6 - units scale as expected
1..6

The functions you will use most:

FunctionPasses when
ok($test, $name)$test is true
is($got, $expected, $name)the two are equal as strings (eq); undef only equals undef
isnt($got, $unexpected, $name)they differ
like($got, qr/.../, $name)$got matches the pattern
cmp_ok($got, $op, $expected, $name)$got $op $expected is true, for any operator such as ==, < or eq
is_deeply($got, $expected, $name)two nested structures have the same shape and contents
subtest $name => sub { ... }every test inside the block passes

Prefer the specific functions over ok: when is fails it shows both values, while a failed ok only says it failed. Here is what a failure looks like. This test file has a deliberately wrong expectation in its second test, and running it with perl t/duration.t printed:

text
ok 1 - hours and minutes
not ok 2 - a bare number means seconds
#   Failed test 'a bare number means seconds'
#   at t/duration.t line 16.
#          got: '60'
#     expected: '90'
1..2
# Looks like you failed 1 test of 2.

and the script exited with status 1. The diagnostic gives the test's name, its line, and both values: everything needed to start debugging.

Testing that code dies uses the eval idiom from lesson 6, as the subtest does: run the code inside eval, check that it failed, and check the message with like. The CPAN modules Test::Fatal and Test::Exception wrap this up. The newer Test2 framework, with its Test2::V0 bundle from CPAN, offers richer comparisons and better diagnostics with a similar style; many new projects use it, and Test::More itself is now built on Test2 internally.

ExplanationH

The professional toolbox

Project layout. A Perl project, whether or not it will ever go to CPAN, usually looks like this:

text
my-tool/
  bin/loganalyse          the program (thin: parses options, calls the library)
  lib/LogAnalyser.pm      the code, in modules
  t/analyser.t            tests
  cpanfile                dependencies (lesson 7)
  README.md

prove runs every test file in t/ and summarises: prove -l t/ (-l adds lib to @INC), prove -lv t/analyser.t for verbose output, prove -lr -j8 to run a large suite recursively in parallel. Run it before every commit and in continuous integration.

Perl::Tidy formats code to a consistent style, configured by a .perltidyrc file in the project; run it before committing, or let your editor run it on save, and style arguments disappear from code review.

Perl::Critic is a linter that checks code against policies drawn from the book Perl Best Practices and from community experience: no two-argument open, no return undef, always use strict, and dozens more, at five levels of severity. Teams pick a severity level and a .perlcriticrc and run perlcritic lib bin in CI.

The debugger is built in: perl -d script.pl lets you step (n, s), set breakpoints (b 42), print values (x $var) and continue (c). Many people get by with warn and Data::Dumper, but for a confusing bug the debugger is faster.

Devel::NYTProf is an excellent profiler: perl -d:NYTProf script.pl then nytprofhtml produces an HTML report of time spent per line and per subroutine. Measure before optimising; the slow part is rarely where you guess.

Documentation is written in POD inside the source file, so it travels with the code and perldoc bin/loganalyse displays it. A =head1 SYNOPSIS section, kept current, is the cheapest documentation you will ever write, and the core module Pod::Usage can print it as the tool's --help.

Language servers and editors. Perl Navigator and the Perl::LanguageServer bring completion, inline Perl::Critic warnings and go-to-definition to editors such as VS Code and Neovim.

Coverage. Devel::Cover, from CPAN, runs your tests and reports which lines, branches and conditions they exercised: cover -test produces an HTML report. Coverage does not prove the tests are good, but uncovered code is certainly untested, and the report is the quickest way to find error paths nobody has tried.

Continuous integration. Run prove (and perlcritic) on every push. Because servers run many different Perl versions, CI for a shared library usually tests a matrix of versions, from the oldest Perl the code promises to support up to the newest release; the official perl Docker images make this easy, one image tag per version. If your code must run on an old system Perl, that version belongs in the matrix too: it is the one that will surprise you.

ComputationC

Project: a web server log analyser

Web servers such as nginx and Apache write one line per request in the "combined" log format: the client's address, the time, the request line, the status code, the number of bytes sent, and more. Every operations team needs quick answers from these files. How many requests? What fraction failed? Which pages and clients are busiest? When was the peak? Here is a complete tool that answers them. It runs on the built-in sample after __DATA__ when given no files, and on real log files otherwise: loganalyse --top 5 access.log. The addresses in the sample are from the ranges reserved for documentation, so they belong to no one.

perl
#!/usr/bin/env perl
# loganalyse: summarise web server access logs in the "combined" format.
use strict;
use warnings;

package LogAnalyser {
    use List::Util qw(sum0);

    my $LINE = qr{
        \A
        (?<ip>     \S+ ) \s+ \S+ \s+ \S+ \s+           # client, identity, user
        \[ (?<day> [^:\]]+ ) : (?<hour> \d\d ) [^\]]* \] \s+
        " (?<method> [A-Z]+ ) \s+ (?<path> \S+ ) \s+ [^"]* " \s+
        (?<status> \d{3} ) \s+
        (?<bytes>  \d+ | - )
    }x;

    sub new {
        my ($class) = @_;
        return bless { records => [], malformed => 0 }, $class;
    }

    sub parse_line {                      # a plain function: easy to test on its own
        my ($line) = @_;
        return unless $line =~ $LINE;
        my %r = %+;                       # copy the named captures
        $r{bytes} = 0 if $r{bytes} eq '-';
        $r{path}  =~ s/\?.*//;            # ignore query strings
        return \%r;
    }

    sub add_line {
        my ($self, $line) = @_;
        my $r = parse_line($line);
        if ($r) { push @{ $self->{records} }, $r } else { $self->{malformed}++ }
        return $self;
    }

    sub summary {
        my ($self, %opt) = @_;
        my $top     = $opt{top} // 3;
        my @records = @{ $self->{records} };
        my (%by_class, %by_path, %by_ip, %by_hour);
        for my $r (@records) {
            $by_class{ substr($r->{status}, 0, 1) . 'xx' }++;
            $by_path{ $r->{path} }++;
            $by_ip{ $r->{ip} }++;
            $by_hour{ $r->{hour} }++;
        }
        my $errors = sum0 map { $by_class{$_} // 0 } qw(4xx 5xx);
        return {
            requests  => scalar @records,
            malformed => $self->{malformed},
            bytes     => sum0(map { $_->{bytes} } @records),
            by_class  => \%by_class,
            error_pct => @records ? 100 * $errors / @records : 0,
            top_paths => [ _top(\%by_path, $top) ],
            top_ips   => [ _top(\%by_ip, $top) ],
            busiest   => (_top(\%by_hour, 1))[0],
        };
    }

    sub _top {                            # keys with the highest counts, ties alphabetical
        my ($count, $n) = @_;
        my @keys = sort { $count->{$b} <=> $count->{$a} or $a cmp $b } keys %$count;
        $#keys = $n - 1 if @keys > $n;
        return map { [ $_, $count->{$_} ] } @keys;
    }
}

package main;
use Getopt::Long qw(GetOptionsFromArray);

sub main {
    my @args = @_;
    my %opt = (top => 3);
    GetOptionsFromArray(\@args, 'top=i' => \$opt{top})
        or die "usage: loganalyse [--top N] [FILE ...]\n";

    my $analyser = LogAnalyser->new;
    if (@args) {
        for my $file (@args) {
            open my $fh, '<', $file or die "Cannot read $file: $!\n";
            $analyser->add_line($_) while <$fh>;
        }
    } else {
        $analyser->add_line($_) while <DATA>;    # the built-in sample
    }
    print report($analyser->summary(top => $opt{top}));
    return 0;
}

sub report {
    my ($s) = @_;
    my $out = sprintf "%d requests, %d malformed lines, %.1f KiB sent\n",
        $s->{requests}, $s->{malformed}, $s->{bytes} / 1024;
    $out .= sprintf "status: %s\n",
        join '  ', map { "$_=$s->{by_class}{$_}" } sort keys %{ $s->{by_class} };
    $out .= sprintf "error rate: %.1f%%\n", $s->{error_pct};
    $out .= "top paths:\n" . join '', map { sprintf "  %-14s %d\n", @$_ } @{ $s->{top_paths} };
    $out .= "top clients:\n" . join '', map { sprintf "  %-14s %d\n", @$_ } @{ $s->{top_ips} };
    $out .= sprintf "busiest hour: %s:00 (%d requests)\n", @{ $s->{busiest} } if $s->{busiest};
    return $out;
}

exit main(@ARGV) unless caller;    # a "modulino": tests can load this file without running it

__DATA__
203.0.113.7 - - [14/Mar/2026:09:58:01 +0000] "GET /index.html HTTP/1.1" 200 5120 "-" "Mozilla/5.0"
203.0.113.7 - - [14/Mar/2026:10:01:12 +0000] "GET /style.css HTTP/1.1" 200 1840 "-" "Mozilla/5.0"
198.51.100.23 - - [14/Mar/2026:10:02:45 +0000] "GET /products?page=2 HTTP/1.1" 200 8100 "-" "curl/8.5"
192.0.2.14 - - [14/Mar/2026:10:05:09 +0000] "POST /login HTTP/1.1" 401 310 "-" "Mozilla/5.0"
192.0.2.14 - - [14/Mar/2026:10:05:30 +0000] "POST /login HTTP/1.1" 302 0 "-" "Mozilla/5.0"
203.0.113.7 - - [14/Mar/2026:10:12:44 +0000] "GET /products HTTP/1.1" 200 7900 "-" "Mozilla/5.0"
198.51.100.23 - - [14/Mar/2026:10:30:00 +0000] "GET /old-page HTTP/1.1" 404 512 "-" "curl/8.5"
this line is garbage from a crashed writer
203.0.113.9 - - [14/Mar/2026:11:15:22 +0000] "GET /index.html HTTP/1.1" 200 5120 "-" "Mozilla/5.0"
203.0.113.9 - - [14/Mar/2026:11:16:03 +0000] "GET /api/report HTTP/1.1" 500 - "-" "Mozilla/5.0"

It printed:

text
9 requests, 1 malformed lines, 28.2 KiB sent
status: 2xx=5  3xx=1  4xx=2  5xx=1
error rate: 33.3%
top paths:
  /index.html    2
  /login         2
  /products      2
top clients:
  203.0.113.7    3
  192.0.2.14     2
  198.51.100.23  2
busiest hour: 10:00 (6 requests)

How it is built

  • One regular expression, readable. $LINE uses /x and named captures to describe a log line field by field. Each piece is specific: [^:\]]+ for the day stops at the colon, [^"]* skips the protocol inside the quotes. Specific patterns are faster and fail cleanly on garbage, which is counted rather than crashing the tool. Copying %+ into a fresh hash keeps the captures safe from the next match.
  • A small class. LogAnalyser keeps the records and the malformed count, and offers add_line and summary. parse_line is a plain function because it needs no object, which also makes it the easiest thing to test.
  • Hashes for counting, a helper for ranking. Four counting hashes are filled in one pass. _top sorts by count and breaks ties alphabetically, so the output is the same on every run despite Perl's random hash order; assigning to $#keys truncates the list. The leading underscore marks it as private by convention.
  • The status class. substr($r->{status}, 0, 1) . 'xx' turns 404 into 4xx, the grouping operations people actually look at.
  • Options with Getopt::Long. 'top=i' declares an option that takes an integer, and bad options produce a usage message and a non-zero exit through die. GetOptionsFromArray works on the array passed to main rather than the global @ARGV, which makes main testable too.
  • Input from files or the sample. while (<$fh>) reads line by line, so a log of several gigabytes uses almost no memory.
  • The modulino. The last line runs main only when the file is run as a program. When another file loads it with require, caller is true and nothing runs, so tests can call LogAnalyser::parse_line directly. exit main(...) passes main's return value out as the exit status.

Its tests

In the project layout, t/analyser.t loads the program and tests the pieces:

perl
use strict;
use warnings;
use Test::More;
use FindBin;

require "$FindBin::Bin/../bin/loganalyse";    # loads the subs; main() does not run

my $line = '192.0.2.14 - - [14/Mar/2026:10:05:09 +0000] "POST /login?next=/home HTTP/1.1" 401 310 "-" "x"';
my $r = LogAnalyser::parse_line($line);
is($r->{ip},     '192.0.2.14', 'client address');
is($r->{method}, 'POST',       'method');
is($r->{path},   '/login',     'query string removed');
is($r->{status}, 401,          'status');
is($r->{hour},   '10',         'hour');

ok(!defined LogAnalyser::parse_line('garbage'), 'malformed line gives undef');
is(LogAnalyser::parse_line(($line =~ s/ 310 / - /r))->{bytes}, 0, 'a dash for bytes counts as 0');

my $analyser = LogAnalyser->new;
$analyser->add_line($line) for 1 .. 3;
$analyser->add_line('not a log line');
my $s = $analyser->summary(top => 1);
is($s->{requests},  3, 'three requests');
is($s->{malformed}, 1, 'one malformed line');
is_deeply($s->{top_paths}, [ [ '/login', 3 ] ], 'top path');
is($s->{error_pct}, 100, 'every request was a 4xx');

done_testing();

With the program saved as bin/loganalyse, running prove -l t/ in the project directory printed:

text
t/analyser.t .. ok
All tests successful.
Files=1, Tests=11,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.07 cusr  0.00 csys =  0.10 CPU)
Result: PASS

As the tool grows, the natural next steps are to move LogAnalyser into lib/LogAnalyser.pm, add POD with a SYNOPSIS and wire it to --help with Pod::Usage, accept - for standard input so it fits into shell pipelines (zcat access.log.gz | loganalyse -), add a --json option that prints the summary with JSON::PP for other programs to consume, and parse timestamps properly with the core module Time::Piece to report per-day trends.

Further reading

Where to go next

You now know the Perl that working Perl programmers use: scalars, arrays, hashes and context; subroutines and references; regular expressions; files, Unicode and errors; modules, CPAN and objects; tests and tools. Some directions from here:

  • Read real code. Pick a well-known CPAN module you used in this course, such as List::Util, Getopt::Long or JSON::PP, open its source on MetaCPAN, and read it. Then read its tests.
  • Go deeper into the language with the official documentation: perlre for every regex feature, perlref and perldsc for data structures, perlootut for objects, and the perldelta page of each new release to see what changed.
  • Web applications. Mojolicious and Dancer2 are the actively developed web frameworks; Plack/PSGI is the interface between Perl web applications and servers.
  • Databases. DBI, with a driver such as DBD::SQLite or DBD::Pg, is the standard database interface, and DBIx::Class is a widely used object-relational mapper.
  • Bioinformatics. BioPerl remains a large library for sequence and genome work, and much published analysis code is Perl.
  • Raku. If Perl's ideas appeal to you and you want to see where its designers went with a clean slate, Raku has grammars, gradual typing, concurrency and much more. Remember that it is a separate language with its own ecosystem.
  • Community. perl.org lists mailing lists, conferences (the Perl and Raku Conference, and many regional workshops) and local groups.
Optional Christian reflection · Software that serves for decades

A reflection from the Christian tradition that inspires Lumenary Academy. It is not part of the mathematics and you are welcome to skip it.

Much of the Perl running today was written fifteen or twenty-five years ago and still quietly does its job: moving files between hospitals and laboratories, checking patches, generating reports someone relies on every morning. Few of its authors are famous. There is a quiet dignity in work done carefully for people you will never meet, tested so that it keeps serving long after you have moved on. The parable of the talents is often read as a story about ambition, but it can also be read as a story about faithfulness with what is entrusted to us, however modest it looks.

Exercises

Exercises

1. Testing, tools and a log analyser

Warm-upComputational

This test script passes. The last line it prints is its plan, 1..N. What is N?

use strict;
use warnings;
use Test::More;
is(2 + 2, 4, 'sum');
subtest 'strings' => sub {
    is(lc 'PERL', 'perl', 'lower case');
    like('camel', qr/^ca/, 'prefix');
    isnt('a', 'b', 'different');
};
ok(length 'abc' == 3, 'length');
done_testing();

2. Testing, tools and a log analyser

StretchComputational

Which of these tests fail? Give their names (for example order).

use strict;
use warnings;
use Test::More;
is('10', 10.0, 'string and number');
is(0.1 + 0.2, 0.3, 'floating point');
is(undef, '', 'undef and empty');
is_deeply([1, 2], [2, 1], 'order');
like('Perl 5', qr/^perl/, 'case');
done_testing();

Write elements separated by commas. Order and repeats don’t matter — just like in a set!

3. Testing, tools and a log analyser

Warm-upHuman

Why do experienced Perl programmers write is($total, 42, 'total') rather than ok($total == 42, 'total')?

Choose one

4. Testing, tools and a log analyser

CoreComputational

What does this program print?

use strict;
use warnings;
use Getopt::Long qw(GetOptionsFromArray);
my @args = ('--top', '5', '--verbose', 'a.log', 'b.log');
my ($top, $verbose) = (3, 0);
GetOptionsFromArray(\@args, 'top=i' => \$top, 'verbose' => \$verbose)
    or die "bad options\n";
print "$top $verbose @args\n";

5. Testing, tools and a log analyser

CoreComputational

This uses the same ranking idea as the project's _top. What does it print?

use strict;
use warnings;
use feature qw(say);
my %count = (pear => 2, fig => 5, kiwi => 2, plum => 1);
my @keys = sort { $count{$b} <=> $count{$a} or $a cmp $b } keys %count;
$#keys = 2 if @keys > 3;
say "@keys";

6. Testing, tools and a log analyser

CoreHuman

The project ends with exit main(@ARGV) unless caller;. What happens when a test file loads it with require?

Choose one

7. Testing, tools and a log analyser

Warm-upHuman

Your team wants CI to reject code that uses two-argument open or forgets use strict. Which tool is built for that?

Choose one

Further reading and resources

  • Test::MoreThe core testing module used by most of CPAN.
  • proveThe command-line test runner that ships with Perl.
  • Getopt::LongThe core module for parsing command-line options.
  • Perl::CriticA configurable linter that checks code against best-practice policies.
  • Perl::TidyThe standard formatter for Perl source code.
  • Devel::NYTProfA powerful profiler that shows where a Perl program spends its time, line by line.
  • perlpodHow to write documentation in POD, Perl's documentation format.