Skip to content
LumenaryAcademy
Practice 60 min

References and nested data structures

How one scalar can point at an array, a hash or a subroutine, and how that lets you build records, tables, trees, closures and dispatch tables.

Code tested · 4AI-draftedAwaiting review by a computer scientist
MotivationH

Data that nests

Real data is rarely flat. A student has a name, a city and a list of scores. A configuration file has sections, each with its own settings. A JSON response from a web service is objects inside arrays inside objects. In the last lesson you saw why Perl's arrays and hashes cannot hold such things directly: they hold only scalars, and lists flatten.

Perl 5's answer, added in 1994 and the single biggest change from Perl 4, is the reference. A reference is a scalar that points to another piece of data: an array, a hash, a scalar, or a subroutine. Because a reference is a scalar, it can go anywhere a scalar can, including inside another array or hash. That one idea gives you every data structure you will need, from a list of records to a tree, and it also gives you functions as values: closures, callbacks and dispatch tables.

If you have used Python, JavaScript or Java, you already know references under another name: in those languages, every list and object variable is a reference all the time. Perl simply makes it visible. @list is the array itself; $ref = \@list is a pointer to it. Seeing the difference is a little more typing, and in return you always know whether you are holding the thing or a pointer to it.

Note

Before Perl 5, programmers faked two-dimensional data by gluing keys together into one string: a table of distances between cities was a single flat hash with keys such as "Lagos,Accra". Perl even has a special variable, $;, holding the separator for that trick, and writing $dist{$from, $to} still uses it. You may meet it in very old scripts. With references there is no need: $dist{$from}{$to} is a real hash of hashes, and you can ask it for all the destinations from one city, which the glued-key version cannot do without scanning every key.

DefinitionsH

Making and following references

Definition— Reference

A scalar value that refers to another value. You take a reference with a backslash: \@array, \%hash, \$scalar, \&subroutine. You create a reference to a brand-new, nameless (anonymous) value with brackets: [ ... ] makes an array, { ... } makes a hash, and sub { ... } makes a subroutine. You dereference (follow the pointer) by putting the right sigil in front of the reference, or with the arrow ->.

The built-in ref tells you what a reference points to: it returns ARRAY, HASH, SCALAR, CODE and so on, or the empty string for a value that is not a reference at all.

There are three syntaxes for following a reference, and you will meet all three:

You wantBlock syntaxArrow syntaxPostfix syntax (5.24+)
the whole array@{$aref} or @$aref$aref->@*
one element${$aref}[0]$aref->[0]
the last index$#{$aref}$aref->$#*
the whole hash%{$href} or %$href$href->%*
one value${$href}{key}$href->{key}
a slice@{$aref}[0, 1]$aref->@[0, 1]
call a subroutine&{$cref}(1, 2)$cref->(1, 2)

The rule behind the block syntax is: wherever you would write the name of an array or hash, you can write a block that returns a reference to one. @primes becomes @{$aref}; $primes[0] becomes ${$aref}[0]. The arrow is shorthand for the common case of one element. Postfix dereferencing, stable since Perl 5.24, reads left to right, which many people find clearer in long chains: $user->{langs}->@* says "user, then langs, then all of it".

perl
use strict;
use warnings;
use feature qw(say);

my @primes = (2, 3, 5, 7);
my %color  = (sky => 'blue', grass => 'green');

my $aref = \@primes;          # a reference to an existing array
my $href = \%color;           # ... and to a hash
my $list = [10, 20, 30];      # an anonymous array, built in place
my $user = { name => 'Aiko', langs => ['Perl', 'Go'] };   # anonymous hash

say "ref types: ", join(' ', ref($aref), ref($href), ref($list), ref(\1), ref(sub {}));

# Three ways to reach through a reference.
say "block syntax:   ${$aref}[0]  @{$aref}";
say "arrow syntax:   $aref->[1]  $href->{sky}";
say "postfix syntax: ", join(',', $list->@*), " (last index ", $list->$#*, ")";

say "$user->{name} knows $user->{langs}[1]";   # arrows between subscripts are optional
push $user->{langs}->@*, 'Lua';
say "now: @{ $user->{langs} }";

$aref->[0] = 'two';            # changes @primes itself: the reference shares it
say "primes: @primes";

# Passing two arrays to a sub without flattening them together.
sub longer_of {
    my ($x, $y) = @_;
    return @$x >= @$y ? $x : $y;
}
my $winner = longer_of([1, 2, 3], ['a', 'b']);
say "longer: @$winner";

It printed:

text
ref types: ARRAY HASH ARRAY SCALAR CODE
block syntax:   2  2 3 5 7
arrow syntax:   3  blue
postfix syntax: 10,20,30 (last index 2)
Aiko knows Go
now: Perl Go Lua
primes: two 3 5 7
longer: 1 2 3

Two details are worth stating as rules. First, between subscripts the arrow is optional: $user->{langs}[1] means $user->{langs}->[1], so deep lookups stay short. The first arrow, right after the variable, is not optional: $user{langs} would mean the hash %user, a different variable. Second, a reference shares the thing it points to: assigning through $aref->[0] changed @primes. That is the point of references, and also the source of the copying trap later in this lesson.

usera scalarname => 'Aiko'langs =>anonymous hash'Perl''Go'anonymous array
The structure the program builds for the user Aiko: a scalar holding a reference to an anonymous hash, one of whose values is a reference to an anonymous array.
Worked examplesH

Records, groups and tables

Three shapes cover most nested data: an array of hashes (a list of records, like rows from a database), a hash of arrays (groups: each key has a list), and a hash of hashes (a table indexed by a key). This program uses all three:

perl
use strict;
use warnings;
use feature qw(say);
use Data::Dumper;
$Data::Dumper::Sortkeys = 1;   # stable output
$Data::Dumper::Indent   = 1;

# An array of hashes: one record per row.
my @students = (
    { name => 'Leila', city => 'Tunis',   scores => [88, 92, 79] },
    { name => 'Kwame', city => 'Kumasi',  scores => [95, 70] },
    { name => 'Ines',  city => 'Tunis',   scores => [60, 85, 90, 75] },
);

# A hash of arrays, built from it: city => list of names.
my %by_city;
for my $s (@students) {
    push @{ $by_city{ $s->{city} } }, $s->{name};   # autovivification creates the array
}
for my $city (sort keys %by_city) {
    say "$city: ", join(', ', @{ $by_city{$city} });
}

# A hash of hashes: name => summary.
my %summary;
for my $s (@students) {
    my @sc = @{ $s->{scores} };
    my $sum = 0;
    $sum += $_ for @sc;
    $summary{ $s->{name} } = { tests => scalar @sc, average => $sum / @sc };
}
for my $name (sort { $summary{$b}{average} <=> $summary{$a}{average} } keys %summary) {
    printf "%-6s %d tests, average %.1f\n", $name, $summary{$name}{tests}, $summary{$name}{average};
}

print Dumper(\%by_city);

It printed:

text
Kumasi: Kwame
Tunis: Leila, Ines
Leila  3 tests, average 86.3
Kwame  2 tests, average 82.5
Ines   4 tests, average 77.5
$VAR1 = {
  'Kumasi' => [
    'Kwame'
  ],
  'Tunis' => [
    'Leila',
    'Ines'
  ]
};

Read the key line slowly: push @{ $by_city{ $s->{city} } }, $s->{name};. From the inside out: $s->{city} is the student's city; $by_city{...} is the value stored for that city, which should be a reference to an array; @{ ... } treats it as an array; push adds the name. The first time a city appears there is no array yet, and Perl creates one for you. That is autovivification, and it is what makes building nested structures in Perl so short.

$sum / @sc divides by the array in scalar context, its length. And Data::Dumper prints any structure as Perl code that would rebuild it, which makes it the everyday tool for looking inside data while debugging. Setting $Data::Dumper::Sortkeys gives stable output, because hash order otherwise changes between runs.

Example— Reading a deep expression

Suppose $data came from a JSON document and you see $data->{orders}[2]{items}[0]{price}. Read it left to right: $data is a reference to a hash; its orders value is a reference to an array; element 2 of that is a reference to a hash; its items value is an array reference; element 0 is a hash reference; and price is the value you want. Every [...] means "this level is an array", every {...} "this level is a hash".

Worked examplesH

Code references, closures and dispatch tables

Subroutines are data too. \&name takes a reference to a named subroutine and sub { ... } creates an anonymous one. You call either with $code->(ARGS).

Definition— Closure

An anonymous subroutine that uses lexical (my) variables from the scope where it was created keeps those variables alive, even after that scope has finished. Each time the sub { ... } expression runs, it captures the variables as they exist at that moment, so two closures made by two calls get two independent sets of variables.

perl
use strict;
use warnings;
use feature qw(say);

# A closure: a sub that keeps the variables it was created with.
sub make_counter {
    my ($start, $step) = @_;
    my $n = $start;
    return sub { my $now = $n; $n += $step; return $now };
}
my $by_one  = make_counter(1, 1);
my $by_ten  = make_counter(0, 10);
say join ' ', map { $by_one->() } 1 .. 3;
say join ' ', map { $by_ten->() } 1 .. 3;
say "first counter again: ", $by_one->();

# A dispatch table: a hash of code references instead of a long if/elsif.
my %op = (
    '+' => sub { $_[0] + $_[1] },
    '-' => sub { $_[0] - $_[1] },
    '*' => sub { $_[0] * $_[1] },
    '/' => sub { $_[1] == 0 ? 'undefined' : $_[0] / $_[1] },
);
for my $expr ('6 * 7', '10 / 4', '3 - 9', '1 / 0', '2 ^ 8') {
    my ($x, $sym, $y) = split ' ', $expr;
    my $code = $op{$sym};
    say $code ? "$expr = " . $code->($x, $y) : "$expr: unknown operator";
}

# Sorting with a code reference passed in.
sub sorted_by {
    my ($key_of, @items) = @_;
    return sort { $key_of->($a) <=> $key_of->($b) } @items;
}
say join ' ', sorted_by(sub { length $_[0] }, qw(banana fig kiwi apple));

It printed:

text
1 2 3
0 10 20
first counter again: 4
6 * 7 = 42
10 / 4 = 2.5
3 - 9 = -6
1 / 0 = undefined
2 ^ 8: unknown operator
fig kiwi apple banana

Each call to make_counter created a fresh $n, and the returned subroutine kept it. That is why the two counters do not interfere, and why $by_one remembers that it had reached 3. Closures are how Perl programs keep private state without objects, and how callbacks carry the context they need.

The dispatch table replaces a chain of if/elsif comparisons with a hash lookup. Adding an operator is one line; an unknown key is simply a missing entry. Command-line tools often use the same pattern to map subcommand names to subroutines.

split ' ', $expr is a special case of split: a single space as the pattern means "split on any run of whitespace, and ignore leading whitespace", which is what you almost always want for words.

Common misconceptionsH

Copies, autovivification and cycles

perl
use strict;
use warnings;
use feature qw(say);
use JSON::PP;

my %config = (name => 'site', ports => [80, 443]);

my %shallow = %config;                   # copies the top level only
push @{ $shallow{ports} }, 8080;         # ... so this changes %config too
say "config ports: @{ $config{ports} }";

my %deep = (%config, ports => [ @{ $config{ports} } ]);   # copy the inner array as well
push @{ $deep{ports} }, 9000;
say "config ports: @{ $config{ports} }";
say "deep ports:   @{ $deep{ports} }";

# Autovivification: looking deep inside can create things.
my %tree;
say "before: ", scalar(keys %tree), " keys";
if (exists $tree{fruit}{mango}) { say "found" }
say "after exists: ", scalar(keys %tree), " keys (", join(',', keys %tree), ")";

# Nested data in and out of JSON, with core JSON::PP.
my $json = JSON::PP->new->canonical;      # canonical: keys in sorted order
my $text = $json->encode({ city => 'Hanoi', temps => [31, 29.5], rain => JSON::PP::true });
say $text;
my $back = $json->decode('{"langs":["Perl","Lua"],"year":1987}');
say "decoded: $back->{langs}[0], $back->{year}";

It printed:

text
config ports: 80 443 8080
config ports: 80 443 8080
deep ports:   80 443 8080 9000
before: 0 keys
after exists: 1 keys (fruit)
{"city":"Hanoi","rain":true,"temps":[31,29.5]}
decoded: Perl, 1987

Misconception

"my %copy = %original; copies the data." It copies the top level only. The ports value is a reference, and copying a reference gives a second pointer to the same array, so pushing through the copy changed the original. To copy deeply, copy each nested level, as %deep does with [ @{ ... } ], or use dclone from the core module Storable for arbitrary structures.

Misconception

"Testing whether something exists cannot change my data." With nested structures it can. exists $tree{fruit}{mango} has to look inside $tree{fruit}, so Perl autovivifies $tree{fruit} as an empty hash to look in. Afterwards %tree has a key fruit that nobody put there. Check each level in turn (exists $tree{fruit} && exists $tree{fruit}{mango}) when this matters. The CPAN module autovivification can also switch the behaviour off.

Misconception

"A reference can be used as a hash key, and an empty array reference is false." Neither is true. Hash keys are always strings, so $seen{$ref} = 1 stores the reference's string form, something like ARRAY(0x55d4c8a1e2f0): you can no longer dereference the key, and the address may be reused once the data is freed. Keep the reference as a value and use a name or an id as the key (or the core module Hash::Util::FieldHash for real object-keyed hashes). And every reference is true, even []: to ask whether an array reference holds anything, test the array, if (@$items), not if ($items).

Memory and cycles. Perl frees data by reference counting: each value knows how many references point at it, and is freed the moment that count drops to zero. This is predictable and immediate, which is why Perl programs rarely think about memory. Its one weakness is a cycle: if a parent node refers to its child and the child refers back to its parent, neither count ever reaches zero, and the pair is never freed while the program runs. For long-running programs, break such cycles by making one direction a weak reference with weaken from Scalar::Util; a weak reference does not count towards keeping its target alive.

JSON. Nested Perl data maps naturally onto JSON: hash references become objects, array references become arrays. The core module JSON::PP encodes and decodes it; the CPAN module Cpanel::JSON::XS (or JSON::XS) is a much faster drop-in when speed matters. canonical sorts keys so that the output is stable, which matters for tests and for diffs.

ExplanationH

How experienced Perl programmers use references

  • Pass and return references for anything bigger than a few scalars. process(\@rows, \%options) avoids flattening and copying, and makes it clear the subroutine receives two things.
  • Prefer anonymous structures built with [ ] and { } when assembling data. You rarely need \@named except to pass an existing array somewhere.
  • Pick one dereferencing style per project. Postfix ($ref->@*) is common in newer code; @{ $ref } is everywhere in older code. Both are fine; mixing them in one function is not.
  • Look at your data. use Data::Dumper; print Dumper($thing); is the first thing to try when a nested lookup returns undef. Set $Data::Dumper::Sortkeys = 1 so two dumps can be compared.
  • Check before you trust. Data from outside (JSON, configuration files) may not have the shape you expect. ref($x) eq 'HASH' before treating $x as a hash reference turns a confusing "Not a HASH reference" crash into a clear error message.
  • Reach for objects when a structure grows behaviour. A hash reference with a few subroutines that all take it as their first argument is halfway to an object already. Lesson 7 makes that final step with bless.
Exercises

Exercises

1. References and nested data structures

Warm-upComputational

What does this program print?

use strict;
use warnings;
use feature qw(say);
my @a = (1, 2, 3);
my $r = \@a;
$r->[1] = 20;
my @b = @a;
$b[2] = 30;
say "@a";

2. References and nested data structures

CoreComputational

What does this program print?

use strict;
use warnings;
use feature qw(say);
my $d = { k => [ 'x', { y => 'z' } ] };
say $d->{k}[1]{y}, ref $d->{k}, ref $d->{k}[1];

3. References and nested data structures

CoreComputational

What number does this program print?

use strict;
use warnings;
use feature qw(say);
sub make {
    my $n = shift;
    return sub { return $n++ };
}
my $f = make(5);
my $g = make(5);
$f->() for 1 .. 3;
say $f->() + $g->();

4. References and nested data structures

StretchComputational

What does this program print?

use strict;
use warnings;
use feature qw(say);
my %h;
my $x = $h{a}{b};
say scalar(keys %h), defined $x ? 'def' : 'undef';

5. References and nested data structures

CoreComputational

$r is a reference to an array. Which expressions give the number of elements in that array? Choose all that apply.

Choose all that apply

6. References and nested data structures

CoreComputational

What does this program print?

use strict;
use warnings;
use feature qw(say);
my %reply = (
    hi  => sub { "hello, $_[0]" },
    bye => sub { "goodbye, $_[0]" },
);
for my $cmd (qw(hi bye yo)) {
    print exists $reply{$cmd} ? $reply{$cmd}->('Ana') : '?', ';';
}
print "\n";

7. References and nested data structures

StretchComputational

What does this program print?

use strict;
use warnings;
use feature qw(say);
my @orig = ([1, 2], [3, 4]);
my @copy = @orig;
$copy[0][0] = 'x';
$copy[1] = ['y'];
say join ' ', map { join(',', @$_) } @orig;

Further reading and resources

  • perlreftutThe official short tutorial on references; read it alongside this lesson.
  • perlrefThe full reference on references, including postfix dereferencing and closures.
  • perldscThe Perl data structures cookbook, with code for building and walking each kind of nested structure.
  • Data::DumperThe core module for printing any data structure as Perl code.
  • JSON::PPThe core, pure-Perl JSON encoder and decoder.
  • Scalar::Utilblessed, reftype, weaken and other utilities for working with references.