Modern Perl Books for modern Perl programming

Suppose you like Dist::Zilla and cpanminus. Suppose you also need to install the new version of a distribution you're working on locally. (I do this all the time.)

Sure, you can use dzil install, which builds, tests, and installs the distribution. That's too slow though—by the time the normal CPAN shell has loaded its indexes, cpanm has often finished.

I've already spent more time writing this post than it took to write this silly little shell alias:

alias dzinstall='dzil build; cpanm install *.tar.gz; dzil clean'

Sure, you have to manage your distro with dzil and you have to have cpanm configured to use the correct Perl and you can't have any other tarballs in your directory, but this works really well for me.

Why I Left Perl is 80% silliness and 20% truth. Some of that truth is "Why did it take seventeen years to make strict mode anything close to a default in Perl 5?" and another part of that truth is "Sometimes you get nagged about unrelated things when you ask a question about Perl".

Not that, of course, you can avoid nagging and insults and bad advice on the Internet in any discussion of programming languages. (Maybe if you only ever program Haskell and can call Brian O'Sullivan and Simon Peyton-Jones on the phone, but while I've done that, I'd feel guilty making it a habit.)

Here's the thing about experts. They're experts because they've made a lot of mistakes building real things and they've learned from those mistakes.

Do you know why most Perl 5 experts don't use inside-out objects? They turned out not to work so well in practice.

Do you know why so many good Perl 5 web developers use PSGI and Plack? They solve real problems elegantly.

Do you know why so many good Perl 5 programmers use automated testing built around Test::Builder? Because it works and it makes them more productive.

Do you know why most good Perl 5 tutorials recommend the use of strict and warnings? Because they ask perl to help you find likely errors and dubious constructs to save you time debugging and to help you write better code.

Do you know how many times your average Perl expert has answered the question "Why isn't my program doing anything?" with "Why aren't you checking to see if your open call succeeded?" Yes, that's an argument for changing Perl 5 so that it has better defaults. Yes, it's a usability problem. Yet it's also something that Modern Perl: the book gets around by recommending the use of autodie.

None of that advice excuses recommending things like strict when they won't help, for example. I've responded to several well-meaning posts on Perlmonks that say "I don't know what your problem is, but you need to use strict and warnings" by asking what that would solve. These pragmas aren't magic pony-colored bandages that make you vomit glitter and candy. (I think that's the Ruby DSL generator called called Hipstr. Download from Github.)

None of that advice necessitates scolding people for not doing things your way. Way back in the mists of time (the late '80s), people like Larry and Randal posted "Here's how you'd do it in Perl!" to Unix administration and programming forae on Usenet.

The tricky part is when someone's obviously doing something the wrong way, say parsing context-free grammars with simple regular expressions. The expert has to find the balance between never condescending ("What is broken in your head that you would ever consider doing such a thing?") and giving the right answer ("Sure, it takes a few minutes to install and read the documentation and learn how to use this tool, but you'll get the right answer and spend much less time debugging frustrating errors none of us wants to debug.")

The novice, of course, has to meet the expert partway and acknowledge that asking for help means, of course, being willing to receive help. Otherwise you're wasting everyone's time. If you want to do that, at least have the decency to explain that you're doing so. (Posting questions without defensive coding squanders a community resource of altruism and good will.)

The best option I've seen is to answer a question directly, and only then explain how to avoid the problem in the first case. Sometimes that works.

I think one of the hallmarks of an expert versus a novice is that the expert knows full well that we always underestimate the time and effort of debugging and so we try to optimize to avoid the need to debug, while the novice thinks that all programming is a matter of juggling magic symbols until things seem like they work and you can escape with only a few bumps, bruises, and scrapes. Thus doing what's obviously extra work now seems like a terrible bargain because you don't know yet everything that can and will go wrong.

Then again, of course I'd say that. I enjoy the act of solving problems far more fun than the act of chasing down bugs (especially bugs I can and should have avoided.)

Reini Urban's ExtUtils::MakeMaker make release post demonstrates how to add a few of the features of Dist::Zilla to the why-does-this-still-exist ExtUtils::MakeMaker. (Spoiler: it still exists because it's not worth rewriting everything that uses it and continues to work.)

While I think everything about MakeMaker's implementation is blepharitic and, quite likely, contagious, you might find it surprising that I agree in spirit with the point both Reini and educated_foo make in the post and its comments. In summary:

That should be staggeringly obvious to everyone with more than six months of serious programming experience, but it's not.

I spent a few days getting my head around dzil and still haven't updated all of my releasable code to use it because the act of conversion takes a few minutes. If I have no plans to release new versions of that code any time soon, there's little value in performing the conversion. Similarly, if I had a good release strategy in place before dzil came about, the cost of switching would have to be less than the cost of keeping things the same for things to work out. (You can make the same argument about switching between technologies such as languages or editors.)

Sometimes a technology is measurably better. For me, a git-based workflow using dzil beats all of the alternatives I've tried. The same goes for Plack for deployment and cpanminus and perlbrew for managing Perl installations. Yet I happily used Subversion until the pain of managing branches and merges and repositories outweighed the pain of figuring out git.

With that said, Aristotle is right (as usual): automating away all of the silly little niggly details that are tedious to remember and get right is almost always worthwhile. Whether that's Reini's EUMM recipe or the dzil ecosystem depends on who has to automate things. That's the same reason make test or dzil test or prove -lr t/ is much better than scanning the output of multiple test files and trying to summarize in your mind.

(Fun fact: in my first few public Perl 5 projects I manually removed all of the CVS directories because I didn't know about CVS export, to say nothing of EUMM. In my defense, this was 1998.)

Sometimes you get lucky and find an automation that lets you share smaller pieces of hard work between lots of other people—there's one advantage of dzil over EUMM. Where a user of EUMM might happily copy Reini's code into every Makefile.PL to add those nice features, a dzil user can install a plugin and use it on every project. (... though I didn't see a NYTProf plugin when I looked yesterday, which surprised me.)

Today's CPAN experiment was 15 minutes with Chart::Clicker. I'd heard great things about this distribution before, but had never tried it.

I've been working lately on financial analysis of publicly traded companies. In particular, I've been analyzing trends in the growth of owner earnings, given a ten year window of SEC reports. (Don't worry if you don't know everything that means yet.) The goal of this work is to find a trendline which smooths out yearly ups and downs and gives a good idea of the company's expected growth.

(That number is particularly important if you want to project the intrinsic value of a company into the future to decide the value of an individual share of that company right now. This is very standard Graham/Dodd/Buffett stuff, but it's also specific domain knowledge interesting only to this post as background information.)

My statistics are a bit rusty, so I wanted to see the resulting information before I trusted my calculations. My first instinct was to copy and paste information into a spreadsheet and create a graph there. Yes, I did that manually a couple of times. Then I remembered I have the full power of Perl available.

Chart::Clicker installed easily. Its documentation is a bit on the thin side, if you need to customize things (and I did), but if you poke around at the various components and their methods, you can make sense of things. (In particular, I wanted to change the underlying grid lines to correspond with the data points on the X axis. They're years, after all.)

My analysis code produces a list of values for free cash flow in thousands of dollars over a ten year range. It also uses the least square fit technique to plot a line representing the change in those values. That line should show the trend in values with as much accuracy as possible. While I have ten points for the free cash flow line, I need only two points for the trend line, because it's a straight line.

Chart::Clicker makes it really easy to add two datasets with different numbers of points. (I'm fortunate that the first and last X coordinates are the same.) Here's the code:

use Chart::Clicker;
use Chart::Clicker::Data::Series;
use Chart::Clicker::Data::DataSet;

my $chart = Chart::Clicker->new;

my $fcf_line = Chart::Clicker::Data::Series->new(
    keys   => [ 0 .. $#{ $fcf_values } ],
    values => $fcf_values,
    name   => 'Free Cash Flow (thousands)',
);

my $trend_line = Chart::Clicker::Data::Series->new(
    keys   => [ 0, $#{ $fcf_values } ],
    values => [ $first_y, $last_y ],
    name   => 'Free Cash Flow trendline',
);
my $dataset = Chart::Clicker::Data::DataSet->new(
    series => [ $fcf_line, $trend_line ],
);

$chart->add_to_datasets( $dataset );
my $context = $chart->get_context('default');
$context->range_axis->format('$%.0f');
$context->domain_axis->hidden(1);
$context->domain_axis->ticks( $#{ $fcf_values } );
$chart->write_output( "${symbol}.png" );

A chart contains one or more datasets, and a dataset contains one or more series. Each series corresponds to a line. Populating a series is easy, given arrays of data; keys represents the X axis and values represents the Y axis.

Most of the rest of my code customizes the display of the data. I haven't found the right way to display the X axis yet, so I've elided that for now. I also had to customize the underlying graph lines, as mentioned before. That customization was the only tricky part of using Chart::Clicker, and that only because it took a few minutes to figure out how to do it.

The results are attractive. Here's a chart showing the earnings for Coca-Cola (NYSE:KO) over the past decade:

Figure 1. Free Cash Flow and Trendline for NYSE:KO.

Chart::Clicker is fast, too. I added this to my analysis step for the 30 stocks in the Dow Jones Industrial Average, and I can't measure the increase in time required to create these images. (This analysis step has network IO as its bottleneck.)

I'm not often wholly impressed by Perl and the CPAN anymore; I expect things to work. I didn't expect things to work as easily as they did today. The whole experiment demonstrates the best the CPAN has to offer.

After a disastrous attempt to write my own templating language as about the third program I ever wrote in Perl (it was the dot-com boom of the '90s, not that that's any excuse), I moved to Template::Toolkit and have been relatively happy with it ever since.

My first real paid programming job was a little GUI app for a customer service group at HP. Customer service agents who needed to escalate to second line support would click on the button for the printer line about which they had to ask a question, and the program recorded the vote, then printed a nice report at the end of the day. It solved a problem. As far as I know, it was still running when I left HP a couple of years later.

A couple of years later, I took a job where we refactored, maintained, and extended a GUI point of sale system.

Since then, I've avoided most graphical programming. Sure, I put together websites for clients once in a while, but most of my work has been emitting the most basic semantically-useful HTML possible such that a Real Designer can manipulate things with CSS (CSS being, of course, one of those so-horrible-it's-almost-good things in that it's the only way to get things done, but you always want to take a shower after you use it, lest you think you start to appreciate it for anything other than its efficacy. See also JavaScript and PHP.).

Most of this meant dropping a big blob of content in a Template Toolkit wrapper in the middle of some HTML, or maybe templatizing some repeated HTML element while iterating over a collection in the toolkit.

Then I decided to redesign a site.

Twitter has its problems (I hope never to understand how or why someone would take a perfectly functional website then try to make it work like a buggy phone app in the same way that I never understood why the first Harry Potter movie hewed so closely to the book it was as boring as a Merchant and Ivory movie and it had wizards in it), but Twitter's Bootstrap CSS framework actually made sense to me, and it let me put together a couple of pages that looked good—far better than the Frankenstein's monster I cobbled together from the "Hey, everything's a blog now, right?" OSWD designs I liked.

(Bootstrap has its problems too, but the worst one is that the Less CSS abstraction layer of CSS is intricately tied to the Lovecraftian bonepile of crazy that is Node.js, because if there's anything I want to do in JavaScript less than write a templating system to perform text substitutions in a cooperative multitasking system, I don't know what it is. It probably involves sharks, skydiving, live volcanoes, and dental work. Yet even only being able to extract repeated colors into named variables and build a static CSS file is a huge improvement, so I put on protective eyeware.)

The experience turned out relatively enjoyable. I had a nice looking wrapper and a decent framework for displaying and managing content.

Then I wanted to change the way I displayed certain elements.

Here's the thing about web programming, or at least the way I'm doing this project. I don't think in terms of pages. I think in terms of components of pages. I have a templates/components/ directory full of reusable Template Toolkit components processed with INCLUDE and PROCESS. The big blurbs of marketing text and instructions and explanations on various pages all live in individual components. Sure, it's a little bit of work to figure out the layout, but this separation of concerns makes editing the site much easier.

It also makes revising the layout more difficult—if changing the layout requires modifying lots of templates with the wrong <div> names and classes and such.

It's possible to write more TT components to abstract away these changes, but the point of diminishing returns appears quickly: TT's syntax and semantics just aren't strong enough to define functions and manage parameters.

Good thing Perl is.

Fewer than 20 minutes after I realized I needed a custom plugin, I had it written:

package MyProject::Template::Plugin::Bootstrap;
# ABSTRACT: basic Bootstrap helpers for the Template system

use Modern::Perl;

use parent 'Template::Plugin';

sub new
{
    my ($class, $context, @params) = @_;

    $class->add_functions( $context );

    return $class->SUPER::new( $context, @params );
}

sub add_functions
{
    my ($class, $context) = @_;
    my $stash             = $context->stash;

    for my $function (qw( row sidebar sideblock maincontent fullcontent span ))
    {
        $stash->set( $function, $class->can( $function ) );
    }

    $stash->set( process => sub { $context->process( @_ ) } );
}

sub row
{
    return <<END_HTML;
<div class="row">
    @_
</div>
END_HTML
}

sub sidebar
{
    return <<END_HTML;
<div class="span4">
    @_
</div>
END_HTML
}

sub sideblock
{
    return <<END_HTML;
<div class="well">
    @_
</div>
END_HTML
}

sub maincontent
{
    return <<END_HTML
<div class="span8 maincontent">
    <div class="hero-unit">
        @_
    </div>
</div>
END_HTML
}

sub fullcontent
{
    return <<END_HTML
<div class="maincontent">
    <div class="hero-unit">
        @_
    </div>
</div>
END_HTML
}

sub span
{
    my $cols = shift;
    return <<END_HTML;
<div class="span$cols">
    @_
</div>
END_HTML
}

1;

This turned my templates into:

[% USE Bootstrap %]

[% row(
    maincontent( process( 'components/content/home_text.tt' ) ),
    sidebar(
        sideblock( process( 'components/forms/login_box.tt' )),
        sideblock( process( 'components/boxes/top_recommendation.tt' ) ),
        sideblock( process( 'components/content/newsletter_text.tt' ) ),
    ),
) %]

This reminds me a bit of Generating HTML from Smalltalk's Seaside or Ruby HAML, except it's less "Wow, lots of tags here!" than Seaside and "So cute people hate you for saying how ugly it is!" than HAML.

I haven't convinced myself I have the right abstractions yet—it's not quite to the point of the semantics I like, it produces its own repetitions, and the idea of returning strings of HTML from a template plugin feels a little wrong to me, but the big advantage is that it's taken a lot of repetitive niggly code and turned it into less code that's much more declarative. The repetition is in the structure of the semantics of the site and not the mechanics of how to produce those semantics.

In other words, this is a step toward thinking in widgets rather than UI primitives.