Just A Summary

I'm working on a web service, and that means that I need to build lots and lots of mildly different looking HTTP requests with various combinations of headers and requested URLs. The camel's back got broken this morning when I realised I didn't want to be writing a method called ssl_request_from_uk_with_bad_cert, which builds me an HTTP::Request with a particular combination of headers, that I can use with Plack::Test to test our webservice. The method name describes what's wanted, but the code is sopping wet and in desperate need of DRYing up.

My first cut was to write a simple request builder:

package RequestBuilder;
use Moose;

with 'MooseX::OneArgNew' => {
  type => 'HTTP::Request',
  init_arg => '_req',
}

has _req => (
  is => 'ro',
  isa => 'HTTP::Request',
  handles => [qw(header headers)],
);

sub ssl_request {
  my $class = shift;
  $class->new(HTTP::Request->new(
    GET => 'https://localhost/',
    [
      SSLSessionID => 'deadbeef',
    ],
  );
}

sub from_uk {
  my $self = shift;
  $self->header('X-IP-Is-UK-Combined' => 'yes');
  return $self;
}

sub with_bad_cert {
  my $self = shift;
  $self->header('SSLClientCertStatus' => 'NoClientCert');
  return $self;
}

sub final {
  my $self = shift;
  $self->_req;
}

Now I have a builder, I can compose meaningful fragments to build a final request. So I might write:

my $r = RequestBuilder->ssl_request
    ->from_uk
    ->with_bad_cert
  ->final;

But things got sticky when I wanted to to take a valid request and bend it out of shape to ensure that it failed correctly. I was writing things like:

use RequestBuilder;

sub valid_request {
  my($self, $is_final) = @_;
  my $r = RequestBuilder->ssl_request->from_uk;
  return $is_final ? $r->final : $r;
}

test "we reject requests from outside the UK" => sub {
  my $self = shift;
  my $r = $self->valid_request(0)->from('Turkmenistan')->final;

  ...
};

which works, but is uglier than a very ugly thing indeed. What I wanted was some way of having the finalization magically happen at the point of use but have some way of extending any intermediate results. The interface I came up with looks like this:

use RequestBuilder;

sub valid_request {
  build_request {
    $_->ssl_request->from_uk;
  }
}

test "we reject requests from outside the UK" => sub {
  my $self = shift;
  my $r = build_request {
    $self->valid_request->from('Turkemenistan');
  };

  ...
};

When the builder is returned from the outermost build_request block, it gets finalized.

"How does that work then?" I hear you ask. It's quite simple, once you know about Moose::Exporter, old fashioned perl prototypes and the Tao of dynamic scope. We just add something like the following to the end of our RequestBuilder:

our $building;

use Moose::Exporter;
Moose::Exporter->setup_import_methods(
  as_is => ['build_request'],
);

sub build_request (&) {
  my $block = shift;
  my $builder = do {
    local $building = 1;
    local $_ = __PACKAGE__;
    $block->();
  };
  return $building ? $builder : $builder->_req;
}

So, build_request has a & prototype, which means it takes a block as its first argument. Perl treats an initial prototyped block argument as slightly magical and doesn't require the use of the sub keyword (though you can use it if you want). It then uses a do block to dynamically set $building to 1 and $_ to be the current __PACKAGE__ name (because $_ is shorter to type than RequestBuilder) before calling the block to get a builder. Then it checks whether we're still building. If we are, it returns the builder. If we aren't, then the block was the outermost block, so build_request returns the built request.

I'm not quite ready to extract this pattern into a parameterized role - I need to make sure it's robust enough, but it's certainly something to think about when you next need to make a builder for your tests.

So, Giles Bowkett asked me on facebook “Why Perl?”. This is the long answer.

I’m a Perl hacker. I have been for around 16 years now. Around 5 years ago, prompted by the Pragmatic Programmer and Adam Turoff, I started looking at Ruby, and Ruby on Rails and sort of fell into maintaining Typo.

Why? I was getting hacked off with Perl.

I was coming to the end of my tenure/tether as Perl 6 Summarizer: watching a language that I still want to use before I die taking forever to get done gets wearing after a while, especially when you’re spending 8 hours a week summarizing the activities of a development community that, in parts, was verging on the toxic (it’s way better now).

I was also getting annoyed by small niggles in the way Perl 5 works. This was in the days before Moose. Stevan Little was just starting work on what became Class::MOP (as a prototype of Perl 6’s metaobject protocol) and way before the days of Devel::Declare.

Ruby seemed to be, as Matz has described it, “Perl, done right”. All the things that were pissing me off about Perl were so much easier in Ruby. I’ve joked, in a Ha Ha, only serious kind of way, that I left Perl for Ruby because I got fed up of unrolling @_, but there’s more to it than that. “Objects everywhere” fits my head. The language is dynamic. Ruby code had less boilerplate in it. Oh, and Ruby on Rails was looking very cool, even if David Heinemeier Hansson didn’t seem to have a clue about RESTful/Resourceful principles (he has rather more of a clue now). The community looked vibrant. What’s not to like?

I was a very happy Ruby programmer. It’s a great language. If you haven’t taken it out for a spin round the block, you should give it a try.

A Perl 5 rennaissance

While I was away on my failed attempt to become a maths teacher and then my Ruby sojourn, Perl 5 was waking up. If Perl 6 were declared a failure tomorrow by the people who are actually working on it (as opposed to the people who aren’t working on it, but bitch about it anyway – see some of chromatic’s more intemperate posts about those people), then it will still have had value for the way it’s inspired members of the Perl 5 community to nick the good bits and make them happen in Perl 5.

Stevan finished Class::MOP and it got used as the basis for Moose, which is essentially a layer of sensible defaults which sits on top of the insanely configurable and slightly more agnostic Class::MOP, which in turn sits on Perl’s weird reduced instruction set object orientation. Moose even got fast enough that it’s usable in polite company (it was always faster than Ruby), so people did. It grew an ecosystem.

Another, more recent development, is Devel::Declare. Devel::Declare is, in accordance with the long history of Perl, completely batshit insane. What it does is lets you trip the perl parser up and, while it’s standing there looking at the pretty birds flying around its head, you can run ahead and rewrite the code that it’s about to parse. Well, I say run, but what I mean is hobble ahead wearing narrow blinkers and mittens. You can accomplish amazing things in such circumstances, but it’s not fun. Devel::Declare enabled my current favourite module on CPAN: MooseX::Declare, which I’ve talked about here:

An introduction to MooseX::Declare from Piers Cawley on Vimeo.

MooseX::Declare lets me kill boilerplate. Instead of writing:

package Something;
use Moose;

has an_attribute => (is => 'rw');

sub a_method {
  my($self, $with, $parameters) 
      = @_;
  ...
}

I can write:

use MooseX::Declare;
class Something {
  has an_attribute => (is => 'rw');

  method a_method($with, $parameters) {
    ...
  }
}

which makes me happy. This is being done by a module which is written in Perl. It relies on a couple of modules that are implemented in C, but the bulk of the work is done in Perl itself.

So, that’s addressed one of the annoyances that led me to Ruby. What about ‘objects everwhere’. I got used writing things like:

5.times { |each| ... }

Surely I’m not going to able to write:

5->times(sub {...})

in perl?

Well…

use Moose::Autobox;
sub Moose::Autobox::SCALAR::times {
    my($count, $block) = @_;
    1->to($count)->each_key($block);
}


5->times(sub {$_->say});

Bingo! I’d like to be able to write 5->times { $_->say } but can’t quite manage it. Give PerlX::MethodCallWithBlock a little more time though…

So, while I was away, most of the issues I had with Perl, issues that had driven me into the arms of Ruby, had been addressed.

It’s not about the bike

The new Perl tech is great, but all it does is removes barriers. I wasn’t attached to the Perl language by some huge bungee cord which was crushing me against these barriers while I sojourned in Ruby land, all the while pining for Perl. I was loving Ruby. It’s a great language.

What I am attached to is the group of friends I’ve made in the Perl community over the years. Many of whom I’ve never actually met face to face.

I felt like such a curmudgeonly old fart in the Ruby community (more specifically, the Rails community). Everything’s shiny and new and awesome. And, I suppose when you’re coming to the language from something like PHP, it is shiny and new and awesome. But it gets wearing after a while. And the arrogance? My dear! The arrogance! Admittedly, some people have a lot to be arrogant about – I still cheer DHH’s “Fuck you!” slide, but others… not so much. The kind of “Look! Ruby is uniquely suited to writing DSLs!” bullshit¹ that sends chromatic off in fits of apoplexy annoyed me too, especially when I was looking at the kind of examples they were presenting as exemplary and remembering Perl from 5 years before that did the same thing but with cleaner language-like interfaces.

Coming back to Perl may well be a straightforward retreat to competence. I may be rationalizing like mad. But right now, it feels like I left Ruby because the ruby community, in the West at least, isn’t a fun place for me to be. The sexism is just the icing on that particular cake.

¹ Here’s my response to Rspec:

testclass exercises Something {
    test that we can create an object {
        isa_ok $test->subject->new, 'Something';
    }
}

Note the lack of perlish furniture in the test that we can create an object part. No need to quote the strings, no meaningless dos scattered about the place, no :symbols appearing at random. Okay, so I have to quote 'Something' in the implementation block, but the implementation block is unadulterated Perl. I don’t claim that only Perl can do this. I do claim that, right now at least, Ruby can’t.

There’s something satisfying about reaching the point when you can’t decompose an object any further and all your methods are tiny and do one thing – it’s especially gratifying when you learn something new in the process. Sadly, it doesn’t happen as often as I’d like, there’s usually annoying bits and pieces where you have to placate the language in some fashion that breaks the flow of what you’re writing.

As I get a better handle on the way MooseX::Declare has changed Perl, I’m finding I have to do much less in the way of placation.

Here’s an example. For context, I’m writing a traffic shaping tool. The basic client interface needs to look something like:

$policy->current_weight # => a percentage between 0 and 100

Not much of an interface really. The requirements state that we should be able to specify weights with 15 minute granularity for every day of the week. Our problem becomes one of mapping from a time to a number between 0 and 7 (days) * 24 (hours) * 4 (quarters) - 1 and looking up the weight in an array.

Here’s my first cut:

use MooseX::Declare;
class WeightVector {
    use DateTime;
    
    has vector => (
        isa => 'Array[Int]',
        is => 'ro',
        required => 1,
    );
    
    method current_weight {
        my $now = DateTime->now;
        my $offset = ($now->wday_0 * 7 * 24 + $now->hours) * 4 + int($now->minutes / 15);
        return $self->vector->[$offset];
    }
}

Which is, I suppose, perfectly respectable. However, current_weight isn’t filling me with delight. First it finds the current time, then it converts the time into an offset, then it uses the offset to lookup the weight in the vector. Let’s introduce a method to find the weight at a specific time"¹":#decomp-motivation, the relevant code becomes:

method current_weight {
    $self->weight_at(DateTime->now);
}

method weight_at (DateTime $time) {
    my $offset = ($now->wday_0 * 7 * 24 + $now->hours) * 4 + int($now->minutes / 15);
    return $self->vector->[$offset];
}

And again, we could rest here, but again, we’re doing two things. We’re converting from a time to an offset, then we’re looking up the value in the vector. Type conversions tend to happen again and again, so it’s good if we can specify them separately. We could write a time_to_offset helper method, but we’re in Mooseland now; there’s a better way. Let’s introduce a formal Moose type and define a coercion for it. Here’s the type definition stanza of the code. I’ve taken the opportunity to add types which do bounds checking for the vector as well, while I’m about it.²

use MooseX::Declare;
class WeightVector {
    use DateTime;
    use Moose::Autobox;
    use MooseX::Types -declare => [qw(SlotOffset VectorOfWeights PercentageInt)];
    use MooseX::Types::Moose qw(ArrayRef Int);
    
    use constant TOTAL_SLOTS = 7 * 24 * 4;
    
    BEGIN {
        subtype PercentageInt,
            as Int,
            where { 0 <= $_ && $_ <= 100 },
            message { "$_ does not is not an integeter between 0 and 100" };
    
        subtype VectorOfWeights,
            as ArrayRef[PercentageInt],
            where { $_->length == TOTAL_SLOTS }
            message { "Vector must have ".TOTAL_SLOTS." entries, not ".$_->length };
    
        subtype SlotOffset,
            as Int,
            where { 0 <= $_ && $_ < TOTAL_SLOTS };
    
        class_type 'DateTime';
    
        coerce SlotOffset,
            from 'DateTime',
            via { ($_->wday_0 * 7 * 24 + $_->hours) * 4 + int($_->minutes / 15) };
     
        # Let's allow clients not to care about using DateTime by allowing 
        # them to simply pass the results of calling 'time()' - It's not like it's
        # still 1970...
    
        coerce SlotOffset
            from subtype(as => Int, where { $_ > TOTAL_SLOTS }
            via { to_SlotOffset(DateTime->from_epoch(epoch => $_) };
    }
    
    has vector => (
        is => 'ro',
        isa => VectorOfWeights,
        required => 1,
    }
    
    ...

Now, if we were programming in plain old Moose, we could rewrite weight_at like so:

sub weight_at {
    my $self = shift;
    my $offset = to_SlotOffset(shift);
    $self->vector->[$offset]
}

Which would be pretty sweet, but we’re using MooseX::Declare; there’s an even better way:

method weight_at (SlotOffset $offset does coerce) {
    $self->vector->[$offset];
}

Sweet!

We could stop there, but I had an insight. What we’ve got here is basically a wrapper around a delegation to our vector, and Moose’s new native types feature let us express the delegation to the vector quite neatly, like so:

has vector => (
    isa => 'VectorOfWeights',
    is => 'ro',
    required => 1,
    traits => ['Array'],
    handles {
        weight_at => 'get',
    },
);

...

around weight_at (SlotOffset $offset does coerce) {
    $self->$orig($offset);
}

This could be overkill when vector is a simple ArrayRef as we have here, but the pattern of delegating declaratively in the attribute definition and then munging arguments in an around handler is applicable to more than just argument transformation. A typical delegation pattern involves having the delegating object passing itself in as an argument to the method delegated to. The nature of Moose’s handles declarations makes that impossible to do within the attribute declaration, but it’s easy to fix with an around helper:

around delegated_method (Any @args) {
    $self->$orig($self, @args);
}

(If you’re wrapping more than one method in this fashion, you should probably consider using a plain old Moose style around handler, which lets you wrap multiple methods with around @delegated_methods => sub {...}

So, at the end of all that, and after we’ve extracted our Type declarations into WeightVector::Types, we have:

use MooseX::Declare;
class WeightVector {
    use WeightVector::Types qw(VectorOfWeights PercentageInt SlotOffset);
    
    has vector => (
        isa => 'VectorOfWeights',
        is => 'ro',
        required => 1,
        traits => ['Array'],
        handles {
            weight_at => 'get',
        },
    );
    
    method current_weight {
        $self->weight_at(time());
    }
    
    around weight_at (SlotOffset $offset does coerce) {
        $self->$orig($offset);
    }
}

And we’ve pushed all knowledge of DateTime off onto our type declarations and gained a boatload of handy bounds checking. We’ve also got a new tool for handling tricky delegation setups in the handles/around combo.

Notes

Motivation

I realise that this looks like a radical decomposition of the class with very little motivation, but it was driven by tests and by some other requirements that I’ve removed from the body of the post. In particular, the type coercions were driven by the need to build particular vectors for testing, a key method being:

method set_weight (PercentageInt $weight, 
                   SlotOffset $from does coerce,
                   SlotOffset $to does coerce)
{
    ...
}

Type coercion is wonderful

Generally, I’m not a fan of static typing. I’m from the “duck type all the way” school of programming,³ so most of my method declarations have no type declarations. But type declarations, especially ones that coerce, make so much sense on methods that make up the public protocol of a class. I only use type declarations on internal methods when I need a narrower coercion, or if I’m using MooseX::Multimethods, which I still haven’t used for anything but exploration.

Updates

Thanks to Chris Dolan for spotting that I’d got the SlotOffset coercion completely wrong. The real code’s doing the right thing, but that’s what comes of recreating code from memory.

¹ This was actually motivated by trying to write tests to verify that the weights were correctly set.

² I’m declaring these in a BEGIN block of the class itself mostly for explanatory purposes – there’s a good case for moving them out into a separate file and pulling it in with use.

³ Except during my periodic attempts to learn Haskell. I’ve learned Haskell at least three times now.

tap tap… Is this thing on?

So, I recently noticed that Test::Class 0.33 got released, which means that Test::Class::Sugar no longer needs to depend on a development release, and I also noticed that it was embarrassingly easy to throw Test::Class::Sugar into an infinite loop by forgetting which way the >> goes when you want to specify the number of subtests in a test method.

So, I’ve done a quick fix of the infinite loop problem as well and uploaded version 0.3 to PAUSE, so now you can write your tests like:

testclass exercises ClassUnderTest {
    test creation of the class under test {
        lives_and {
            isa_ok ClassUnderTest->new, $test->subject;
        }
    }
    ...
}

without having to jump through the hoops of downloading a development version of Test::Class or worry about accidental infinite loops…

Next up, fix the syntax to either allow both << and >> as test count specifiers, or come up with a more memorable way of separating the count from the test name.

Update:

Shortly after I released 0.3, Joel Bernstein asked if I’d be interested in a topic branch to make Test::Class::Sugar work with perl 5.8.

“Of course!” I said.

One day later, there it was. Thank you to Joel and to his employers, NET-A-PORTER for sponsoring his work. So now, Test::Class::Sugar 0.4 is winging its way to CPAN and now I have no excuse for not using it at work.

At last! Start your compilers everybody.

Back in January, I wrote that I was choosing Perl to revisit as my language for recreational programming.

Quite a bit has happened since then. I gave a talk on MooseX::Declare to the London Perl Mongers and will be delivering an extended version at this year’s YAPC::Europe in August.

Then, a week ago, I accepted an offer to work full time as a Perl programmer again. In London.

Sing “Ho!” for the life of the long distance commuter.

Back in (crikey) February, I gave a talk at the London Perl Mongers’ technical meeting about Moose for Ruby Programmers and wrote it up here. Mike Whittaker was in the front row of the audience with his iPhone and, a couple of minutes in, started a voice recording and gave me a copy.

So… finally… I’ve taken the time I should have been using to write another article for The H and wrestled the slides and the audio into something like sync and uploaded the results to Vimeo for your viewing pleasure.

An introduction to MooseX::Declare from Piers Cawley on Vimeo.

I’ve just pushed the second version of Test::Class::Sugar (first discussed here). It’s pretty much as discussed in the original article, but after some discussion with David Wheeler, I’ve dropped the +uses clause from the testclass declaration in favour of less DWIMmy (and thus easier to explain behaviour).

I’ve also introduced a defaults argument at use time. The only working default in this release is one which lets you specify the prefix used to derive the test class name from the class/module under test. I’ve documented a couple of extra and so far unimplemented defaults as well.

Have a play, you might like it.

I got a bit of stick on IRC last night for some of the choices I’d made when I was writing Test::Class::Sugar, in particular because one of the prerequisites is chromatic’s handy and opinionated Modern::Perl module. The ‘controversial’ aspect of Modern::Perl is that, when you use it, your code won’t run on any Perl before Perl 5, version 10.

The thing is, I don’t care about older perls any more. Version 10 features like ~~ and // are too convenient to fart about writing circumlocutions just to run on a version of Perl that I have no intention of ever using again.

Actually, that’s not quite true, those version 10 features are too convenient for me to fart about working around their absence. If you think that a module I write is useful enough that you want it to work on version 8, then of course I’ll accept your patch. But don’t be surprised if, when I start adding new features, I break the backward compatibility.

Also, on the happy day that version 10.1 escapes the pumpking patch, I’ll be setting that as my minimum perl version even if chromatic doesn’t bump the version number in Modern::Perl.

It’s all about the virtues

Your context is different from mine. I’m writing in Perl again for my own amusement more than anything. There are developments in modern Perl – tools like Moose and Devel::Declare – that I think are exciting and important. The Announcements project I started was as much about playing with the new tools as it was about trying to write something of wider utility. Test::Class::Sugar arose as a direct result of attempting to write Announcements and the desire to write test classes without hoopage. My principle drive then, is impatience to get Test::Class::Sugar to the point where I can get back to writing Announcements.

But then laziness and hubris kick in. So the code needs some polish. The parser and the code generator need to be disentangled, I need to get Adrian Howard to apply the little patch I had to make to Test::Class. Laziness demands I document it.

Impatience tells me to lean on the features of modern perl – that way I can get back to being a user of the new library as quickly as possible. Laziness tells me that I’m not going to need backwards compatibilty. Hubris tells me my work is good enough that someone who does will like it enough to send me a patch.

Everybody wins.

In which your correspondent does magical battle with the guts of Perl and emerges bloodied, but unbowed with a useful principle to code by.

Skip to the conclusion if you’re uncomfortable with the guts of the Perl runtime

Test::Class had me tearing my hair out earlier. There I was, happily transforming

test something {
  ok 1;
};

into something very like¹:

*test_something = 
  Sub::Name::subname(
      'test_something' 
    => sub : Test { ok 1 }
  );

through the magic of Devel::Declare, but Test::Class didn’t seem to be playing fair. Instead of letting my tests run happily, it was complaining that it:

The thing is, I wasn’t loading Test::Class too late. The problem is that, at the point I applied the Test attribute to my sub, the sub didn’t have a name and, because of the constraints you’re operating under when you’re using Devel::Declare to do code transformation, there was no obvious way to give it a name in time.

Incompatible magics

The trouble is, Test::Class does what it does through the magic of compile time code attributes, and, further, it relies on the fact that if a perl subroutine that gets inserted into the symbol table like this:

sub has_a_name {...}

Then, when you get hold of a reference to that code by other means (say, in the subroutine that handles the setting of an attribute, that code ref knows its own name. However, if a subroutine that ends up in the symbol table like this:

*anonymous_ref = sub {...};

Doesn’t know its name, unless you take advantage of the Sub::Name module.

So, in my generated code, I was giving my coderef a name, but it was happening to late. At the point that Test::Class::Test method was seeing the coderef, the coderef was anonymous.

My magic and Test::Class’s magic were incompatible.

The thing is, both sorts of magic are really just sugar for some pretty mundane donkey work. Test::Class does what it does through attributes because no flesh and blood programmer in their right mind would want to write something like this every time they wanted to write a test method:

sub test_something {
  ...
}
__PACKAGE__->mark_as_special_method('test_something', 'test', '3');

In fact, mark_as_special_method doesn’t even exist as its own subroutine. The code that marks a method as special is just part of the body of the Test attribute handler.

Conclusion

Which brings me neatly to my conclusion.

When you’re designing a module that does anything magical, consider starting with a mundane core API that handles the business side of things. Then layer your magic on top of that API. Then document the API and the magic. Obviously the magic bits go up front in the docs, and the API goes in its own section (or even podfile) down at the bottom, where only eejits like me, who want their magic to work slightly different to yours, will bother reading it.

Obviously, I’m motivated by an issue I’m having with a particular module from CPAN, but the principle of separating the magic and the mundane is applicable everywhere. It’s called Separation of Concerns, or The Single Responsiblity Pattern. I call it a Just Story.

You’ll find the pattern in well designed websites that are using unobtrusive javascript to wave an AJAX wand over the site. You’ll see it woven through books like The Structure and Interpretation of Computer Programmers – where it’s called an Abstraction Barrier.

Patches sent

It turns out to be very easy to add mark_as_special_method (though I actually wrote it as ‘add_testinfo’ in the patch) to Test::Class. It’s about as straightforward an Extract Method refactoring as I’ve ever done – even without automated tools, I managed not to fuck it up. There’s a patch in Adrian Howard’s inbox, and I’m hopeful that it’ll be applied soon.

¹ Not exactly – that’s the result of calling the shadowed test subroutine which was the result of the code transformation.

Check out the osfameron fork of Devel::Declare for the beginnings of some decent documentation which explains what’s going on.