Planet Bazaar

testtools's sister project, subunit, was using a private API that we deleted in the 0.9.13 release.  This release restores that API in order to smooth out the upgrade path.

If you don't use subunit, then this release won't matter very much to you.

We’re publishing an initial version of the Ubuntu Business Desktop Remix today, based on Ubuntu 11.10.

Deployment teams have long been modifying their Ubuntu installs to remove features like music players or games and add components that are a standard part of their business workflow.

This remix takes the most common changes we’ve observed among institutional users and bundles them into one CD which can be installed directly or used as a basis for further customization. Before anyone gets all worked up and conspiratorial: everything in the remix is available from the standard Software Centre.  Packages out, packages in. No secret sauce for customers only; we’re not creating a RHEL, we already have an enterprise-quality release cadence called LTS and we like it just the way it is. This is a convenience for anyone who wants it. Having a common starting point, or booting straight into a business-oriented image makes it easier for institutional users to evaluate Ubuntu Desktop for their specific needs.

This work was first discussed at the Ubuntu Developer Summit in October. We consulted with the Ubuntu Technical Board and Ubuntu Release Team, to ensure that the finished product met the standards of the Ubuntu project.  Doing so resulted in a commitment to enable community participation in the packaging of some of the pieces that are important to enterprise users.

Ubuntu makes a point of openness to heterogeneous environments. We celebrate the point that the Ubuntu desktop can be highly useful, beautiful, functional and complete without any proprietary applications at all, while recognising that some people need to work with proprietary software on occasion, making sure that software is available and certified for Ubuntu, and making it easy to install. Remixes can include non-free software and still retain the Ubuntu name, as long as they can be brought back to the standard Ubuntu experience with straightforward package management tools and no risk of divergence on the hardware and security front.

Since we established the system of remixes, the Technical Board has defined guidelines for additional package archives which are exposed to Ubuntu users through the Software Centre. We’ve clarified with the Technical Board that remixes can draw from any such archives.

<blink>Registration required</blink> ;-) Some applications like VMWare View are included in this release under a proprietary license so download is covered by an EULA, and this image can’t be mirrored unless you make prior arrangements with the relevant ISVs. Boring, but better to do it once than for every individual app. We will ask users who download it to provide feedback on how we might improve the product, and provide them with details of Canonical’s deployment services and management solutions.

Get it here.

It has been a while, but testtools 0.9.13 is finally out! Lots more matchers and bug fixes, as well as improved error reporting. Full release notes on Launchpad.

Thanks to James Westby, Graham Binns, Francesco Banconi and Robert Collins for making this release possible.

Rich Hickey did a great talk at Strange Loop called "Simple Made Easy". You should watch it.

When I tried to explain the talk to someone, I stumbled a lot and it was obvious to me that I didn't really understand it. So I'm going through it again and turning it into a blog post, purely for my own gain.

This is roughly the first half of the talk. Not much of my own analysis or opinion is inserted, and I've pretty much stuck with Hickey's illustrations and phrasings. Thus this post is pretty derivative. Oops.

Simple vs Easy

"Simple" means one thing, "easy" another. Simple is the opposite of complex. A thing is simple if it has no interleaving, if it has one purpose, one concept, one dimension, one task. Being simple does not imply one instance or one operation: it's about interleaving, not cardinality. Importantly, this means that simplicity is objective.

Easy is the opposite of hard, or difficult. A thing is easy if it's near to hand, if it's easy to get at (location), if it's near to our understanding (familiarity) or skill set or if it's within our capabilities. This means that ease is relative.

Speaking English is dead easy for me, but that doesn't mean that speaking English is intrinsically simple. I find French quite difficult. Little French children speak French all the time, and there's always a part of me that thinks, "Boy, those kids are clever, being able to speak a foreign language at that age", but that's silly. It's easy for them, it lies near to them.

This distinction between simple & easy is good one, and is useful in all sorts of areas. But how does it relate to software?

Constructs vs Artefacts

As programmers, when we make software we are working on constructs: source code, libraries, language concepts and so forth. Rich contends that we focus on the ease of use of those constructs: How many lines of code? How much boilerplate? Will new developers be familiar with our technology?

But all of this is secondary. What actually matters is the artefact, the running programs that users actually use. Does it do what it's supposed to do? Does it do it well? Can we rely on it working well? Can we fix problems when they occur? Can we change it? You know, the interesting problems.

Thus we need to be assessing our constructs – our code, our technology choices – based on the attributes of the artefacts that we'll create, not based on the experience of typing code in.

Limits

We can't make something reliable if we don't understand it. And, actually, everyone's understanding is pretty limited. We can all only hold a small number of things in our head at once.

When things are complex, many parts are tied together by definition. You can't pull out just one piece and consider it because it's intertwined with other pieces. This creates an extra burden to understanding a system and thus makes it difficult to reason about the system.

You do need to reason about a system, both to know what to change and to be able to do so without introducing defects. Tests, refactoring, rapid deployment and all that are great, but to make a change to the system safely & without fear still requires you to be able to reason about it. Every bug in your product that was found in the field passed the type checker and passed all of the tests.  Your type system doesn't tell you what change to make next in order to get the software you want any more than guard rails on a highway tell you how to get to Grandma's.

Speed

Focusing on ease and ignoring simplicity means that you'll go really fast in the beginning, but will become slower and slower as the complexity builds.

Focusing on simplicity will mean that you'll go slower in the beginning, because you'll have to do some work to simplify the problem space, but making sure that you only have intrinsic complexity means that your rate of development will remain at a high constant.

There are no actual numbers for this.

Complicating constructs

Many complicating constructs are available, familiar, succinctly described and easy to use. But none of that matters to end users. What matters is the complexity they yield. This complexity is incidental, it's not intrinsic to the problem.

If we build things simply, then the resulting system is easier to debug, easier to change and easier to understand.

Compare a knitted castle to a castle made of Lego. The knitted castle might have been great fun to make, and might have been really easy if knitted using a loom and cutting edge knitting tools, but there's no way that it's easier to change than a Lego castle. It's not about the ease of construction, it's about the simplicity of the artefact.

How can we make software easier?

Well, we can install it to make it easier by location. We can learn it and try it to make it easier by familiarity. We can't do much about our capabilities though. If we want software to be easier to comprehend, we are going to have to bring it down to our level. We have to make it simpler.

Take Lisp as an example. It's hard for many people because they don't have a Lisp installed, or their editor doesn't support paren matching, but they can make it easier by installing a Lisp and getting a plugin for their editor. It's also hard because it's unfamiliar. Who'd have thought that parens could go on that side of the function? But you can gain that familiarity quickly enough.

But parens in Lisp are used for functions and for grouping data. That's hard to get your head around, and that's because it's complex. It braids together two distinct notions.

One of the lesser-sung attributes of Test-Driven Development is that it can actually save you development effort.

In TDD, you start by creating a failing test. Then you write code to make it pass. Sometimes the test will not fail. Perhaps you're written the test incorrectly, but other times, the functionality you planned to implement is already implemented elsewhere (e.g. in the framework / library you're using). That means you can skip implementing useless code. Remember, functionality is an asset, but code is a liability.

This happened to me yesterday, when I was refactoring some Launchpad code. Turned out Zope sets the Content-length http header if you don't. It could have happened today, because Zope also encodes Unicode strings to utf-8 for you, but I was unit-testing at a different level, so I implemented, then removed, manual utf-8 encoding.

The desktop remains central to our everyday work and play, despite all the excitement around tablets, TV’s and phones. So it’s exciting for us to innovate in the desktop too, especially when we find ways to enhance the experience of both heavy “power” users and casual users at the same time. The desktop will be with us for a long time, and for those of us who spend hours every day using a wide diversity of applications, here is some very good news: 12.04 LTS will include the first step in a major new approach to application interfaces.

This work grows out of observations of new and established / sophisticated users making extensive use of the broader set of capabilities in their applications. We noticed that both groups of users spent a lot of time, relatively speaking, navigating the menus of their applications, either to learn about the capabilities of the app, or to take a specific action. We were also conscious of the broader theme in Unity design of leading from user intent. And that set us on a course which led to today’s first public milestone on what we expect will  be a long, fruitful and exciting journey.

The menu has been a central part of the GUI since Xerox PARC invented ‘em in the 70′s. It’s the M in WIMP and has been there, essentially unchanged, for 30 years.

We can do much better!

Say hello to the Head-Up Display, or HUD, which will ultimately replace menus in Unity applications. Here’s what we hope you’ll see in 12.04 when you invoke the HUD from any standard Ubuntu app that supports the global menu:

The intenterface – it maps your intent to the interface

This is the HUD. It’s a way for you to express your intent and have the application respond appropriately. We think of it as “beyond interface”, it’s the “intenterface”.  This concept of “intent-driven interface” has been a primary theme of our work in the Unity shell, with dash search as a first class experience pioneered in Unity. Now we are bringing the same vision to the application, in a way which is completely compatible with existing applications and menus.

The HUD concept has been the driver for all the work we’ve done in unifying menu systems across Gtk, Qt and other toolkit apps in the past two years. So far, that’s shown up as the global menu. In 12.04, it also gives us the first cut of the HUD.

Menus serve two purposes. They act as a standard way to invoke commands which are too infrequently used to warrant a dedicated piece of UI real-estate, like a toolbar button, and they serve as a map of the app’s functionality, almost like a table of contents that one can scan to get a feel for ‘what the app does’. It’s command invocation that we think can be improved upon, and that’s where we are focusing our design exploration.

As a means of invoking commands, menus have some advantages. They are always in the same place (top of the window or screen). They are organised in a way that’s quite easy to describe over the phone, or in a text book (“click the Edit->Preferences menu”), they are pretty fast to read since they are generally arranged in tight vertical columns. They also have some disadvantages: when they get nested, navigating the tree can become fragile. They require you to read a lot when you probably already know what you want. They are more difficult to use from the keyboard than they should be, since they generally require you to remember something special (hotkeys) or use a very limited subset of the keyboard (arrow navigation). They force developers to make often arbitrary choices about the menu tree (“should Preferences be in Edit or in Tools or in Options?”), and then they force users to make equally arbitrary effort to memorise and navigate that tree.

The HUD solves many of these issues, by connecting users directly to what they want. Check out the video, based on a current prototype. It’s a “vocabulary UI”, or VUI, and closer to the way users think. “I told the application to…” is common user paraphrasing for “I clicked the menu to…”. The tree is no longer important, what’s important is the efficiency of the match between what the user says, and the commands we offer up for invocation.

In 12.04 LTS, the HUD is a smart look-ahead search through the app and system (indicator) menus. The image is showing Inkscape, but of course it works everywhere the global menu works. No app modifications are needed to get this level of experience. And you don’t have to adopt the HUD immediately, it’s there if you want it, supplementing the existing menu mechanism.

It’s smart, because it can do things like fuzzy matching, and it can learn what you usually do so it can prioritise the things you use often. It covers the focused app (because that’s where you probably want to act) as well as system functionality; you can change IM state, or go offline in Skype, all through the HUD, without changing focus, because those apps all talk to the indicator system. When you’ve been using it for a little while it seems like it’s reading your mind, in a good way.

We’ll resurrect the  (boring) old ways of displaying the menu in 12.04, in the app and in the panel. In the past few releases of Ubuntu, we’ve actively diminished the visual presence of menus in anticipation of this landing. That proved controversial. In our defence, in user testing, every user finds the menu in the panel, every time, and it’s obviously a cleaner presentation of the interface. But hiding the menu before we had the replacement was overly aggressive. If the HUD lands in 12.04 LTS, we hope you’ll find yourself using the menu less and less, and be glad to have it hidden when you are not using it. You’ll definitely have that option, alongside more traditional menu styles.

Voice is the natural next step

Searching is fast and familiar, especially once we integrate voice recognition, gesture and touch. We want to make it easy to talk to any application, and for any application to respond to your voice. The full integration of voice into applications will take some time. We can start by mapping voice onto the existing menu structures of your apps. And it will only get better from there.

But even without voice input, the HUD is faster than mousing through a menu, and easier to use than hotkeys since you just have to know what you want, not remember a specific key combination. We can search through everything we know about the menu, including descriptive help text, so pretty soon you will be able to find a menu entry using only vaguely related text (imagine finding an entry called Preferences when you search for “settings”).

There is lots to discover, refine and implement. I have a feeling this will be a lot of fun in the next two years

Even better for the power user

The results so far are rather interesting: power users say things like “every GUI app now feels as powerful as VIM”. EMACS users just grunt and… nevermind . Another comment was “it works so well that the rare occasions when it can’t read my mind are annoying!”. We’re doing a lot of user testing on heavy multitaskers, developers and all-day-at-the-workstation personas for Unity in 12.04, polishing off loose ends in the experience that frustrated some in this audience in 11.04-10. If that describes you, the results should be delightful. And the HUD should be particularly empowering.

Even casual users find typing faster than mousing. So while there are modes of interaction where it’s nice to sit back and drive around with the mouse, we observe people staying more engaged and more focused on their task when they can keep their hands on the keyboard all the time. Hotkeys are a sort of mental gymnastics, the HUD is a continuation of mental flow.

Ahead of the competition

There are other teams interested in a similar problem space. Perhaps the best-known new alternative to the traditional menu is Microsoft’s Ribbon. Introduced first as part of a series of changes called Fluent UX in Office, the ribbon is now making its way to a wider set of Windows components and applications. It looks like this:

You can read about the ribbon from a supporter (like any UX change, it has its supporters and detractors ) and if you’ve used it yourself, you will have your own opinion about it. The ribbon is highly visual, making options and commands very visible. It is however also a hog of space (I’m told it can be minimised). Our goal in much of the Unity design has been to return screen real estate to the content with which the user is working; the HUD meets that goal by appearing only when invoked.

Instead of cluttering up the interface ALL the time, let’s clear out the chrome, and show users just what they want, when they want it.

Time will tell whether users prefer the ribbon, or the HUD, but we think it’s exciting enough to pursue and invest in, both in R&D and in supporting developers who want to take advantage of it.

Other relevant efforts include Enso and Ubiquity from the original Humanized team (hi Aza &co), then at Mozilla.

Our thinking is inspired by many works of science, art and entertainment; from Minority Report to Modern Warfare and Jef Raskin’s Humane Interface. We hope others will join us and accelerate the shift from pointy-clicky interfaces to natural and efficient ones.

Roadmap for the HUD

There’s still a lot of design and code still to do. For a start, we haven’t addressed the secondary aspect of the menu, as a visible map of the functionality in an app. That discoverability is of course entirely absent from the HUD; the old menu is still there for now, but we’d like to replace it altogether not just supplement it. And all the other patterns of interaction we expect in the HUD remain to be explored. Regardless, there is a great team working on this, including folk who understand Gtk and Qt such as Ted Gould, Ryan Lortie, Gord Allott and Aurelien Gateau, as well as designers Xi Zhu, Otto Greenslade, Oren Horev and John Lea. Thanks to all of them for getting this initial work to the point where we are confident it’s worthwhile for others to invest time in.

We’ll make sure it’s easy for developers working in any toolkit to take advantage of this and give their users a better experience. And we’ll promote the apps which do it best – it makes apps easier to use, it saves time and screen real-estate for users, and it creates a better impression of the free software platform when it’s done well.

From a code quality and testing perspective, even though we consider this first cut a prototype-grown-up, folk will be glad to see this:

Overall coverage rate:
   lines......: 87.1% (948 of 1089 lines)
   functions..: 97.7% (84 of 86 functions)
   branches...: 63.0% (407 of 646 branches)

Landing in 12.04  LTS is gated on more widespread testing.  You can of course try this out from a PPA or branch the code in Launchpad (you will need these two branches). Or dig deeper with blogs on the topic from Ted Gould, Olli Ries and Gord Allott. Welcome to 2012 everybody!

Here's a thing that happens a lot to me: I'm doing some work, and as part of that work I need to run a command in my terminal that takes a little while. I run the command, look at it for about a second and then switch to doing something else – checking email, perhaps. I get deeply involved in my email checking, and then about twenty minutes later I switch back to the terminal and see the command has finished. For all I know, it finished nineteen minutes ago, and I was just too engrossed to notice it.

This is a big productivity sink for me, especially if the command happened to fail and need retrying. I'm not disciplined enough to just sit and watch the command, and I'm not prescient enough to add something to each invocation telling me when a command is done. What I want is something that alerts me whenever long running commands finish.

Well, that thing now exists, thanks to glyph's script that provides precmd and postcmd support to bash and a lot of help from Chris Jones of Terminator.

To use it right now:
 $ bzr co lp:~jml/+junk/shell-tools
 $ . shell-tools/long-running.bash
 $ notify_when_long_running_commands_finish_install


You'll see that if you run a command that takes over 30 seconds to complete, it will pop up a notification, which should hopefully take you away from whatever it was you are doing and back to the task at hand.

If you look at the code, you'll see that it installs two hooks: precmd and preexec. preexec runs just before the shell launches a command, and precmd runs just before it prompts for the next command. Our preexec stores when the command was launched and the precmd checks to see if it finished within a certain time frame. If not, it sends out a notification.

Currently, you'll get a notification when you finish reading a long document, since the command finishes a long time after the command starts. Obviously this isn't ideal. I think the fix is to only send notifications when the shell doesn't have focus. Unfortunately, that's a little tricky and I think is going to be highly terminal specific.

Anyway, I'm a total shell newbie, so I'd love to know if there's any way this could be done better.  Also let me know if you find this useful, or you know of someone who has already done this.

Jelmer writes:

bzr-builddeb 2.8.1 has just landed on Debian Sid and Ubuntu Precise. This version contains some of my improvements from late last year for the handling of quilt patches in packaging branches. Most of these improvements depend on bzr 2.5 beta 5, which is also in Sid/Precise.

The most relevant changes (enabled by default) are:

Furthermore, you can now specify whether you would like bzr to automatically apply all patches for stored data and whether you would like to automatically have them applied in your working tree by setting ‘quilt-tree-policy‘ and ‘quilt-commit-policy‘ to either ‘applied‘ or ‘unapplied‘. This means that you can have the patches unapplied in the repository, but automatically have them applied upon checkout or update. Setting these configuration options to an empty string causes bzr to not touch your patches during commits, checkout or update.

We’ve done some testing of it, as well as running through a package merge involving patches with Barry, but none of us do package merges regularly. If you do run into issues or if you think there are ways we can improve the quilt handling further, please comment here or file a bug report against the UDD project.

Caveats:

To disable the automatic unapplying of patches and fall back to the previous behaviour, set the following in your builddeb configuration:

quilt-smart-merge = False

I was recently asked to estimate how long I’d been working on a particular project. Unfortunately I hadn’t been keeping track of my time in any organized way.

Fortunately I realized that, since I like to commit frequently (though nothing like Stephen Turnbull’s commit-on-save!), I could come up with an estimate based on my commit dates.

But I quickly realized that bzr log --line puts the author name before the commit date:

  $ bzr log --line -r -3..
  150: Max Bowsher 2011-02-12 [merge] Fix invalid version_info.
  149: Jelmer Vernooij 2010-12-20 [merge] Fixes most of the remaining test fai...
  148: Gary van der Merwe 2010-10-20 [merge] Ignore build folder created by se...
  147: Martin 2010-09-09 [merge] Import xml escaping function through local mo...

The spaces could make extracting the date a bit fragile.

Fortunately I remembered the bzr-xmloutput plugin, which makes processing this kind of information really easy. bzr-xmloutput adds an “–xml” option to many of the standard bzr commands that encodes the output as an XML document. Combined with XMLStarlet, a command-line XML tool that provides XSLT/XPath processing (amongst other things), I was able to cook up a recipe in a matter of minutes:

  $ bzr log --xml \
  | xml sel -t -m '/logs' -m '//log' \
    -v 'substring-before(substring-after(timestamp," ")," ")' -n \
  | sort -u \
  | wc -l

The substring() is required to pull out the date; as bzr-xmloutput prints dates as ‘Day YYYY-MM-DD HH:MM:SS offset’. awk would have worked just as well too.

Too easy, thanks to Guillermo and the other bzr-xmloutput contributors! Now I’m thinking of other questions that can be answered…

This is a tiny PSA prompted by my digging into a deadlock condition in the Launchpad application servers.

We were observing a small number of servers stopping cold when we did log rotation, with no particularly rhyme or reason.

tl;dr: do not call any non-reentrant code from a Python signal handler. This includes the signal handler itself, queueing tools, multiprocessing, anything with locks (including RLock).

Tracking this down I found we were using an RLock from within the signal handler (via a library…) – so I filed a bug upstream: http://bugs.python.org/issue13697

Some quick background: when a signal is received by Python, the VM sets a status flag saying that signal X has been received and returns. The next chance that thread 0 gets to run bytecode, (and its always thread 0) the signal handler in Python itself runs. For builtin handlers this is pretty safe – e.g. for SIGINT a KeyboardInterrupt is raised. For custom signal handlers, the current frame is pushed and a new stack frame created, which is used to execute the signal handler.

Now this means that the previous frame has been interrupted without regard for your code: it might be part way through evaluating a multi-condition if statement, or between receiving the result of a function and storing it in a variable. Its just suspended.

If the code you call somehow ends up calling that suspended function (or other methods on the same object, or variations on this theme), there is no guarantee about the state of the object; it becomes very hard to reason about.

Consider, for instance, a writelines() call, which you might think is safe. If the internal implementation is ‘for line in lines: foo.write(line)’, then a signal handler which also calls writelines, could have what it outputs appear between any two of the lines in writelines.

True reentrancy is a step up from multithreading in terms of nastiness, primarily because guarding against it is very hard: a non-reentrant lock around the area needing guarding will force either a deadlock, or an exception from your reentered code; a reentrant lock around it will provide no protection. Both of these things apply because the reentering occurs within the same thread – kindof like a generator but without any control or influence on what happens.

Safe things to do are:

In Launchpad’s case, we will be setting a flag variable unconditionally from the signal handler, and the next log write that occurs will lock out other writers, consult the flag, and if needed do a rotation, resetting the flag. Writes after the rotation signal, which don’t see the new flag, would be ok. This is the only possible race, if a write to the variable isn’t seen by an in-progress or other-thread log write.

That is all.

I upgraded my primary laptop to Precise yesterday. Very smoooooth! Kudos to the Ubuntu team for the way they are running this cycle; their commitment to keeping the Precise Pangolin usable from opening to release as 12.04 LTS is very evident.

The three legs of our engineering practice are cadence, quality and design. For those teams which maintain their own codebases (unity, juju, bzr, lp and many more) the quality position is a easier to define, because we can make test coverage and continuous tested integration standard practices. It’s more challenging for the platform team and Ubuntu community, who integrate thousands of packages from all sorts of places into one product: Ubuntu. We’ve traditionally focused on items like security, where participation in a global security process helps us ensure Ubuntu gets world-class security support and has established a world-leading track record of security patches and proactive security.

Nevertheless, the last year has seen some amazing leaps forward in our ability to manage quality across the entire platform. In large part, that’s thanks to the leadership of Rick Spencer and Pete Graner, who made smoke-testing and benchmarking a rigorous part of the process for every change to the platform, and lead the work to make that commitment sane in practice across all the hundreds of people, inside and outside Canonical, who needed to be on board with it. And it’s thanks to tools like Jenkins and LAVA which automate the testing and reporting across a vast array of problem spaces, architectures and packages.

So we have a daily weather report for Precise, which gives you a feeling for where things stand right now, as well as tighter integration of the test suites being run by Canonical upstreams on code destined for Precise with the test harness used by the platform team integrating that work into the distribution.  I’ll take the liberty of repeating some of Rick’s core points here:

This isn’t limited to Canonical team members; didrocks and the French Musketeers have built a Unity SRU testing process which should let us crowdsource perspectives on the quality improvements or regressions of changes in Unity. Ara’s ongoing work around component and system testing is giving us a very useful database of known issues at the hardware level. Work on Checkbox and related tools continues to ensure that people can contribute data and help prioritise the issues which will have the widest benefit for millions of community adopters.

Upstream quality

Where upstreams have test suites, we’re integrating those into the automated QA framework. In an ideal world, whenever a package is changed, we’d have an upstream test suite to run for that package AND for every package which depends on it. That way, we’d catch breakage in the package itself, but more importantly, we’d catch consequential damage elsewhere, which is much harder for upstreams to catch themselves.

We’re already running that program, and as upstreams start to take testing more seriously, coverage across the whole platform will improve significantly. It’s been Canonical practice to have test suites for several years, and it’s very encouraging to see other upstreams adopting TDD and at the least rigorous unit and functional testing, one at a time. Open source projects love to talk about quality – but it’s important to back that with measurable practices and data. As an example in a complex case, we run the LTP against every kernel SRU, in addition to our own kernel and hardware cert tests.

In future, it should be possible to link this to the existing daily builds of tip (we have over 500 upstreams running daily builds on Launchpad, which is fantastic). THAT would give upstreams the ability to know when commits to their tip break tests in dependent packages. It would suck a large amount of compute, but it would provide a fantastic early warning system of collisions between independent changes in diverse but related projects.

There’s a lot more we will do, by integrating Apport for crash data collection, and routing those reports through a big data sieve we should be able to identify the issues which are having the biggest impact on the most users. But that’s a blog for another day. For now, well done, team Ubuntu!

Launchpad doesn't really have any good mechanism for letting you review a list of "your" projects.

That's partly because there are a lot of different ways that it could do it. You could be the maintainer of a project, or its driver, or its bug supervisor, or you might have commit access to its trunk branch. All of this could be direct, or through membership of a team. It's tough.

Since it's that time of year when I review my projects, responsibilities, goals and the like and start to figure out what I want to do next year, I want to figure out what are my projects on Launchpad.

Luckily, Launchpad has an API. I can't use it to figure out what projects I'm the maintainer of, but I can use it to figure out what trunk branches I have commit access to.  The link has the Python code for the script.

Really, Launchpad should allow me to curate my own list of projects, with input to that list coming from all of the sources mentioned above, as well as arbitrarily selecting projects.

This mo might gro!

It’s the last day of Movember.

Inspired by your comments, and frankly, more inspired by your donations to Movember in the name of this metrosexual disaster, I’ll take on the challenge of wearing this mating handicap for an extra week for every additional USD 1,000 donated today. About USD 2,100 raised so far, and I’ll match the total when the event wraps up.

Go on, make me do it, at  mobro.co/smonaut.

Update

Your generosity was incredible – thank you everybody. All that hard currency meant I had to wear the mo for an extra week (Claire was a little worried that you’d have her looking around for somebody else to kiss at New Years eve!) and the folks at ZA.Movember seem delighted. I matched all donations made by the end of the month, and will match one other challenge that came in shortly after. Would never have thought to top the charts; kudos is all yours.

- Playback of physical media (USB cd/dvd/bluray drive)

- Installable image

- Easy configuration of new devices (eg. installing same plugins, mounting same network shares)

- Ubuntu One Accounts

- Push media to/from other Ubuntu devices / Media syncing capabilities (Pause on one device, resume from same spot on another device)

- Collaborate with other Ubuntu devices (context: https://lists.launchpad.net/ubuntu-phone/msg00006.html )

- Control from portable devices (phones/tablets/web interface/PC) (collaboration with Ubuntu Phone/Tablet?)

Medium priority

- Sharing media with friends (social network connectivity)

- Purchasing media through online stores (Ubuntu one/Amazon/Netflix)

 

Thanks to radix, exarkun & dash, my branch to pyflakes to warn about duplicate definitions of classes finally landed. I did the work a year ago as an outrage-powered, opportunistic fix after I saw a co-worker struggle with tests weirdly not failing (Turned out it was a huge test module and there was another class at the bottom with the same name). I'm very happy to see it landed.

For those who haven't been paying attention, official pyflakes development is now taking place on Launchpad as part of the divmod.org project. The trunk of that project now has the best version of pyflakes known to man.

pyflakes is the best static Python checker. It's fast, and has very few false positives.

Update: A couple of people have asked me about lp:pyflakes. It's dead. It died when the divmorg.org trac instance died ages ago. Don't use it. To get pyflakes or any other divmod project, use lp:divmod.org.

Today, since it’s impossible to thank everyone individually who helps make Ubuntu such a wonderful project, I want to express public appreciation for Daniel Holbach’s amazing contribution to our community.

A smooth-running community is not a miracle, it’s the result of dedication, energy, organisation and empathy. Daniel has all of those gifts and qualities in abundance, and the impact he’s had on the governance of Ubuntu is profound.

We pride ourselves on being a meritocracy and a do-ocracy. Those who have the capacity and the will and the commitment and the values and the energy to lead, and who are recognised as leaders by their peers in the project, get the opportunity to take on responsibilities in one of the many parts of the project, or overall in the CC or TB. But it takes insight and effort to recognise those with that potential, and it takes leadership to encourage them to step forward to shine, and it takes organisation to coordinate diverse efforts so as to ensure a harmonious result. For all of the work and play he brings to that, I thank him.

In a galaxy of many stars, it’s perhaps impolite to single out one in particular. So please counterbalance that impertinence by picking your own star to thank; in our community, the depth and strength don’t come from the headline acts so much as the diversity of contributions from an enormous number of people. Thank you all.

I’ve donated my face to charity. It’s not a tattoo, but it’s nevertheless a commitment. Really.

You can contribute to the cause (male health, grin and bare it etc) via the excellent Movember and if you do it as a result of my embarrassment then I’ll match it up to a somewhat unreasonable figure. I think they take plastic, and for the globally challenged $1 is about R8, so your donation will look much more impressive than it is. Just like my tache. Think big

If however you would like to substitute wit for cash, and this is a rare occasion when that is a possibility, add your caption for this 70′s throwback to the comments below.

I just blogged on developer.ubuntu.com about the work we're doing with automatic packaging. Read it!

Canonical has a company scheme where after working there for a few years you can rotate to work at another part of the company for 6 months. Having worked on the desktop team for over five years I decided to do a rotation to Bazaar. My hopes for this were to build up my own programming skills by learning more Python and by experiencing different programming practices from the ones I’m used to in KDE.

I started off with some fixes to the developer documentation. This got me used to the process that you can not commit directly to bzr’s trunk, instead all committers are required to make merge proposals on Launchpad, have those approved by a fellow developer, then send it to a programme called Patch Queue Manager which will integrate the patch and run the test suite to check everything still works.

Next I started fixing a few easy command line UI bugs, improving error messages or stopping exception output and so forth. This got me into the world of writing test cases. Everything in bzr needs a test case, merge proposals will not be accepted without them. Like much of bzr I find that the test cases lack API documentation and comments but it turns out they are easy enough to read and similarly easy to write. There are both internal test cases, which run a small part of the code within bzr, and blackbox test cases which run a bzr command.

Bazaar is the version control system used by top open source project hosting site Launchpad so I was surprised to come across a bug which prevented bzr from talking to Launchpad properly on errors. “This is really important to fix. We need error reporting.” said Jonathan Lange over 2 years before. Pleasingly I could fix it, very satisfying. I had to learn about the hooks mechanism in bzr which shows up some of the downside of Python, you have to guess the arguments to send the hook. But who needs API documentation when you can just read the code?

Bazaar’s main GUI is qbzr (which provides GUIs for individual commands) and Bazaar Explorer (which provides a complete GUI). I worked with Martin (gz) to make these two talk to the normal Ubuntu crash system, Apport, rather than showing a nasty crash backtrace to the user.

Then I noticed that Bazaar Explorer has a lot of “Refresh” toolbar buttons about the place, any time you make a change to the file you have to click one before the UI will update. Not very user friendly. So I added file watchers about the place to make it magically update. Nifty, except that after release it turns out this breaks horribly when doing some commands outside of Bazaar Explorer, oops. Quick fix and message to packagers, hang head in shame.

The first large feature I worked on was GPG signing of commits. The documentation for Bazaar promised that this was implemented and all you need do was set the various options in the config file. Alas it lied. I fixed up the documentation and started looking into the GPG python bindings, which turn out to be completely undocumented on the Python side and surprisingly badly documented on the C side. Security critical code which is badly documented seems scary to me, mistakes could easily be made which go unnoticed until it appears on full-disclosure. But I manage to implement signing and adding a GUI to Bazaar Explorer being cautious as I go.

Bazaar has a scheme called patch pilot where we review patches submitted by the community and help them on their way to being integrated. I started out with this by following John Meinel who can write code faster than I can write English prose. We made small changes to some patches and integrated them, we gave feedback to newer patches that needed some work and we chased up contributors who had not responded. The barrier to entry in Bazaar is pleasingly small, if you don’t have the skills to write a perfect patch it’s encouraged to say so and someone else will finish it off.

Why, I wondered, is bzr (the command line UI to Bazaar) not translated? There were parts of gettext scattered around the code, and some code to extract strings but it didn’t get used. Turns out this code was a half completed feature that had never been taken to completion. I finished off translations by adding gettext()s throughout the code, ensuring tests still pass, fix the installation of .mos and enable the generation of .pot. This missed the 2.4 release so I’m still waiting to see how it works for 2.5, I suspect some strings will be missing context needed to do a good translation and of course the occasionally technical output of bzr might need some thought on how to translate but it should make bzr easier to use for non-English speakers.

Ubuntu Distributed Development is the project to put all of Ubuntu’s packages and history into Bazaar branches and change our packages processes to use Bazaar. This makes a lot of sense, the Ubuntu archive is already a primitive revision control system (you upload for each new version, often its useful to look at older versions). This project has been a long time coming and is one of the original reasons why Canonical started Bazaar back in the day. It suffers from a number of problems, notably the failure of quite a lot of packages to import into Bazaar including currently the whole of KDE due to a patch into openSUSE’s bz2 package. Also the quilt patch system we use tends to clash with being held within a revision control system so you end up with diffs of diffs. I tend to think that would have been an easier win to import only the debian/ packaging into Bazaar branches.

I tidied up the new Ubuntu Packaging Guide which is a guide to packaging with UDD branches (named in the hope that UDD will soon become the definitive way to do packaging). I also added a new command bzr get-orig-source to make it easier to do packaging in the current directory rather than a separate directory as used by bzr builddeb. I also added a hook to set the bzr changelog from the debian/changelog entry which is the current behaviour with debcommit. I got mixed feedback on this so I added a config option to disable it too. I also tidied up some of the bzr-builddeb code by removing weird terms like “larstiq” and removing acronyms by default.

My Python programming has improved a lot and I’m a convert to the cause of unit tests. Python is a fun and productive language but the lack of culture for documenting APIs is disappointing and being dynamic it’s that much easier to make mistakes without realising it. My productivity is nothing like as high as others on the Bazaar team but it seems I’m better at improving (graphical and command) user interfaces than my colleagues who can memorise internal data structures trivially. My six months is now up, I’ve enjoyed them and now I’m looking forward to getting back into Kubuntu and KDE.

A few months back I gave a lightning talk at the Launchpad Thunderdome about how I do refactoring.  It's very opinionated, and mostly applies to big, old code bases, but worth writing up anyway.


The core idea here is that very few things make me feel as good as deleting code. I love cleaning up code and the clean code base that results, and I'm sure that many others feel like me. As such, this is a guide on how to feel better.

1. Know your enemy

“Functionality is an asset, code is a liabilityâ€. Truer words were never spoken. Every line of code is a potential source of bugs and a barrier to understanding, and thus carries a maintenance cost.

Maintain an awareness of things that need refactoring. Here's a quick and incomplete list:

2. Keep a "yak stack"

Yak shaving is "any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you're working on." (Jargon File)

A few of us have extended the concept beyond the "actually necessary" to include anything that's making the task at hand more difficult and less fun but is not worth fixing right now. Hence the yak stack. Here's how it works:

Whenever you come across something in your code base that is difficult to understand or that slows you down: make a note of it. When you've finished the task at hand, fix the problem. If you encounter other things that slow you down, write them down. Work through the list.

3. Every option is wrong

In a big, old code base, there are probably many, many areas that need refactoring.  Don't worry about which is the "best" place to start – there is no best place.

4. Start from green, stay green

Never, ever refactor while your tests are failing. Refactoring is about changing the shape of code while preserving its behaviour. It's much harder to be sure you're keeping the behaviour if you are comparing one set of thirty tracebacks with another set of thirty tracebacks. Better to compare a passing ("green") test run with another passing test run.

Run tests frequently. More often then you think you should. Commit often – think of it like quick save in a tough level of a video game. It frees you up to experiment more and means you have less in your head at any one time.

5. Do not nest

Don't begin a refactoring while you are in the middle of another refactoring. If you find you must, use tools like 'bzr shelve' to store your current diff and then work from the clean head of your branch.

6. Keep moving, leave a trail

Don't get bogged down in details, otherwise you'll never finish. Literally. Someone will come along and distract you and before you know it, three months will pass and your refactoring branch will be full of conflicts. If you see something you are unsure of, mark it with a XXX or a FIXME or a TODO or whatever works for you and then continue with what you are doing.

Tools like 'bzr todo' can make it really easy to check to see if you've added any XXX comments.

7. Translate tests with care

As said above, refactoring is about changing the shape of code while preserving its behaviour. When you update tests, you risk changing your definition of the system's behaviour – so be careful.

8. Confront uncertainty with destruction

If you see some code and you are not sure if it's needed, delete it. Doesn't matter if it's a whole function or just an odd line. If you have a test suite, and it was important, that will catch the failure. If you have version control, and it was important, one of your collaborators will notice and revert the change.

If it was important and neither of these happened, then your whole project has learned something new about itself, and that's probably worth the hassle. (Oh, add tests & better docs after this happens.)

9. Good grep tools

Remember that symbols aren't only referenced from files that match *.py.  In big code bases there are often other sorts of files that refer to symbols in the main code. In Launchpad, for example, we have ZCML and doctest files that refer to symbols. When you want to know how something is used or you want to rename something, make sure you use a grep command that actually finds everything.

Ideally, you should be able to run this command faster than you can think about wanting to do it.

Personally, I use 'bzr grep' a lot. Others recommend 'ack'.

10. There will be failures

Mentally prepare yourself for the fact that the first two or three full test runs after your refactoring will fail.  This is especially important for code bases that have multi-hour test run times.

If you think this way, then you won't be as discouraged when it actually happens.

11. Finish the job

Busy people refactoring a big code base are often tempted to apply a refactoring to only a single part.  For example, some useful function is extracted from repeated boilerplate in a few other functions. However, many, many other instances in the code base continue to use the repeated boilerplate.

This is almost worse than just leaving the repeated boilerplate.  There are now two idiomatic ways of doing the activity.  Further, other developers who work on other parts of the code base probably won't find out about it, and might end up repeating your refactoring. Ugh. How is anyone new expected to get to grips with this code?

Similarly, if a class, function or concept is renamed, rename it everywhere, especially in the documentation.

It's difficult and often tedious, but it really is worth taking refactorings to completion. Apply them to the whole code base, or not at all.

Note that I'm referring to completeness, not perfection. If you block on perfection, you never get anything useful done. If you aim for frequent incremental improvements, you soar.

12. Read these books

I highly recommend "Refactoring" by Martin Fowler and "TDD by Example" by Kent Beck. I stole many of these ideas from them.