Mon, 01 Feb 2010
Project Cambria
David, it's interesting that you posted about that, as it's something I've been toying with for the last couple of years. For the last few months I've been (very) slowly experimenting in my free time with an approach that I think works well, and I think it's time to tell more people about it and to ask for contributions.
Opportunistic programmers are useful to cater for here, as Debian/Ubuntu development isn't trivial, and so we are simplifying something existing, which means that it will still be powerful, which is also important. I'm not only interested in improving the experience for the opportunistic programmer though, why should they get all the cool stuff? I'm interested in producing something that I can use for doing Ubuntu development too (though not every last detail).
The project I am talking about has been christened "cambria" and is now available on Launchpad. It's a library that aims to provide great APIs for working with packages throughout the lifecycle, including things like Bazaar, PPAs, local builds, testing, lintian, etc. It should be pleasurable to use and also allow you to build tools on top that are also pleasurable. It should also allow for easy extension in to different GUI toolkits and for command-line tools, though I've only been working with GTK so far.
In addition, there is a gedit plugin that allows you to perform common tasks from within gedit. I chose gedit as it has a pleasant Python API for plugins, isn't so complicated that it takes much learning, and will already be installed on most Ubuntu desktop systems. As I said though, the libarary allows you to implement in anything you like (that can use a python library.)
I've put together some mockups that suggest some of the things that I would like to do:
The RATIONALE file includes some more reasons for the project:
Project cambria is about wrapping the existing tools for Debian/Ubuntu development to allow a more task-based workflow. Depending on the task the developer is doing there may be several things that must be done, but they must currently work each one out individually. We have documentation to help with this, but it's much simpler if your tools can take care of it for you.
Project cambria aims to make Ubuntu development easier to get started with. There are several ways that it will help. Providing a task-based workflow where you are prompted for the information that is needed to complete the task, and other things are done automatically, or defaults chosen helps as it means you can concentrate on completing the task, rather than learning about all the possible changes you could make and deciding which applies.
Project cambria aims to make Ubuntu development easier for everyone by automating common tasks, and alleviating some of the tool tax that we pay. It won't just be a beginner tool, but will provide tools and APIs that experienced developers can use, or can build upon to build tools that suit them.
Project cambria will help to take people from novice to experienced developer by providing documentation that allows you to learn about the issues related to your current task. This provides an easier way in to the documentation than a large individual document (but it can still be read that way if you like).
Project cambria will make Ubuntu development more pleasurable by focusing on the user experience. It will aim to pull together disparate interfaces in to a single pleasing one. Where it needs to defer to a different interface it should provide the user with an explanation of what they will be seeing to lessen the jarring effect.
I'm keen for others to contribute, there is some information about this in the project's CONTRIBUTING file. I'm looking for all sorts of contributions from all kinds of people and keen to help you get started if you aren't confident with the type of contribution you would like to make.
There's a mailing list as part of the ~cambria team on Launchpad and IRC channel if you are interested in discussing it more.
Wed, 16 Dec 2009
Ubuntu Distributed Development Overview
You may well have heard about it (on this blog especially), but though I spend lots of my time involved with it and talking to people about it, there may be some people who aren't entirely sure what we are doing with the Ubuntu Distributed Development initiative, or what we are trying to achieve. To try and help this I wrote up an overview of what we are doing.
If this project interests you and you would like to help, or just observe, then you can subscribe to the mailing list. There's lots of fun projects that you could take on: there's far more that is possible and would be hugely useful to Ubuntu developers than we can currently work on. If you want to work on something then feel free to talk to me about it and we can see if there is something that would suit you.
Without further ado...
The aim
The TL;DR version:
- Version Control rocks.
- Distributed version control rocks even more.
- Bazaar rocks particularly well.
- Let's use Bazaar for Ubuntu.
Or, if you prefer a more verbose version...
Ubuntu is a global project with many people contributing to the development of it in many ways. In particular development/packaging involves many people working on packages, and much of this requires more than one person to work on the change that it is being made, for e.g.
- Working on the problem together
- Sponsoring
- Other review
etc.
These things usually require the code to be passed backwards and forwards, and in particular, merged. In addition, we sometimes have to do things like merge the patch in the bug with a later version of the Ubuntu package. In fact, Ubuntu is a derivative of Debian, and we expend a huge effort every cycle merging the two.
Distributed version control systems have to be good at merging, it's a fundamental property. We currently do without, but we have tools such as MoM that use version control techniques to help us with some of the merging. We could carry on in this fashion, or we could move to use a distributed version control system and make use of its features, and gain a lot of other things in the process.
Tasks such as viewing history, and annotating to find who made a particular change and why, also become much easier than when you have to download and unpack lots of tarballs.
This isn't to say that there aren't costs to the transition, and tools and processes we currently use that don't currently have an obvious analogue in the bzr world. That just means we have to identify those things and put the work in to provide an alternative, or to port, where it makes sense.
The aim is therefore to help make Ubuntu developers more productive, and enable us to increase the number of developers, by making use of modern technologies, in particular Bazaar, though there are several other things that are also being used to do this.
What it isn't
This isn't a project to overhaul all the Ubuntu development tools. While there are many things I would like to fix about some of our tools (see some of the things that Barry had to get his head around in the "First Impressions" thread), that can go ahead without having to tie it in to this project. I hope that when me make some common tasks easier, it will focus attention on others that are still overly complex, and encourage people to work on those too.
We are not replacing the entire stack. We are building upon the lower layers, and replacing some of the higher ones. We aim for compatibility where possible, and not breaking existing workflows until it makes sense.
The plan
You can read the original overall specification for this work at
https://wiki.ubuntu.com/DistributedDevelopment/Specification
It is rather dry and lacking in commentary, and also a little out of date as we drill down in to each of the phases. Therefore I'll say a little more about the plan here.
The plan is to work from the end of the Ubuntu developers, converting the things that we work most directly with first. This should give the biggest impact. We will then work to pull in other things that improve the system.
This means that we start by making all packages available in bzr, and make it possible to use bzr to do packaging tasks. In addition to this we are working with the LP developers to make it possible for Soyuz to build a source package from the branch, so that you don't have to leave bzr to make a change to a package. This work is underway.
After that we make all of Debian available in bzr in the same way. This allows us to merge from Debian directly in bzr. At a first cut, this just allows us to replace MoM, but in fact allows for more than that. Have a conflict? You have much more information available as to why the changes were made, which should help when deciding what to do.
The next step after that is to also bring the Vcs-* branches in to the history. These are the branches used by the Debian maintainer, and so allow you to work directly with the Debian maintainer without switching out of the system that you have learnt.
In a similar way we then want to pull in the upstream branches themselves. Again, this will allow you to work closely with upstream, without having to step out of the normal workflow you know.
The last point deserves some more explanation. The idea is that you will be able to grab a package as you normally do, work on a patch, and then when you are happy run a command or three that does something like the following:
- Merges your change in to the tip of upstream, allowing you to resolve any conflicts.
- Provide a cover letter for the change (seeded with the changelog entry and/or commit message(s).
- Send the change off to upstream in their preferred format and location (LP merge proposal, patch in the bugtracker, mailing list etc.)
As you can imagine, there are a fair number of prerequisites that we need to complete before we can get to that stage, but I think of that as the goal. This will smooth some of the difficulties that arise in packaging from having to deal with a variety of upstreams. Finding the upstream VCS, working out their preferred form and location for submission, rebasing your change on their tip etc. I hope this will make Ubuntu developers more efficient, make forwarding changes easier to do and do well, and save new contributors from having to learn too many things at once.
Where we are now
We currently have all of Ubuntu imported (give or take), you can
bzr branch lp:ubuntu/<source package name>
which is great in itself for many people.
We also have all of Debian imported, and similarly available with
bzr branch lp:debian/<source package name>
which naturally allows
bzr merge lp:debian/<source package name>
so you can make use of that right now.
We are also currently looking at the sponsorship process around bzr branches, and once we have that cracked it will be much easier for upstream developers who know bzr to submit a bugfix, and that's a large constituency.
In addition, this means that a new contributor can start without having to learn debdiff etc., we can pass code around without having to merge two diffs and the like.
This is great in itself, but we are still some way from the final goal.
We are currently working on the VCS-* branches, to make them mergeable, but their are a number of prerequisites.
In addition the Launchpad team are also working on making it possible to build from a branch.
Where we can go
As I said, building on top of bzr makes a number of things easier.
For instance, once LP can build from branches, we could have a MoM-a-like that very cheaply tries to merge from Debian every time there is an upload there, and if it succeeds build the package. This could then tell you not only if there were any conflicts in the merge, but any build failures, even before you download the code.
In addition, we are currently talking a lot about Daily Builds, building the latest code every day (or commit, week, whatever). There are a number of things this brings. It doesn't strictly require version control, but as it's basically a merging problem having everything in Bazaar makes it much easier to do. We have a system now built on "recipes" that we are working to add to LP.
Parts of the work
There are a number of parts to the work, and you will see these and others being discussed on the list:
- bzr (obviously), which we sometimes need to change to make this work possible, either bug fixes, or sometimes new features.
- bzr-builddeb, which is a bzr plugin that knows how to go from branch to package and vice-versa.
- bzr-builder, the bzr plugin that implements "recipes."
- Launchpad, which hosts the branches, provides the merge prosals, and will allow building from branches and daily builds.
- The bzr importer, this is the process that mirrors the Ubuntu and Debian archives in to bzr and pushes the branches to LP.
and probably others that I have forgotten right now.
Wed, 23 Sep 2009
Kerneloops enabled by default in Karmic
One of the new things that is going to be in karmic is that the kerneloops daemon will be installed and running by default. This tool, created by Arjan van de Ven, watches the kernel logs for problems. It has a companion service, kerneloops.org which aggregates reports of these problems, and can sort by kernel version and the like. This allows kernel developers to spot the most commonly encountered problems, areas of the code which are prone to bugs etc. When the kerneloops daemon catches a problem it allows you to send the problem to kerneloops.org.
We however, are not using the applet that comes with kerneloops to do this, we are making use of the brilliant Apport. There are a couple of reasons for this. We also want to make it easy for you to report these issues as bugs to Launchpad, and we don't want to prompt you with two different interfaces to do that.
The changes mean that if your machine has a kernel issue you will get an apport prompt as usual. As well as asking if you would like to report the problem to Launchpad like it does for other crashes it will ask if you would also like to report it to kerneloops.org. Passing the information through apport means that it can also be used on servers as well without running X.
Hopefully you will never see this improvement, but it's now going to be there for when those bugs do creep in.
Tue, 18 Aug 2009
Command support in bzr-builder
I've just implemented the most requested feature in bzr-builder (Hi Ted), command support.
Sometimes you need to run a particular command to prepare a branch of your project for packaging (e.g. autoreconf). I think this should generally go in your build target, but not everyone agrees, and sometimes there is just no other way.
Therefore I added a new instruction to bzr-builder recipes, "run". If you put
run some command here
in your recipe then it will run "some command here" at that point when assembling.
Note that running commands that require arbitrary network access is still to be discouraged, as you don't know in what environment someone may assemble the recipe. I'd also advise against using commands unless you really need them, but that's obviously your call.
Distributed Development Video
I recently gave a talk to some fellow Canonical employees about where we are with the "Distributed Development" project. For that I made a screencast showing some of the Launchpad Codehosting features that you can now use for Ubuntu. Thanks to the Launchpad team for making this happen. We're still ironing out the remaining kinks that make it a pain to use, and getting all the packages imported, but it's possible to use them now.
The screencast has no audio unfortunately, but you can watch it and try and guess what I was saying. There's documentation available on the wiki as well.
One of the things the video shows is how to request someone review your change, i.e. how to get a change sponsored in to Ubuntu. I'm keen to have people test this, as it's not something I do very often now that I am a core-dev. Therefore if you want to help test then propose a merge and set the appropriate sponsor team as the reviewer, and I will prioritise it and you can give me feedback in return.
Note that a bug in Launchpad means that I won't get a notification when they are created, so feel free to drop me a line via email or on IRC until that bug is fixed next month. I'll continue to poll the lists though, so nothing will get dropped.
Tue, 04 Aug 2009
Daily Builds
As well as seeing use of PPAs for providing bug fixes, new upstream versions, proposed packages, testing etc., we are also seeing them used for providing daily builds of packages. For instance Fabien Tassin provides daily builds of lots of Mozilla-related packages and snapshots of Chromium in his various PPAs. Also, there is Project Neon, to provide daily builds of Amarok.
They massively lessen the barrier to using and testing code that is fresh from the fingers of the developers. They avoid you having to build a project from source every day, making sure to keep up with changes in dependencies. They allow you to be testing code almost as it is written, speeding up the feedback cycle to the developers, and potentially increasing the number of people involved in that feedback cycle.
In addition they allow you to verify bugs against the latest code, so that bug reports are of more relevance to the developers. If you so choose they can also be set up so that bugs are also tested with fewer distribution patches, further increasing the developers' confidence in the bug reports.
Mark had an idea for an elegant way to describe how to combine the code to produce the package, and we worked on producing a tool to follow the steps. You can find the result of this in the bzr plugin bzr-builder. I've documented how to use it on the wiki.
There's still more we can do to improve the process, and we have a lot to discuss about what makes a good daily package, and what the limits of them are. If you are interested in discussing this then please join the list of the dailydebs team in Launchpad.
I'm currently running the bzr-nightly-ppa using this tool, and have improved some things based on this, but more testing, feedback, and patches are always welcome.
Tue, 21 Apr 2009
Banshee as the default mediaplayer?
Jo Shields posts about his intention to propose switching Rhythmbox for Banshee in Karmic. (I'm trying to convince him to apply to be a MOTU, or at least an Ubuntu member, so he can be on Planet Ubuntu.) He's not the first to suggest it, but he is in the right place to make the proposal for this release.
I'm personally quite happy with Rhythmbox, and haven't really tried Banshee, though for my use I imagine they are pretty similar. (Two things that would be definite benefits for me would be remembering what I was doing when I closed it when it starts, and understanding mix CDs better). I can certainly understand some of the arguments for switching as well, so I wouldn't be against it.
There is an idea on Brainstorm about this, and while Brainstorm can't capture the intricacies of the debate, it suggests that Rhythmbox is quite popular with the people that voted (though it's not clear how many had tried the alternatives.)
This post isn't really to argue one way or the other, or to attempt to cover all the criteria by which a decision will be made. My point is to emphasise that the Banshee developers have done themselves a great favour in this debate by making one aspect of switching easier. The have implemented importing from Rhythmbox. This means that any switch wouldn't mean that all users had to re-import their collection.
That's not everything that is involved in switching, and indeed, many of the issues around trying to change a default don't have good answers, and that's something we should work to improve as a community.
You don't have to spend time implementing importers for every similar application out there, but easing migration from commonly used apps can help users switch, and is a big benefit when trying to switch a large number of users painlessly. Also, while importing is useful, it's not the ideal solution. A common storage format, and shared storage would be superior in many ways for this purpose.
Jaunty's almost ready
Jaunty just froze a little bit more, with the last few normal uploads being done. From here on in it's mainly about getting the CDs perfect for release, which will hopefully go smoothly.
Over the last few days there have been a number of people working on Universe to get it in to the best shape we could in the remaining time. We did a pretty good job of it too, towards the end we were scavenging around for any more fixes that were ready to upload. As always, with more people we could have done more, but it seemed to be a very smooth landing this time.
The sponsorship queue is virtually all things that were not appropriate for Jaunty, with just a couple of desirable fixes not making the cut (we'll work to have those in jaunty-updates ASAP). In addition to that, NBS was clear, meaning that there were no outstanding library transitions or similar, and there are very few uninstallables. Obviously we would want all of these numbers to be zero, but you can't have that with a time-based release schedule. Unfortunately the FTBFS list is rather long (mainly due to toolchain changes), but it's generally infrequently updated packages on there, which will tend to be of less interest.
The MOTUs also did a fantastic job of the python 2.6 transition, which was a huge job, and a compressed timeframe to do it in. Unfortunately there are going to be some issues with the change in the default python for some time to come, but given the state of python a couple of months ago this is a great acheivement.
Also, I'll make special mention of the Mono 2.0 transition. Co-ordinated by Jo Shields, and thanks to a lot of people on both the Debian and Ubuntu sides, this was completed with very little fuss. It was a great example of co-ordinating work on a large number of packages, and of collaboration between Debian and Ubuntu. I also think that it showed some of the advantages of the Ubuntu method of development over the Debian one, but the shared work trumps that.
If you are reading this thinking "You might think you did a good job, but what about this bug that I provided a patch for 3 months ago, why didn't you fix that?" then all I can really do is point you to the sponsorship process. Yes, it sucks that not knowing about this cost you, but reviewing every bug with an attachment tagged "patch" is currently a little out of our reach. I'm always looking for ways to improve this, and I hope one day we can do that, but in the meantime using the sponsorship process will help get your patch included.
Thu, 26 Feb 2009
Developer News
The second installment of Developer News went out on Monday, and boy was it hard work. It's great to see so much going on, but it does make preparing the summary time consuming. Thanks a lot to Stefan Lesicnik for his help in preparing this one. As well as having more to report I also broadened the intended audience this month.
For the first month the intended audience was just Ubuntu developers, so I didn't include anything from ubuntu-devel-announce, as I assumed that everyone would already have seen those announcements. The first issue was picked up by both the fridge and LWN though, so it was more widely read, and showed there was an interest in having more news for external people. Therefore I wrote this edition to include those people as well.
I'm pretty sure that there was plenty more that we could have included as well, it's just I didn't know about it, or there was nothing to link to. I think we need to do a better job as a community to communicate what we are doing. My process was simply to trawl the archives looking for announcements and discussions of things, so all anyone had to do was write one email.
Therefore I think we have two problems to solve, firstly getting everything that is going on announced in the right places, and secondly making it easier to summarise all of the activity.
I hope that the first will become more a part of our culture, and the Developer News will help by getting more exposure for those who do communicate about what they are doing.
I'm not too sure how to improve the second though. We have a defined process for submitting items for the news, but to date there have been no submissions using that process, and one submission by editing the wiki page as I was preparing the second issue. Why is this? Why have you not submitted anything? Is it that you didn't know about the process? Is it too much work? Is it that you never think to use it? That you are not sure that your item counts? That you are not sure that someone else hasn't alread submitted it?
The feedback I have got about the Developer News has been both frequent and entirely positive, so I believe it is a valuable service that should be carried on, but I fear that the current way of doing it won't really scale. I don't think it would be worth a day of my time a month to complete it.
Any suggestions for improving the situation would be appreciated.
Thu, 22 Jan 2009
Fixing an Ubuntu bug with Bazaar
Yesterday as part of Ubuntu Developer Week I gave a session entitled "Bazaar for Packaging". At the last minute I decided to change the session somewhat, so that it would show how things would work if you were to use bzr to modify an Ubuntu package once Distributed Development is fully up and running.
The session went ok, and while I was showing some fairly experimental things it all worked quite well. The biggest problem was when we grabbed a change from SVN using the bzr-svn plugin. The rather simple step of extracting a patch took up quite a lot of the session as bzr-svn initialised it's metatdata about the Subversion branch. bzr-svn is amazing, it allows you to store a bzr branch inside an svn repository, while still making it readable to svn. However, to do this it has to do some fairly intensive transformations to maintain the mapping. The biggest impact of this is when you access the SVN repository for the first time though, so it wasn't the smartest idea for an IRC tutorial. That's the problem with changing your session 10 minutes before it starts.
You can go and read the transcript of the session if you want to see how all of this worked. I'd like to skip ahead a little bit and show you how it will work in a short while when launchpad hosts the branches and all the bits are in place.
First we need to grab the source for the package we want to work on. We'll grab a whole branch here, but you could just as well use a lightweight checkout or a stacked branch to transfer less data.
$ bzr branch lp:ubuntu/jaunty/gnome-utils gnome-utils.jaunty
Will give us a local copy of that branch in gnome-utils.jaunty. We can now make our changes in that branch.
$ cd gnome-utils.jaunty
You can run bzr log (or better bzr viz from bzr-gtk or bzr qlog from qbzr) to see the history if that is interesting for the change we are making.
We're just going to apply the patch from SVN though. This is what we did in the session:
$ bzr diff -c svn:8378 http://svn.gnome.org/svn/gnome-utils/trunk | bzr patch
(where bzr patch is supplied by bzrtools. Try it it's cool, it works over any transport, so you can apply a remote patch file without downloading it)
What we are doing here is in fact a "cherry-pick". bzr will happily do these for you, but it does the equivalent of diff + patch, it is hoped to improve this and record which revisions were merged, and use that information to help you understand what changes have and haven't been merged.
To more directly do a cherry-pick you can run
$ bzr merge -c svn:8378 http://svn.gnome.org/svn/gnome-utils/trunk
(merge the changes introduced in svn revision 8378 of this branch please, the svn: is neccesary as bzr and svn count their revisions differently)
However, this won't currently work, as the branches have what is called "different rich-root support", so we have to use the explicit "diff and patch" for now. This is a pain, and will hopefully go away sometime soon. This method would work fine with most bzr branches though.
Once we have applied the patch we write the changelog entry for it. We run "dch -i -D UNRELEASED", which will create us a new changelog entry, and mark it as "UNRELEASED" so that it is clear it still needs to be uploaded. Obviously if there is an existing UNRELEASED changelog entry then we want to add to that. I would like to write a wrapper than did the right thing here.
For that changelog entry we write the usual thing, something like:
* Don't crash when asked to show a path that has been excluded. (LP: #301952)
Now we are ready to build and test our changes. Running
$ bzr builddeb -S
will spit out a source package in the parent directory, in the same way as debuild -S. We can then build this package in our normal fashion, in pbuilder say, or upload it to a PPA.
Once we are happy then we can commit our changes. The easiest way to do this is to run
$ debcommit
which uses our changelog entry as the commit message, saving us from typing the same thing again.
There's one extra bit of magic that goes on here. bzr supports the --fixes option to commit. This marks the resulting revision as fixing the specified bug, for example --fixes lp:301952 would indicate that we closed the bug that we are working on in this revision. In Intrepid (thanks to the idea from Colin Watson) I implemented support for this in debcommit. If debcommit sees you closing a bug in the changelog message that it is using it will automatically add the corresponding --fixes argument (it works for Debian bugs too). We'll see where this comes in useful in a minute.
The last step is to get our changes in to the distribution. If we have upload rights for the package then we can dput our source package that we created a minute ago, and then run
$ bzr push lp:ubuntu/jaunty/gnome-utils
to push the bzr branch back. (Yes, launchpad plans to support building directly from a branch, so you just need to push along with some undecided mechanism to request it be included in Ubuntu)
If you don't have upload rights for the package then you need someone to sponsor the change for you. To do this you first push your branch to launchpad somewhere under your name. For instance I would run
$ bzr push lp:~james-w/ubuntu/jaunty/fix-301952
Note that thanks to the launchpad and bazaar developers implementing support for "stacked branches" and automatic stacking in launchpad this will be a very cheap operation, only pushing a single revision.
Next we would create a merge proposal for this change. You can either do this from the branch page on launchpad, or you can use bzr send. Just running
$ bzr send
should do the right thing. It will open up a new message in your mail client. You then enter your "cover letter" for the change, and hit send. It will mail the request to launchpad which will interpret the machine-readable attachment and turn it in to a merge request. The developers can then review the changes and either ask for improvements to be done, or upload the package.
Remember the --fixes information that was stored? That was also used by launchpad. The bug that we were fixing now has a link to our branch on it, so that anyone that wants to test the fix can find the right place to get the change from. This currently does not generate any bugmail though, so you have to go to the page to see it. I think this is something we need to improve.
Some of the things that I have explained here haven't been fully decided, so this isn't documentation, that will come later, the intent is to give an idea of how this may work.
I've been asked a few times about an IRC channel where we can discuss this sort of thing, so I created #ubuntu-bzr today. If you are interested in shaping how this will work then please join it and we can discuss it. Support can continue anywhere though.
Tue, 06 Jan 2009
ppamadison
After an idea from bigon on #launchpad today I threw together a tool using the Launchpad API. I've christened this tool ppamadison. It does the same thing as rmadison, but for PPAs. You tell it who's PPA to examine, and what source package to get the information for, and it tells you what versions are available.
$ ppamadison james-w bzr-builddeb bzr-builddeb | 2.0~0ubuntu1~ppa1 | intrepid | source bzr-builddeb | 2.0~ppa1~hardy1 | hardy | source
There's still some things left to do, such as replicating rmadison's odd output formatting, some things missing from the Launchpad API and some interesting things you could add, but the idea is there. One thing missing from the Launchpad APIs as far as I can see is an efficient way to find out which PPAs contain a certain source package name. This would be quite an interesting thing to know.
Would ppamadison be a useful thing to have in ubuntu-dev-tools? If it is worthwhile then I will integrate it. Because this blog post is something that developers might not see, but might be interested in I would then pass it on to the Developer News service, as all it would take would be a quick email, as little as a link to the blog post would do.
(Yes, I am being facetious, but we haven't had a single submission yet)
As an aside, I play with the Launchpad APIs every couple of months and they are getting better, to the point now where most data I want for things I do is available. Thanks to the Launchpad team for their work on it. There are some real problems for some use-cases, such as a cache hit requiring a https connection, but ways can be found to deal with them. In any case, the APIs will allow us to do some really useful things.
P.S. Thank you all for your kind words.
Thu, 27 Nov 2008
Bazaar branches of Ubuntu
This morning I announced the availability of bzr branches for every source package in Ubuntu.
This is exciting, and I hope the first step in revolutionising the way we develop Ubuntu. I believe it will lead to an increase in efficiency, and give us interesting new ways to collaborate.
There's a long way to go yet though, this is just the first milestone. Over the next few development cycles we will be improving the branches themselves so you can work more effectively with people outside Ubuntu using them, as well as improving the tools to help us use them, and many things we haven't thought of yet.
I'd like to thank the Bazaar developers for the tool that all of this is built on, as well as Robert Collins, Colin Watson who helped massively with this project, and our hard working sysadmins who keep everything running smoothly.
Thu, 06 Nov 2008
REVU
Quality of packages not in Debian
While Ubuntu takes most of its packages from Debian, it does contain a few that are not in Debian, for one reason or another. One common reason is simply that someone wanted a package that wasn't in Debian, and packaged it and requested its inclusion.
The fact that these packages are not in Debian means that they are slightly different to the rest of the packages in Ubuntu, as they are completely Ubuntu's responsibility to maintain. By including them we are making a different sort of contract with our users and the authors of the software.
I was interested in how well we do at keeping this contract. A quick bit of scripting and I found some basic numbers on this. There are 886 packages in Ubuntu's universe component that are not in Debian. One possible measure of quality is the number of open bugs against these packages, which is shown in the following table as frequency counts.
| Number of open bug reports | Number of packages |
|---|---|
| 0 | 595 |
| 1 | 146 |
| 2 | 49 |
| 3 | 23 |
| 4 | 17 |
| 5 | 14 |
| 6 | 8 |
| 7 | 4 |
| 8 | 3 |
| 9 | 6 |
| 10 | 5 |
| 11 | 4 |
| 12 | 1 |
| 13 | 2 |
| 15 | 2 |
| 17 | 1 |
| 21 | 2 |
| 22 | 1 |
| 26 | 1 |
| 30 | 1 |
| 82 | 1 |
So there are few open bugs on these packages, and around two-thirds have none. However, there are more than 200 open bugs in total, which we would want to do something about.
To look at how well these bugs are being triaged I looked more closely at those packages with open bugs, and looked at what percentage of those bugs were still in the "New" status with an "Undecided" priority, i.e. completely
| Percent of open bugs untriaged, at most | Number of packages |
|---|---|
| 20 | 107 |
| 40 | 35 |
| 60 | 37 |
| 80 | 21 |
| 100 | 91 |
So again, this isn't too bad, with many packages with less than 5% completely untriaged bugs. Again though, the numbers are too high, and as this only counts those bugs that are completely untriaged, I'm sure that most are not triaged to the level that we would want.
Lastly I wanted to see how much visibility we have in to problems these packages may have. For this I looked at the number of bug subscribers for the source package. While this is certainly not everything, I would certainly fell better if we had one or two people subscribed to the bugs for every one of these packages.
| Subscribers | Number of packages |
|---|---|
| 0 | 596 |
| 1 | 232 |
| 2 | 42 |
| 3 | 7 |
| 4 | 6 |
| 5 | 3 |
This shows that two thirds of the packages don't have anyone subscribed to the bugs for these packages.
What about keeping up with upstream? Shipping old versions of packages means they are likely to be more buggy, and won't be popular with the authors of the software. The Ubuntu External Health Status site attempts to track this using the debian/watch files in the source packages. I attempted to pull the information about these packages from there.
A bit of explanation is in order for those not intimately familiar with Debian packaging. The debian/watch file specifies the location of the upstream source in such a way that new versions can be checked for, and the update of the package be done semi-automatically. This can't be automated for a package that has no watch file, so I list the number of those. Also those packages that have a watch file, but it doesn't work for some reason are listed, as the watch file is not usable for this, though sometimes that is a transient problem.
The figures for the number of packages in each of the three states are:
| Out of date | 46 |
| No watch file | 495 |
| Broken watch file | 32 |
The number of packages that are out of date is not that bad, especially considering we've been frozen for the last few months. However, there is a large number of packages without a watch file. This means that there's no automated way to find out about new versions of the package being released. It's possible to do without that, but I doubt that all 495 packages have someone watching over them.
Conclusion
This is in no way scientific, and I'm very clearly adding some of my bias in to interpreting the results, but here are my conclusions from the investigation.
The packages that we have in Ubuntu universe but not Debian don't have too many bugs, but the ones that they do have are under-triaged, and we aren't that aware of what bugs we have.
Also, we could do more to allow our automated tools for finding out about new upstream releases work better, again making sure we are aware when a package is out of date.
Actions
Firstly, I am going to discuss with the QA team ways in which we can improve the QA on these packages. I will add an item to the QA team's agenda for the next meeting that I am able to attend to do this. A couple of ideas I have are:
- a hug day to triage the bugs currently open on these packages.
- a team that is subscribed to all packages not in Debian. I would like to discuss making this the MOTU team so that all reports end up on the mailing list, but that would require more discussion.
- Secondly, we should improve the situation with the watch files. I will add the UEHS page with the list of packages without a watch file to a wiki page of easy tasks for people to work on, or perhaps as a harvest data source. The UEHS page also lists packages maintained by the Debian QA team on that page, and while it would be good to fix those in Debian, it may be a good idea to split the lists, not least because the Debian QA team may not appreciate an un-coordinated flood of watch files. I will talk to the maintainers of UEHS about the feasibility of doing this.
Why Ubuntu?
The reason we allow packages directly in to Ubuntu is that it brings benefit to our users. Most packages that enter Ubuntu will be of benefit to someone, and that's one of our aims, to give our users a good experience, isn't it? To give our users a good experience we also want high quality though. So we have a balancing act, splitting our workload between fixing what we have and spending time bringing in and maintaining new packages.
I would argue that we want to limit the flow of new packages quite severely, as we are not exactly short of work with our current set of packages. However, the argument isn't quite as simple as that.
A common way for people to get involved with development is to package something new and get it uploaded, and later branch out in to working on existing packages. This means that bringing in lots of new packages may lead to an influx of new contributors that more than makes up for the new workload. I'm not sure how strong this link will be, and whether the ratios will be such that there is a net gain in developer time.
Actions
- Discuss whether the rate of new developers coming from getting a new package in to Ubuntu is high enough to warrant the activity.
- I would also like to discuss ways to encourage people to start their involvement by working on existing packages instead.
Why not Debian?
So, some people will be wondering why all these packages aren't in Debian. The overarching reason for this is that contributors would rather get their packages directly in to Ubuntu.
There are several possible reasons for this, and I have heard most of these stated by someone trying to get their package in to Ubuntu.
One is a simple one, they don't run Debian, and so it's difficult to test in that environment. Yes, it's not impossible, but it is more work.
There is also a perception among many contributors that getting your package in to Debian is hard work, with long delays trying to find a sponsor. That's not something that most of us can really do much about. We can try and tackle the perception, but without upload right we can't really fix the problem. There was the utnubu team that tried to streamline this process, but that is now defunct as far as I know.
Another reason is that they may not wish to do this is that they use Ubuntu and don't really care enough about Debian to do the extra work. This is something we can try and do something about, explaining the virtues of getting the package in to Debian, more than just it being the right thing to do in most cases.
As a counterpoint to this, if the package is going to be useful to a lot of people then even if the person proposing it for Ubuntu does not want to try and get it in to Debian then there is likely to be someone with the interest and skills to maintain it in Debian. For me this means that the packages that we want to be pulling in to Ubuntu should be easy to find Debian maintainers for anyway.
Actions
- To tackle the first issue we could have documentation of the best way to run Debian to test your packages, and links to the important places to keep an eye on the status of your package in Debian. While not solving the problem it may convince some to do the extra work as they don't have to learn a second way of doing things.
- For the second issue it would be great to get the utnubu team going again, but I can't start this, as I am not a DD.
- The third issue could again be tackled with documentation, we could have a wiki page explaining some of the virtues, and link this from places like REVU.
- Discuss requiring people proposing packages for Ubuntu to at least file a request for packaging bug in Debian. This will give a much better chance that those interested in packaging for Debian are aware of the existence of the package. There are a lot of people interested in packaging for Debian that are just looking for something to work on.
REVU and queues
Now, I want to talk a little bit about REVU, and dealing with queues.
Again, a quick bit of background for those not involved in Ubuntu development. REVU is the tool we generally use in Ubuntu to review new packages. Anyone can propose a package for inclusion in to Ubuntu simply by uploading it there. Developers can then review it and provide feedback, asking for things to be changed where necessary. Once it is to a satisfactory standard and has the support of two people with upload privileges it is uploaded to the archive.
REVU is generally a nice platform for doing this work, and I'm not necessarily criticising its design here, I would just like to examine some of the effects of some design decisions.
REVU works with three queues. The first is the queue for packages that have one advocation from an Ubuntu developer. This queue is normally very short and fast moving, as the package is in good shape and just needs double-checking. The second queue is all of the packages waiting for a review. The third queue is for packages that need some work done on them before they can be uploaded. This means that for developers there is primarily one queue that contains things they can look at.
This is similar to the sponsorship queue, which is used in Ubuntu as a way for people without upload rights to make a change to a package in the archive. This can be seen as just one queue where the developer looks for things to review and upload.
The sponsorship queue is for changes to packages we already have in the archive, and we generally want to upload everything on the list, even if not at a certain point in the development cycle. The difference is that it's generally easier to give a reason for saying "no" for the rare times that it happens. This changes the complexion of what it means to keep the queue small.
If an item on the sponsorship queue is incomplete and the person who submitted it doesn't follow through then we should be picking up the item and ensuring the problem is fixed. If a package on REVU is incomplete and the person who submitted it doesn't follow through then there is no real problem. If the package is popular someone else will eventually pick it up and do the work.
We don't have (or at least haven't had) the time to review enough packages on REVU, so the queue is pretty long. Assuming that doesn't change keeping the queue small will mean removing things because of lack of interest from the person submitting them. We can work to review more things, but unless we reach the point where we are accepting more packages than are being proposed that will always be the case.
There was recently a proposal to help clear the queue of things where the submitter has given up, so that effort can be focused on the packages where problems are likely to be fixed. While doing this is makes best use of developer time I feel that if we feel the need to do this then we have already lost.
If that is the case then there is a worrying aspect to it. We have had a bunch of people show an interest in Ubuntu development, and take the first steps towards becoming a developer, only to get discouraged and give up.
I have heard complaints about it being difficult to find a reviewer, and I'm sure the people that gave up would not speak fondly of the experience. However, this hasn't become widespread enough that it has stopped people giving it a go, and it would be terrible if it did, as having that reputation may cause potential contributors to look elsewhere.
I think this indicates that we should reconsider the way REVU is presented within our community. We are presenting a great service to people who are getting started with packaging and pointing them there saying, put your package here, "we will review it and you will get it in to Ubuntu." However, this unfortunately doesn't happen all that often. We are channeling potential new developers there, knowing that there is a high chance of them getting discouraged and giving up. On top of that we give the reason that many of them do give up as one reason that we're not better at reviewing the packages in the first place.
To me this almost feels like we are trying to put people off from getting started with Ubuntu development.
It looks as though the proposal to help clear the queue for the start of this cycle is going to go ahead. That's fine with me, we might as well make the most of a bad situation and help the reviewers this cycle. If we are in a position where we feel we need to do the same at the start of next cycle then I would say that it is clear we have not fixed the underlying issues, and we really need to stop and reconsider how we present REVU.
There is also a slim possibility that the clearing of the queue may have a different effect. If some contributors are put of by the number of packages waiting for review, then clearing that may cause them to put their own up there. While this could imply more potential developers, it would also mean more packages to review, making it harder to keep up.
If we keep up this cycle and have a good process going in six months time then I will be happy, but it will make the rest of this post pretty much obsolete. I think now is a reasonable time to discuss the "what if?" though.
Changing things
If we are in the same situation in six months then I think it will demonstrate that even with a clean slate and a will to fix the problem we were not able to do enough reviews to keep up with the supply. In that case we will need to look for ways to turn down the taps, to reduce the supply.
In no way do I want to send the message that we shouldn't be welcoming to new contributors, I would just want to explore ways to get them started with working on our existing packages. It may involve making it harder to get a package in to Ubuntu though. We may just have to live with that.
In my opinion there would be two ways to target this, and the solution may come from some combination of the two. The first would be to make working on existing packages more attractive, easier to start with, and the default choice. The second would be to make proposing a new package more difficult, less attractive, or at least not demoralising if your package isn't reviewed.
If I had any bright ideas for the first then I would already be trying to implement them, so let's focus on ways to do the second. Keep thinking about the first though, and feel free to discuss any ideas with me.
I'll take three ideas from the recent mailing list thread and one I just had and look at what effects they may have.
Restricting access to REVU
Michael Casadevall proposed restricting those who could upload to REVU. In particular he proposed restricting it to the "Ubuntu Universe Contributors" team. These are people that have been given Ubuntu membership in recognition of their contribution to Ubuntu development. This includes all Ubuntu developers, but also others who haven't been given upload rights yet.
This would restrict REVU to those who have contributed to Ubuntu development in other ways, and so would make it clear that to get started you should work on existing packages.
In my opinion it also helps with the QA issue, as these people have shown sustained contribution, and are not going to disappear as soon as the package is uploaded. When looking for packages on REVU I first look for names that I know partly for this reason.
As a concession to those that haven't yet reached this stage but want to get a package in for some reason we could have a kind of sponsorship process where a member of the team can put a package there for someone who isn't if they think it is worthy. (While the normal sign and upload process would work I imagine we would only want to do this at most once, with the person that proposed the package being able to upload that one to make changes according to feedback.)
Asking for a rationale
REVU could grow a field for the person that proposed the package to give their reasons why they think the package should be included in Ubuntu.
While it is currently feasible to reject a package from REVU because it is not deemed worthwhile I'm not sure that it ever happens. In my opinion we could use this field as a basis for doing so, perhaps allowing the proposer to counter and give more reasons.
A more subtle effect I believe it would have would be to make the proposer think about these issues. There could even be several fields asking about various different things, such as responsiveness of upstream, potential number of users, etc. While it probably wouldn't stop anyone from uploading we could change the REVU help pages to explain that we upload packages based on their merit, and so if their package is not reviewed they may understand.
A related idea was to link the packages to brainstorm, so that we could gauge user interest.
Making REVU per-cycle
My second proposal was to make REVU per-cycle. This would mean that you don't propose a package for inclusion in to Ubuntu, but for inclusion in to the next release. REVU would open the day after a release with a clean slate, and then close sometime around Feature Freeze, with packages that were still on there being archived with a message that they were not successful this time.
I imagine that this would actually include the rationale field, asking why it should be included this cycle.
This goes further than the previous proposal though, in that it ensures a clean slate to work from.
More importantly in my opinion is that it focuses on the rationale even more in my opinion. It asks the question, "For the next release, would you rather we spent time reviewing this package, or fixing those annoying bugs that you hit?"
However, I'm not sure that this proposal does enough to encourage people to work on existing packages to counteract its harshness.
Order packages by uploads
We could change the ordering of the packages from chronological order to ordering by the number of uploads to Ubuntu done by the submitter. We would then work top down on that list (obviously being able to pick packages from anywhere if we like).
This would draw an obvious link, upload a bug fix, get more chance of a review, but do so in a more subtle way, and not penalise brand-new contributors too much (there are plenty of bite-size tasks that are easy if you're able to create a package from scratch).
This wouldn't work that well if there was little distribution in the number of uploads, or the top of the list wasn't reviewed very often, but it could be a good way to work.
We could obviously substitute number of uploads for something else, for instance launchpad karma, or mix the numbers somehow. This would mean that the link was less direct, but it would reward the fantastic triagers, translators, etc. that want to get in to packaging.
Conclusion
I don't really have a conclusion, but I have plenty to think about, and I hope you have too. I'll be more than happy to discuss these issues at any time, so just give me a shout.
Fri, 12 Sep 2008
Some advantages of having packages in Bazaar
After some discussions in the last week or so I decided to finally get around to something that I have been promising for ages: making a screencast about what is good about having packages in bzr. It's basically just me rambling for almost 20 minutes, so don't expect anything slick or compelling. Also, I have to apologies for speaking too quickly and for the skips in the audio. I'll improve this for next time.
If there are areas that you would like me to go in to in more detail in this format then leave a comment and I'll see what I can do. As usual feel free to grab me if you would like to discuss anything, and there are lots of interesting things that could be prototyped if you are in to that.
I'm not sure where else to put this so that those who don't read planet could find out about it. It doesn't seem like something that should be sent to a mailing list, and just dropping a note on IRC isn't that reliable. Any suggestions would be welcome.
Thu, 28 Aug 2008
Making Intrepid Solid
Making Intrepid Solid
With feature freeze now in effect the bulk of the big changes in Intrepid should now be done. There will still be new features entering the archive with the appropriate exceptions, but the rate will slow as we move forward.
Now we really need to focus on making Intrepid solid. We want to squash as many bugs as possible, so that when we deliver the final release it is something we can be proud of.
This is something that everyone can help with. There are plenty of ways to help out, so there should be something for everyone.
Testing
Simply running Intrepid and reporting bugs is a great start. It's still not recommended to run it if you aren't able to fix a system that doesn't boot, or where X doesn't start, but if you are then upgrading now will be a great help.
You can do more than just using the system though, pick an application and start testing all of the functionality, and report the bugs that you find. Some bugs only show up in certain locales, with certain hardware, or with certain combinations of packages, so try different things and look for serious problems.
Upgrade testing is an area that is under-tested until the last minute when floods of users upgrade. Also the easiest testing to do lots of is upgrading pretty standard installations, but this doesn't catch a lot of problems. So, when you are comfortable with running Intrepid upgrade and let apport file any upgrade problems that you find. You should also be on the lookout for unnecessary prompts that happen while upgrading, or packages that are left broken by the upgrade.
Even if you are not happy running Intrepid yet you can still potentially help upgrade testing, thanks to the unstoppable Michael Vogt. He has written a tool that will clone your system in to a kvm virtual machine, and then upgrade that. This means you can test a real world upgrade without risk to your system. If you do this a few times during the remaining time for Intrepid and file bugs, then you will have a much better chance of a hassle free upgrade to the final release. You can find more details on Michael's work here. (Not everyone has kvm capable hardware though unfortunately.)
Looking at bugs
As well as trying to find your own bugs you can look at the ones that other people have already found. There are several important things here.
The first is bug triage, trying to make sure that a bug report has all the information that it needs, and trying to set an appropriate priority. This is really important work, and we always need more help doing it, so consider joining the bugsquad and helping out.
At this time important bugs should also be milestoned so that they can be concentrated on for the release if possible. Deciding the different classes of bugs here is really tricky, and there can often be disagreements. It is important work though. If you see a bug that should probably become release critical then work with the bugsquad to triage it, and make sure to suggest that it is considered for release-critical status.
Developers can help by actually trying to fix these bugs. Some can be easy, for instance if they are known to be fixed elsewhere. Some can be really complex, and take a lot of effort. Fixing things from the release-critical bug list, and lists of other important bugs is always valuable.
Doing the easy things
There are some ways to improve the quality that are actually fairly easy. Though there is a freeze in effect in Ubuntu there is still loads of work going on elsewhere, and many, many bugs being fixed elsewhere. Pulling these fixes in to Ubuntu will improve the quality, while in theory being easier than coming up with a fix for a bug.
We should keep an eye on upstream projects and pick up bug fix point releases to the versions that we have. If the project doesn't do this then look out for important fixes going in to trunk and back-port them. If you are doing that then it can be worthwhile looking at the versions in other distributions that plan to release soon, and if they carry the same version suggesting that you share the workload of creating these point release, or at least collaborate on fixes and share them.
Adding external bug watches in launchpad is also a great way to help. Jorge explained this recently. This helps easily spot when there is a bug fix that we could pull in. When there is they will appear on harvest
Harvest (now with a new look) is another easy way to do things. It lists opportunities to fix things that should be fairly easy, such as bugs fixed elsewhere, or bugs with patches attached.
Making Intrepid+1 rock
In parallel with all the above now is the time to start thinking about what you want to achieve in the next cycle. If any of that requires changes to an upstream then speaking to them early can be a good idea, as you can get their feedback and see how it fits in to their plan. I'm sure everyone has loads of ideas, and a bit of preparation now can help you hit the ground running in the next cycle.