Quality of packages not in Debian

While Ubuntu takes most of its packages from Debian, it does contain a few that are not in Debian, for one reason or another. One common reason is simply that someone wanted a package that wasn't in Debian, and packaged it and requested its inclusion.

The fact that these packages are not in Debian means that they are slightly different to the rest of the packages in Ubuntu, as they are completely Ubuntu's responsibility to maintain. By including them we are making a different sort of contract with our users and the authors of the software.

I was interested in how well we do at keeping this contract. A quick bit of scripting and I found some basic numbers on this. There are 886 packages in Ubuntu's universe component that are not in Debian. One possible measure of quality is the number of open bugs against these packages, which is shown in the following table as frequency counts.

So there are few open bugs on these packages, and around two-thirds have none. However, there are more than 200 open bugs in total, which we would want to do something about.

To look at how well these bugs are being triaged I looked more closely at those packages with open bugs, and looked at what percentage of those bugs were still in the "New" status with an "Undecided" priority, i.e. completely

So again, this isn't too bad, with many packages with less than 5% completely untriaged bugs. Again though, the numbers are too high, and as this only counts those bugs that are completely untriaged, I'm sure that most are not triaged to the level that we would want.

Lastly I wanted to see how much visibility we have in to problems these packages may have. For this I looked at the number of bug subscribers for the source package. While this is certainly not everything, I would certainly fell better if we had one or two people subscribed to the bugs for every one of these packages.

This shows that two thirds of the packages don't have anyone subscribed to the bugs for these packages.

What about keeping up with upstream? Shipping old versions of packages means they are likely to be more buggy, and won't be popular with the authors of the software. The Ubuntu External Health Status site attempts to track this using the debian/watch files in the source packages. I attempted to pull the information about these packages from there.

A bit of explanation is in order for those not intimately familiar with Debian packaging. The debian/watch file specifies the location of the upstream source in such a way that new versions can be checked for, and the update of the package be done semi-automatically. This can't be automated for a package that has no watch file, so I list the number of those. Also those packages that have a watch file, but it doesn't work for some reason are listed, as the watch file is not usable for this, though sometimes that is a transient problem.

The figures for the number of packages in each of the three states are:

The number of packages that are out of date is not that bad, especially considering we've been frozen for the last few months. However, there is a large number of packages without a watch file. This means that there's no automated way to find out about new versions of the package being released. It's possible to do without that, but I doubt that all 495 packages have someone watching over them.

Why Ubuntu?

The reason we allow packages directly in to Ubuntu is that it brings benefit to our users. Most packages that enter Ubuntu will be of benefit to someone, and that's one of our aims, to give our users a good experience, isn't it? To give our users a good experience we also want high quality though. So we have a balancing act, splitting our workload between fixing what we have and spending time bringing in and maintaining new packages.

I would argue that we want to limit the flow of new packages quite severely, as we are not exactly short of work with our current set of packages. However, the argument isn't quite as simple as that.

A common way for people to get involved with development is to package something new and get it uploaded, and later branch out in to working on existing packages. This means that bringing in lots of new packages may lead to an influx of new contributors that more than makes up for the new workload. I'm not sure how strong this link will be, and whether the ratios will be such that there is a net gain in developer time.

Why not Debian?

So, some people will be wondering why all these packages aren't in Debian. The overarching reason for this is that contributors would rather get their packages directly in to Ubuntu.

There are several possible reasons for this, and I have heard most of these stated by someone trying to get their package in to Ubuntu.

One is a simple one, they don't run Debian, and so it's difficult to test in that environment. Yes, it's not impossible, but it is more work.

There is also a perception among many contributors that getting your package in to Debian is hard work, with long delays trying to find a sponsor. That's not something that most of us can really do much about. We can try and tackle the perception, but without upload right we can't really fix the problem. There was the utnubu team that tried to streamline this process, but that is now defunct as far as I know.

Another reason is that they may not wish to do this is that they use Ubuntu and don't really care enough about Debian to do the extra work. This is something we can try and do something about, explaining the virtues of getting the package in to Debian, more than just it being the right thing to do in most cases.

As a counterpoint to this, if the package is going to be useful to a lot of people then even if the person proposing it for Ubuntu does not want to try and get it in to Debian then there is likely to be someone with the interest and skills to maintain it in Debian. For me this means that the packages that we want to be pulling in to Ubuntu should be easy to find Debian maintainers for anyway.

REVU and queues

Now, I want to talk a little bit about REVU, and dealing with queues.

Again, a quick bit of background for those not involved in Ubuntu development. REVU is the tool we generally use in Ubuntu to review new packages. Anyone can propose a package for inclusion in to Ubuntu simply by uploading it there. Developers can then review it and provide feedback, asking for things to be changed where necessary. Once it is to a satisfactory standard and has the support of two people with upload privileges it is uploaded to the archive.

REVU is generally a nice platform for doing this work, and I'm not necessarily criticising its design here, I would just like to examine some of the effects of some design decisions.

REVU works with three queues. The first is the queue for packages that have one advocation from an Ubuntu developer. This queue is normally very short and fast moving, as the package is in good shape and just needs double-checking. The second queue is all of the packages waiting for a review. The third queue is for packages that need some work done on them before they can be uploaded. This means that for developers there is primarily one queue that contains things they can look at.

This is similar to the sponsorship queue, which is used in Ubuntu as a way for people without upload rights to make a change to a package in the archive. This can be seen as just one queue where the developer looks for things to review and upload.

The sponsorship queue is for changes to packages we already have in the archive, and we generally want to upload everything on the list, even if not at a certain point in the development cycle. The difference is that it's generally easier to give a reason for saying "no" for the rare times that it happens. This changes the complexion of what it means to keep the queue small.

If an item on the sponsorship queue is incomplete and the person who submitted it doesn't follow through then we should be picking up the item and ensuring the problem is fixed. If a package on REVU is incomplete and the person who submitted it doesn't follow through then there is no real problem. If the package is popular someone else will eventually pick it up and do the work.

We don't have (or at least haven't had) the time to review enough packages on REVU, so the queue is pretty long. Assuming that doesn't change keeping the queue small will mean removing things because of lack of interest from the person submitting them. We can work to review more things, but unless we reach the point where we are accepting more packages than are being proposed that will always be the case.

There was recently a proposal to help clear the queue of things where the submitter has given up, so that effort can be focused on the packages where problems are likely to be fixed. While doing this is makes best use of developer time I feel that if we feel the need to do this then we have already lost.

If that is the case then there is a worrying aspect to it. We have had a bunch of people show an interest in Ubuntu development, and take the first steps towards becoming a developer, only to get discouraged and give up.

I have heard complaints about it being difficult to find a reviewer, and I'm sure the people that gave up would not speak fondly of the experience. However, this hasn't become widespread enough that it has stopped people giving it a go, and it would be terrible if it did, as having that reputation may cause potential contributors to look elsewhere.

I think this indicates that we should reconsider the way REVU is presented within our community. We are presenting a great service to people who are getting started with packaging and pointing them there saying, put your package here, "we will review it and you will get it in to Ubuntu." However, this unfortunately doesn't happen all that often. We are channeling potential new developers there, knowing that there is a high chance of them getting discouraged and giving up. On top of that we give the reason that many of them do give up as one reason that we're not better at reviewing the packages in the first place.

To me this almost feels like we are trying to put people off from getting started with Ubuntu development.

It looks as though the proposal to help clear the queue for the start of this cycle is going to go ahead. That's fine with me, we might as well make the most of a bad situation and help the reviewers this cycle. If we are in a position where we feel we need to do the same at the start of next cycle then I would say that it is clear we have not fixed the underlying issues, and we really need to stop and reconsider how we present REVU.

There is also a slim possibility that the clearing of the queue may have a different effect. If some contributors are put of by the number of packages waiting for review, then clearing that may cause them to put their own up there. While this could imply more potential developers, it would also mean more packages to review, making it harder to keep up.

If we keep up this cycle and have a good process going in six months time then I will be happy, but it will make the rest of this post pretty much obsolete. I think now is a reasonable time to discuss the "what if?" though.

Changing things

If we are in the same situation in six months then I think it will demonstrate that even with a clean slate and a will to fix the problem we were not able to do enough reviews to keep up with the supply. In that case we will need to look for ways to turn down the taps, to reduce the supply.

In no way do I want to send the message that we shouldn't be welcoming to new contributors, I would just want to explore ways to get them started with working on our existing packages. It may involve making it harder to get a package in to Ubuntu though. We may just have to live with that.

In my opinion there would be two ways to target this, and the solution may come from some combination of the two. The first would be to make working on existing packages more attractive, easier to start with, and the default choice. The second would be to make proposing a new package more difficult, less attractive, or at least not demoralising if your package isn't reviewed.

If I had any bright ideas for the first then I would already be trying to implement them, so let's focus on ways to do the second. Keep thinking about the first though, and feel free to discuss any ideas with me.

I'll take three ideas from the recent mailing list thread and one I just had and look at what effects they may have.

Conclusion

I don't really have a conclusion, but I have plenty to think about, and I hope you have too. I'll be more than happy to discuss these issues at any time, so just give me a shout.

Making Intrepid Solid

With feature freeze now in effect the bulk of the big changes in Intrepid should now be done. There will still be new features entering the archive with the appropriate exceptions, but the rate will slow as we move forward.

Now we really need to focus on making Intrepid solid. We want to squash as many bugs as possible, so that when we deliver the final release it is something we can be proud of.

This is something that everyone can help with. There are plenty of ways to help out, so there should be something for everyone.

Making Intrepid+1 rock

In parallel with all the above now is the time to start thinking about what you want to achieve in the next cycle. If any of that requires changes to an upstream then speaking to them early can be a good idea, as you can get their feedback and see how it fits in to their plan. I'm sure everyone has loads of ideas, and a bit of preparation now can help you hit the ground running in the next cycle.

Revision numbers vs. revision ids

One thing that Bazaar does a little different to the other distributed systems is to give every revision a revision number. Some people don't like this as the revision numbers are global, that means that the revision number of a revision in my branch does not necessarily match the number that it was given in your branch. Some people say that this makes bzr somehow "less distributed." This is not the case at all, you just need to be careful to be clear what branch you are referring to, i.e. say "revision 315 on branch http://...", rather than just "revision 315".

This is a little dangerous in that the branch may have it's revision history changed, for instance by uncommitting and then committing again. If that may happen then you should use revision ids, which you can find from "bzr log --show-ids".

Why do number the revisions at all then? One reason is simply that some people find revision ids ugly, and may be scared off by them.

Another reason is that they are shorter to type. git folks will tell you can use the first few characters of the revision id, and git will work out what revision you mean. However these shortened ids are not necessarily stable over a long period of time, and so again, if you are worried you should use the whole thing.

The third reason is that the numbers can give some sense of the order of the revisions in your branch. If I say talk about revisions "3445abe" and "b27ac9" then you don't know which is earlier in history. If I refer to revisions 345 and 532 then it is immediately obvious (providing that I am only referring to a single branch).

The advantages I have outlined are small, but they could be valuable at times, and you always have the revision ids to fall back on if you require them.

Numbering merged revisions

Along with numbering the mainline bzr also numbers merged revisions using a dotted numbering scheme. This means that your mainline revisions are "1", "2", "3", as you would expect, but any merged revisions are given three digit numbers, e.g. "2.1.3".

The numbering scheme has a couple of nice properties, the most notable of which is that it is "stable", this means that once I have numbered a merged revision with respect to a certain mainline it cannot be effected by any addition I make to the revision history. This means that any commits, pulls or merges that I do will not change any of the existing revision numbers, but they will add numbers to any new merged revisions such that they will not be the same as any number already used.

The current algorithm used for this involves looking at the whole history of the branch to number the revisions, which is obviously undesirable.

On the last evening of the sprint last week myself and John were discussing the numbering scheme, and thinking about possible algorithms to do the numbering that would be more efficient.

We had the following inputs:

1. The revsision id you are trying to give a dotted number for.
2. The tip of the mainline that you are numbering against.
3. The revision number of that mainline revision.
4. A map of revision_id -> parents.

And you are asked to provide the revision number for the specified revision. Any other numbering that you may be able to do along the way would be a bonus.

The fact that all we are given to get the information we need is a map telling us the parents of a revision id means that we cannot ask the question "what are all the children of this revision?"

I don't really want to explain the numbering scheme here, as it is a little long-winded to do so. The outline is that for the first digit you find the intersection of the target revision's left hand ancestry with the mainline, and use its revision number. For the second digit you find all of the branches that originated at the revision found in the first part, and then number them by the order that they merged back in to mainline. The third digit is then just the place of the revision in its own part of one of these branches.

Notice the second step there. Remember that we are not able to retrieve the children of any revision? That means that we must work backwards from our mainline to do this. This is where the real complexity comes in, and it appears as though it is necessary to search a reasonable amount of history to calculate this part.

After the discussion with John I had a reasonable idea of how such an algorithm would work, and yesterday I posted a first draft of that to the mailing list. We have found some problems with it, and haven't benchmarked it yet to see if it is actually an improvement, but hopefully it will evolve and prove to be faster.

Displaying logs, and history emphasis

The revision numbering code has a very close relationship, and also interacts with it in an awkward way from a performance standpoint. This lead to John explaining to me how the logs are generated in more depth.

When bzr produces logs by default it emphasises the left hand parent to produce your mainline. It then indents any revisions that you merged:

>       -----------------------------------------------------------
>       revno: 3270
>       committer: Canonical.com Patch Queue Manager <pqm@pqm.ubuntu.com>
>       branch nick: +trunk
>       timestamp: Thu 2008-03-13 00:40:30 +0000
>       message:
>         (Adeodato Simo) Add a space after "revision-id:" in log output.
>          -----------------------------------------------------------
>          revno: 3257.2.1
>          committer: Adeodato Simó <dato@net.com.org.es>
>          branch nick: foo
>          timestamp: Sun 2008-03-09 23:06:47 +0100
>          message:
>            Add a space after "revision-id:" in log output.
>       -----------------------------------------------------------
>       revno: 3269
>       committer: Canonical.com Patch Queue Manager <pqm@pqm.ubuntu.com>
>       branch nick: +trunk
>       timestamp: Wed 2008-03-12 23:08:34 +0000
>       message:
>         (Daniel Watkins) Add a --revision option to 'bzr push'
>           -----------------------------------------------------------
>           revno: 3256.1.5
>           committer: Daniel Watkins <D.M.Watkins@warwick.ac.uk>
>           branch nick: push-r
>           timestamp: Sun 2008-03-09 18:41:31 +0000
>           message:
>             Added NEWS entry.

To do this it must decide which revisions are present in the history of one revision, but not in the history of its left hand parent. To do this it starts off two history walkers in parallel, one walking the history of the first revision, the second walking the history of the parent. The first walker then stops walking down a particular line of history when the second "claims" it, once the first walker has no more lines of history to walk it returns its group of revisions, and the log formatter code then displays them indented as necessary to match the history.

This is a much more complex process than that you get with "git log", in which the revisions are produced in just date order. There is a "--topo-order" option to git log, but that just ensures that all parents are output before their children. It doesn't ensure that all parents not in the ancestry of the left-hand parent are shown before the left-hand parent. The work to ensure that is significantly more than that done to provide "--topo-order".

This display makes it easy to see what work was done on a branch, and when those changes entered your branch. This is one reason why bzr's merge doesn't fast-forward by default ("bzr merge --pull" will do this for you if you like). This means that you can always instantly identify which work came from another branch and have them tied together.

Always having merge commits means that "bzr log --short" and "bzr log --line" can give you a good summary of what happened on your branch, the commits you did, and the things that you merged. It preserves a mainline for you in the left hand ancestry, which means that you can always see what happened in that particular branch. "bzr pull" then gives you a mirror of another branch, and the left hand ancestry tells you what happened in that branch.

The indentation of the merged commits (and the fact they disappear with "--short" and "--line") means that mentally they become of lesser importance. You see "merged performance work from Emma's branch", rather than all of the commits that you got from her. They are still there to look at if you want, but they can be ignored at most times.

This means that you don't have to spend time rewriting history to be clean if you don't want to. You don't have the history right in your face either way, though there can still be value in having a clean history. However rewriting history is not what some people want to do, and causes problems for those who base their work on yours.