REVU

Quality of packages not in Debian

While Ubuntu takes most of its packages from Debian, it does contain a few that are not in Debian, for one reason or another. One common reason is simply that someone wanted a package that wasn't in Debian, and packaged it and requested its inclusion.

The fact that these packages are not in Debian means that they are slightly different to the rest of the packages in Ubuntu, as they are completely Ubuntu's responsibility to maintain. By including them we are making a different sort of contract with our users and the authors of the software.

I was interested in how well we do at keeping this contract. A quick bit of scripting and I found some basic numbers on this. There are 886 packages in Ubuntu's universe component that are not in Debian. One possible measure of quality is the number of open bugs against these packages, which is shown in the following table as frequency counts.

So there are few open bugs on these packages, and around two-thirds have none. However, there are more than 200 open bugs in total, which we would want to do something about.

To look at how well these bugs are being triaged I looked more closely at those packages with open bugs, and looked at what percentage of those bugs were still in the "New" status with an "Undecided" priority, i.e. completely

So again, this isn't too bad, with many packages with less than 5% completely untriaged bugs. Again though, the numbers are too high, and as this only counts those bugs that are completely untriaged, I'm sure that most are not triaged to the level that we would want.

Lastly I wanted to see how much visibility we have in to problems these packages may have. For this I looked at the number of bug subscribers for the source package. While this is certainly not everything, I would certainly fell better if we had one or two people subscribed to the bugs for every one of these packages.

This shows that two thirds of the packages don't have anyone subscribed to the bugs for these packages.

What about keeping up with upstream? Shipping old versions of packages means they are likely to be more buggy, and won't be popular with the authors of the software. The Ubuntu External Health Status site attempts to track this using the debian/watch files in the source packages. I attempted to pull the information about these packages from there.

A bit of explanation is in order for those not intimately familiar with Debian packaging. The debian/watch file specifies the location of the upstream source in such a way that new versions can be checked for, and the update of the package be done semi-automatically. This can't be automated for a package that has no watch file, so I list the number of those. Also those packages that have a watch file, but it doesn't work for some reason are listed, as the watch file is not usable for this, though sometimes that is a transient problem.

The figures for the number of packages in each of the three states are:

The number of packages that are out of date is not that bad, especially considering we've been frozen for the last few months. However, there is a large number of packages without a watch file. This means that there's no automated way to find out about new versions of the package being released. It's possible to do without that, but I doubt that all 495 packages have someone watching over them.

Why Ubuntu?

The reason we allow packages directly in to Ubuntu is that it brings benefit to our users. Most packages that enter Ubuntu will be of benefit to someone, and that's one of our aims, to give our users a good experience, isn't it? To give our users a good experience we also want high quality though. So we have a balancing act, splitting our workload between fixing what we have and spending time bringing in and maintaining new packages.

I would argue that we want to limit the flow of new packages quite severely, as we are not exactly short of work with our current set of packages. However, the argument isn't quite as simple as that.

A common way for people to get involved with development is to package something new and get it uploaded, and later branch out in to working on existing packages. This means that bringing in lots of new packages may lead to an influx of new contributors that more than makes up for the new workload. I'm not sure how strong this link will be, and whether the ratios will be such that there is a net gain in developer time.

Why not Debian?

So, some people will be wondering why all these packages aren't in Debian. The overarching reason for this is that contributors would rather get their packages directly in to Ubuntu.

There are several possible reasons for this, and I have heard most of these stated by someone trying to get their package in to Ubuntu.

One is a simple one, they don't run Debian, and so it's difficult to test in that environment. Yes, it's not impossible, but it is more work.

There is also a perception among many contributors that getting your package in to Debian is hard work, with long delays trying to find a sponsor. That's not something that most of us can really do much about. We can try and tackle the perception, but without upload right we can't really fix the problem. There was the utnubu team that tried to streamline this process, but that is now defunct as far as I know.

Another reason is that they may not wish to do this is that they use Ubuntu and don't really care enough about Debian to do the extra work. This is something we can try and do something about, explaining the virtues of getting the package in to Debian, more than just it being the right thing to do in most cases.

As a counterpoint to this, if the package is going to be useful to a lot of people then even if the person proposing it for Ubuntu does not want to try and get it in to Debian then there is likely to be someone with the interest and skills to maintain it in Debian. For me this means that the packages that we want to be pulling in to Ubuntu should be easy to find Debian maintainers for anyway.

REVU and queues

Now, I want to talk a little bit about REVU, and dealing with queues.

Again, a quick bit of background for those not involved in Ubuntu development. REVU is the tool we generally use in Ubuntu to review new packages. Anyone can propose a package for inclusion in to Ubuntu simply by uploading it there. Developers can then review it and provide feedback, asking for things to be changed where necessary. Once it is to a satisfactory standard and has the support of two people with upload privileges it is uploaded to the archive.

REVU is generally a nice platform for doing this work, and I'm not necessarily criticising its design here, I would just like to examine some of the effects of some design decisions.

REVU works with three queues. The first is the queue for packages that have one advocation from an Ubuntu developer. This queue is normally very short and fast moving, as the package is in good shape and just needs double-checking. The second queue is all of the packages waiting for a review. The third queue is for packages that need some work done on them before they can be uploaded. This means that for developers there is primarily one queue that contains things they can look at.

This is similar to the sponsorship queue, which is used in Ubuntu as a way for people without upload rights to make a change to a package in the archive. This can be seen as just one queue where the developer looks for things to review and upload.

The sponsorship queue is for changes to packages we already have in the archive, and we generally want to upload everything on the list, even if not at a certain point in the development cycle. The difference is that it's generally easier to give a reason for saying "no" for the rare times that it happens. This changes the complexion of what it means to keep the queue small.

If an item on the sponsorship queue is incomplete and the person who submitted it doesn't follow through then we should be picking up the item and ensuring the problem is fixed. If a package on REVU is incomplete and the person who submitted it doesn't follow through then there is no real problem. If the package is popular someone else will eventually pick it up and do the work.

We don't have (or at least haven't had) the time to review enough packages on REVU, so the queue is pretty long. Assuming that doesn't change keeping the queue small will mean removing things because of lack of interest from the person submitting them. We can work to review more things, but unless we reach the point where we are accepting more packages than are being proposed that will always be the case.

There was recently a proposal to help clear the queue of things where the submitter has given up, so that effort can be focused on the packages where problems are likely to be fixed. While doing this is makes best use of developer time I feel that if we feel the need to do this then we have already lost.

If that is the case then there is a worrying aspect to it. We have had a bunch of people show an interest in Ubuntu development, and take the first steps towards becoming a developer, only to get discouraged and give up.

I have heard complaints about it being difficult to find a reviewer, and I'm sure the people that gave up would not speak fondly of the experience. However, this hasn't become widespread enough that it has stopped people giving it a go, and it would be terrible if it did, as having that reputation may cause potential contributors to look elsewhere.

I think this indicates that we should reconsider the way REVU is presented within our community. We are presenting a great service to people who are getting started with packaging and pointing them there saying, put your package here, "we will review it and you will get it in to Ubuntu." However, this unfortunately doesn't happen all that often. We are channeling potential new developers there, knowing that there is a high chance of them getting discouraged and giving up. On top of that we give the reason that many of them do give up as one reason that we're not better at reviewing the packages in the first place.

To me this almost feels like we are trying to put people off from getting started with Ubuntu development.

It looks as though the proposal to help clear the queue for the start of this cycle is going to go ahead. That's fine with me, we might as well make the most of a bad situation and help the reviewers this cycle. If we are in a position where we feel we need to do the same at the start of next cycle then I would say that it is clear we have not fixed the underlying issues, and we really need to stop and reconsider how we present REVU.

There is also a slim possibility that the clearing of the queue may have a different effect. If some contributors are put of by the number of packages waiting for review, then clearing that may cause them to put their own up there. While this could imply more potential developers, it would also mean more packages to review, making it harder to keep up.

If we keep up this cycle and have a good process going in six months time then I will be happy, but it will make the rest of this post pretty much obsolete. I think now is a reasonable time to discuss the "what if?" though.

Changing things

If we are in the same situation in six months then I think it will demonstrate that even with a clean slate and a will to fix the problem we were not able to do enough reviews to keep up with the supply. In that case we will need to look for ways to turn down the taps, to reduce the supply.

In no way do I want to send the message that we shouldn't be welcoming to new contributors, I would just want to explore ways to get them started with working on our existing packages. It may involve making it harder to get a package in to Ubuntu though. We may just have to live with that.

In my opinion there would be two ways to target this, and the solution may come from some combination of the two. The first would be to make working on existing packages more attractive, easier to start with, and the default choice. The second would be to make proposing a new package more difficult, less attractive, or at least not demoralising if your package isn't reviewed.

If I had any bright ideas for the first then I would already be trying to implement them, so let's focus on ways to do the second. Keep thinking about the first though, and feel free to discuss any ideas with me.

I'll take three ideas from the recent mailing list thread and one I just had and look at what effects they may have.

Conclusion

I don't really have a conclusion, but I have plenty to think about, and I hope you have too. I'll be more than happy to discuss these issues at any time, so just give me a shout.