Normally when you write some code using launchpadlib you end up with Launchpad showing your users something like this:

This isn't great, how is the user supposed to know which option to click? What do you do if they don't choose the option you want?

Instead it's possible to limit the choices that the user has to make to only those that your application can use, plus the option to deny all access, by changing the way you create your Launchpad object.

from launchpadlib.launchpad import Launchpad

lp = Launchpad.get_token_and_login("testing", allow_access_levels=["WRITE_PUBLIC"])

This will present your users with something like this:

which is easier to understand. There could be further improvements, but they would happen on the Launchpad side.

This approach works for both Launchpad.get_token_and_login and Launchpad.login_with.

The values that you can pass here aren't documented, and should probably be constants in launchpadlib, rather than hardcoded in every application, but for now you can use:

• READ_PUBLIC
• READ_PRIVATE
• WRITE_PUBLIC
• WRITE_PRIVATE

Dear Mr Neary, thanks for your thought provoking post, I think it is a problem we need to be aware of as Free Software matures.

Firstly though I would like to say that the apparent ageism present in your argument isn't helpful to your point. Your comments appear to diminish the contributions of a whole generation of people. In addition, we shouldn't just be concerned with attracting young people to contribute, the same changes will have likely reduced the chances that people of all ages will get involved.

Aside from that though there is much to discuss. You talk about the changes in Free Software since you got involved, and it mirrors my observations. While these changes may have forced fewer people to learn all the details of how the system works, they have certainly allowed more people to use the software, bringing many different skills to the party with them.

I would contend that often the experience for those looking to do the compilation that you rate as important has parallels to the experience of just using the software you present from a few years ago. If we can change that experience as much as we have the installation and first use experience then we will empower more people to take part in those activities.

It is instructive then to look at how the changes came about to see if there are any pointers for us. I think there are two causes of the change that are of interest to this discussion.

Firstly, one change has been an increased focus on user experience. Designing and building software that serves the users needs has made it much more palatable for people, and reduced the investment that people have to make before using it. In the same way I think we should focus on developer experience, making it more pleasant to perform some of the tasks needed to be a hobbyist. Yes, this means hiding some of the complexity to start with, but that doesn't mean that it can't be delved in to later. Progressive exposure will help people to learn by not requiring them to master the art before being able to do anything.

Secondly, there has been a push to make informed decisions on behalf of the user when providing them with the initial experience. You no longer get a base system after installation, upon which you are expected to select from the thousands of packages to build your perfect environment. Neither are you led to download multiple CDs that contain the entire contents of a distribution, much of which is installed by default. Instead you are given an environment that is already equipped to do common tasks, where each task is covered by an application that has been selected by experts on your behalf.

We should do something similar with developer tools, making opinionated decisions for the new developer, and allowing them to change things as they learn, similar to the way in which you are still free to choose from the thousands of packages in the distribution repositories. Doing this makes documentation easier to write, allows for knowledge sharing, and reduces the chances of paralysis of choice.

There are obviously difficulties with this given that often the choice of tool that one person makes on a project dicatates or heavily influences the choice other people have to make. If you choose autotools for your projects then I can't build it with CMake. Our development tools are important to us as they shape the environment in which we work, so there are strong opinions, but perhaps consistency could become more of a priority. There are also things we can do with libraries, format specifications and wrappers to allow choice while still providing a good experience for the fledgling developer.

Obviously as we are talking about free software the code will always be available, but that isn't enough in my mind. It needs to be easier to go from code to something you can install and remove, allowing you to dig deeper once you have achieved that.

I believe that our effort around things like https://dev.launchpad.net/BuildBranchToArchive will go some way to helping with this.

The deadline for students to submit their applications to Google for Summer of Code is imminent.

If you were waiting for the last minute to submit, that is now!

If you are mentor and have the perfect student you have been working with, check with them that they have submitted the application to Google, otherwise you will be stuck.

Next week we'll start to process the huge number of applications that we have for Ubuntu.

If you don't want to read this article, then just steer clear of python-multiprocessing, threads and glib in the same application. Let me explain why.

There's a rather famous bug in Gwibber in Ubuntu Lucid, where a gwibber-service process will start taking 100% of the CPU time of one of your cores if it can. While looking in to why this bug happened I learnt a lot about how multiprocessing and GLib work, and wanted to record some of this so that others may avoid the bear traps.

Python's multiprocessing module is a nice module to allow you to easily run some code in a subprocess, to get around the restriction of the GIL for example. It makes it really easy to run a particular function in a subprocess, which is a step up from what you had to do before it existed. However, when using it you should be aware how the way it works can interact with the rest of your app, because there are some possible nasties lurking there.

GLib is a set of building blocks for apps, most notably used by GTK+. It provides an object system, a mainloop and lots more besides. What we are most interested here is the mainloop, signals, and thread integration that it provides.

Let's start the explanation by looking at how multiprocessing does its thing. When you start a subprocess using multiprocessing.Process, or something that uses it, it causes a fork(2), which starts a new process with a copy of the programs current memory, with some exceptions. This is really nice for multiprocessing, as you can just run any code from that program in the subprocess and pass the result back without too much difficulty.

The problems occur because there isn't an exec(3) to accompany the fork(2). This is what makes multiprocessing so easy to use, but doesn't insert a clean process boundary between the processes. Most notably for this example, it means the child inherits the file descriptors of the parent (critically even those marked FD_CLOEXEC).

The other piece to this puzzle is how the GLib mainloop communicates between threads. It requires some mechanism where one thread can alert another that something of interest happened. To do this when you tell GLib that you will be using threads in your app by calling g_thread_init (gobject.threads_init() in Python) then it will create a pipe for use by glib to alert other threads. It also creates a watcher thread that polls one end of this pipe so that it can act when a thread wishes to pass something on to the mainloop.

The final part of the puzzle is what your app does in a subprocess with mutliprocessing. If you purely do something such as number crunching then you won't have any issues. If however you use some glib functions that will cause the child to communicate with the mainloop then you will see problems.

As the child inherits the file descriptors of the parent it will use the same pipe for communication. Therefore if a function in the child writes to this pipe then it can put the parent in to a confused state. What happens in gwibber is that it uses some gnome-keyring functions and that puts the parent in to a state where the watcher thread created by g_thread_init busy-polls on the pipe, taking up as much CPU time as it can get from one core.

In summary, you will see issues if you use python-multiprocessing from a thread and use some glib functions in the children.

There are some ways to fix this, but no silver bullet:

• Don't use threads, just use multiprocessing. However, you can't communicate with glib signals between subprocesses, and there's no equivalent built in to multiprocessing.
• Don't use glib functions from the children.
• Don't use multiprocessing to run the children, use exec(3) a script that does what you want, but this isn't as flexible or as convenient.

It may be possible to use the support for different GMainContexts for different threads to work around this, but:

• You can't access this from Python, and
• I'm not sure that every library you use will correctly implement it, and so you may still get issues.

Note that none of the parties here are doing anything particularly wrong, it's a bad interaction caused by some decisions that are known to cause issues with concurrency. I also think there are issues when using DBus from multiprocessing children, but I haven't thoroughly investigated that. I'm not entirely sure why the multiprocessing child seems to have to be run from a non-main thread in the parent to trigger this, any insight would be welcome. You can find a small script to reproduce the problem here.

Or, to put it another way, global state bad for concurrency.

As you've probably heard by now, Ubuntu has been accepted to Google Summer of Code this year. We're currently at the point where we are looking for students to take part and the mentors to pair with them to make the proposal. We have some ideas on the wiki, but there's nothing to stop you coming up with your own if you have a great idea. The only requirement is that you find a mentor to help you with it.

The best way to do this is to write up a proposal on your wiki page on the Ubuntu wiki, and then to email the Ubuntu Summer of Code mailing list about it. You can also ask for possible mentors on IRC and on other Ubuntu mailing lists related to your idea.

I have a couple of ideas on the wiki page, but I am happy to consider ideas from students that fall in my area of expertise.

I spend most of my time working on developer tools and infrastructure. These are things that users of Ubuntu won't see, but are used every day by developers of Ubuntu. Improvements we can make in this area can in turn improve Ubuntu by giving us happier, more productive, developers. It's also an interesting area to work in, as there are usually different constraints to developing user software, as developers have different demands.

If you think that sounds interesting and you have a great idea that falls in to that area, or you like one of my ideas on the wiki page, then get in touch with me. I will be happy to discuss your ideas and help you flesh them out in to a possible proposal, but I won't be able to mentor everyone.

I would consider mentoring any idea that either improved existing tools used by Ubuntu developers (bzr, pbuilder, devscripts, ubuntu-dev-tools, etc.) or created a new one that would make things easier. In the same spirit, anything that makes it easier for someone to get started with Ubuntu development, such as Harvest, helpers for creating packages, etc. could be a possible project. The last category would be infrastructure-type projects such as the idea to automate test-merging-and-building of new upstreams, or similar ideas.

I've also posted about some ideas that I would like to see previously on my blog, which might be a source of inspriation.

If this interests you then you can find out how to contact me on my Launchpad profile.

As my contribution to Ada Lovelace Day 2010 I would like to mention Emma Jane Hogbin.

Emma is an Ubuntu Member, published author, documentation evangelist, conference organiser, Drupal theming expert, tireless conference presenter, and many more things as well.

Her enthusiasm is infectious, and her passion for solving problems for people is admirable. She is a constant source of inspiration to me, and that continues even as she branches out in to new things.

(Hat tip for the title to the ever excellent Sharrow)

The Bazaar package importer is a service that we run to allow people to use Bazaar for Ubuntu development by importing any source package uploads in to bzr. It's not something that most Ubuntu developers will interact with directly, but is of increasing importance.

I've spent a lot of time working in the background on this project, and while the details have never been secret, and in fact the code has been available for a while, I'm sure most people don't know what goes on. I wanted to rectify that, and so started with some wiki documentation on the internals. This post is more abstract, talking about the archtecture.

While it has a common pattern of requirements, and so those familiar with the architecture of job systems will recognise the solution, the devil is in the details. I therefore present this as a case-study of one such system that can be used to constrast other similar sytstems as an aid to learning how differing requirements affect the finished product.

write_failure(package, "died")
try:
    import(package)
except:
    write_failure(packge, stderr)
finally:
    remove_failure(package)

distro.main_archive.getPublishedSources()

for s in distro.series:
    distro.main_archive.getPublishedSources(distro_series=s)

Made this up after buying a nice piece of locally caught freshwater trout. I think that it would be even better if you were to hot-smoke it. Apply the rub between two and twelve hours before cooking.

Mix up the following then rub on to the flesh of the fish (enough for four servings):

• 1 tbsp sea/rock salt.
• 1 tbsp black peppercorns crushed.
• 1 tbsp ground cumin.
• 1 tbsp ground coriander.
• 2 tsp caraway seed.
• 2 tsp dried tarragon.
• 2 tsp dried thyme.
• 2 tsp chilli powder.
• Zest of one lemon.

To drizzle on top when cooked melt some butter in a pan, add the juice of the lemon you used above, a pinch of salt, one crushed clove of garlic, and a handful of chopped coriander. Simmer for a couple of minutes.

Enjoy!

David, it's interesting that you posted about that, as it's something I've been toying with for the last couple of years. For the last few months I've been (very) slowly experimenting in my free time with an approach that I think works well, and I think it's time to tell more people about it and to ask for contributions.

Opportunistic programmers are useful to cater for here, as Debian/Ubuntu development isn't trivial, and so we are simplifying something existing, which means that it will still be powerful, which is also important. I'm not only interested in improving the experience for the opportunistic programmer though, why should they get all the cool stuff? I'm interested in producing something that I can use for doing Ubuntu development too (though not every last detail).

The project I am talking about has been christened "cambria" and is now available on Launchpad. It's a library that aims to provide great APIs for working with packages throughout the lifecycle, including things like Bazaar, PPAs, local builds, testing, lintian, etc. It should be pleasurable to use and also allow you to build tools on top that are also pleasurable. It should also allow for easy extension in to different GUI toolkits and for command-line tools, though I've only been working with GTK so far.

In addition, there is a gedit plugin that allows you to perform common tasks from within gedit. I chose gedit as it has a pleasant Python API for plugins, isn't so complicated that it takes much learning, and will already be installed on most Ubuntu desktop systems. As I said though, the libarary allows you to implement in anything you like (that can use a python library.)

I've put together some mockups that suggest some of the things that I would like to do:

Build

Package list

Download

The RATIONALE file includes some more reasons for the project:

Project cambria is about wrapping the existing tools for Debian/Ubuntu development to allow a more task-based workflow. Depending on the task the developer is doing there may be several things that must be done, but they must currently work each one out individually. We have documentation to help with this, but it's much simpler if your tools can take care of it for you.

Project cambria aims to make Ubuntu development easier to get started with. There are several ways that it will help. Providing a task-based workflow where you are prompted for the information that is needed to complete the task, and other things are done automatically, or defaults chosen helps as it means you can concentrate on completing the task, rather than learning about all the possible changes you could make and deciding which applies.

Project cambria aims to make Ubuntu development easier for everyone by automating common tasks, and alleviating some of the tool tax that we pay. It won't just be a beginner tool, but will provide tools and APIs that experienced developers can use, or can build upon to build tools that suit them.

Project cambria will help to take people from novice to experienced developer by providing documentation that allows you to learn about the issues related to your current task. This provides an easier way in to the documentation than a large individual document (but it can still be read that way if you like).

Project cambria will make Ubuntu development more pleasurable by focusing on the user experience. It will aim to pull together disparate interfaces in to a single pleasing one. Where it needs to defer to a different interface it should provide the user with an explanation of what they will be seeing to lessen the jarring effect.

I'm keen for others to contribute, there is some information about this in the project's CONTRIBUTING file. I'm looking for all sorts of contributions from all kinds of people and keen to help you get started if you aren't confident with the type of contribution you would like to make.

There's a mailing list as part of the ~cambria team on Launchpad and IRC channel if you are interested in discussing it more.

You may well have heard about it (on this blog especially), but though I spend lots of my time involved with it and talking to people about it, there may be some people who aren't entirely sure what we are doing with the Ubuntu Distributed Development initiative, or what we are trying to achieve. To try and help this I wrote up an overview of what we are doing.

If this project interests you and you would like to help, or just observe, then you can subscribe to the mailing list. There's lots of fun projects that you could take on: there's far more that is possible and would be hugely useful to Ubuntu developers than we can currently work on. If you want to work on something then feel free to talk to me about it and we can see if there is something that would suit you.

Without further ado...

Many projects that I work on, or follow the development of, and granted there may be a large selection bias here, are showing some of the same tendencies. Combined these indicate to me that we need to change the way we look at becoming a trusted member of the project.

The obvious change here is the move to distributed version control. I'm obviously a fan of this change, and for many reasons. One of those is the democratisation of the tools. There is no longer a special set of people that gets to use the best tools, with everyone else having to make do. Now you get to use the same tools whether you were the founder of the project, or someone working on your first change. That's extremely beneficial as it means that we don't partition our efforts to improve the tools we use. It also means that new contributors have an easier time getting started, as they get to use better tools. These two influences combine as well: a long time contributor can describe how they achieve something, and the new contributor can directly apply it, as they use the same tools.

This change does mean that getting "commit access" isn't about getting the ability to commit anymore; everyone can commit anytime to their own branch. Some projects, e.g. Bazaar, don't even hand out "commit access" in the literal sense, the project blessed code is handled by a robot, you just get the ability to have the robot merge a branch.

While it is true that getting "commit access" was never really about the tools, it was and is about being trusted to shepherd the shared code, a lot of projects still don't treat it that way. Once a developer gets "commit access" they just start committing every half-cooked patch they have to trunk. The full use of distributed version control, with many branches, just emphasises the shared code aspect. Anyone is free to create a branch with their half-baked idea and see if anyone else likes it. The "blessed" branch is just that, one that the project as a whole decides they will collaborate on.

This leads to my second change, code review. This is something that I also deeply believe in; it is vital to quality, and a point at which open source software becomes a massive advantage, so something we should exploit to the full. I see it used increasingly in many projects, and many moving up jml's code review "ladder" towards pre-merge review of every change. There seems to be increasing acceptance that code review is valuable, or at least that it is something a good project does.

Depending on the project the relationship of code review and "commit access" can vary, but at the least, someone with "commit access" can make their code review count. Some projects will not allow even those with "commit access" to act unilaterally, requiring multiple reviews, and some may even relegate the concept, working off votes from whoever is interested in the change.

At the very least, most projects will have code review when a new contributor wishes to make a change. This typically means that when you are granted "commit access" you are able or expected to review other people's code, even though you may never have done so before. Some projects also require every contribution to be reviewed, meaning that "commit access" doesn't grant you the ability to do as you wish, it instead just puts the onus on you to review the code of others as well as write your own.

As code review becomes more prevalent we need to re-examine what we see as "commit access," and how people show that they are ready for it. It may be that the concept becomes "trusted reviewer" or similar, but at the least code review will be a large part of it. Therefore I feel that we shouldn't just be looking at a person's code contributions, but also their code review contributions. Code review is a skill, some people are very good at it, some people are very very bad at it. You can improve with practice and teaching, and you can set bad examples for others if you are not careful. We will have to make sure that review runs through the blood of a project, everyone reviews the code of everyone else, and the reviews are reviewed.

The final change that I see as related is that of empowering non-code contributors. More and more projects are valuing these contributors, and one important part of doing that is trusting them with responsibility. It may be that sometimes trusting them means giving them "commit access", if they are working on improving the inline help for instance. Yes, it may be that distributed version control and code review mean that they do not have to do this, but those arguments could be made for code contributors too.

This leads me to another, and perhaps the most important, aspect of the "commit access" idea: trust. The fundamental, though sometimes unspoken, measure we use to gauge if someone should get "commit access" is whether we believe them to be trustworthy. Do we trust them to introduce code without review? Do we trust them to review other people's changes? Do we trust them to change only those areas they are experts in, or to speak to someone else if they are not? This is the rule we should be applying when making this decision, and we should be sure to be aware that this is what we are doing. There will often be other considerations as well, but this decision will always factor.

These ideas are not new, and the influences described here did not create them. However the confluence of them, and the changes that will likely happen in our projects over the next few years, mean that we must be sure to confront them. We must discard the "commit access" idea as many projects have seen it, and come up with new responsibilities that better reflect the tasks people are doing, the new ways projects operate, and that reward the interactions that make our projects stronger.

One of the new things that is going to be in karmic is that the kerneloops daemon will be installed and running by default. This tool, created by Arjan van de Ven, watches the kernel logs for problems. It has a companion service, kerneloops.org which aggregates reports of these problems, and can sort by kernel version and the like. This allows kernel developers to spot the most commonly encountered problems, areas of the code which are prone to bugs etc. When the kerneloops daemon catches a problem it allows you to send the problem to kerneloops.org.

We however, are not using the applet that comes with kerneloops to do this, we are making use of the brilliant Apport. There are a couple of reasons for this. We also want to make it easy for you to report these issues as bugs to Launchpad, and we don't want to prompt you with two different interfaces to do that.

The changes mean that if your machine has a kernel issue you will get an apport prompt as usual. As well as asking if you would like to report the problem to Launchpad like it does for other crashes it will ask if you would also like to report it to kerneloops.org. Passing the information through apport means that it can also be used on servers as well without running X.

Hopefully you will never see this improvement, but it's now going to be there for when those bugs do creep in.

I've just implemented the most requested feature in bzr-builder (Hi Ted), command support.

Sometimes you need to run a particular command to prepare a branch of your project for packaging (e.g. autoreconf). I think this should generally go in your build target, but not everyone agrees, and sometimes there is just no other way.

Therefore I added a new instruction to bzr-builder recipes, "run". If you put

run some command here

in your recipe then it will run "some command here" at that point when assembling.

Note that running commands that require arbitrary network access is still to be discouraged, as you don't know in what environment someone may assemble the recipe. I'd also advise against using commands unless you really need them, but that's obviously your call.

I recently gave a talk to some fellow Canonical employees about where we are with the "Distributed Development" project. For that I made a screencast showing some of the Launchpad Codehosting features that you can now use for Ubuntu. Thanks to the Launchpad team for making this happen. We're still ironing out the remaining kinks that make it a pain to use, and getting all the packages imported, but it's possible to use them now.

The screencast has no audio unfortunately, but you can watch it and try and guess what I was saying. There's documentation available on the wiki as well.

One of the things the video shows is how to request someone review your change, i.e. how to get a change sponsored in to Ubuntu. I'm keen to have people test this, as it's not something I do very often now that I am a core-dev. Therefore if you want to help test then propose a merge and set the appropriate sponsor team as the reviewer, and I will prioritise it and you can give me feedback in return.

Note that a bug in Launchpad means that I won't get a notification when they are created, so feel free to drop me a line via email or on IRC until that bug is fixed next month. I'll continue to poll the lists though, so nothing will get dropped.

As well as seeing use of PPAs for providing bug fixes, new upstream versions, proposed packages, testing etc., we are also seeing them used for providing daily builds of packages. For instance Fabien Tassin provides daily builds of lots of Mozilla-related packages and snapshots of Chromium in his various PPAs. Also, there is Project Neon, to provide daily builds of Amarok.

They massively lessen the barrier to using and testing code that is fresh from the fingers of the developers. They avoid you having to build a project from source every day, making sure to keep up with changes in dependencies. They allow you to be testing code almost as it is written, speeding up the feedback cycle to the developers, and potentially increasing the number of people involved in that feedback cycle.

In addition they allow you to verify bugs against the latest code, so that bug reports are of more relevance to the developers. If you so choose they can also be set up so that bugs are also tested with fewer distribution patches, further increasing the developers' confidence in the bug reports.

Mark had an idea for an elegant way to describe how to combine the code to produce the package, and we worked on producing a tool to follow the steps. You can find the result of this in the bzr plugin bzr-builder. I've documented how to use it on the wiki.

There's still more we can do to improve the process, and we have a lot to discuss about what makes a good daily package, and what the limits of them are. If you are interested in discussing this then please join the list of the dailydebs team in Launchpad.

I'm currently running the bzr-nightly-ppa using this tool, and have improved some things based on this, but more testing, feedback, and patches are always welcome.