I have a problem that I believe will be easy for someone with a bit of UNIX coding knowledge to solve, so I appeal to those that can to help.

I'm trying to write a DBus service that will spawn a command, and provide the output to the user. The service runs on the system bus as root, and so it is a form of privilege escalation. However, the command may be long running, and produce a lot of output as it works, so I want to allow the calling process to get this output before the command completes.

My current approach uses gobject.spawn_async and so gets file descriptors back, one for stdout and one for stderr. I currently have a thread that uses select to wait for output, and then uses DBus signals to allow the client to access it. This works great, except that stdout and stderr can become interleaved in the middle of lines.

I believe that I can't just wait for full lines before signalling, as a command might do something like print "Username: " and then wait for input. I could normally do full lines, and then if the child blocks on stdin send whatever it has written so far, but that doesn't seem ideal. (I haven't implemented anything about proving input on stdin so far, but I don't want a solution that makes it difficult to do so).

It seems to me that this is something that will be implemented somewhere, for instance my shell can run commands and then interleave the output in a desirable manner, but I haven't found how yet. Any suggestions are welcome, but this is from python, so system calls that I can't make directly from python would be a pain, though I'm not that bothered about portability.

I came across Anthony Raijekov the other day, and was treated to some of his Trip-Hop, which is outstanding. That was an added bonus though, as I sought him out to download his Piano track: "Photo Theme: Window Like". You can find him on Jamendo and on ccmixter. I would highly recommending going to listen, and donating via Jamendo if you like what you hear.

This also led me to go back and listen to some new stuff from Amether, who I found a few years ago thanks to Rob Da Bank. Definitely worth checking out, especially their remix of "Artisan - Hold my breath".

On the freely available, if not freely licensed side I noticed a new station from the excellent SomaFM today: Lush. It is said to be "Sensuous and mellow vocals, mostly female, with an electronic influence," and so far I am enjoying it though it is rather similar to Groove Salad. I find that I can't keep SomaFM on all day every day while working though, as I find that it repeats tracks just a little too often.

I've also been listening a lot to Ombilikal which covers the harder edge very well, with some breaks, drum and bass, and dubstep amongst other things.

While in Barcelona I had the pleasure of meeting Karl Fogel (and hearing him play, which was a treat), and talking a bit about free content. He explained to me some of the things that they are trying to do with QuestionCopyright.org, and some of their methods. It's great to see more projects working on the issue in a very constructive manner, and I hope that they succeed. So that we can have many more artists like Anthony Raijekov that I can discover and reward for their work more directly.

This was our dinner last night. There's nothing particularly original here, but it was damn tasty. There are no pictures, so you'll have to take my word that it looked great.

First chop the veg for roasting. Anything that roasts well will work. I used sweet potato, squash and courgette. Stick it in the tray, with some oil, garlic cloves, salt, pepper, and ground cumin and coriander. Put that in the oven to roast and prepare the rest of the meal. When you remove it to serve then stir in a large handful of roughly chopped coriander, it tastes great and the green looks great with the orange of the veg.

Next prepare the lamb. Make patties from minced lamb, chopped fresh mint (half the bunch), the juice of half a lemon, chopped red onion, and finely chopped garlic with some spices: ground cinnamon, coriander, cumin, garam masala and chilli powder. Grill or barbecue the patties (or make them in to kebabs if you like).

Chop the other half of the mint, and add it and the juice from the rest of the lemon to some fresh yoghurt. Then stir in a little maple syrup and a little Cajun or jerk spice. Add only a little at first then add more to taste. You're aiming for the sweetness and spice to be very subtle and to soften the yoghurt flavour.

Serve with toasted pitta and shredded gem lettuce.

Jo Shields posts about his intention to propose switching Rhythmbox for Banshee in Karmic. (I'm trying to convince him to apply to be a MOTU, or at least an Ubuntu member, so he can be on Planet Ubuntu.) He's not the first to suggest it, but he is in the right place to make the proposal for this release.

I'm personally quite happy with Rhythmbox, and haven't really tried Banshee, though for my use I imagine they are pretty similar. (Two things that would be definite benefits for me would be remembering what I was doing when I closed it when it starts, and understanding mix CDs better). I can certainly understand some of the arguments for switching as well, so I wouldn't be against it.

There is an idea on Brainstorm about this, and while Brainstorm can't capture the intricacies of the debate, it suggests that Rhythmbox is quite popular with the people that voted (though it's not clear how many had tried the alternatives.)

This post isn't really to argue one way or the other, or to attempt to cover all the criteria by which a decision will be made. My point is to emphasise that the Banshee developers have done themselves a great favour in this debate by making one aspect of switching easier. The have implemented importing from Rhythmbox. This means that any switch wouldn't mean that all users had to re-import their collection.

That's not everything that is involved in switching, and indeed, many of the issues around trying to change a default don't have good answers, and that's something we should work to improve as a community.

You don't have to spend time implementing importers for every similar application out there, but easing migration from commonly used apps can help users switch, and is a big benefit when trying to switch a large number of users painlessly. Also, while importing is useful, it's not the ideal solution. A common storage format, and shared storage would be superior in many ways for this purpose.

Jaunty just froze a little bit more, with the last few normal uploads being done. From here on in it's mainly about getting the CDs perfect for release, which will hopefully go smoothly.

Over the last few days there have been a number of people working on Universe to get it in to the best shape we could in the remaining time. We did a pretty good job of it too, towards the end we were scavenging around for any more fixes that were ready to upload. As always, with more people we could have done more, but it seemed to be a very smooth landing this time.

The sponsorship queue is virtually all things that were not appropriate for Jaunty, with just a couple of desirable fixes not making the cut (we'll work to have those in jaunty-updates ASAP). In addition to that, NBS was clear, meaning that there were no outstanding library transitions or similar, and there are very few uninstallables. Obviously we would want all of these numbers to be zero, but you can't have that with a time-based release schedule. Unfortunately the FTBFS list is rather long (mainly due to toolchain changes), but it's generally infrequently updated packages on there, which will tend to be of less interest.

The MOTUs also did a fantastic job of the python 2.6 transition, which was a huge job, and a compressed timeframe to do it in. Unfortunately there are going to be some issues with the change in the default python for some time to come, but given the state of python a couple of months ago this is a great acheivement.

Also, I'll make special mention of the Mono 2.0 transition. Co-ordinated by Jo Shields, and thanks to a lot of people on both the Debian and Ubuntu sides, this was completed with very little fuss. It was a great example of co-ordinating work on a large number of packages, and of collaboration between Debian and Ubuntu. I also think that it showed some of the advantages of the Ubuntu method of development over the Debian one, but the shared work trumps that.

If you are reading this thinking "You might think you did a good job, but what about this bug that I provided a patch for 3 months ago, why didn't you fix that?" then all I can really do is point you to the sponsorship process. Yes, it sucks that not knowing about this cost you, but reviewing every bug with an attachment tagged "patch" is currently a little out of our reach. I'm always looking for ways to improve this, and I hope one day we can do that, but in the meantime using the sponsorship process will help get your patch included.

As well as the new notification bubbles, jaunty now has something called the "indicator-applet", or various other names. If you are running Jaunty you may have seen it as an envelope in your panel. Not much is taking advantage of it yet, with only Evolution and Pidgin having the necessary support until recently.

The purpose of this applet is to give you access to messages that are waiting for you, regardless of the source. You can see some of Ted's early sketches on the idea, or the early draft of the spec for some information. Having this place gives you one place to look to respond to a message you received, and is part of the strategy to solve the issues caused by having notifications without actions. It also allows you to launch the main windows of the applications so you don't need to have an icon for each in the notification area.

As I don't use pidgin the only thing taking advantage of the menu was evolution, so I went to find which of the other apps I use had support. I first grabbed a more recent version of gwibber which had support, and sent a quick merge request to improve the support. There was little else I could do there though, as Ryan had done a great job already, and all I had to do was hook up a callback so that gwibber would open when you clicked on a message you received from someone.

I then moved on to Empathy, and found a bug requesting support be added. I decided to relax this weekend by working on something completely different and trying to add this support.

I found the libindicate API easy to work with, though it is un-documented currently there is not too much too it, and it has quite some similarity with libnotify. After finding my way around the empathy internals, learning more about gtk+ and encountering a bug in libindicate that I spent some time investigating I had something mostly working. It's not ready to merge yet, but it is most of the way there, and covers the most common cases.

You can see a quick video of the interaction, which shows the concept behind the menu as well. (Yes, I am talking to myself in that video, as Ted said, working on IM apps can be quite lonely as you continually send yourself messages).

This was supposed to be an Ada Lovelace Day post, but jet lag got the better of me last night before I could write it, so it's a Lady Day post instead, which is strangely apt. We don't have to restrict ourselves to a single day to highlight great people anyway.

The woman that I would like to write about is Sarah Bird. Sarah is an engineer by training, and spent much of her time working in the development, thanks in part to MIT's D-Lab. One of her University projects was a cheap land mine clearing device, which looked like a big drum on wheels, because that is essentially what it was. She always talks about it with great passion, as it meant that she got to test it by blowing things up.

In conjunction with Amy Smith (who is apparently also made of awesome) she has been working on "balls" which allow samples to be refrigerated so that they can be tested when there is no power. There work has the potential to save many lives, and they won a Deshpande grant to continue the research.

Last time I spoke to Sarah she was heading out to Pakistan with her new company SaafWater, trying to ensure that Pakistani families drank clean water, and using an innovative strategy to achieve that.

I know Sarah as I did a Summer internship at Aptivate, a UK NGO, at the same time as she was working there. She had a knack for analysis, and for explaining things, and she gave me the language to talk about many things, such as concerns about the OLPC project.

In particular though she taught me that simply considering yourself to be non-discriminatory is not enough. Trying to treat everyone in the same manner doesn't help to remove inequality, it will just perpetuate it. Without concerted effort the solution will just reflect the needs of the majority, and their needs are already well enough accounted for. If you want to have a solution that works for everyone, then everyone needs to be represented in those designing the solution. For that I am very grateful.

In addition to all that she's also great fun, and we spent many a great evening in the pub.

I'm also glad to say that there are also many inspiring women in the communities I am part of. It was great to see so many of the posts yesterday being about women that are currently active in Free Software, it's great to have so many peers that you can look up to, as well as the pioneers.

I'm a crap maintainer, as well as being bloody lazy.

I'm relieved I've finally been found out, I can desist with this charade. I mean, it must be true, someone said it on the Internet.

The second installment of Developer News went out on Monday, and boy was it hard work. It's great to see so much going on, but it does make preparing the summary time consuming. Thanks a lot to Stefan Lesicnik for his help in preparing this one. As well as having more to report I also broadened the intended audience this month.

For the first month the intended audience was just Ubuntu developers, so I didn't include anything from ubuntu-devel-announce, as I assumed that everyone would already have seen those announcements. The first issue was picked up by both the fridge and LWN though, so it was more widely read, and showed there was an interest in having more news for external people. Therefore I wrote this edition to include those people as well.

I'm pretty sure that there was plenty more that we could have included as well, it's just I didn't know about it, or there was nothing to link to. I think we need to do a better job as a community to communicate what we are doing. My process was simply to trawl the archives looking for announcements and discussions of things, so all anyone had to do was write one email.

Therefore I think we have two problems to solve, firstly getting everything that is going on announced in the right places, and secondly making it easier to summarise all of the activity.

I hope that the first will become more a part of our culture, and the Developer News will help by getting more exposure for those who do communicate about what they are doing.

I'm not too sure how to improve the second though. We have a defined process for submitting items for the news, but to date there have been no submissions using that process, and one submission by editing the wiki page as I was preparing the second issue. Why is this? Why have you not submitted anything? Is it that you didn't know about the process? Is it too much work? Is it that you never think to use it? That you are not sure that your item counts? That you are not sure that someone else hasn't alread submitted it?

The feedback I have got about the Developer News has been both frequent and entirely positive, so I believe it is a valuable service that should be carried on, but I fear that the current way of doing it won't really scale. I don't think it would be worth a day of my time a month to complete it.

Any suggestions for improving the situation would be appreciated.

Please consider the following hypothetical situation. gnome-screensaver keeps crashing for people on Jaunty, which is pretty annoying, so it sends lots of people to Launchpad to report a bug.

It's pretty quickly established by the amazing desktop team triagers that it's actually a problem in Gtk+, so they duly reassign the bug there.

However, it's a little while before the fix is forthcoming, and it that time many users are still getting hit by the bug, so they are still heading to launchpad. As most of us aren't Gtk+ hackers we don't see that the issue is there, so we go to the obvious place to report the bug, the gnome-screensaver package, and not seeing any existing reports about the bug we file a new one.

This means that the desktop team has to spend time shovelling the bugs over to Gtk+ and duplicating them to the first one. While we have some tools to help with this sort of thing it would be great if we could try and avoid it all together.

One way this could be solved in Launchpad is to leave a task open on the gnome-screensaver package, but this isn't ideal, as the bug doesn't need to be fixed there. You could mark a gnome-screensaver bug on the package as Invalid, which would make the dupe search as you report a bug show it, but it wouldn't show up for those that just trawled the open gnome-screensaver bug reports looking for the crash.

My idea for something that would help in this case is Ghost bugs. These are bugs that are the ghosts of a bug somewhere else. In the above case a ghost bug could be created against gnome-screensaver pointing to the Gtk+ bug. It would then show up in the bug lists, but not have a status etc., and be somewhat "greyed out", hence the name ghost bug.

As the bug affects gnome-screensaver it makes sense for it to show up against that, but as it doesn't need to be fixed there it shouldn't have the rest of the information.

It doesn't just work for packages. See for example launchpad bug #174539. This is a bug that should be fixed in bzr, but it only manifests itself in bzr-builddeb. I am keeping a task against bzr-builddeb so that it shows up in my list to fix to have bzr-buiddeb work great, but I don't really need a status as nothing needs to change in bzr-builddeb for it to work.

With Launchpad having the concept of bug tasks this could be easily done by adding a new status Ghost, or perhaps Fix Elsewhere or something. This would be handled differently to the other statuses, giving behaviour such as that I described above.

Having a whole bug task may not be the right thing though. It could instead be a separate list, similar to the list of bug tasks, but just listing the things that should have ghost tasks.

Does anyone else think this would be useful? Is there prior art?

Mr LeSage, I just happened to run bzr plugins today, and noticed this:

.
.
sandwich
    (no description)
.
.

Curious as to what the hell that was, I ran the command again with -v and saw that it was installed in my ~/.bazaar/plugins/ directory. I opened the file and found this that I wrote a few months ago:

from bzrlib.commands import Command, register_command

class cmd_make_me_a_sandwich(Command):

    def run(self):
        self.outf.write("What? Make it yourself.\n")

class cmd_sudo_make_me_a_sandwich(Command):

    def run(self):
        self.outf.write("Okay.\n")

register_command(cmd_make_me_a_sandwich)
register_command(cmd_sudo_make_me_a_sandwich)

Not quite what you were after though. That would be this plugin:

class SandwichCommand(Command):

    def run(self, **kwargs):
        for name, arg in kwargs.items():
            if arg != name:
                self.outf.write(self.fail_message + "\n")
                return 1
        self.outf.write(self.success_message + "\n")
        return 1

class cmd_make(SandwichCommand):

    takes_args = ["me", "a", "sandwich"]
    fail_message = "Make you a what?"
    success_message = "What? Make it yourself."

class cmd_sudo(SandwichCommand):

    takes_args = ["make", "me", "a", "sandwich"]
    fail_message = "Of course, but what do you want me to do?"
    success_message = "Okay."

register_command(cmd_make)
register_command(cmd_sudo)

I'm still none the wiser as to why I wrote that plugin in the first place though.

I spent today hacking on bzr-builddeb for the first time in a while. The first order of business was to review and merge all of the proposed merges in launchpad. Thanks especially to Jelmer Vernooij for all his work.

After that I tackled a long-standing annoyance in bzr-builddeb. This limitation made it difficult to pass extra options to the underlying build command. For instance, when building a merge you must use the -v option to debuild. Previously this meant that you would have to run something like:

$ bzr builddeb -S --builder "debuild -S -v0.1-1"

Using the tradition of -- to end option parsing this now becomes:

$ bzr builddeb -- -S -v0.1-1

which is much better.

The -- isn't exactly obvious, but it does mean that I don't have to implement support for every option of dpkg-buildpackage.

It may be possible to add support in bzr to make this work without the --, I haven't looked yet, and the current implementation doesn't prevent us from doing that.

One other thing I did was to make the build command default to debuild, rather than dpkg-buildpackage -rfakeroot -uc -us, so that we get all the debuild goodness. The obvious change here will be that it tries to sign the package when you build. You can define your builder to be debuild -uc -us to avoid this.

One last break was to remove support for source-builder from the configuration. It now assumes that adding -S to the build command will make it build a source package.

Comments on any of this are welcome.

Yesterday as part of Ubuntu Developer Week I gave a session entitled "Bazaar for Packaging". At the last minute I decided to change the session somewhat, so that it would show how things would work if you were to use bzr to modify an Ubuntu package once Distributed Development is fully up and running.

The session went ok, and while I was showing some fairly experimental things it all worked quite well. The biggest problem was when we grabbed a change from SVN using the bzr-svn plugin. The rather simple step of extracting a patch took up quite a lot of the session as bzr-svn initialised it's metatdata about the Subversion branch. bzr-svn is amazing, it allows you to store a bzr branch inside an svn repository, while still making it readable to svn. However, to do this it has to do some fairly intensive transformations to maintain the mapping. The biggest impact of this is when you access the SVN repository for the first time though, so it wasn't the smartest idea for an IRC tutorial. That's the problem with changing your session 10 minutes before it starts.

You can go and read the transcript of the session if you want to see how all of this worked. I'd like to skip ahead a little bit and show you how it will work in a short while when launchpad hosts the branches and all the bits are in place.

First we need to grab the source for the package we want to work on. We'll grab a whole branch here, but you could just as well use a lightweight checkout or a stacked branch to transfer less data.

$ bzr branch lp:ubuntu/jaunty/gnome-utils gnome-utils.jaunty

Will give us a local copy of that branch in gnome-utils.jaunty. We can now make our changes in that branch.

$ cd gnome-utils.jaunty

You can run bzr log (or better bzr viz from bzr-gtk or bzr qlog from qbzr) to see the history if that is interesting for the change we are making.

We're just going to apply the patch from SVN though. This is what we did in the session:

$ bzr diff -c svn:8378 http://svn.gnome.org/svn/gnome-utils/trunk | bzr patch

(where bzr patch is supplied by bzrtools. Try it it's cool, it works over any transport, so you can apply a remote patch file without downloading it)

What we are doing here is in fact a "cherry-pick". bzr will happily do these for you, but it does the equivalent of diff + patch, it is hoped to improve this and record which revisions were merged, and use that information to help you understand what changes have and haven't been merged.

To more directly do a cherry-pick you can run

$ bzr merge -c svn:8378 http://svn.gnome.org/svn/gnome-utils/trunk

(merge the changes introduced in svn revision 8378 of this branch please, the svn: is neccesary as bzr and svn count their revisions differently)

However, this won't currently work, as the branches have what is called "different rich-root support", so we have to use the explicit "diff and patch" for now. This is a pain, and will hopefully go away sometime soon. This method would work fine with most bzr branches though.

Once we have applied the patch we write the changelog entry for it. We run "dch -i -D UNRELEASED", which will create us a new changelog entry, and mark it as "UNRELEASED" so that it is clear it still needs to be uploaded. Obviously if there is an existing UNRELEASED changelog entry then we want to add to that. I would like to write a wrapper than did the right thing here.

For that changelog entry we write the usual thing, something like:

* Don't crash when asked to show a path that has been excluded. (LP: #301952)

Now we are ready to build and test our changes. Running

$ bzr builddeb -S

will spit out a source package in the parent directory, in the same way as debuild -S. We can then build this package in our normal fashion, in pbuilder say, or upload it to a PPA.

Once we are happy then we can commit our changes. The easiest way to do this is to run

$ debcommit

which uses our changelog entry as the commit message, saving us from typing the same thing again.

There's one extra bit of magic that goes on here. bzr supports the --fixes option to commit. This marks the resulting revision as fixing the specified bug, for example --fixes lp:301952 would indicate that we closed the bug that we are working on in this revision. In Intrepid (thanks to the idea from Colin Watson) I implemented support for this in debcommit. If debcommit sees you closing a bug in the changelog message that it is using it will automatically add the corresponding --fixes argument (it works for Debian bugs too). We'll see where this comes in useful in a minute.

The last step is to get our changes in to the distribution. If we have upload rights for the package then we can dput our source package that we created a minute ago, and then run

$ bzr push lp:ubuntu/jaunty/gnome-utils

to push the bzr branch back. (Yes, launchpad plans to support building directly from a branch, so you just need to push along with some undecided mechanism to request it be included in Ubuntu)

If you don't have upload rights for the package then you need someone to sponsor the change for you. To do this you first push your branch to launchpad somewhere under your name. For instance I would run

$ bzr push lp:~james-w/ubuntu/jaunty/fix-301952

Note that thanks to the launchpad and bazaar developers implementing support for "stacked branches" and automatic stacking in launchpad this will be a very cheap operation, only pushing a single revision.

Next we would create a merge proposal for this change. You can either do this from the branch page on launchpad, or you can use bzr send. Just running

$ bzr send

should do the right thing. It will open up a new message in your mail client. You then enter your "cover letter" for the change, and hit send. It will mail the request to launchpad which will interpret the machine-readable attachment and turn it in to a merge request. The developers can then review the changes and either ask for improvements to be done, or upload the package.

Remember the --fixes information that was stored? That was also used by launchpad. The bug that we were fixing now has a link to our branch on it, so that anyone that wants to test the fix can find the right place to get the change from. This currently does not generate any bugmail though, so you have to go to the page to see it. I think this is something we need to improve.

Some of the things that I have explained here haven't been fully decided, so this isn't documentation, that will come later, the intent is to give an idea of how this may work.

I've been asked a few times about an IRC channel where we can discuss this sort of thing, so I created #ubuntu-bzr today. If you are interested in shaping how this will work then please join it and we can discuss it. Support can continue anywhere though.

After an idea from bigon on #launchpad today I threw together a tool using the Launchpad API. I've christened this tool ppamadison. It does the same thing as rmadison, but for PPAs. You tell it who's PPA to examine, and what source package to get the information for, and it tells you what versions are available.

$ ppamadison james-w bzr-builddeb
bzr-builddeb | 2.0~0ubuntu1~ppa1 | intrepid | source
bzr-builddeb | 2.0~ppa1~hardy1 | hardy | source

There's still some things left to do, such as replicating rmadison's odd output formatting, some things missing from the Launchpad API and some interesting things you could add, but the idea is there. One thing missing from the Launchpad APIs as far as I can see is an efficient way to find out which PPAs contain a certain source package name. This would be quite an interesting thing to know.

Would ppamadison be a useful thing to have in ubuntu-dev-tools? If it is worthwhile then I will integrate it. Because this blog post is something that developers might not see, but might be interested in I would then pass it on to the Developer News service, as all it would take would be a quick email, as little as a link to the blog post would do.

(Yes, I am being facetious, but we haven't had a single submission yet)

As an aside, I play with the Launchpad APIs every couple of months and they are getting better, to the point now where most data I want for things I do is available. Thanks to the Launchpad team for their work on it. There are some real problems for some use-cases, such as a cache hit requiring a https connection, but ways can be found to deal with them. In any case, the APIs will allow us to do some really useful things.

P.S. Thank you all for your kind words.

I have a small request to make. When writing a changelog entry please remember you are writing it for other people to read as well. Please document why you are doing something in enough detail that I can work it out later on.

A changelog is not just for describing what you changed, it should be for describing what you changed and why you changed it.

Just documenting what was changed is not that helpful, as I can simply look at the diff to see that. What I can't get from the diff is what was in your head at the time, the motivation, the reasons why you chose that solution, the stuff that I care about when I am looking at the change.

Oh, and one more thing, your immediate motivation may not be the one that I care about. "Because somebody told me to" isn't the reason that will help me understand the change.

Thanks in advance,

James