At UDS last week there was another "Testing in Ubuntu" session. During the event I gave a brief presentation on monitoring and testability. The thesis was that there are a lot of parallels between monitoring and testing, so many that it's worth thinking of monitoring as a type of testing at times. Due to that great monitoring requires a testable system, as well as thinking about monitoring right at the start to build a monitorable system as well as a testable one.

You can watch a video of the talk here. (Thanks to the video team for recording it and getting it online quickly.)

I have two main questions. Firstly, what are the conventional names for the "passive" and "active" monitoring that I describe? Seecondly, do you agree with me about monitoring?

I recently finished reading "All Art is Propaganda" by George Orwell, a collection of some of his critical essays. It was a fascinating read, and would recommended it. Each of the essays is thought-provoking and enlightening, and the topics covered are numerous and varied.

The most interesting feature of the book though wasn't the subject matter of the essays, but the organisation of them. The editor decided to put them in chronological order, meaning that you see some development of ideas over the essays, and different topics rise and fall in prominence.

While that's certainly not novel, the effect of structuring the book in this way was very noticeable in this case for me. I saw a lot of parallels to the impressions I've had from following @RealTimeWWII on twitter. This account is "live tweeting" the Second World War as if it were happening today (currently in 1940).

This artifice brings a whole fresh appreciation of this period that I have learnt so much about. Consuming the events at the pace they occured gives time to reflect on each one, and forgetting that I know the events that followed allows one to get a greater understanding of what it would have been like to live at that time. The time-compressing effect of looking back tends to obscure the uncertainty and fear of that time, the slowness with which some events were unfolding accentuating it. Consuming this via twitter, with its headline-like format mixing in with the news of today, heightens this effect.

While it's something of a loss that we aren't able to know what Orwell would say about the events of today, or what he would have changed in these essays with the addition of hindsight, there's an undeniable value in reading this primary source. While hindsight adds, it also takes away, blurring memories and changing perspectives. Reading the essays allows you to pick up on the thinking of those living through the events that we think we know so well.

For instance Orwell seemed to believe that Soviet Russia was a greater threat than Nazism. The essays in the book run from 1940 to 1949, and there are many more words devoted to the Soviets than the Nazis throughout. His writing suggests that he thought the techniques the Soviets used to achieve and maintain power were less well known and understood, and would be more effective over a long period.

After better understanding these benefits I plan to redouble my efforts to choose books from a varied set of sources, including from different times, and avoid falling in to a trap of thinking that more recent must be better as knowledge is always on the increase.

Today is the first day of the Linaro Connect in Cambridge. Linaro has gathered to spend a week talking, coding and having fun.

The Infrastructure team is spending most of the week coding, on a few select topics, chosen to make good use of the time that we have together.

In order to help us focus on our goals for the week I've put together a hard copy version of status.linaro.org.

We'll be updating it during the week as we make progress. I'll report back on how it looks at the end of the week.

If you are an application developer and you want to distribute your new application for a linux distribution, then you currently have several hurdles in your path. Beyond picking which one to start with, you either have a learn a packaging format well enough that you can do the work yourself, or find someone that can do it for you.

At the early stages though neither of these options is particularly compelling. You don't want to learn a packaging format, as there is lots of code to write, and that's what you want to focus on. Finding someone to do the work for you would be great, but there are far more applications than skilled packagers, and convincing someone to help you with something larval is tough: there are going to be a lot of updates, with plenty of churn, to stay on top of, and it may be too early for them to tell if the application will be any good.

This is where pkgme comes in. This is a tool that can take care of the packaging for you, so that you can focus on writing the code, and skilled packagers can focus on packages that need high-quality packaging as they will have lots of users.

This isn't a new idea, and there are plenty of tools out there to generate the packaging for e.g. a Python application. I don't think it is a particularly good use of developer time to produce tools like that for every language/project type out there.

Instead, a few of us created pkgme. This is a tool in two parts. The first part knows about packaging, and how to create the necessary files to build a working package, but it doesn't know anything about your application. This knowledge is delegated to a backend, which doesn't need to understand packaging, and just needs to be able to tell pkgme certain facts about the application.

pkgme is now at a stage where we would like to work with people to develop backends for whatever application type you would like (Python/ Ruby On Rails/GNOME/KDE/CMake/Autotools/Vala etc.) You don't have to be an expert on packaging, or indeed on the project type you want to work on. All it takes is writing a few scripts (in whatever language makes sense), which can introspect an application and report things such as the name, version, dependencies, etc.

If this sounds like something that you would like to do then please take a look at the documentation, write the scripts, and then submit your backend for inclusion in pkgme.

You can also contact the developers, see the nascent website at pkgme.net, or visit the Launchpad page. (We are also very interested in help with the website and documentation if that is where you skills or interests lie.)

This was the confusing part when I first ran couchapp to create a new app, I couldn't really see where the "entry point" of the app was. In the hope that it might help someone else I'm going to present a quick overview of the default setup.

jml just announced testtools 0.9.8 and in it mentioned the soupmatchers project that I started. Given that I haven't talked about it here before, I wanted to do a post to introduce it, and explain some of the rationale behind it.

soupmatchers is a library for unit testing HTML, allowing you to assert that certain things are present or not within an HTML string. Asserting this based on substring matching is going to be too fragile to be usable, and so soupmatchers works on a parsed representation of the HTML. It uses the wonderful BeautifulSoup library for parsing the HTML, and allows you to assert the presence or not of tags based on the attributes that you care about.

self.assertThat(some_html,
                HTMLContains(Tag('testtools link', 'a',
                attrs={'href': 'https://launchpad.net/testtools'})))

You can see more examples in the README.

Basing this on the testtools matchers frameworks allows you to do this in a semi-declarative way. I think there is a lot of potential here to improve your unit tests. For instance you can start to build a suite of matchers tailored to talking about the HTML that your application outputs. You can have matchers that match areas of the page, and then talk about other elements relative to them ("This link is placed within the sidebar"). One thing that particularly interests me is to create a class hierarchy that allows you test particular things across your application. For instance, you could have an ExternalLink class that asserts that a particular class is set on all of your external links. Assuming that you use this at the appropriate places in your tests then you will know that the style that is applied to class will be on all external links. Should you wish to change the way that external links are represented in the HTML you can change the one class and your tests should tell you all the places that the code has to be updated.

Please go ahead and try the library and let me know how it could be improved.

[ Apologies to those that saw this half-finished when I published rather than saving a draft ]

This is the part that it took me a long time to understand: how the different parts of the default couchapp collaborate to present data to the user.

In this post I'm just going to deal with client-side couchapps using the default technologies. As explained in the previous post you can use any combination of HTML and javascript in a couchapp, and you can also do some of the work server-side in couchdb. However, I'm going to explain what the couchapp tool gives you when you create a new project, as that is where you are likely to be starting, and once you understand that you can choose where to deviate from that model.

{click: function() {
        alert("You clicked!");
    }
}

{click:
    {
        mustache: "<div>You clicked!</div>"
    }
}

{click:
    {
        mustache: "<div>You clicked!</div>",
        render: "append"
    }
}

{click:
    {
        data: function(e) {
            return {name, "Bob"};
        },
        mustache: "<div>Hi {{name}}!</div>"
    }
}

{click:
    {
        async: function(callback) {
            /* some code that does an async request, and then calls callback with the result */
        },
        data: function(resp) {
            /* Some code that processes the data from the async function to ready it for the template */
        },
        mustache: "A tempate that will be rendered with the result of the data function"
    }
}

Today I would like to talk about the couchapp tool. This is something that you can use when working on couchapps, and provides a way to quickly iterate your design.

However, rather confusingly, the couchapp tool isn't actually required for couchapps. If you get a design document with HTML attachments in to your database then you have a couchapp.

Why would you want to use such a tool then? Firstly because it will generate a skeleton couchapp for you, so that you don't have to remember how to organise it, and if it is your first couchapp it's good to start from something working.

More importantly though, the couchapp tool is useful as you develop as it allows you to edit the parts of your app in their native format. This means that if you are writing a HTML snippet you can just put some HTML in a file, rather than having to write it as part of a deeply nested JSON structure and deal with the errors that you would make if you did that. Also it means that you can use things like syntax highlighting in your preferred text editor without any special magic.

couchapp generate myapp

cd myapp
ls

couchapp push http://localhost:5984/mydb

couchapp push desktopcouch://

http://localhost:5984/_utils

xdg-open ~/.local/share/desktop-couch/couchdb.html

couchapp push http://localhost:5984/mydb

Couchapps are a particular way of using couchdb that allow you to serve web applications directly from the database. These applications generate HTML and javascript to present data from couchdb to the user, and then update the database and the UI based on their actions.

Of course there are plenty of frameworks out there that do this sort of thing, and more and more of them are adding couchdb support. What makes couchapps particularly interesting are two things. Firstly, the ease with which they can be developed and deployed. As they are served directly from couchdb they require little infrastructure, and the couchapp tool allows for a rapid iteration. In addition, the conveniences that are provided mean that simple things can be done very quickly with little code.

The other thing that makes couchapps attractive is that they live inside the database. This means that the code lives alongside the data, and will travel with it as it is replicated. This means that you can easily have an app that you have fast, local access to on your desktop, while at the same time replicating to a server so that you can access the same data from your phone while you are out. Again, this doesn't require couchapps, and they won't be suitable for all needs, but they are certainly an interesting idea.

You can read more about couchapps at http://couchapp.org.

Intrigued by couchapps I set out to play with them over a weekend. Unfortunately the documentation is rather lacking currently, so I wouldn't recommend experimenting yourself if you are not happy digging around for answers, and sometimes not finding them outside the code. In order to go a little way to rectifying this, I intend to write a few posts about the things I wish I had known when I started out. I found everything to be a little strange at first, and it wasn't even clear where the entry point of a couchapp was for instance. Hopefully these posts will be found using google by others who are struggling in a similar way.

The examples for Django testing point you towards hardcoding a username and password for a user to impersonate in tests, and the API of the test client encourages this too.

However, Django has a nice pluggable authentication system that means you can easily use something such as OpenID instead of passwords.

Putting the passwords in your tests ties you to having the password support enabled, and while you could do this for just the tests, it's completely out of the scope of most tests (I'm not talking about any tests for the actual login process here.)

When I saw this while reviewing code recently I worked with Zygmunt to write a Client subclass that didn't have this restriction. With this subclass you can just choose a User object, and have that client login as that user, without them having to have a password at all. Doing this decoples your tests from the implementation of the authentication system, and makes them target the code you want to test more precisely.

Here's the code:

from django.conf import settings
from django.contrib.auth import login
from django.http import HttpRequest
from django.test.client import Client


class TestClient(Client):

    def login_user(self, user):
        """
        Login as specified user, does not depend on auth backend (hopefully)

        This is based on Client.login() with a small hack that does not
        require the call to authenticate()
        """
        if not 'django.contrib.sessions' in settings.INSTALLED_APPS:
            raise AssertionError("Unable to login without django.contrib.sessions in INSTALLED_APPS")
        user.backend = "%s.%s" % ("django.contrib.auth.backends",
                                  "ModelBackend")
        engine = import_module(settings.SESSION_ENGINE)

        # Create a fake request to store login details.
        request = HttpRequest()
        if self.session:
            request.session = self.session
        else:
            request.session = engine.SessionStore()
        login(request, user)

        # Set the cookie to represent the session.
        session_cookie = settings.SESSION_COOKIE_NAME
        self.cookies[session_cookie] = request.session.session_key
        cookie_data = {
            'max-age': None,
            'path': '/',
            'domain': settings.SESSION_COOKIE_DOMAIN,
            'secure': settings.SESSION_COOKIE_SECURE or None,
            'expires': None,
        }
        self.cookies[session_cookie].update(cookie_data)

        # Save the session values.
        request.session.save()

Then you can use it in your tests like this:

from django.contrib.auth.models import User


client = TestClient()
user = User(username="eve")
user.save()
client.login_user(user)

Then any requests you make with that client will be authenticated as the user that was created.

Ticket submitted with Django to have this available for everyone in future.

I'm keen to try and write more about the things that I work on as part of my job at Canonical. In order to get started I wanted to write a summary of some of the things that I have done, as well as a little about what I am working on now.

Normally when you write some code using launchpadlib you end up with Launchpad showing your users something like this:

This isn't great, how is the user supposed to know which option to click? What do you do if they don't choose the option you want?

Instead it's possible to limit the choices that the user has to make to only those that your application can use, plus the option to deny all access, by changing the way you create your Launchpad object.

from launchpadlib.launchpad import Launchpad

lp = Launchpad.get_token_and_login("testing", allow_access_levels=["WRITE_PUBLIC"])

This will present your users with something like this:

which is easier to understand. There could be further improvements, but they would happen on the Launchpad side.

This approach works for both Launchpad.get_token_and_login and Launchpad.login_with.

The values that you can pass here aren't documented, and should probably be constants in launchpadlib, rather than hardcoded in every application, but for now you can use:

• READ_PUBLIC
• READ_PRIVATE
• WRITE_PUBLIC
• WRITE_PRIVATE

Dear Mr Neary, thanks for your thought provoking post, I think it is a problem we need to be aware of as Free Software matures.

Firstly though I would like to say that the apparent ageism present in your argument isn't helpful to your point. Your comments appear to diminish the contributions of a whole generation of people. In addition, we shouldn't just be concerned with attracting young people to contribute, the same changes will have likely reduced the chances that people of all ages will get involved.

Aside from that though there is much to discuss. You talk about the changes in Free Software since you got involved, and it mirrors my observations. While these changes may have forced fewer people to learn all the details of how the system works, they have certainly allowed more people to use the software, bringing many different skills to the party with them.

I would contend that often the experience for those looking to do the compilation that you rate as important has parallels to the experience of just using the software you present from a few years ago. If we can change that experience as much as we have the installation and first use experience then we will empower more people to take part in those activities.

It is instructive then to look at how the changes came about to see if there are any pointers for us. I think there are two causes of the change that are of interest to this discussion.

Firstly, one change has been an increased focus on user experience. Designing and building software that serves the users needs has made it much more palatable for people, and reduced the investment that people have to make before using it. In the same way I think we should focus on developer experience, making it more pleasant to perform some of the tasks needed to be a hobbyist. Yes, this means hiding some of the complexity to start with, but that doesn't mean that it can't be delved in to later. Progressive exposure will help people to learn by not requiring them to master the art before being able to do anything.

Secondly, there has been a push to make informed decisions on behalf of the user when providing them with the initial experience. You no longer get a base system after installation, upon which you are expected to select from the thousands of packages to build your perfect environment. Neither are you led to download multiple CDs that contain the entire contents of a distribution, much of which is installed by default. Instead you are given an environment that is already equipped to do common tasks, where each task is covered by an application that has been selected by experts on your behalf.

We should do something similar with developer tools, making opinionated decisions for the new developer, and allowing them to change things as they learn, similar to the way in which you are still free to choose from the thousands of packages in the distribution repositories. Doing this makes documentation easier to write, allows for knowledge sharing, and reduces the chances of paralysis of choice.

There are obviously difficulties with this given that often the choice of tool that one person makes on a project dicatates or heavily influences the choice other people have to make. If you choose autotools for your projects then I can't build it with CMake. Our development tools are important to us as they shape the environment in which we work, so there are strong opinions, but perhaps consistency could become more of a priority. There are also things we can do with libraries, format specifications and wrappers to allow choice while still providing a good experience for the fledgling developer.

Obviously as we are talking about free software the code will always be available, but that isn't enough in my mind. It needs to be easier to go from code to something you can install and remove, allowing you to dig deeper once you have achieved that.

I believe that our effort around things like https://dev.launchpad.net/BuildBranchToArchive will go some way to helping with this.

The deadline for students to submit their applications to Google for Summer of Code is imminent.

If you were waiting for the last minute to submit, that is now!

If you are mentor and have the perfect student you have been working with, check with them that they have submitted the application to Google, otherwise you will be stuck.

Next week we'll start to process the huge number of applications that we have for Ubuntu.

If you don't want to read this article, then just steer clear of python-multiprocessing, threads and glib in the same application. Let me explain why.

There's a rather famous bug in Gwibber in Ubuntu Lucid, where a gwibber-service process will start taking 100% of the CPU time of one of your cores if it can. While looking in to why this bug happened I learnt a lot about how multiprocessing and GLib work, and wanted to record some of this so that others may avoid the bear traps.

Python's multiprocessing module is a nice module to allow you to easily run some code in a subprocess, to get around the restriction of the GIL for example. It makes it really easy to run a particular function in a subprocess, which is a step up from what you had to do before it existed. However, when using it you should be aware how the way it works can interact with the rest of your app, because there are some possible nasties lurking there.

GLib is a set of building blocks for apps, most notably used by GTK+. It provides an object system, a mainloop and lots more besides. What we are most interested here is the mainloop, signals, and thread integration that it provides.

Let's start the explanation by looking at how multiprocessing does its thing. When you start a subprocess using multiprocessing.Process, or something that uses it, it causes a fork(2), which starts a new process with a copy of the programs current memory, with some exceptions. This is really nice for multiprocessing, as you can just run any code from that program in the subprocess and pass the result back without too much difficulty.

The problems occur because there isn't an exec(3) to accompany the fork(2). This is what makes multiprocessing so easy to use, but doesn't insert a clean process boundary between the processes. Most notably for this example, it means the child inherits the file descriptors of the parent (critically even those marked FD_CLOEXEC).

The other piece to this puzzle is how the GLib mainloop communicates between threads. It requires some mechanism where one thread can alert another that something of interest happened. To do this when you tell GLib that you will be using threads in your app by calling g_thread_init (gobject.threads_init() in Python) then it will create a pipe for use by glib to alert other threads. It also creates a watcher thread that polls one end of this pipe so that it can act when a thread wishes to pass something on to the mainloop.

The final part of the puzzle is what your app does in a subprocess with mutliprocessing. If you purely do something such as number crunching then you won't have any issues. If however you use some glib functions that will cause the child to communicate with the mainloop then you will see problems.

As the child inherits the file descriptors of the parent it will use the same pipe for communication. Therefore if a function in the child writes to this pipe then it can put the parent in to a confused state. What happens in gwibber is that it uses some gnome-keyring functions and that puts the parent in to a state where the watcher thread created by g_thread_init busy-polls on the pipe, taking up as much CPU time as it can get from one core.

In summary, you will see issues if you use python-multiprocessing from a thread and use some glib functions in the children.

There are some ways to fix this, but no silver bullet:

• Don't use threads, just use multiprocessing. However, you can't communicate with glib signals between subprocesses, and there's no equivalent built in to multiprocessing.
• Don't use glib functions from the children.
• Don't use multiprocessing to run the children, use exec(3) a script that does what you want, but this isn't as flexible or as convenient.

It may be possible to use the support for different GMainContexts for different threads to work around this, but:

• You can't access this from Python, and
• I'm not sure that every library you use will correctly implement it, and so you may still get issues.

Note that none of the parties here are doing anything particularly wrong, it's a bad interaction caused by some decisions that are known to cause issues with concurrency. I also think there are issues when using DBus from multiprocessing children, but I haven't thoroughly investigated that. I'm not entirely sure why the multiprocessing child seems to have to be run from a non-main thread in the parent to trigger this, any insight would be welcome. You can find a small script to reproduce the problem here.

Or, to put it another way, global state bad for concurrency.