Thu, 30 Sep 2010
Couchapp Walkthrough: Part 2: The couchapp tool
Today I would like to talk about the couchapp tool. This is something that you can use when working on couchapps, and provides a way to quickly iterate your design.
However, rather confusingly, the couchapp tool isn't actually required for couchapps. If you get a design document with HTML attachments in to your database then you have a couchapp.
Why would you want to use such a tool then? Firstly because it will generate a skeleton couchapp for you, so that you don't have to remember how to organise it, and if it is your first couchapp it's good to start from something working.
More importantly though, the couchapp tool is useful as you develop as it allows you to edit the parts of your app in their native format. This means that if you are writing a HTML snippet you can just put some HTML in a file, rather than having to write it as part of a deeply nested JSON structure and deal with the errors that you would make if you did that. Also it means that you can use things like syntax highlighting in your preferred text editor without any special magic.
How it works
At it's core couchapp is a rather simple tool. It walks a filesystem tree and assembles the things that it finds there in to a JSON document.
For instance, if it finds a directory at the root of the tree called _attachments it puts the content of each file there in to a list which it puts in to the dict with an "_attachments" key, Which is one of the ways that couchdb accepts document attachments. Therefore if you have an _attachments/index.html file in your tree it will be attached to your design document when the JSON structure is sent to couchdb.
This continues across the tree, so the contents of the "views" directory will become the "views" key of the documents, which is how you do map/reduce queries on the database.
couchapp has various conventions for dealing with files. For instance if it finds a ".js" file it treats it as a javascript snippet which will be encoded in to a string in the resulting document. ".html" files outside of the "_attachments" directory will also be encoded as strings. If it finds a ".json" file then it treats it as literal JSON that will be embebedded.
This way it builds up the JSON structure that a couchapp expects, and will send it to the couchdb of your choice when it is done.
In addition to this functionality the tool can also generate you a skeleton app, and also add new pieces to your app, such as new views.
Getting It
couchapp is a python tool, so you can install it using pip or similar. However, Ubuntu users can install it from a PPA (yay for daily builds with recipes!).
Using It
To use it run
couchapp generate myapp
which will create you a new skeleton in myapp.
cd myapp ls
You will see for instance the _attachments and views directories, and an _attachments/index.html.
To get your app in to couchdb you can run
couchapp push http://localhost:5984/mydb
and it will tell you the URL to visit to see your new app.
If you want to use desktopcouch you can run
couchapp push desktopcouch://
though I think it has a bug that it prints the wrong URLs when pushing to desktopcouch.
Once you have looked at the HTML generated by your app you should look at the design document that couchapp created. Go to
http://localhost:5984/_utils
or
xdg-open ~/.local/share/desktop-couch/couchdb.html
if you are using desktopcouch.
Click to the mydb database and you will see a document called _design/myapp. Click on this and you will see the content of the design document; you are looking at a couchapp in its raw form.
If you compare what is in that design document with what is in the myapp directory that the tool created you should start to see how it generates it from the filesystem.
Now try making a change on the filesystem, for instance edit _attachments/index.html and put your name somewhere in the body. Then push again, running
couchapp push http://localhost:5984/mydb
and refresh the page in your browser and you should see the change. (Just click on index.html in the design document to get back to viewing your app from there).
I will go in to more detail about the content of the couchapp that was generated for you in another post.
See the previous installment.
Wed, 29 Sep 2010
Couchapp Walkthrough - Part 1
Couchapps are a particular way of using couchdb that allow you to serve web applications directly from the database. These applications generate HTML and javascript to present data from couchdb to the user, and then update the database and the UI based on their actions.
Of course there are plenty of frameworks out there that do this sort of thing, and more and more of them are adding couchdb support. What makes couchapps particularly interesting are two things. Firstly, the ease with which they can be developed and deployed. As they are served directly from couchdb they require little infrastructure, and the couchapp tool allows for a rapid iteration. In addition, the conveniences that are provided mean that simple things can be done very quickly with little code.
The other thing that makes couchapps attractive is that they live inside the database. This means that the code lives alongside the data, and will travel with it as it is replicated. This means that you can easily have an app that you have fast, local access to on your desktop, while at the same time replicating to a server so that you can access the same data from your phone while you are out. Again, this doesn't require couchapps, and they won't be suitable for all needs, but they are certainly an interesting idea.
You can read more about couchapps at http://couchapp.org.
Intrigued by couchapps I set out to play with them over a weekend. Unfortunately the documentation is rather lacking currently, so I wouldn't recommend experimenting yourself if you are not happy digging around for answers, and sometimes not finding them outside the code. In order to go a little way to rectifying this, I intend to write a few posts about the things I wish I had known when I started out. I found everything to be a little strange at first, and it wasn't even clear where the entry point of a couchapp was for instance. Hopefully these posts will be found using google by others who are struggling in a similar way.
Architecture
Firstly something about the pieces that make up a couchapp (or at least those that the tool and documentation recommend,) and the way that they all fit together.
At the core is the couchdb database itself. It is a collection of "documents", each of which can have attachments. Some of these documents are known as "design documents," and they start with a prefix of "_design." Design documents can have "view" functions, and various other special fields that can be used to query or manipulate other documents.
A couchapp is a design document with an attachment, usually called index.html. These attachments are served directly by couchdb and can be accessed at a known URL. You can put anything you like in that html file, and you could just have a static page if you wanted. Usually however though it is is an HTML page that uses javascript in order to display the results of queries on the database. The user will then access the attachment on the design document, and will interact with the resulting page.
In theory you can do anything you like in that page, but it is usual to make use of standard tools in order to query the database and provide information and opportunity for interaction to the user.
The first standard tool is jQuery, with a couple of plugins for working with couchdb and couchapps specifically. These allow for querying views in the database and acting on the results, retrieving and updating documents, and plenty more.
In addition the couchapp tool sets you up with another jQuery plugin called "evently", which is a way to structure interactions with jQuery, and change the page based on various events. I will go in to more detail about how evently works in a later post.
In addition to all the client-side tools for interacting with the database, it is also possible to make use of couchdb features such as shows, lists, update handlers validation functions in order to move some of the processing server-side. This is useful for various reasons, including being more accessible, allowing search engines to index the content, and not having to trust the client not to take malicious actions.
The two approaches can be combined, and you can prototype with the client-side tools, and then move some of the work to the server-side facilities later.
Stay tuned for more on how a simple couchapp generates content based on what is in the db.
Tue, 28 Sep 2010
Directly logging in a user in Django tests
The examples for Django testing point you towards hardcoding a username and password for a user to impersonate in tests, and the API of the test client encourages this too.
However, Django has a nice pluggable authentication system that means you can easily use something such as OpenID instead of passwords.
Putting the passwords in your tests ties you to having the password support enabled, and while you could do this for just the tests, it's completely out of the scope of most tests (I'm not talking about any tests for the actual login process here.)
When I saw this while reviewing code recently I worked with Zygmunt to write a Client subclass that didn't have this restriction. With this subclass you can just choose a User object, and have that client login as that user, without them having to have a password at all. Doing this decoples your tests from the implementation of the authentication system, and makes them target the code you want to test more precisely.
Here's the code:
from django.conf import settings
from django.contrib.auth import login
from django.http import HttpRequest
from django.test.client import Client
class TestClient(Client):
def login_user(self, user):
"""
Login as specified user, does not depend on auth backend (hopefully)
This is based on Client.login() with a small hack that does not
require the call to authenticate()
"""
if not 'django.contrib.sessions' in settings.INSTALLED_APPS:
raise AssertionError("Unable to login without django.contrib.sessions in INSTALLED_APPS")
user.backend = "%s.%s" % ("django.contrib.auth.backends",
"ModelBackend")
engine = import_module(settings.SESSION_ENGINE)
# Create a fake request to store login details.
request = HttpRequest()
if self.session:
request.session = self.session
else:
request.session = engine.SessionStore()
login(request, user)
# Set the cookie to represent the session.
session_cookie = settings.SESSION_COOKIE_NAME
self.cookies[session_cookie] = request.session.session_key
cookie_data = {
'max-age': None,
'path': '/',
'domain': settings.SESSION_COOKIE_DOMAIN,
'secure': settings.SESSION_COOKIE_SECURE or None,
'expires': None,
}
self.cookies[session_cookie].update(cookie_data)
# Save the session values.
request.session.save()
Then you can use it in your tests like this:
from django.contrib.auth.models import User client = TestClient() user = User(username="eve") user.save() client.login_user(user)
Then any requests you make with that client will be authenticated as the user that was created.
Ticket submitted with Django to have this available for everyone in future.
Tue, 14 Sep 2010
What I work on
I'm keen to try and write more about the things that I work on as part of my job at Canonical. In order to get started I wanted to write a summary of some of the things that I have done, as well as a little about what I am working on now.
Ubuntu Distributed Development
This isn't the catchiest name for a project ever, and has an unfortunate collision with an Debian project, also shortened to "UDD." However, the aim is for this title to become a thing of the past, and this just to be the way things are done.
This effort is firstly about getting Ubuntu to use Bazaar, and a suite of associated tools, to get the packaging work done. There are multiple reasons for this.
First, and most simply, is to give developers the power of version control as they are working on Ubuntu packages. This is useful for both the large things and the small. For instance I sometimes appreciate being able to walk through the history of a package, comparing diffs here and files there when debugging a complex problem. Sometimes though it's just being able to "bzr revert" a file, rather than having to unpack the source again somewhere else, extracting the file and copying it over the top.
There are higher purposes with the work too. The goal is to link the packaging with the upstream code at the version control level, so that one flows in to the other. This has practical uses, such as being able to follow changes as they flow upstream and back down again, or better merging of new upstream versions. I believe it has some other benefits too, such as being able to see the packages more clearly as what they are, a branch of upstream. We won't just talk about them being that, but they truly will be.
Some of you will be thinking "that's all well and good, but <project> uses git," and you are absolutely right. Throughout this work we have had two principles in mind, to work with multiple systems outside of Ubuntu, and to provide a consistent interface within Ubuntu.
Due to the way that Ubuntu works an Ubuntu developer could be working on any package next. I would really like it if the basics of working with that package were the same regardless of what it was. We have a lot of work to do on the packaging level to get there, but this project gets this consistency on the version control level.
We can't get everyone outside of Ubuntu to follow us in this though. We have to work with the system that upstream uses, and also to work with Debian in the middle. This means that we have to design systems that can interface between the two, so we rely a lot on Launchpad's bzr code imports. We also want to interface at the other end as well, at "push" time. This means that if an Ubuntu developer produces a patch that they want to send upstream they can do that without having to reach for a possibly different VCS.
Thanks mainly to the work of Jelmer Vernooij we are doing fairly well at being able to produce patches in the format appropriate for the upstream VCS, but we still have a way to go to close the loop. The difficultly here is more around the hundreds of ways that projects like to have patches submitted, whether it is a mailing list or a bug tracker, or in some other form. At this stage I'd like to provide the building blocks that developers can put together as appropriate for that project.
Daily package builds
Relatedly, but with slightly different aims, I have been working on a project in conjunction with the Launchpad developers to allow people to have daily builds of their projects as packages.
Currently there is too often a gap between using packaged versions of a project, and running the tip of that project daily. I believe that there are lots of people that would like to follow the development of their favourite projects closely, but either don't feel comfortable building from the VCS, or don't want to go through the hassle.
Packages are of course a great way to distribute pre-compiled software, so it was natural to want to provide builds in this format, but I'm not aware of many projects doing that, aside from those which fta provides builds for. Now that Launchpad provides PPAs and code imports, and the previous project provides imports of the packaging of all Debian and Ubuntu packages in to bzr, all the pieces are there in order to allow you to produce packages of a project automatically every day.
This is currently available in beta in Launchpad, so you can go and try it out, though there are a few known problems that we are working on until it will be as pleasant as we want.
This has the potential to do great things for projects if used correctly. It can increase the number of people testing fresh code and giving feedback by orders of magnitude. Also, just building the packages acts as a kind of continuous integration, and can provide early warning of problems that will affect the packaging of the project. Also, they provide an easy way for people to test the latest code if a bug is believed to be fixed.
Obviously there are some dangers associated with automatic builds, but if they are used by people who know what they are doing then it can help to close the loop between users and developers.
There are also many more things that can be done with this feature by people with imagination, so I'm excited to see what directions people will take it in.
Fixing things
Aside from these projects, I was also given some time to work on Ubuntu itself, but without long-term projects to ship. That meant that I was able to fix things that were standing in my way, either in the way of the above projects, or just hampering my use of Ubuntu, or fix important bugs in the release.
In addition I took on smaller projects, such as getting kerneloops enabled by default in Ubuntu. While doing this I realised that the user experience of that tool could be improved a lot for Ubuntu users, as well as allowing us to report the problems caught by the tool as bugs in Launchpad if we wished.
I really enjoyed having this flexibility, as it allowed me to learn about many areas of the Ubuntu system, and beyond, and also played to my strengths of being able to quickly dive in to a new codebase and diagnose problems.
I think that in my own small way, each of these helped to improve Ubuntu releases, and in turn the projects that Ubuntu is built from.
Sponsoring
While I'm sorry to say that other demands have pulled my code review time in to other projects, I did used to spend a lot of time reviewing and sponsoring changes in to Ubuntu.
I highlight this mainly as another chance to emphasise how important I think code review is, especially when it is review of code from people new to the project. It improves code quality, but is also a great opportunity for mentoring, encouraging good habits, and helping new developers join the project. I hope that my efforts in this are had a few of these characteristics and helped increase the number of free software developers. Oh how I wish there were more time to continue doing this.
Linaro
I've now been started working on the Linaro project, specifically in the Infrastructure team, working on tools and Infrastructure for Linaro developers and beyond. I'm not one to be all talk and no action, so I won't talk to much about what I am working on, but I would like to talk about why it is important.
Firstly I think that Linaro is an important project for Free Software, as it has the opportunity to lead to more devices being sold that are built on or entirely free software, some in areas that have historically been home to players that have not been good open source citizens.
Also, I think tools are an important area to work on, not just in Linaro. They pervade the development experience, and can be a huge pain to work with. It's important that we have great tools for developing free software so as not to put people off. Developers, volunteers and paid, aren't going to carry on too long with tools that cause them more problems than they are worth, and not all are going to persist because they value Free Software over their own enjoyment of what they do.