debian tips/ tips
.........................

Instead of making blog posts about how to do something, I have decided to use the wiki.

Most of the tips will be about Debian, but there will also be more general things.

Tips

RSS Add a new post titled:
Compiling a Debian package with debug symbols

Today I was experiencing a SIGSEGV while using mutt, and running in gdb was useless as there are were no debug symbols in the binary.

This is standard practice for Debian packages, so if you want to provide a backtrace for a bug report you will need to recompile the package. Here I demonstrate a quick and dirty way to do that. This method is not recommended if you want to make changes to the code as well.

So first check if the package has a debug package for it. More and more libraries are providing these packages to make debugging easier. If you are experiencing a problem with a library just installing the associated -dbg package should provide all the debugging symbols. (Marc Haber wrote this script to automatically install any -dbg packages of dependencies if they exist. Requires debfoster and grep-dctrl packages. This should help fill in gaps in the backtrace if possible).

If there is no debug package then you will have to recompile. First get the necessary dependencies (I will use mutt as my example).

     apt-get build-dep mutt

Then get the source of the package

     apt-get source mutt

Then enter the source directory

     cd mutt-1.5.13/

Most (all?) packages that use C code should look at the environment variable DEB_BUILD_OPTIONS to see whether to include debug symbols. The other use is to turn off optimisations. This can be useful, so try it, but it is possible that turning off optimistations solves the problem, in which case you should not use noopt.

     export DEB_BUILD_OPTIONS=nostrip,noopt

Now recompile the package (requires dpkg-dev and fakeroot),

     dpkg-buildpackage -rfakeroot -uc -us

This should complete normally, and leav a .deb in the parent directory, so then (as root):

     dpkg -i ../mutt_1.5.13-1_i386.deb

This will install the new package. Now try running under gdb again, and the debug symbols should be present.

Posted Tue Aug 29 14:30:52 2006
Hacking on an open source project using bzr

I have recently been providing patches to a couple of open source projects that use svn to keep their work versioned.

The difficulty for me is that I will create a patch, and there is no guarantee it will be accepted. However I want to create multiple patches and be able to provide them separately.

A good way to solve this is by using branches. They allow changes to be developed in parallel, without affecting one another. The problem is that with svn you need commit access to the repository to be able to branch easily.

One workaround would be to have several checkouts, and create a patch in each, but this is quite expensive, requiring a full checkout for each branch. This takes up both disk space and network bandwidth.

I have recently started using bzr to manage my work, and one of it's features is that branching is cheap and can be done locally. So I would like to use bzr to manage my work in svn.

To do this I used the following method.

      # create a directory to hold all of the work
      mkdir project
      cd project
      # Create a bzr repository (reduces disk usage later)
      bzr init-repo --trees .
      # Get the upstream source
      svn co http://wherever/some/project upstream

This creates a directory containing the upstream source. The idea is to leave this as untouched as possible. Try not to make any changes to this code, and don't build from there, as this might confuse you later.

Now create some bzr branches to manage your work

      # Create a bzr branch for the upstream code.
      cd upstream
      bzr init
      bzr add .
      bzr ci -m'Import of project'
      # Create the master branch
      cd ../
      bzr branch upstream main

In the main branch you should make any local modifications that are necessary for your system, but that wont be sent upstream. 

      cd main
      # make local changes
      bzr ci -m'Add local changes'

Now whenever you want to work on a patch (say add a feature), create a new branch for it. This will keep the changes independent.

      # Create the working branch
      cd ..
      bzr branch main feature
      cd feature
      # Implement the feature
      bzr ci -m'Added feature'

Now you should make sure the code is up to date with upstream to provide a clean patch (you can do this often to reduce the amount of conflict resolution that you have to do)

      cd ../upstream
      svn update

This will pull upstream's changes. There should not be any conflicts as you didn't change any files. You must watch out for any added files though (A lines of the update output). bzr unknowns can help you here. If there were any added files you must add them to bzr.

      bzr add file
      # or bzr add $(bzr unknowns)
      bzr ci -m'Merge upstream svn revision 1234'

This means that your upstream branch is up to date. You now must propogate these to your feature branch. You should go through your main branch first though to maintain your local changes.

      cd ../main
      # Pull in the upstream changes
      bzr merge
      # If you have any conflicts then edit the files and
      bzr resolved file
      # When all is well (bzr status and bzr diff can tell 
      # you what is going on)
      bzr ci -m'Merge upstream svn revision 1234'

You can now do the same for your feature branch.

      cd ../feature
      bzr merge
      # bzr resolved if needed
      bzr ci -m'Merge upstream svn revision 1234'

This might seem like a lot of work, but it can be done quickly, especially if development is quiet. However I find the benifits of this to outweigh this work. It is also said the bzr has superior merging ability to svn, so perhaps doing this will reduce the amount of manual conflict resolution you will have to do.

So now you have a feature implemented in a branch and you want to send a patch upstream. This is very easy. Due to the separation of feature branches you will always have clean diffs, and branching from a bzr branch containing your local modifications means that your diffs will not contain your changes. So to create these diffs do

      bzr diff -r branch:../main > ../feature.diff

You can then review the diff and send it upstream.

Now you can leave your feature branch alone, and create another branch if you want to do more work. If upstream asks for more work, you can just go back to the branch and make the necessary changes, and then send another diff.

Now, if the patch gets accepted then your changes will come down in the next update from svn, and then will propogate to all of your branches. You will not get any conflicts, even if upstream makes some changes to your code.

If your patch is not accepted, but you would like to keep it locally, you can do this easily. Just merge your branch back in to your main branch, which will then propogate to all of your branches.

      cd ../main
      bzr merge ../feature
      # If upstream partially included your patch you may have to
      # deal with conflicts here.
      bzr ci -m'Add feature'

Now when you merge from any of your feature branches they will get these changes, and so their diffs will not contain it, keeping the patches separate.

If you want to split up a feature implementation, so that you can work with upstream to get it included in parts you can create a feature branch that depends on another one. 

     bzr branch main small.change
     bzr branch small.change large.change

Now you propogate changes to large.change through small.change. You can then generate 3 patches.

     # Create a diff to implement small.change
     cd small.change
     bzr diff -r branch:../main
     # Create a diff to implement large.change
     cd large.change
     bzr diff -r branch:../main
     # Create a diff to implement large.change that is dependent on 
     # small.change. Useful if you would like to discuss the second
     # part of the feature on it's own.
     cd large.change
     bzr diff -r branch:../small.change

So you can see here how bzr can give some flexibilty when hacking on a project, no matter what vcs they use.

Posted Tue Aug 29 01:06:09 2006
Using the Debian Bug Tracking Sytem

Debian makes heavy use of it's bug-tracking system, (BTS) to coordinate work, and for developers to know that a problem needs fixing.

This article is aimed at those who haven't seen the BTS before, or know who have used it a little, but would like to know about how to manipulate bugs.

Viewing the bugs

If you want to see what bugs have been reported to the BTS this is easy to do using the web interface. This is accessible at http://www.debian.org/Bugs/. If you go to this page you can see a form for searching the bug reports. The system is package based, so you can enter the name of a package and search for all bugs related to that package.

For example, enter apt in to the search box, and make sure the package option is selected just above it. Then click the find button. This will then show the page with all of apt's bugs on it. You can click on any of the bug titles to see more information on the bug.

There are also shortcuts that you can use e.g. http://bugs.debian.org/apt for all apt's bugs and http://bugs.debian.org/383487 for bug #383487.

States, Severities and Tags.

You can see that the bugs are divided up in to sections. These sections are based on the states that the bugs are in, and the severities that were assigned to them. This helps the maintainer of the package to see what needs doing, and to prioritise their work.

The states of the bugs are:

  • Open - The bug is not fixed.
  • Forwarded - The bug has been reported to the upstream maintainer as it requires work on their part.
  • Resolved - The bug has been fixed.

The severities are:

  • wishlist - The submitter would like some extra feature to be added to the package.
  • minor - There is something wrong with the package that doesn't really matter, but should be fixed anyway. This could be for a typo in the package description, for example.
  • normal - This is where most bugs belong. There is a problem with the package, but it doesn't break the whole thing, or cause too much trouble.
  • important - This is a bug that has a big impact on the usability of the package.

These are the severities that you will normally use. There are three more severities, and they have special meanings. They also get special treatment when they are not being fixed. You should only really report a bug at one of these severities if you understand what they mean. Don't worry though, if you report a bug that deserves to get one of these severities it will be upgraded.

  • serious - A violation of the Debian Policy. The rules set out in the Policy aim to make sure that a package will not break a system it is installed on, as well as trying to make packages consistent. Have a look at the policy document if you are interested.
  • grave - A bug that renders the package unusable in all or nearly all situations. This would cover things like uninstallable packages, segmentation faults on startup, missing files. It should only be used when it breaks the software for most users though, it can be tricky to know when this is the case, but experience helps. This severity also covers some security holes.
  • critical - Installing the package breaks other unrelated software on the system, or indeed the while system. This also covers programs that cause data loss and the rest of the security bugs.

Tags are used by developers to keep track of what is happening to the bugs. A full list is available at http://www.debian.org/Bugs/Developer#tags. Some of the most common ones you will see are:

  • patch - someone has supplied a patch that fixes the problem. This does not apply for workarounds.
  • wontfix - the maintainer or upstream developer has said they will not fix the bug for some reason. There will usually be an explanation in the bug report.
  • moreinfo - the submitter did not provide enough information to the maintainer. If you are hit by a bug with a moreinfo tag then you should follow up (see below) with the information the maintainer wants.
  • unreproducible - the maintainer can't reproduce the bug on his system. This is similar to the moreinfo tag, in that if you can reproduce you should help the maintainer by sending more information to the bug report.
  • help - the maintainer can't fix the bug by themselves (for example it only occurs on an architecture they do not have access to), and would like other people to help come up with a solution.
  • pending - The problem is solved and an upload should happen soon to fix it.

Some of the other tags are either obvious (e.g. security) or are used more for tracking things across the distribution (e.g. ipv6).

Reporting a bug

If you find a bug that you would like to see fixed then you have to use the BTS to make the maintainer aware, and work with them to fix it. So to start off you need to report the bug.

Before you report it though there are a few things you need to do.

  1. Work out what package the bug comes from.

    • Sometimes this is very easy, and you will instantly know the package.
    • If you only know that it occurs because of a certain file or program, you can use dpkg -S with the filename to find out the package that it belongs to.
    • If you can not work it out, then you will have to make a best guess, and you can report it to that package. You should note in the report that you are only guessing, and the maintainer of that package will try and help you work it out.

  2. Find if the bug has already been reported.

    • Use the package search to see the bugs for the package and look for your bug in the list. If it is there then do not report a new bug, you can add more information to it, see below.
    • You may need to check a few associated packages to make sure that you didn't get the responsible package slightly wrong.

  3. Gather as much information as possible. Ideally the report should contain:

    • The steps to reproduce, i.e. the complete sequence of commands used to trigger the problem.
    • Any relevant configuration files, or the options from the program.
    • Any relevant log files. You can attach them to your report (see below).
    • Any other information you think might help.
    • Any workarounds you may have found, or anything interesting you have noticed about the bug.
    • If you know how you can include a backtrace from gdb, or the output of strace. These can be invaluable in some cases.

Now you have all this information you are ready to report a bug. The preferred way to do this is to use the report bug program, so 

    # apt-get install reportbug

And the simply run

    $ reportbug

There are many options you can pass to reportbug (man reportbug will help you there), but reportbug will walk you through the steps needed to send the report.

So first you are asked for the name of the package that you wish to report a bug against.

    Please enter the name of the package in which you have found a problem, or type
    'other' to report a more general problem.
    >

Once you have entered the name of the package you will be shown the list of bug reports already filed against the package. Make sure your bug is not listed and continue. If it is listed then you can send more information (see below).

When you have checked the bug reports, you will be prompted for the subject of the bug report. 

     Please briefly describe your problem (you can elaborate in a moment; an empty
     response will stop reportbug). This should be a concise summary of what is
     wrong with the package, for example, "fails to send email" or "does not start
     with -q option specified."
     >

Some people like to start this with the name of the package (though it is not essential), e.g.

     apt: does not download correct version of dpkg

You will then be prompted for the severity of your report. Select a number from the list that you think best applies. If you are unsure just hit Enter and you will get the default normal and the maintainer can change it if need be.

reportbug may then ask you if it can include some of your configuration files in the report. You should answer yes, unless the files contain sensitive information (e.g. passwords).

An editor will then be spawned for you to write you bug report. Fill in all the information that you found above, then save the file and close the editor.

You will then be shown the final menu

    Report will be sent to "Debian Bug Tracking System" <submit@bugs.debian.org>
    Submit this report on apt (e to edit) [y|n|a|c|E|i|l|m|p|q|?]?

There are many options here. Type ? to get a list. The interesting ones are

  • e - edit the report again
  • a - attach another file (e.g. log file, gdb backtrace)
  • y - send the report via email.

When you have finished choose y to send it (or if you require you can use the other options to send it your self). That is then it, you have submitted your first report.

You will receive a copy of the report straight away, and the confirmation from the BTS within the hour.

It is possible to report bugs via email as well if you want. Instructions are given at http://www.debian.org/Bugs/Reporting. Not using reportbug has the disadvantage that you must do more by yourself, as you don't have reportbug's checks, or automatic inclusion of some information.

Following up to bugs

If you wish to send more information to a bug report there are two ways to do it.

  • Use reportbug again.
  • Use email.

I wont describe the reportbug way in too much detail, as it provides you with help, and is similar to the procedure above.

To use report bug to send a followup to a bug start reportbug and enter the package name. Then when the list of bugs is displayed press y, which will prompt for the bug number. You can then proceed as before. If you see the bug in the list it shows you enter the number from the left hand side, and reportbug will show you the bug report. If you enter x at the prompt then you will provide a follow-up to that bug.

To send information by email you just need to send an email to bug-number@bugs.debian.org, e.g. 22334@bugs.debian.org. The content of the email will then become part of the bug report.

Whichever method you use it is useful if you don't simply send an email like

    Subject: Bug #12345

    I am seeing this bug as well, can I help fixing it?

as, while it is good to offer your help, it requires anyone who receives that email (and the emails do go to a few places), to go and look up the bug. It would be better to send something like:

    Subject: Re: Bug#13245: apt doesn't download correct version of dpkg

    I am also seeing this bug. I have tried giving the version on the
    command line to apt, but it still downloads version 1.2.3 of dpkg.

    Is there anything else I can do to help?

It says the same thing, but provides much more context to those who receive the email, so they can simply reply to your email if they want, without having to go and look up the report.

Manipulating bugs

You can manipulate bugs using the email interface. This involves sending and email to control@bugs.debian.org with a certain format. The message is then processed, and the instructions in it are executed.

The full documentation on how to do this is at http://www.debian.org/Bugs/server-control and is good for a reference.

A sample email might look like

     #only consider bugs that are assigned to apt
     package apt
     #change the severity of a bug
     severity 12345 normal
     #reassign a bug
     reassign 12345 dpkg
     #stop processing commands
     stop

The comments are optional and only used here to try and illustrate what is happening. The stop command is useful so that the server doesn't process any more of the email and get confused.

Closing and reopening bugs

If you wish to close a bug report (normally only the submitter or the maintainer do this), you can send an email to bugnumber-done@bugs.debian.org and include a Version: pseudo-header that tells the system in what version the bug was fixed. e.g.

     To: 12345-done@bugs.debian.org

     Version: 1.2.3

     This is fixed in the latest upload. Thanks for your patience.

The version is used so that it is possible to track when a bug was opened and closed, so that it is known which versions a given bug applies to.

If you discover a bug that has been closed, but re-occurs in a later version, you can send an email with a found command to control@bugs.debian.org (see above), e.g.

     found 1234 1.2.4
     stop

     I have found this in version 1.2.4, it occurs in the same conditions
     as before.

You should only do this if it is the same bug though. Submit a new one if you find a new bug.

Other tricks

There are many other things you can do with the BTS. I will let you figure out how to use them. The documentation at http://www.debian.org/Bugs/ covers most of them.

  • bts tool - The devscripts package contains a tool called bts. This is a quick way to send commands to the BTS. e.g. bts severity 12345 normal
  • subscriptions - It is possible to subscribe to a bug report, so that you receive all information sent to the report, and can find out when it gets closed.
  • usertags - allow you to tag bugs for yourself. Other people looking at reports wont normally see them. These are most often used by maintainers who look after a lot of bugs so they can be categorised or prioritised.
Posted Sat Aug 26 14:53:00 2006
Installing IkiWiki from source

ikiwiki is still young, and so you may need to get hacking to get it to work.

The ikiwiki setup page is a good start.

Make sure you have all of the perl modules installed to save you some cryptic error messages later.

It is a good idea to use the ikiwiki.setup file, but name it something different and do not place it in svn (or whatever). Also do not put it under the source directory of your wiki, as it will then be viewable by anybody.

I was a little unsure with the instructions for using cgi, but you simply uncomment the cgi section in the example file, and then run ikiwiki --setup, which will generate the script for you. It really is that simple.

I had to hack a few things to get ikiwiki to install and work. Firstly with perl 5.8.3 there is a bug in taint mode that stops it compiling. This appears to be fixed in 5.8.8. It can be worked around by making the "pagespec_match" of IkiWiki.pm function read:

    sub pagespec_match ($$) { #{{{
      my $page=shift;
      my $spec=shift;

      my $pagespec = pagespec_translate($spec);
      my $newpagespec;
      local($1);
      if ($pagespec =~ /(.*)/) {
        $newpagespec = $1;
      } else {
        die "oh";
      }

      return eval $newpagespec;
    } #}}}

I also had to change the two occurences of "Markdown" to "Text::Markdown" in IkiWiki/Plugin/mdwn.pm

To use the search plugin you may need to edit line 105 of IkiWiki/Plugin/search.pm to point to where estseek.cgi is on your system.

To get RecentChanges link to work i had to change (line 38 of IkiWiki/Rcs/svn.pm) from

    my $info=`LANG=C svn info $file`;

to 

    my $info=`svn info $file`;

I'm not sure why though.

I also had errors in my error_log saying "could not access /home/user/.subversion" or similar. This was fixed by using 

    svn $svn_limit --xml -v log '$svn_url' --config-dir /tmp

on line 145 of IkiWiki/Rcs/svn.pm.

Posted Fri Aug 25 17:16:48 2006