Tue, 09 Jun 2009

Merging file descriptor output

I have a problem that I believe will be easy for someone with a bit of UNIX coding knowledge to solve, so I appeal to those that can to help.

I'm trying to write a DBus service that will spawn a command, and provide the output to the user. The service runs on the system bus as root, and so it is a form of privilege escalation. However, the command may be long running, and produce a lot of output as it works, so I want to allow the calling process to get this output before the command completes.

My current approach uses gobject.spawn_async and so gets file descriptors back, one for stdout and one for stderr. I currently have a thread that uses select to wait for output, and then uses DBus signals to allow the client to access it. This works great, except that stdout and stderr can become interleaved in the middle of lines.

I believe that I can't just wait for full lines before signalling, as a command might do something like print "Username: " and then wait for input. I could normally do full lines, and then if the child blocks on stdin send whatever it has written so far, but that doesn't seem ideal. (I haven't implemented anything about proving input on stdin so far, but I don't want a solution that makes it difficult to do so).

It seems to me that this is something that will be implemented somewhere, for instance my shell can run commands and then interleave the output in a desirable manner, but I haven't found how yet. Any suggestions are welcome, but this is from python, so system calls that I can't make directly from python would be a pain, though I'm not that bothered about portability.

Posted at: 00:38 | category: /tech | Comments (6)


Hey James,

I think that dbus-test-runner does what you want.  Check out lp:dbus-test-runner, I think the code is short enough that it's explainitory :)  Don't use select though, you can use all the GLib main loop.

Ted

Posted by Ted Gould at Tue Jun 9 02:27:16 2009

Oh, another thought.  I have heard that the PulseAudio folks are working on patches to DBus to be able to pass open file descriptors across the bus.  I'm not quite sure how this works, but I'll assume they do :)  I know that they're using it for passing open sockets, but it may work for passing STDOUT and STDERR as well.

Ted

Posted by Ted Gould at Tue Jun 9 02:32:58 2009

Two solutions I can think of: You can dup2 the stderr descriptor onto the stdout descriptor (this is like running cmd 2>&1 | tee logfile in bash) or you can mess around with ptys (which is what your shell is doing).  The main difference, aiui, is in how buffering works, though there are others.

HTH,
mwh

Posted by Michael Hudson at Tue Jun 9 05:33:51 2009

Use the Python Pexpect. With it, you can pragmatically drive other processes. And it is in main, with Canonical support. I use it all the time to encode DVDs and build packages.

Posted by Matt Jones at Tue Jun 9 07:36:36 2009

Also. Pexpect (and its parent Expect) were designed to specifically solve the stdout and stderr reading tangle problem.

Posted by Matt Jones at Tue Jun 9 07:39:34 2009

I think gobject.spawn_async() in combination with gobject.IOChannel is perfect for this kind of tasks.

I'm using this to get stdout and stderr of an external script and use it in an gtk widget. It should be easy to add dbus signal things
http://paste.ubuntu.com/191521/

Also expanding this to watch stdin shouldn't be hard.

Posted by Markus Korn at Tue Jun 9 10:19:04 2009


Name:


E-mail: (optional, will not be disclosed)


URL: (optional)


Comment: