Job control

This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite

How often do you do this:

• open service_foo.conf,
• edit,
• save and close service_foo.conf,
• restart the service foo,
• get a syntax error,
• reopen service_foo.conf,
• navigate to the same position you were at,
• edit,
• save,
• try restart,
• etc.

It’s pretty common.

Or:

$ long_running_command
# Darn, should've started it in the background instead!
CTRL-C
$ long_running_command &

All of these situations can be dealt with using basic job control in your shell. Most proper shells have some job control, but since it’s by far the most common shell, we’ll talk about how bash handles it.

It’s actually very simple. Here’s what you need to know:

A job can be any command that would normally run in the foreground. You can also use %prefix instead of the job number, where the prefix is the command you started. For instance if you run man bash to read up on job control, then stop it, you could resume the job with fg %man.

Stopping a job is not the same as putting it in the background. When you stop a job, it really does stop running. For your editor, this doesn’t matter. Here’s a simple example where I just have a script output the time::

$ ( while sleep 1; do date ; done )
Thu Dec  3 03:05:16 CET 2015
Thu Dec  3 03:05:17 CET 2015
Thu Dec  3 03:05:18 CET 2015
Thu Dec  3 03:05:20 CET 2015
Thu Dec  3 03:05:21 CET 2015
Thu Dec  3 03:05:22 CET 2015
Thu Dec  3 03:05:23 CET 2015
^Z
[2]+  Stopped                 ( while sleep 1; do
    date;
done )
kristian@luke:~$ date
Thu Dec  3 03:05:33 CET 2015
kristian@luke:~$ jobs
[1]-  Stopped                 man bash
[2]+  Stopped                 ( while sleep 1; do
    date;
done )
kristian@luke:~$ fg 2
( while sleep 1; do
    date;
done )
Thu Dec  3 03:05:42 CET 2015
Thu Dec  3 03:05:43 CET 2015
Thu Dec  3 03:05:44 CET 2015
Thu Dec  3 03:05:45 CET 2015
^C

Notice how there are no time stamps printed for the time the command was stopped.

If you wanted that, you would have to put the job in the background. When you do put jobs in the background their output will generally pop up in your shell, just like what would happen if you use & without redirecting output.

There are a few shortcuts to job control too, though I personally don’t use them. Take a look at the Job Control chapter in man bash for more.

screen

Using your shell’s job control is great for manipulating jobs within a single open shell. But it has many limitations too. And it doesn’t allow you to stop a job in one shell and open it up again in an other (perhaps at a later time from an other machine).

Screen is most famous for allowing you to keep programs running even if you lose your connection.

Screen is a simple wrapper around any command you run. You typically start screen with just screen and end up in a plain shell. You can also start a single command directly, for instance using screen irssi. Under the hood you’ve now created a screen “server” which is what your applications are connected to, and a screen “client” which is what your terminal is looking at. If you close your terminal, the client will stop, but the server will keep running and the applications inside it will be unaware of the disappearance of the terminal. You can also detach from the server manually by hitting ctrl-a d. All screen-bindings start with ctrl-a. I’ll have a little list further down.

Here’s a demo:

The basics of screen are:

• screen starts screen with a regular shell.
• screen app starts screen running app. The app-argument can include arguments. screen irssi -! will start screen and irssi -!.
• All screen-commands start with CTRL-a (^A).
• CTRL-a d (^A d) detaches from screen. This happens automatically if you close the terminal or your SSH connection breaks or similar.
• screen -r re-attaches to a screen session. If you have multiple screens running you will have to specify which one (it will prompt you to).
• screen -r -d re-attaches to a screen session that you are still attached to somewhere else. This means that if you SSH to a server at work and open screen but forget to close it, you can take over that screen session when you get home for example.
• screen -x attaches to a screen session without detaching any other screen clients. A good use case is ssh’ing to a server, starting screen and having your customer do the same with screen -x so he can see exactly what you’re doing and even type himself. It’s quite cool, so try it out!

Screen can also have multiple ‘windows’ inside a session. I mostly use “full screen windows” as they are simplest. Try it out while running screen:

• Hit ^A c to create a new window.
• Hit ^A n to go to the next window.
• Hit ^A p to go to the previous window.
• Hit ^A a to go to the window you were at last.

You can also show multiple windows at the same time (split screen) and jump to specific windows if you have many (e.g: jump from window 1 to 6 without going through window 2, 3, 4 and 5.). Check the screen manual page for more.

Screen has some quirks with regards to scrolling, though, so you may want to check out the man page for that too.

Ever need to re-configure network stuff over ssh? Run the commands in screen.

What I often do is something along the lines of: ifdown eth0; sleep 5; ifup eth0; sleep 60 && ifconfig eth0 some-safe-ip for instance. This ensures that the commands run even if the connection drops. It also allows you to regain your old session if you have to reconnect.

Minor tips

• Use tee file if you both want to write the out to a file (echo blatti > foo) and want to see the output at the same time. tee -a will append instead of overwrite, similar to what >> does.
• Simple bash loops are wonderful for testing. I use variations of while true; do blatti; done, while sleep 1; do blatti; done; and for a in foo bar bat; do echo $a; done frequently.