about summary refs log tree commit diff
path: root/nq.1
blob: a881d4e8ef7fa8bbb6882a503ff8a6a87b2c0cf7 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
.Dd March 13, 2021
.Dt NQ 1
.Os
.Sh NAME
.Nm nq
.Nd job queue utility
.Sh SYNOPSIS
.Nm
.Op Fl c
.Op Fl q
.Ar command\ line ...
.Nm
.Fl t
.Ar job\ id ...
.Nm
.Fl w
.Ar job\ id ...
.Sh DESCRIPTION
The
.Nm
utility provides a very lightweight queuing system without
requiring setup,
maintenance,
supervision
or any long-running processes.
.Pp
Job order is enforced by a timestamp
.Nm
gets immediately when started.
Synchronization happens on file-system level.
Timer resolution is milliseconds.
No sub-second file system time stamps are required.
Polling is not used.
Exclusive execution is maintained strictly.
.Pp
You enqueue(!) new jobs into the queue by running
.Pp
.Dl nq Ar command line ...
.Pp
The job id (a file name relative to
.Ev NQDIR ,
which defaults to the current directory) is
output (unless suppressed using
.Fl q )
and
.Nm
detaches from the terminal immediately,
running the job in the background.
Standard output and standard error are redirected into the job id file.
.Xr fq 1
can be used to conveniently watch the log files.
.Pp
The options are as follows:
.Bl -tag -width Ds
.It Fl c
Clean up job id file when process exited with status 0.
.It Fl q
Suppress output of the job id after spawning new job.
.It Fl t
Enter
.Em test mode :
exit with status 0 when
.Em all
of the listed job ids are already done, else with status 1.
.It Fl w
Enter
.Em waiting mode :
wait in the foreground until
.Em all
listed job ids are done.
.El
.Sh ENVIRONMENT
.Bl -hang -width "NQDONEDIR"
.Bl -hang -width "NQFAILDIR"
.It Ev NQDIR
Directory where lock files/job output resides.
Each
.Ev NQDIR
can be considered a separate queue.
The current working directory is used when
.Ev NQDIR
is unset.
.Ev NQDIR
is created if needed.
.It Ev NQDONEDIR
When set, specifies a directory
.Po
must be on the same file system as
.Ev NQDIR
.Pc
where lock files/job output is moved
to after successful execution of the job.
.Pp
Ignored when
.Fl c
is used.
.It Ev NQFAILDIR
When set, specifies a directory
.Po
must be on the same file system as
.Ev NQDIR
.Pc
where lock files/job output is moved
to after unsuccessful execution of the job.
.It Ev NQJOBID
The job id of the currently running job,
exposed to the job itself.
.El
.Sh FILES
.Nm
expects to control all files in
.Ev NQDIR
(respectively
.Pa \&. )
which start with
.Dq Li \&,
or
.Dq Li \&., .
These files are created according to the following scheme:
.Pp
.Dl ,hexadecimal-time-stamp.pid
.Sh EXIT STATUS
The
.Nm
utility exits 0 on success, and >0 if an error occurs;
unless
.Em test mode
is used, in which case exit status 1 means there is a job running.
.Pp
On fatal errors, exit codes 111 and 222 are used.
.Sh EXAMPLES
Build
.Xr make 1
targets
.Ic clean ,
.Ic depends ,
.Ic all ,
without occupying the terminal:
.Bd -literal -offset indent
% nq make clean
% nq make depends
% nq make all
% fq
\&... look at output, can interrupt with C-c any time
without stopping the build ...
.Ed
.Pp
Simple download queue, accessible from multiple terminals:
.Bd -literal -offset indent
% alias qget='NQDIR=/tmp/downloads nq wget'
% alias qwait='NQDIR=/tmp/downloads fq -q'
window1% qget http://mymirror/big1.iso
window2% qget http://mymirror/big2.iso
window3% qget http://mymirror/big3.iso
% qwait
\&... wait for all downloads to finish ...
.Ed
.Pp
As
.Xr nohup 1
replacement
(The benchmark will run in background,
every run gets a different output file,
and the command line you ran is logged too.):
.Bd -literal -offset indent
% ssh remote
remote% nq ./run-benchmark
,14f6f3034f8.17035
remote% ^D
% ssh remote
remote% fq
\&... see output, fq exits when job finished ...
.Ed
.Sh TRICKS
The "file extension" of the log file is actually the PID of the job.
.Nm
runs all jobs in a separate process group,
so you can kill an entire job process tree at once using
.Xr kill 1
with a negative PID.
Before the job is started, it is the PID of
.Nm ,
so you can cancel a queued job by killing it as well.
.Pp
Thanks to the initial
.Li exec
line in the log files, you can resubmit a
job by executing it as a shell command file,
i.e. running
.Pp
.Dl sh Em job\ id
.Pp
Creating
.Nm
wrappers setting
.Ev NQDIR
to provide different queues for different purposes is encouraged.
.Sh INTERNALS
Enforcing job order works like this:
.Bl -dash -compact
.It
every job has an
output file locked using
.Xr flock 2
and named according to
.Sx FILES .
.It
every job starts only after all earlier
flocked files are unlocked.
.It
the lock is released by the kernel after the job terminates.
.El
.Sh ASSUMPTIONS
.Nm
will only work correctly when:
.Bl -dash
.It
.Ev NQDIR
(respectively
.Pa \&. )
is writable.
.It
.Xr flock 2
works correctly in
.Ev NQDIR
(respectively
.Pa \&. ) .
.It
.Xr gettimeofday 2
behaves monotonic (using
.Dv CLOCK_MONOTONIC
would create confusing file names after reboot).
.It
No other programs put files matching
.Li ,*
into
.Ev NQDIR
(respectively
.Pa \&. ) .
.El
.Sh SEE ALSO
.Xr fq 1 ,
.Xr tq 1 .
.Pp
Alternatives to the
.Nm
system include
.Xr batch 1 ,
.Xr qsub 1 ,
.Xr schedule 1 ,
.Xr srun 1 ,
and
.Xr ts 1 .
.\" .Sh STANDARDS
.\" .Sh HISTORY
.Sh AUTHORS
.An Leah Neukirchen Aq Mt leah@vuxu.org
.Sh CAVEATS
All reliable queue status information is in main memory only,
which makes restarting a job queue after a reboot difficult.
.Sh LICENSE
.Nm
is in the public domain.
.Pp
To the extent possible under law,
the creator of this work
has waived all copyright and related or
neighboring rights to this work.
.Pp
.Lk http://creativecommons.org/publicdomain/zero/1.0/
.\" .Sh BUGS