diff options
Diffstat (limited to 'ppmtompeg.html')
| -rw-r--r-- | ppmtompeg.html | 1291 |
1 files changed, 1291 insertions, 0 deletions
diff --git a/ppmtompeg.html b/ppmtompeg.html new file mode 100644 index 00000000..ccafd14f --- /dev/null +++ b/ppmtompeg.html @@ -0,0 +1,1291 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2//EN"> +<HTML> +<HEAD> +<TITLE>Ppmtompeg User Manual</TITLE> +</HEAD> +<BODY> +<H1>Ppmtompeg</H1> +Updated: 23 July 2006 +<BR> +<A HREF="#index">Table Of Contents</A> + +<H2>NAME</H2> +ppmtompeg - encode an MPEG-1 bitstream + +<H2 id="synopsis">SYNOPSIS</H2> + +<B>ppmtompeg</B> +[<I>options</I>] +<I>parameter-file</I> + +<H2 id="description">DESCRIPTION</H2> + +<p>This program is part of <a href="index.html">Netpbm</a>. + +<P><B>ppmtompeg</B> produces an MPEG-1 video stream. MPEG-1 is the +first great video compression method, and is what is used in Video CDs +(VCD). <b>ppmtompeg</b> originated in the year 1995. DVD uses a more +advanced method, MPEG-2. There is an even newer method called MPEG-4 +which is also called Divx. I don't know where one finds that used. + +<p>There's technically a difference between a compression method for +video and an actual file (stream) format for a movie, and I don't know +if it can be validly said that the format of the stream +<b>ppmtompeg</b> produces is MPEG-1. + +<p>Mencoder from the <a href="http://www.mplayerhq.hu">Mplayer +package</a> is probably superior for most video format generation +needs, if for no other reason than that it is more popular. + +<p>The programming library <a href="http://pm2v.free.fr"><b>PM2V</b></a> +generates MPEG-2 streams. + +<p>Use <a href="http://www.mplayerhq.hu">Mplayer</a> (not part of Netpbm) +to do the reverse conversion: to create a series of PNM files from an MPEG +stream. + +<p><i>param_file</i> is a parameter file which includes a list of +input files and other parameters. The file is described in detail +below. + +<P>To understand this program, you need to understand something about +the complex MPEG-1 format. One source of information about this +standard format is the section Introduction to MPEG in the<a +href="http://www.faqs.org/faqs/compression-faq">Compression FAQ</a>. + +<H2 id="options">OPTIONS</H2> + +<p>The <b>-gop</b>, <b>-combine_gops</b>, <b>-frames</b>, and +<b>-combine_frames</b> options are all exclusive. + +<DL COMPACT> +<DT><B>-stat stat_file</B> + +<DD>This option causes <b>ppmtompeg</b> to append the statistics that +it write to Standard Output to the file <I>stat_file</I> as well. The +statistics use the following abbreviations: bits per block (bpb), bits +per frame (bpf), seconds per frame (spf), and bits per second (bps). + +<p>These statistics include how many I, P, and B frames there were, +and information about compression and quality. + + +<DT><B>-quiet</b> <i>num_seconds</i> + +<DD> causes <b>ppmtompeg</b> not to report remaining time more often +than every <i>num_seconds</i> seconds (unless the time estimate rises, +which will happen near the beginning of the run). A negative value +tells <b>ppmtompeg</b> not to report at all. 0 is the default +(reports once after each frame). Note that the time remaining is an +estimate and does not take into account time to read in frames. + +<DT><B>-realquiet</B> <DD> causes <b>ppmtompeg</b> to run silently, +with the only screen output being errors. Particularly useful when +reading input from stdin. + +<DT> +<B>-no_frame_summary</B> + +<DD> This option prevents <b>ppmtompeg</b> from printing a summary +line for each frame + +<DT><B>-float_dct</B> + +<DD> forces <b>ppmtompeg</b> to use a more accurate, yet more +computationally expensive version of the DCT. + +<DT><B>-gop</b> <i>gop_num</i> +<DD> +causes <b>ppmtompeg</b> to encode only the numbered GOP (first GOP is 0). The +parameter file is the same as for normal usage. The output file will be +the normal output file with the suffix <b>.gop.</b><i>gop_num</i>. +<b>ppmtompeg</b> does not output any sequence information. + +<DT><B>-combine_gops</B> + +<DD> causes <b>ppmtompeg</b> simply to combine some GOP files into a +single MPEG output stream. <b>ppmtompeg</b> inserts a sequence header +and trailer. In this case, the parameter file needs only to contain +the SIZE value, an output file, and perhaps a list of input GOP +files (see below). + +If you don't supply a list of input GOP files is used, then +<b>ppmtompeg</b> assumes you're using the same parameter file you used +when you created the input (with the <b>-gop</b> option) and +calculates the corresponding gop filenames itself. If this is not the +case, you can specify input GOP files in the same manner as normal +input files -- except instead of using INPUT_DIR, INPUT, and +END_INPUT, use GOP_INPUT_DIR, GOP_INPUT, and GOP_END_INPUT. If no +input GOP files are specified, then the default is to use the output +file name with suffix <b>.gop.</b><i>gop_num</i>, with <i>gop_num</i> +starting from 0, as the input files. + +<p>Thus, unless you're mixing and matching GOP files from different +sources, you can simply use the same parameter file for creating the +GOP files (<b>-gop</b>) and for later turning them into an MPEG stream +(<b>-combine_gops</b>). + + +<DT><B>-frames <i>first_frame</i> <i>last_frame</i></B> + +<DD>This option causes <b>ppmtompeg</b> to encode only the frames numbered +<i>first_frame</i> to <i>last_frame</i>, inclusive. The parameter +file is the same as for normal usage. The output will be placed in +separate files, one per frame, with the file names being the normal +output file name with the suffix <b>.frame.</b><i>frame_num</i>. No +GOP header information is output. (Thus, the parameter file need not +include the GOP_SIZE value) + +<p>Use <b>ppmtompeg -combine_frames</b> to combine these frames later into +an MPEG stream. + + +<DT><B>-combine_frames</B> + +<DD> This option causes <b>ppmtompeg</b> simply to combine some +individual MPEG frames (such as you might have created with an earlier +run of <b>ppmtompeg -frames</b>) into a single MPEG stream. Sequence +and GOP headers are inserted appropriately. In this case, the +parameter file needs to contain only the SIZE value, the GOP_SIZE +value, an output file, and perhaps a list of frame files (see below). + +<p>The parameter file may specify input frame files in the same manner +as normal input files -- except instead of using INPUT_DIR, INPUT, and +END_INPUT, use FRAME_INPUT_DIR, FRAME_INPUT, and FRAME_END_INPUT. If +no input frame files are specified, then the default is to use the +output file name with suffix <b>.frame.</b><i>frame_num</i>, with +<i>frame_num</i> starting from 0, as the input files. + + + +<DT><B>-nice</B> + +<DD>This option causes <b>ppmtompeg</b> to run any remote processes +"nicely," i.e. at low priority. (This is relevant only if you are +running <b>ppmtompeg</b> in parallel mode. Otherwise, there are no +remote processes). See 'man nice.' + +<DT><B>-max_machines <i>num_machines</i></B> + +<DD>This option causes <b>ppmtompeg</b> to use no more than +<i>num_machines</i> machines as slaves for use in parallel encoding. + +<DT><B>-snr</B> + +<DD>This option causes <b>ppmtompeg</b> to include the signal-to-noise +ratio in the reported statistics. Prints SNR (Y U V) and peak SNR (Y +U V) for each frame. In summary, prints averages of luminance only +(Y). SNR is defined as 10*log(variance of original/variance of +error). Peak SNR is defined as 20*log(255/RMSE). Note that +<b>ppmtompeg</b> runs a little slower when you use this option. + +<DT><B>-mse</B> + +<DD>This option causes <b>ppmtompeg</b> to report the mean squared +error per block. It also automatically reports the quality of the +images, so there is no need to specify <b>-snr</b> then. + +<DT><B>-bit_rate_info</b> <i>rate_file</i> + +<DD> This option makes <b>ppmtompeg</b> write bit rate information +into the file <i>rate_file</i>. Bit rate information is bits per frame, and +also bits per I-frame-to-I-frame. + +<DT><B>-mv_histogram</B> + +<DD> This option causes <b>ppmtompeg</b> to print a histogram of the +motion vectors as part of statistics. There are three histograms -- +one for P frame, one for forward B frame, and one for backward B frame +motion vectors. + +<p>The output is in the form of a matrix, each entry corresponding to one +motion vector in the search window. The center of the matrix +represents (0,0) motion vectors. + +<dt><b>-debug_sockets</b> + +<dd>This option causes <b>ppmtompeg</b> to print to Standard Output +messages that narrate the communication between the machines when you run +<b>ppmtompeg</b> in <a href="#parallel">parallel mode</a>. + +<dt><b>-debug_machines</b> + +<dd>This option causes <b>ppmtompeg</b> to print to Standard Output +messages that narrate the progress of the conversion on the various +machines when you run <b>ppmtompeg</b> in <a href="#parallel">parallel +mode</a>. + +</DL> + +<H2 id="parmfile">PARAMETER FILE</H2> + +<P>The parameter file <strong>must</strong> contain the following +lines (except when using the <b>-combine_gops</b> or <b>-combine_frames</b> +options): + +<DL COMPACT> + +<DT><B>PATTERN</b> <i>pattern</i> + +<DD>This statement specifies the pattern (sequence) of I frames, P frames, +and B frames. <i>pattern</i> is just a sequence of the letters I, P, and +B with nothing between. Example: + +<pre> + PATTERN IBBPBBPBBPBBPBB +</pre> + +<p>See <a href="#ipb">I Frames, P Frames, B Frames</a>. + +<DT><B>OUTPUT</b> <i>output file</i> +<DD>This names the file where the output MPEG stream goes. + +<DT><B>INPUT_DIR</b> <i>directory</i> + +<DD>This statement tells where the input images (frames) come from. +If each frame is in a separate file, <i>directory</i> is the directory +where they all are. You may use <b>.</b> to refer to the current +directory. A null <i>directory</i> refers to the root directory of the +system file tree. + +<p>To have <b>ppmtompeg</b> read all the frames serially from Standard +Input, specify +<pre> + INPUT_DIR stdin +</pre> + +<DT><B>INPUT</b> +<DD> +This line must be followed by a list of the input files (in display order) +and then the line <B>END_INPUT</B>. + +<p>There are three types of lines between INPUT and END_INPUT. First, +a line may simply be the name of an input file. Second, the line +may be of the form <i>single_star_expr</i> +<b>[</b><i>x</i><b>-</b><i>y</i><b>]</b>. +<i>single_star_expr</i> can have a single <b>*</b> in it. It is +replaced by all the numbers between x and y inclusive. So, for +example, the line <b>tennis*.ppm [12-15]</b> refers to the files +tennis12.ppm, tennis13.ppm, tennis14.ppm, tennis15.ppm. + +<p>Uniform zero-padding occurs, as well. For example, the line +<b>football.*.ppm [001-130]</b> refers to the files football.001.ppm, +football.002.ppm, ..., football.009.ppm, football.010.ppm, ..., +football.130.ppm. + +<p>The third type of line is: <i>single_star_expr</i> +<b>[</b><i>x</i><b>-</b><i>y</i><b>+</b><i>s</i><b>]</b>, where the +line is treated exactly as above, except that we skip by <i>s</i>. Thus, the +line <b>football.*.ppm [001-130+4]</b> refers to the files +football.001.ppm, football.005.ppm, football.009.ppm, +football.013.ppm, etc. + +<p>Furthermore, a line may specify a shell command to execute to generate +lines to be interpreted as described above, as if those lines were in the +parameter file instead. Use back ticks, like in the +Bourne Shell, like this: + +<pre> + `cat myfilelist` +</pre> + +<p> +If input is from Standard Input (per the <b>INPUT_DIR</b> statement), +<b>ppmtompeg</b> ignores the <B>INPUT</b>/<b>END_INPUT</b> block, but +it still must be present. + +<DT><b>BASE_FILE_FORMAT</b> {<b>PPM</b> | <b>PNM</b> | <b>YUV</b> | + <b>JPEG</b> | <b>JMOVIE</b>} + +<DD><B>ppmtompeg</b> must convert all input files to one of the +following formats as a first step of processing: PNM, YUV, JPEG(v4), +or JMOVIE. (The conversion may be trivial if your input files are +already in one of these formats). This line specifies which of the +four formats. PPM is actually a subset of PNM. The separate +specification is allowed for backward compatibility. Use PNM instead +of PPM in new applications. + +<DT><b>INPUT_CONVERT</b> <i>conversion_command</i> + +<DD>You must specify how to convert a file to the base file format. +If no conversion is necessary, then you would just say: + + <pre> + INPUT_CONVERT * + </pre> + +<p>Otherwise, <i>conversion_command</i> is a shell command that causes +an image in the format your specified with <B>BASE_FILE_FORMAT</b> to +be written to Standard Output. <b>ppmtompeg</b> executes the command +once for each line between <b>INPUT</b> and <b>END_INPUT</b> (which is +normally, but not necessarily, a file name). In the conversion +command, <b>ppmtompeg</b> replaces each '*' with the contents of that +line. + + If you had a bunch of gif files, you might say: + <pre> + INPUT_CONVERT giftopnm * + </pre> + + If you have a bunch of separate a.Y, a.U, and a.V files (where + the U and V have already been subsampled), then you might say: + + <pre> + INPUT_CONVERT cat *.Y *.U *.V + </pre> + +<p>Input conversion is not allowed with input from stdin, so use + + <pre> + INPUT_CONVERT * + </pre> + +as described above. + +<DT><b>SIZE</b> <i>width</i><b>x</b><i>height</i> + +<dd> + +<p><i>width</i> and <i>height</i> are the width and height of each +frame in pixels. + +<p>When <b>ppmtompeg</b> can get this information from the input image +files, it ignores the <b>SIZE</b> parameter and you may omit it. + +<p>When the image files are in YUV format, the files don't contain +dimension information, so <b>SIZE</b> is required. + +<p>When <b>ppmtompeg</b> is running in parallel mode, not all of the +processes in the network have access to the image files, so +<b>SIZE</b> is required and must give the same dimensions as the +input image files. + +<DT><b>YUV_SIZE</b> <i>width</i><b>x</b><i>height</i> + +<dd>This is an obsolete synonym of <b>SIZE</b>. + +<DT><b>YUV_FORMAT</B> {<b>ABEKAS</b> | <b>PHILLIPS</b> | <b>UCB</B> | + <b>EYUV</b> | <i>pattern</i>} + +<DD>This is meaningful only when <b>BASE_FILE_FORMAT</b> specifies +YUV format, and then it is required. It specifies the sub-format of +the YUV class. + + +<DT><b>GOP_SIZE</b> <i>n</i> + +<DD><i>n</i> is the number of frames in a Group of Pictures. Except that +because a GOP must start with an I frame, <b>ppmtompeg</b> makes a GOP as +much longer than <i>n</i> as it has to to make the next GOP start with an +I frame. + +<p>Normally, it makes sense to make your GOP size a multiple of your +pattern length (the latter is determined by the PATTERN parameter file +statement). + +<p>See <a href="#gop">Group Of Pictures</a>. + +<DT><b>SLICES_PER_FRAME</b> <i>n</i> +<dd><i>n</i> is roughly the number of slices per frame. Note, at +least one MPEG player may complain if slices do not start at the left +side of an image. To ensure this does not happen, make sure the +number of rows is divisible by SLICES_PER_FRAME. + +<DT><b>PIXEL</b> {<b>FULL</b> | <b>HALF</b>} + +<dd>use half-pixel motion vectors, or just full-pixel ones It is +usually important that you use half-pixel motion vectors, because it +results in both better quality and better compression. + + +<DT><b>RANGE</b> <i>n</i> +<dd>Use a search range of <i>n</i> pixels in each of the four directions +from a subject pixel. (So the search window is a square <i>n</i>*2 pixels +on a side). + +<DT><b>PSEARCH_ALG</b> {<b>EXHAUSTIVE</B> | <b>TWOLEVEL</b> | + <b>SUBSAMPLE</b> | <b>LOGARITHMIC</b>} + +<dd>This statement tells <b>ppmtompeg</b> what kind of search + technique (algorithm) to use for P frames. You select the desired + combination of speed and compression. <b>EXHAUSTIVE</b> gives the + best compression, but <b>LOGARITHMIC</B> is the fastest. + <B>TWOLEVEL</B> is an exhaustive full-pixel search, followed by a + local half- pixel search around the best full-pixel vector (the + PIXEL option is ignored for this search technique). + +<DT><b>BSEARCH_ALG</b> {<b>SIMPLE</B> | <B>CROSS2</B> | <B>EXHAUSTIVE</B>} + +<dd>This statement tells <b>ppmtompeg</b> what kind of search + technique (algorithm) to use for B frames. <b>SIMPLE</B> means + find best forward and backward vectors, then interpolate. + <B>CROSS2</B> means find those two vectors, then see what backward + vector best matches the best forward vector, and vice versa. + <b>EXHAUSTIVE</b> does an n-squared search and is + <em>extremely</em> slow in relation to the others (<b>CROSS2</b> + is about half as fast as <b>SIMPLE</B>). + +<DT><b>IQSCALE</b> <i>n</i> +<dd>Use <i>n</i> as the qscale for I frames. + See <a href="#qscale">Qscale</a>. + +<DT><b>PQSCALE</b> <i>n</i> +<dd>Use <i>n</i> as the qscale for P frames. + See <a href="#qscale">Qscale</a>. + +<DT><b>BQSCALE</b> <i>n</i> +<dd>Use <i>n</i> as the qscale for B frames. + See <a href="#qscale">Qscale</a>. + +<DT><b>REFERENCE_FRAME</b> {<B>ORIGINAL</B> | <b>DECODED</b>} <dd>This +statement determines whether <b>ppmtompeg</b> uses the original images +or the decoded images when computing motion vectors. Using decoded +images is more accurate and should increase the playback quality of +the output, but it makes the encoding take longer and seems to give +worse compression. It also causes some complications with parallel +encoding. (see the section on parallel encoding). One thing you can +do as a trade-off is select <b>ORIGINAL</b> here, and lower the +qscale (see <b>QSCALE</b> if the quality is not good enough. + +<table border="1" cellpadding="5" cellspacing="0" +summary="comparison of original to decoded"> + <caption>Original or Decoded? (Normalized)</caption> +<?makeman r c c c c c. ?> +<?makeman _ ?> + <tr align="center" bgcolor="#CCCCCC"> + <th>Reference</th> + <th>Compression</th> + <th>Speed</th> + <th>Quality I</th> + <th>Quality P</th> + <th>Quality B</th> + </tr> + <tr> + <td align="right">Decoded</td> + <td align="center">1000</td> + <td align="center">1000</td> + <td align="center">1000</td> + <td align="center">969</td> + <td align="center">919</td> + </tr> + <tr> + <td align="right">Original</td> + <td align="center">885</td> + <td align="center">1373</td> + <td align="center">1000</td> + <td align="center">912</td> + <td align="center">884</td> + </tr> + </table> + + + +</dl> + +<p>The following lines are optional: + +<DL> + +<DT><B>FORCE_ENCODE_LAST_FRAME</B> + +<dd>This statement is obsolete. It does nothing. + +<p>Before Netpbm 10.26 (January 2005), <b>ppmtompeg</b> would drop +trailing B frames from your movie, since a movie can't end with a B +frame. (See <a href="#ipb">I Frames, P Frames, B Frames</a>. +You would have to specify <b>FORCE_ENCODE_LAST_FRAME</b> to stop +that from happening and get the same function that <b>ppmtompeg</b> +has today. + + +<DT><b>NIQTABLE</b> + +<dd>This statement specifies a custom non-intra quantization table. +If you don't specify this statement, <b>ppmtompeg</b> uses a default +non-intra quantization table. + +<p> +The 8 lines immediately following <b>NIQTABLE</b> specify the quantization +table. Each line defines a table row and consists of 8 integers, +whitespace-delimited, which define the table columns. + +<DT><B>IQTABLE</b> + +<dd>This is analogous to NIQTABLE, but for the intra quantization table. + +<DT><b>ASPECT_RATIO</b> <i>ratio</i> + +<dd>This statement specifies the aspect ratio for <b>ppmtompeg</b> to +specify in the MPEG output. I'm not sure what this is used for. + +<p><i>ratio</i> must be 1.0, 0.6735, 0.7031, 0.7615, 0.8055, 0.8437, +0.8935, 0.9157, 0.9815, 1.0255, 1.0695, 1.0950, 1.1575, or 1.2015. + +<DT><b>FRAME_RATE</b> <i>rate</i> +<dd>This specifies the frame rate for <b>ppmtompeg</b> to specify in the +MPEG output. Some players use this value to determine the playback rate. + +<p><i>rate</i> must be 23.976, 24, 25, 29.97, 30, 50, 59.94, or 60. + +<DT><b>BIT_RATE</b> <i>rate</i> +<DD>This specifies the bit rate for Constant Bit Rate (CBR) encoding. + +<p><i>rate</i> must be an integer. + +<DT><b>BUFFER_SIZE</b> <i>size</i> + +<dd>This specifies the value +<b>ppmtompeg</b> is to specify in the MPEG output for the Video +Buffering Verifier (VBV) buffer size needed to decode the sequence. + +<p>A Video Verifying Buffer is a buffer in which a decoder keeps the +decoded bits in order to match the uneven speed of the decoding with +the required constant playback speed. + +<p>As <b>ppmtompeg</b> encodes the image, it simulates the decoding +process in terms of how many bits would be in the VBV as each frame gets +decoded, assuming a VBV of the size you indicate. + +<P>If you specify the <b>WARN_VBV_UNDERFLOW</b> statement, +<b>ppmtompeg</b> issues a warning each time the simulation underflows +the buffer, which suggests that an underflow would occur on playback, +which suggests the buffer is too small. + +<P>If you specify the <b>WARN_VBV_OVERFLOW</b> statement, +<b>ppmtompeg</b> issues a warning each time the simulation overflows +the buffer, which suggests that an overflow would occur on playback, +which suggests the buffer is too small. + +<DT><B>WARN_VBV_UNDERFLOW</B> +<DT><B>WARN_VBV_OVERFLOW</B> + +<dd>See <b>BUFFER_SIZE</b>. + +<p>These options were new in Netpbm 10.26 (January 2005). Before that, +<b>ppmtompeg</b> issued the warnings always. + +</DL> + + +The following statements apply only to parallel operation: + +<DL> + +<DT><b>PARALLEL</b> + +<dd>This statement, paired with <b>END PARALLEL</B>, is what causes +<b>ppmtompeg</b> to operate in parallel mode. See <a +href="#parallel">Parallel Operation</a>. + +<dt><b>END PARALLEL</b> + +<DD>This goes with <b>PARALLEL</b>. + +<DT><b>PARALLEL_TEST_FRAMES</b> <i>n</i> + +<dd>The master starts off by measuring each slave's speed. It does +this by giving each slave <i>n</i> frames to encode and noting how +long the slave takes to finish. These are not just test frames, +though -- they're real frames and the results become part of the +output. +<b>ppmtompeg</b> is old and measures time in undivided seconds, so +to get useful timings, specify enough frames that it will take at +least 5 seconds to process them. The default is 10. + +<p>If you specify <b>FORCE_I_ALIGN</b>, <b>ppmtompeg</b> will increase +the test frames value enough to maintain the alignment. + +<p>If there aren't enough frames for every slave to have the indicated +number of test frames, <b>ppmtompeg</b> will give some slaves fewer. + + +<DT><b>PARALLEL_TIME_CHUNKS</b> <i>t</i> + +<dd>When you specify this statement, the master attempts to feed work +to the slaves in chunks that take <i>t</i> seconds to process. It uses +the speed measurement it made when it started up (see PARALLEL_TEST_FRAMES) +to decide how many frames to put in the chunk. This statement obviously +doesn't affect the first batch of work sent to each slave, which is the +one used to measure the slave's speed. + +<p>Smaller values of <i>t</i> increase communication, but improve load +balancing. The default is 30 seconds. + +<p>You may specify only one of PARALLEL_TIME_CHUNKS, PARALLEL_CHUNK_TAPER, +and PARALLEL_PERFECT. PARALLEL_CHUNK_TAPER is usually best. + +<DT><b>PARALLEL_CHUNK_TAPER</b> + +<DD>When you specify this statement, the master distributes work like +with PARALLEL_TIME_CHUNKS, except that the master chooses the number +of seconds for the chunks. It starts with a large number and, as it +gets closer to finishing the job, reduces it. That way, it reduces +scheduling overhead when precise scheduling isn't helpful, but still +prevents a slave from finishing early after all the work has already +been handed out to the other slaves, and then sitting idle while +there's still work to do. + +<p>You may specify only one of PARALLEL_TIME_CHUNKS, PARALLEL_CHUNK_TAPER, +and PARALLEL_PERFECT. PARALLEL_CHUNK_TAPER is usually best. + + +<DT><b>PARALLEL_PERFECT</b> + +<dd>If this statement is present, <b>ppmtompeg</b> schedules on the +assumption that each machine is about the same speed. The master will +simply divide up the frames evenly between the slaves -- each +slave gets the same number of frames. If some slaves are faster than +others, they will finish first and remain idle while the slower slaves +continue. + +<p>This has the advantage of minimal scheduling overhead. Where slaves +have different speeds, though, it makes inefficient use of the fast +ones. Where slaves are the same speed, it also has the disadvantage +that they all finish at the same time and feed their output to the +single Combine Server in a burst, which makes less efficient use of +the Combine Server and thus can increase the total elapsed time. + +<p>You may specify only one of PARALLEL_TIME_CHUNKS, PARALLEL_CHUNK_TAPER, +and PARALLEL_PERFECT. PARALLEL_CHUNK_TAPER is usually best. + +<DT><b>RSH</b> <i>remote_shell_command</i> + +<DD><b>ppmtompeg</b> executes the shell command +<i>remote_shell_command</i> to start a process on another machine. +The default command is <b>rsh</b>, and whatever command you specify +must have compatible semantics. <b>ssh</b> is usually compatible. +The command <b>ppmtompeg</b> uses is one like this: +<b>ssh remote.host.com -l username shellcommand</b>. + +<p>Be sure to set up <b>.rhosts</b> files or SSH key authorizations +where needed. Otherwise, you'll have to type in passwords. + +<p>On some HP machines, <b>rsh</b> is the restricted shell, and you want +to specify <b>remsh</b>. + +<DT><b>FORCE_I_ALIGN</b> + +<dd>This statement forces each slave to encode a chunk of frames which +is a multiple of the pattern length (see <b>PATTERN</b>). Since the +first frame in any pattern is an I frame, this forces each chunk +encoded by a slave to begin with an I frame. + +<p>This document used to say there was an argument to +<b>FORCE_I_ALIGN</b> which was the number of frames <b>ppmtompeg</b> +would use (and was required to be a multiple of the pattern length). +But <b>ppmtompeg</b> has apparently always ignored that argument, and +it does now. + +<DT><B>KEEP_TEMP_FILES</B> + +<dd>This statement causes <b>ppmtompeg</b> not to delete the temporary +files it uses to transmit encoded frames to the combine server. This +means you will be left with a file for each frame, the same as you +would get with the <b>-frames</b> option. + +<p>This is mostly useful for debugging. + +<p>This works only if you're using a shared filesystem to communicate +between the servers. + +<p>This option was new in Netpbm 10.26 (January 2005). + +</DL> + + +<H3>Parameter File Notes</h3> + +<P> If you use the <b>-combine_gops</b> option, then you need to specify +only the SIZE and OUTPUT values in the parameter file. In +addition, the parameter file may specify input GOP files in the same +manner as normal input files -- except instead of using INPUT_DIR, +INPUT, and END_INPUT, use GOP_INPUT_DIR, GOP_INPUT, and GOP_END_INPUT. +If you specify no input GOP files, then <b>ppmtompeg</b> uses by default the +output file name with suffix <b>.gop.</b><i>gop_num</i>, with <i>gop_num</i> +starting from 0, as the input files. + +<p>If you use the <b>-combine_frames</b> option, then you need to +specify only the SIZE, GOP_SIZE, and OUTPUT values in the +parameter file. In addition, the parameter file may specify input +frame files in the same manner as normal input files -- except instead +of using INPUT_DIR, INPUT, and END_INPUT, use FRAME_INPUT_DIR, +FRAME_INPUT, and FRAME_END_INPUT. If no input frame files are +specified, then the default is to use the output file name with suffix +<b>.frame.</b><i>frame_num</i>, with <i>frame_num</i> starting from 0, +as the input files. + +<p>Any number of spaces and tabs may come between each option and value. Lines +beginning with <b>#</b> are ignored. Any other lines are ignored except for +those between INPUT and END_INPUT. This allows you to use the same +parameter file for normal usage and for <b>-combine_gops</b> and +<b>-combine_frames</b>. + +<P>The file format is case-sensitive so all keywords should be in +upper case. + +<P>The statements may appear in any order, except that the order within +a block statement (such as INPUT ... END INPUT) is significant. + +<P><b>ppmtompeg</b> is prepared to handle up to 16 B frames between +reference frames when encoding with input from stdin. (To build a +modified <b>ppmtompeg</b> with a higher limit, change the constant +B_FRAME_RUN in frame.c and recompile). + +<H2 id="general">GENERAL USAGE INFORMATION</H2> + +<H3 id="qscale">Qscale</h3> + +<p>The quantization scale values (qscale) give a trade-off between +quality and compression. Using different Qscale values has very little +effect on speed. The qscale values can be set separately for I, P, and +B frames. + +<p>You select the qscale values with the <B>IQSCALE</b>, +<b>PQSCALE</b>, and <b>BSCALE</b> parameter file statements. + +<p>A qscale value is an integer from 1 to 31. Larger numbers give +better compression, but worse quality. In the following, the quality +numbers are peak signal-to-noise ratio, defined as: +<img src="ppmtompeg-snr.gif" alt="signal-to-noise formula" height="52" width="302"> +where MSE is the mean squared error. + + +<p>Flower garden tests: + +<table border="1" cellpadding="5" cellspacing="0" summary="Qscale vs Quality"> + <caption>Qscale vs Quality</caption> +<?makeman r r r r. ?> +<?makeman _ ?> + <tr align="center"> + <th>Qscale</th> + <th>I Frames</th> + <th>P Frames</th> + <th>B Frames</th> + </tr> + <tr> + <td align="right">1</td> + <td align="right">43.2</td> + <td align="right">46.3</td> + <td align="right">46.5</td> + </tr> + <tr> + <td align="right">6</td> + <td align="right">32.6</td> + <td align="right">34.6</td> + <td align="right">34.3</td> + </tr> + <tr> + <td align="right">11</td> + <td align="right">28.6</td> + <td align="right">29.5</td> + <td align="right">30.0</td> + </tr> + <tr> + <td align="right">16</td> + <td align="right">26.3</td> + <td align="right">26.8</td> + <td align="right">28.6</td> + </tr> + <tr> + <td align="right">21</td> + <td align="right">24.7</td> + <td align="right">25.0</td> + <td align="right">27.9</td> + </tr> + <tr> + <td align="right">26</td> + <td align="right">23.5</td> + <td align="right">23.9</td> + <td align="right">27.5</td> + </tr> + <tr> + <td align="right">31</td> + <td align="right">22.6</td> + <td align="right">23.0</td> + <td align="right">27.3</td> + </tr> +</table> + +<table border="1" cellpadding="5" cellspacing="0" +summary="Qscale vs Compression"> + <caption>Qscale vs Compression</caption> +<?makeman r r r r. ?> +<?makeman _ ?> + <tr align="center"> + <th>Qscale</th> + <th>I Frames</th> + <th>P Frames</th> + <th>B Frames</th> + </tr> + <tr> + <td align="right">1</td> + <td align="right">2</td> + <td align="right">2</td> + <td align="right">2</td> + </tr> + <tr> + <td align="right">6</td> + <td align="right">7</td> + <td align="right">10</td> + <td align="right">15</td> + </tr> + <tr> + <td align="right">11</td> + <td align="right">11</td> + <td align="right">18</td> + <td align="right">43</td> + </tr> + <tr> + <td align="right">16</td> + <td align="right">15</td> + <td align="right">29</td> + <td align="right">97</td> + </tr> + <tr> + <td align="right">21</td> + <td align="right">19</td> + <td align="right">41</td> + <td align="right">173</td> + </tr> + <tr> + <td align="right">26</td> + <td align="right">24</td> + <td align="right">56</td> + <td align="right">256</td> + </tr> + <tr> + <td align="right">31</td> + <td align="right">28</td> + <td align="right">73</td> + <td align="right">330</td> + </tr> +</table> + + +<h3>Search Techniques</h3> + +<p>There are several different motion vector search techniques +available. There are different techniques available for P frame +search and B frame search. Using different search techniques present +little difference in quality, but a large difference in compression +and speed. + +<p>There are 4 types of P frame search: Exhaustive, TwoLevel, +SubSample, and Logarithmic. + +<p>There are 3 types of B frame search: Exhaustive, Cross2, and +Simple. + +The recommended search techniques are TwoLevel and Logarithmic for +P frame search, and Cross2 and Simple for B frame search. Here are +some numbers comparing the different search methods: + +<table border="1" cellpadding="5" cellspacing="0" +summary="P frame motion vector search"> + <caption>P frame Motion Vector Search (Normalized)</caption> +<?makeman r c c c. ?> +<?makeman _ ?> + <tr align="center"> + <th>Technique</th> + <th>Compression<a href="#smallbetter"><sup>1</sup></a></th> + <th>Speed <a href="#largefaster"><sup>2</sup></a></th> + <th>Quality <a href="#largebetter"><sup>3</sup></a></th> + </tr> + <tr> + <td align="right">Exhaustive</td> + <td align="center">1000</td> + <td align="center">1000</td> + <td align="center">1000</td> + </tr> + <tr> + <td align="right">SubSample</td> + <td align="center">1008</td> + <td align="center">2456</td> + <td align="center">1000</td> + </tr> + <tr> + <td align="right">TwoLevel</td> + <td align="center">1009</td> + <td align="center">3237</td> + <td align="center">1000</td> + </tr> + <tr> + <td align="right">Logarithmic</td> + <td align="center">1085</td> + <td align="center">8229</td> + <td align="center">998</td> + </tr> +</table> + +<table border="1" cellpadding="5" cellspacing="0" +summary="B frame motion vector search"> + <caption>B frame Motion Vector Search (Normalized)</caption> +<?makeman r c c c. ?> +<?makeman _ ?> + <tr align="center"> + <th>Technique</th> + <th>Compression<a href="#smallbetter"><sup>1</sup></a></th> + <th>Speed<a href="#largefaster"><sup>2</sup></a></th> + <th>Quality<a href="#largebetter"><sup>3</sup></a></th> + </tr> + <tr> + <td align="right">Exhaustive</td> + <td align="center">1000</td> + <td align="center">1000</td> + <td align="center">1000</td> + </tr> + <tr> + <td align="right">Cross2</td> + <td align="center">975</td> + <td align="center">1000</td> + <td align="center">996</td> + </tr> + <tr> + <td align="right">Simple</td> + <td align="center">938</td> + <td align="center">1765</td> + <td align="center">991</td> + </tr> +</table> + +<a name="smallbetter"> </a><sup>1</sup>Smaller numbers are better +compression. + +<a name="largefaster"> </a><sup>2</sup>Larger numbers mean faster +execution. + +<a name="largebetter"> </a><sup>3</sup>Larger numbers mean better quality. + +<p>For some reason, Simple seems to give better compression, but it +depends on the image sequence. + +<p>Select the search techniques with the <B>PSEARCH_ALG</B> and +<B>BSEARCH_ALG</b> parameter file statements. + + +<a name="gop"></a> +<h3>Group Of Pictures (GOP)</h3> + +<p>A Group of Pictures (GOP) is a roughly independently decodable +sequence of frames. An MPEG video stream is made of one or more +GOP's. You may specify how many frames should be in each GOP with the +<b>GOP_SIZE</b> parameter file statement. A GOP always starts with an +I frame. + +<p>Instead of encoding an entire sequence, you can encode a single +GOP. To do this, use the <b>-gop</b> command option. You can later +join the resulting GOP files at any time by running <b>ppmtompeg</b> +with the <b>-combine_gops</b> command option. + + +<h3>Slices</h3> + +<p>A slice is an independently decodable unit in a frame. It can be +as small as one macroblock, or it can be as big as the entire frame. +Barring transmission error, adding slices does not change quality or +speed; the only effect is slightly worse compression. More slices are +used for noisy transmission so that errors are more recoverable. Since +usually errors are not such a problem, we usually just use one slice +per frame. + +<p>Control the slice size with the <B>SLICES_PER_FRAME</B> parameter +file statement. + +<p>Some MPEG playback systems require that each slice consist of whole +rows of macroblocks. If you are encoding for this kind of player, if +the height of the image is H pixels, then you should set the +SLICES_PER_FRAME to some number which divides H/16. For example, if +the image is 240 pixels (15 macroblocks) high, then you should use +only 15, 5, 3, or 1 slices per frame. + +<p>Note: these MPEG playback systems are really wrong, since the MPEG +standard says this doesn't have to be so. + + + +<h3>Search Window</h3> + +<p>The search window is the window in which <b>ppmtompeg</b> searches +for motion vectors. The window is a square. You can specify the size +of the square, and whether to allow half-pixel motion vectors or not, +with the <b>RANGE</b> and <b>PIXEL</B> parameter file statements. + +<h3 id="ipb">I Frames, P Frames, B Frames</h3> + +<p>In MPEG-1, a movie is represented as a sequence of MPEG frames, +each of which is an I Frame, a P Frame, or a B Frame. Each represents +an actual frame of the movie (don't get confused by the dual use of +the word "frame." A movie frame is a graphical image. An MPEG frame +is a set of data that describes a movie frame). + +<p>An I frame ("intra" frame) describes a movie frame in isolation -- +without respect to any other frame in the movie. A P frame +("predictive" frame) describes a movie frame by describing how it +differs from the movie frame described by the latest preceding I or +P frame. A B frame ("bidirectional" frame) describes a movie frame by +describing how it differs from the the movie frames described by the +nearest I or P frame before <em>and</em> after it. + +<p>Note that the first frame of a movie must be described by an I +frame (because there is no previous movie frame) and the last movie +frame must be described by an I or P frame (because there is no +subsequent movie frame). + +<p>Beyond that, you can choose which frames are represented by which +types. You specify a pattern, such as IBPBP and <b>ppmtompeg</b> +simply repeats it over and over throughout the movie. The pattern +affects speed, quality, and stream size. Here is a chart which shows +some of the trade-offs: + +<table border="1" cellpadding="5" cellspacing="0" +summary="Comparison of I/P/B Frames"> + <caption>Comparison of I/P/B Frames (Normalized)</caption> +<?makeman r c c c. ?> +<?makeman _ ?> + <tr align="center"> + <th>Frame Type</th> + <th>Size</th> + <th>Speed</th> + <th>Quality</th> + </tr> + <tr> + <td align="right">I frames</td> + <td align="center">1000</td> + <td align="center">1000</td> + <td align="center">1000</td> + </tr> + <tr> + <td align="right">P frames</td> + <td align="center">409</td> + <td align="center">609</td> + <td align="center">969</td> + </tr> + <tr> + <td align="right">B frames</td> + <td align="center">72</td> + <td align="center">260</td> + <td align="center">919</td> + </tr> + </table> + +(this is with constant qscale) + +<p>A standard sequence is IBBPBBPBBPBBPBB. + +<p>Select the sequence with the <B>PATTERN</B> parameter file statement. + +<p>Since the last MPEG frame cannot be a B frame (see above), if the +pattern you specify indicates B frames for the last movie frames, +<b>ppmtompeg</b> makes it an I frame instead. + +<p>Before Netpbm 10.26 (January 2005), <b>ppmtompeg</b> instead drops +the trailing B frames by default, and you need the +<b>FORCE_ENCODE_LAST_FRAME</b> parameter file statement to make it do +this. + +<p>The MPEG frames don't appear in the MPEG-1 stream in the same order that +the corresponding movie frames appear in the movie -- the B frames come after +the I and P frames on which they are based. For example, if the movie is +4 frames that you will represent with the pattern IBBP, the MPEG-1 stream +will start with an I frame describing movie frame 0. The next frame in +the MPEG-1 stream is a P frame describing movie frame 3. The last two +frames in the MPEG-1 stream are B frames describing movie frames 1 and 2, +respectively. + + +<h3>Specifying Input and Output Files</h3> + +<p>Specify the input frame images with the <B>INPUT_DIR</B>, +<B>INPUT</B>, <B>END_INPUT</B>, <B>BASE_FILE_FORMAT</B>, +<B>SIZE</B>, <B>YUV_FORMAT</B> and <b>INPUT_CONVERT</B> parameter +file statements. + +<p>Specify the output file with the <b>OUTPUT</b> parameter file statement. + + +<h3>Statistics</h3> + +<p><b>ppmtompeg</b> can generate a variety of statistics about the +encoding. See the <b>-stat</b>, <b>-snr</b>, <b>-mv_histogram</b>, +<b>-quiet</b>, <b>-no_frame_summary</b>, and <b>-bit_rate_info</b> +options. + + +<H2 id="parallel">PARALLEL OPERATION</H2> + +<P>You can run <b>ppmtompeg</b> on multiple machines at once, encoding +the same MPEG stream. When you do, the machines are used as shown in +the following diagram. We call this "parallel mode." + +<p><img src="ppmtompeg-par.gif" alt="ppmtompeg-par.gif"> + +<p>To do parallel processing, put the statement + +<pre> + PARALLEL +</pre> + +in the parameter file, followed by a listing of the machines, one +machine per line, then + +<pre> + END_PARALLEL +</pre> + +Each of the machine lines must be in one of two forms. If the machine +has filesystem access to the input files, then the line is: + +<p> +<i>machine</i> <i>user</i> <i>executable</i> + +<P>The executable is normally <b>ppmtompeg</b> (you may need to give +the complete path if you've built for different architectures). If +the machine does not have filesystem access to the input files, the line +is: + +<P><b>REMOTE</b> <i>machine</i> <i>user</i> <i>executable</i> +<i>parameter file</i> + +<p>The <b>-max_machines</b> command option limits the number of +machines <b>ppmtompeg</b> will use. If you specify more machines in +the parameter file than <b>-max_machines</b> allows, <b>ppmtompeg</b> +uses only the machines listed first. This is handy if you want to +experiment with different amounts of parallelism. + +<p>In general, you should use full path file names when describing +executables and parameter files. This <em>includes</em> the parameter +file argument on the original invocation of <b>ppmtompeg</b>. + +<p>All file names must be the same on all systems (so if e.g. you're +using an NFS filesystem, you must make sure it is mounted at the same +mountpoint on all systems). + +<P>Because not all of the processes involved in parallel operation +have easy access to the input files, you must specify the <B>SIZE</B> +parameter file statement when you do parallel operation. + +<p>The machine on which you originally invoke <b>ppmtompeg</b> is the +master machine. It hosts a "combine server,", a +"decode server," and a number of "i/o servers," +all as separate processes. The other machines in the network (listed +in the parameter file) are slave machines. Each hosts a single +process that continuously requests work from the master and does it. +The slave process does the computation to encode MPEG frames. It +processes frames in batches identified by the master. + +<p>The master uses a remote shell command to start a process on a +slave machine. By default, it uses an <b>rsh</b> shell command to do +this. But use the <b>RSH</b> parameter file statement to control +this. The shell command the master executes remotely is +<b>ppmtompeg</b>, but with options to indicate that it is to perform +slave functions. + +<p>The various machines talk to each other over TCP connections. Each +machine finds and binds to a free TCP port number and tells its +partners the port number. These port numbers are at least 2048. + +<p>Use the PARALLEL_TEST_FRAMES, PARALLEL_TIME_CHUNKS, and +PARALLEL_PERFECT parameter file statements to control the way the +master divides up work among the slaves. + +<p>Use the <b>-nice</b> command option to cause all slave processes to run +"nicely," i.e. as low priority processes. That way, this substantial and +long-running CPU load will have minimal impact on other, possibly +interactive, users of the systems. + +<A NAME="speed"> </A> +<H2>SPEED</h2> + +<p>Here is a look at <b>ppmtompeg</b> speed, in single-node (not parallel) +operation: + +<table border="1" cellpadding="5" cellspacing="0" summary="Compression speed"> + <caption>Compression Speed</caption> +<?makeman r c. ?> +<?makeman _ ?> + <tr align="center"> + <th>Machine Type</th> + <th>Macroblocks per second<sup>1</sup></th> + </tr> + <tr> + <td align="right">HP 9000/755</td> + <td align="center">280</td> + </tr> + <tr> + <td align="right">DEC 3000/400</td> + <td align="center">247</td> + </tr> + <tr> + <td align="right">HP 9000/750</td> + <td align="center">191</td> + </tr> + <tr> + <td align="right">Sparc 10</td> + <td align="center">104</td> + </tr> + <tr> + <td align="right">DEC 5000</td> + <td align="center">68</td> + </tr> +</table> +<sup>1</sup>A macroblock is a 16x16 pixel square + +<p>The measurements in the table are with inputs and outputs via a +conventional locally attached filesystem. If you are using a network +filesystem over a single 10 MB/s Ethernet, that constrains your speed more +than your CPU speed. In that case, don't expect to get better than 4 +or 5 frames per second no matter how fast your CPUs are. + +<p>Network speed is even more of a bottleneck when the slaves do not +have filesystem access to the input files -- i.e. you declare them +REMOTE. + +<p>Where I/O is the bottleneck, size of the input frames can make a big +difference. So YUV input is better than PPM, and JPEG is better than +both. + +<p>When you're first trying to get parallel mode working, be sure to +use the <b>-debug_machines</b> option so you can see what's going on. +Also, <b>-debug_sockets</b> can help you diagnose communication +problems. + + +<H2 id="authors">AUTHORS</H2> + +<UL> + +<LI>Kevin Gong - University of California, Berkeley, <A +HREF="mailto:keving@cs.berkeley.edu">keving@cs.berkeley.edu</A> + +<LI>Ketan Patel - University of California, Berkeley, <A +HREF="mailto:kpatel@cs.berkeley.edu">kpatel@cs.berkeley.edu</A> + +<LI>Dan Wallach - University of California, Berkeley, <A +HREF="mailto:dwallach@cs.berkeley.edu">dwallach@cs.berkeley.edu</A> + +<LI>Darryl Brown - University of California, Berkeley, <A +HREF="mailto:darryl@cs.berkeley.edu">darryl@cs.berkeley.edu</A> + +<LI>Eugene Hung - University of California, Berkeley, <A +HREF="mailto:eyhung@cs.berkeley.edu">eyhung@cs.berkeley.edu</A> + +<LI>Steve Smoot - University of California, Berkeley, <A +HREF="mailto:smoot@cs.berkeley.edu">smoot@cs.berkeley.edu</A> + +</UL> + +<HR> +<A NAME="index"> </A> +<H2>Table Of Contents</H2> +<UL> +<LI><A HREF="#synopsis">SYNOPSIS</A></LI> +<LI><A HREF="#description">DESCRIPTION</A></LI> +<LI><A HREF="#options">OPTIONS</A></LI> +<LI><A HREF="#parmfile">PARAMETER FILE</A></LI> +<LI><A HREF="#general">GENERAL USAGE INFORMATION</A></LI> +<LI><A HREF="#parallel">PARALLEL OPERATION</A></LI> +<LI><A HREF="#speed">SPEED</A></LI> +<LI><A HREF="#authors">AUTHORS</A></LI> +</UL> +</BODY> +</HTML> |