In-memory record sort
aq_ord [-h] Global_Opt Input_Spec Sort_Spec Output_Spec
Global_Opt:
[-verb] [-stat] [-bz ReadBufSiz]
Input_Spec:
[-f[,AtrLst] File [File ...]] [-d ColSpec [ColSpec ...]]
Sort_Spec:
-sort[,AtrLst] ColTyp:ColNum | -sort[,AtrLst] ColName [ColName ...]
Output_Spec:
[-o[,AtrLst] File] [-c ColName [ColName ...]]
aq_ord sorts input records according to the value of the sort columns.
Sort is done in memory, so it is fast.
However, the entire data set must fit into a single machine’s main memory.
The program offers two sort modes. One is fast and simple but less flexible.
The other requires more processing overhead but is more versatile.
Raw sort mode
In this mode, raw input rows are stored in memory as-is. Column values are not interpreted except for the sort column. Advantages are:
Disadvantages are:
Parsed sort mode
In this mode, a column spec must be defined. Columns are converted before they are stored in memory - numeric and IP address types are stored in binary forms, string type is hashed and the pointer to the hash entry is stored. Advantages are:
Disadvantages are:
-verb-statPrint a record count summary line to stderr at the end of processing. The line has the form:
aq_ord: rec=Count err=Count
-bz ReadBufSizReadBufSiz is a number in bytes.-f[,AtrLst] File [File ...]Set the input attributes and files.
If the data come from stdin, set File to ‘-‘ (a single dash).
Optional AtrLst is described under Input File Attributes.
If this option is not given, stdin is assumed.
Example:
$ aq_ord ... -f,+1l,eok file1 -f file2 ...
-d ColSpec [ColSpec ...] or -d [SepSpec] ColSpec [[SepSpec] ColSpec ...]Define the columns in the input records from all the -f specs.
Only needed in Parsed sort mode.
Up to 2048 ColSpec can be defined (excluding X type columns).
ColSpec has the form Type[,AtrLst]:ColName.
Supported Types are:
S - String.F - Double precision floating point.L - 64-bit unsigned integer.LS - 64-bit signed integer.I - 32-bit unsigned integer.IS - 32-bit signed integer.IP - v4/v6 address.X[Type] - marks an unwanted input column.
Type is optional. It can be one of the above (default is S).
ColName is also optional. Such a name is simply discarded.Optional AtrLst is used in conjunction with the input file attributes
to determine how column data are to be extracted from the input.
It is a comma separated list containing:
n=Len - Extract exactly Len source bytes. Use this for a fixed
length data column.esc - Denote that the input field uses ‘\’ as escape character. Data
exported from databases (e.g. MySQL) sometimes use this format. Be careful
when dealing with multibyte character set because ‘\’ can be part of a
multibyte sequence.clf - Denote that the input field uses these encoding methods:HH is the hex value of
the byte.noq - Denote that the input field is not quoted. Any quotes in or around
the field are considered part of the field value.hex - For numeric type. Denote that the input field is in hexdecimal
notation. Starting 0x is optional. For example, 100 is
converted to 256 instead of 100.trm - Trim leading/trailing spaces from input field value.lo, up - For S type. Convert input field to lower/upper case.ColName is case insensitive. It can have up to 31 alphanumeric and ‘_’
characters. The first character must not be a digit.
The alternate column definition involving SepSpec is designed for
input data that have multibyte separators and/or varying separators from
field to field. In these cases, all the separators must be individually
specified. SepSpec has the form SEP:SepStr where SEP
(case insensitive) is a keyword and SepStr is a literal separator of one
or more bytes. A SepSpec is generally needed between two adjacent
ColSpec unless the former column has a length spec.
Example:
$ aq_ord ... -d s:Col1 s,lo:Col2 i,trm:Col3 ...
trm
attribute removes blanks around the value before it is converted to
an internal number.$ aq_ord ... -d sep:' [' s:time_s sep:'] "' s,clf:url sep:'"' ...
-sort[,AtrLst] ColTyp:ColNumDefine the Raw sort mode sort column.
ColTyp specifies the sort column’s data type. See -d for a list of
types,``X`` is not supported.
ColNum specifies the column number (one-based) of the sort column in
each row.
Optional AtrLst is a comma separated list containing:
dec - Sort in descending order. Default order is ascending.
Descending sort is done by inverting the ascending sort result.Example:
$ aq_ord ... -sort s:2
-sort[,AtrLst] ColName [ColName ...]Define the Parsed sort mode sort columns.
ColNames must already be defined under -d.
Optional AtrLst is a comma separated list containing:
dec - Sort in descending order. Default order is ascending.
Descending sort is done by inverting the ascending sort result.Example:
$ aq_ord ... -d i:Col1 s:Col2 ... -sort Col2 Col1
[-o[,AtrLst] File] [-c ColName [ColName ...]]Output data rows.
Optional “-o[,AtrLst] File” sets the output attributes and file.
If File is a ‘-‘ (a single dash), data will be written to stdout.
Optional AtrLst is described under Output File Attributes.
In the Raw sort mode, most output attributes have no effect since
the records are not altered (only their order).
The -c option is not applicable either.
In the Parsed sort mode,
optional “-c ColName [ColName ...]” selects the columns to output.
ColName refers to a column in the data set.
Without -c, all columns are selected by default.
If -c is specified without a previous -o, output to stdout is
assumed.
Multiple sets of “-o ... -c ...” can be specified.
Example:
$ aq_ord ... -d s:Col1 s:Col2 s:Col3 ... -o,esc,noq - -c Col2 Col1
If successful, the program exits with status 0. Otherwise, the program exits with a non-zero status code along error messages printed to stderr. Applicable exit codes are:
Each input option can have a list of comma separated attributes that control input processing.
Positioning the start of input:
+Num[b|r|l] - Specifies the number of bytes (b suffix), records (r
suffix) or lines (l suffix) to skip before processing.
Line is the default.Error handling:
eok - Make input error non-fatal. If there is an input parse error,
program will try to skip over bad/broken record. If there is an input data
processing error, program will just discard the record.qui - Quiet; i.e., do not print any input error message.Input formats - these attributes are mutually exclusive except for
sep=c and csv that can be used together:
sep=c or sep=\xHH - Input is in ‘c’ (single byte) separated value
format. ‘\xHH’ is a way to specify ‘c’ via its HEX value HH.csv - Input is in CSV format. This is the only format that supports
quoted data fields. Although CSV implies comma separated,
sep=c can be used to override this.fix - Input columns are all fixed width without any separator.
Individual column widths are set in the n=Len
column spec attribute.tab - Input is in HTML table format. Columns must be enclosed in
“<td>data</td>” or “<td ...>data</td>” and rows must be terminated
by a “</tr>”.bin - Input is in aq_tool’s internal binary format.These are used in conjunction with the column spec attributes:
esc - ‘\’ is an escape character in input fields.noq - No quotes around fields in csv format.If no input format attribute is given, csv is assumed.
Each output option can have a list of comma separated attributes:
notitle - Suppress the column name label row from the output.
A label row is normally included by default.app - When outputting to a file, append to it instead of overwriting.sep=c or sep=\xHH - Output in ‘c’ (single byte) separated value
format. ‘\xHH’ is a way to specify ‘c’ via its HEX value HH.csv - Output in CSV format. Strings will be quoted. The default
separator is comma, but sep=c can be used to override this.bin - Output in aq_tool’s internal binary format.esc - Use ‘\’ to escape the field separator, ‘”’ and ‘\’ (non binary).noq - Do not quote string fields in csv format.fmt_g - Use “%g” as print format for F type columns. Only use this
to aid data inspection (e.g., during integrity check or debugging).If no output format attribute is given, csv is assumed.