loginf

Synopsis

loginf [-h] Global_Opt Input_Spec Output_Spec

Global_Opt:
    [-test] [-verb] [-bz ReadBufSiz]

Input_Spec:
    [-f[,AtrLst] File [File ...]] [-lim Num]
    [-f_raw File [File ...]]

Output_Spec:
    [-o File]
    [-o_raw File]
    [-o_pp_col File]

Description

loginf is used to collect data characteristics related information from the input. It can process log files and/or previously generated raw result files as input. On output, it produces a data characteristics report in a JSON text format and/or a raw result format. Characteristics compiled include row counts, column counts, column min/max length/value, column type guesses, column uniquness estimates, and so on.

Options

-test

Test command line arguments and exit.

  • If all specs are good, the exit code will be 0.
  • If there is an error, the exit code will be non-zero. Usually, an error message will also be printed to stderr.
-verb
Verbose - print program progress to stderr while processing. Usually, a marker is printed for each 10,000,000 records processed.
-bz ReadBufSiz
Set input buffer length. It is also the maxium record length. If a record exceeds this length, it is considered broken and will cause the program to abort or the record to be discarded. Default length is 64KB. Use this option if a longer record is expected. ReadBufSiz is a number in bytes.
-f[,AtrLst] File [File ...] [-lim Num]

Set the input attributes and log files to analyze. If the data come from stdin, set File to ‘-‘ (a single dash). Optional AtrLst is described under Input File Attributes. If no -f or -f_raw is given, log from stdin is assumed.

The -lim option sets the maximum number of records to load from each input file to Num.

Example:

$ loginf ... -f file1 file2 ...
  • Load and analyze logs file1 and file2.
-f_raw File [File ...]

Set the input raw result files to load. Files must be previously generated by this program via the -o_raw option. If the data come from stdin, set File to ‘-‘ (a single dash).

Example:

$ loginf ... -f_raw file1.raw file2.raw ...
  • Load and combine file1.raw and file2.raw.
$ loginf ... -f file3 file4 -f_raw file1.raw file2.raw ...
  • Load and combine file1.raw and file2.raw, then further load and analyze logs file3 and file4 and combine all the results together.
-o File

Output a text report of the result. Report is written in JSON format. If File is a ‘-‘ (a single dash), data will be written to stdout. Note that the file will be overwritten if it contains any data. If no -o, -o_raw or -o_pp_col is given, a report will be written to stdout.

Example:

$ loginf ... -f file1 ... -o file1.report
  • Save the JSON report to file1.report.
-o_raw File

Output raw result. This raw result can be used in a later run using the -f_raw option. If File is a ‘-‘ (a single dash), data will be written to stdout.

Example:

$ loginf ... -f file1 ... -o_raw file1.raw -o file1.report
  • Save raw result to file1.raw and a report of the same result to file1.report.
-o_pp_col File

Output aq_pp column spec based on the charasteristics of the processed data. The output is line oriented, with one column spec per line. If File is a ‘-‘ (a single dash), data will be written to stdout.

Example:

$ loginf ... -f file1 -lim 1000 ... -o_pp_col file1.col
  • Analyze the first 1000 records in file1 and output aq_pp column spec to file1.col.

Exit Status

If successful, the program exits with status 0. Otherwise, the program exits with a non-zero status code along error messages printed to stderr. Applicable exit codes are:

  • 0 - Successful.
  • 1 - Memory allocation error.
  • 2 - Command option spec error.
  • 3 - Initialization error.
  • 11 - Input open error.
  • 13 - Input processing error.
  • 21 - Output open error.
  • 22 - Output write error.

Input File Attributes

Each input file can have these comma separated attributes:

  • notitle - The first record from the input is not a label line.
  • tsv - Input is tab separated (default is comma separated).
  • sep=c - Use separator ‘c’ (single byte) as column separactor.
  • +Num[b|l] - Specifies the number of bytes (b suffix) or lines (no suffix or l suffix) to skip before processing.

See Also

  • aq_pp - Record preprocessor
  • udbd - User (Bucket) Database server
  • aq_udb - Interface to Udb server