loginf

Log analyzer

Synopsis

loginf [-h] Global_Opt Input_Spec Output_Spec

Global_Opt:
    [-test] [-verb] [-bz ReadBufSiz]

Input_Spec:
    [-f[,AtrLst] File [File ...]] [-lim Num]
    [-f_raw File [File ...]]

Output_Spec:
    [-o File]
    [-o_raw File]
    [-o_pp_col File]

Description

loginf is used to collect data characteristics related information from the input. It can process log files and/or previously generated raw result files as input. On output, it produces a data characteristics report in a JSON text format and/or a raw result format. Characteristics compiled include row counts, column counts, column min/max length/value, column type guesses, column uniquness estimates, and so on.

Options

-test

Test command line arguments and exit.

  • If all specs are good, the exit code will be 0.
  • If there is an error, the exit code will be non-zero. Usually, an error message will also be printed to stderr.
-verb
Verbose - print program progress to stderr while processing. Usually, a marker is printed for each 10,000,000 records processed.
-bz ReadBufSiz
Set input buffer length. It is also the maxium record length. If a record exceeds this length, it is considered broken and will cause the program to abort or the record to be discarded. Default length is 64KB. Use this option if a longer record is expected. ReadBufSiz is a number in bytes.
-f[,AtrLst] File [File ...] [-lim Num]

Set the input attributes and log files to analyze. If the data come from stdin, set File to ‘-‘ (a single dash). Optional AtrLst is described under Input File Attributes. If no -f or -f_raw is given, log from stdin is assumed.

The -lim option sets the maximum number of records to load from each input file to Num.

Example:

$ loginf ... -f file1 file2 ...
  • Load and analyze logs file1 and file2.
-f_raw File [File ...]

Set the input raw result files to load. Files must be previously generated by this program via the -o_raw option. If the data come from stdin, set File to ‘-‘ (a single dash).

Example:

$ loginf ... -f_raw file1.raw file2.raw ...
  • Load and combine file1.raw and file2.raw.
$ loginf ... -f file3 file4 -f_raw file1.raw file2.raw ...
  • Load and combine file1.raw and file2.raw, then further load and analyze logs file3 and file4 and combine all the results together.
-o File

Output a text report of the result. Report is written in JSON format. If File is a ‘-‘ (a single dash), data will be written to stdout. Note that the file will be overwritten if it contains any data. If no -o, -o_raw or -o_pp_col is given, a report will be written to stdout.

Example:

$ loginf ... -f file1 ... -o file1.report
  • Save the JSON report to file1.report.
-o_raw File

Output raw result. This raw result can be used in a later run using the -f_raw option. If File is a ‘-‘ (a single dash), data will be written to stdout.

Example:

$ loginf ... -f file1 ... -o_raw file1.raw -o file1.report
  • Save raw result to file1.raw and a report of the same result to file1.report.
-o_pp_col File

Output aq_pp column spec based on the charasteristics of the processed data. The output is line oriented, with one column spec per line. If File is a ‘-‘ (a single dash), data will be written to stdout.

Example:

$ loginf ... -f file1 -lim 1000 ... -o_pp_col file1.col
  • Analyze the first 1000 records in file1 and output aq_pp column spec to file1.col.

Exit Status

If successful, the program exits with status 0. Otherwise, the program exits with a non-zero status code along error messages printed to stderr. Applicable exit codes are:

  • 0 - Successful.
  • 1 - Memory allocation error.
  • 2 - Command option spec error.
  • 3 - Initialization error.
  • 11 - Input open error.
  • 12 - Input read error.
  • 13 - Input processing error.
  • 21 - Output open error.
  • 22 - Output write error.

Input File Attributes

Each input option can have a list of comma separated attributes:

  • notitle - The first record from the input is not a label line.
  • csv - Input is in CSV format. This is the default.
  • sep=c or sep=\xHH - Input is in ‘c’ (single byte) separated value format. ‘xHH’ is a way to specify ‘c’ via its HEX value HH. Note that sep=, is not the same as csv because CSV is a more advanced format.
  • auto - Determine input data format automatically. Supported formats are:
    • Delimiter-separated columns. May not work if the number of columns in not fixed.
    • Blank padded fixed-width columns. Individual columns can be left or right adjusted (but not both on the same column).
    • JSON, detection only, no further analysis.
    • XML, detection only, no further analysis.
    • Default to a line separated format with a single column.
  • +Num[b|l] - Specifies the number of bytes (b suffix) or lines (no suffix or l suffix) to skip before processing.

See Also

  • aq_pp - Record preprocessor
  • udbd - Udb server
  • aq_udb - Udb server interface