Change Log – V2.0.1

You are here:
← All Topics

Replaces: V2.0.0
Revision: V2.0.1-0 2019-07-02
Revision: V2.0.1-1 2019-08-05
Revision: V2.0.1-2 2019-08-14
Revision: V2.0.1-3 2020-02-26

Introduction

The following sections list the category/command specific changes. There are 4 kind of changes:

  • [New]
    Describe a new feature.
  • [Warn]
    An existing feature has changed. The old usage is still supported; however, applications should switch to the new usage as soon as possible. The old usage will be discontinued in the next release.
  • [Critical]
    An existing feature has changed in a non backward compatible manner. Applications must be ported to the new usage in order to use the new version.
  • [Bug]
    A bug was discovered and fixed.

Summary

Incompatible changes:

  • The RxMap() function’s default behavior has changed. Specifically, when the input does not match the given pattern, the old function would return without changing the output columns’ values. That was a design error. Now, the default behavior is to update the output columns’ values anyway. The old behavior can still be selected by setting the any attribute (new feature).
  • The % (remainder) evaluation operator has been extended to support floating-point operands. The results from the old and new versions may be different if one or more operands are floats. For example, 3.3 % 2.2 yields 1 in the old version and 1.1 in the new one.
  • The all attribute of aq_pp -sub has been removed. It had never worked as intended anyway (due to a design bug).
  • The lookup-file column specs of aq_pp -sub|-grep must use the X[Type] spec to mark irrelevant columns explicitly. The old design would automatically exclude unknown columns, but that was not good.
  • The descending sort of aq_ord and aq_udb may return records having the same keys in a different order as before.

Bugs:

  • Input columns that are implicitly excluded may not be parsed correctly.
  • loginf … -o_pp_col may output duplicated column names.
  • The all attribute of aq_pp -sub did not work as intended due to a design bug.
  • Fixed a bug in the -f servers@DbName feature when parsing the Udb spec file.
  • Fixed a bug in aq_rst that causes the program to crash when there are duplicated labels.
  • Fixed a bug in the column name parser when column names are used in evaluation/filter expressions.
  • Fixed a filter expression parser bug that printed an unintended error message when ! and () were used together in certain ways – e.g., -filt ‘(!0).
  • Fixed a filter evaluation bug that would produce unpredictable results when certain failures occurred during the evaluation – e.g., -filt ‘toi(“x”)>0, -filt ‘toi(“x”)>0 and so on.

New features:

  • loginf can optionally output column names containing multi-byte UTF8 characters.
  • New date comparison functions DateDiff() and DateDiff2().
  • Enhanced DateToTime() and related functions to parse the %Z (timezone name) info. The functions can also take an optional UTC time offset argument.
  • Enhanced the RxMap() function to handle multiple map-from inputs.
  • New mapping function PatMap() that works like RxMap() but uses RT-style mapping.
  • New IP comparison function IPCmp().
  • New built-in evaluation variables $CurSec, $CurMSec and $CurUSec for the current time.
  • Automatically strip BOM mark at the beginning of each input.
  • Support outputting a BOM. Can also output PC-style <CR><LF> .
  • Support fixed output column width with -c … ColName,n=[-]Width …. Also added a new fix output type for the all-fixed-width-columns cases.
  • aq_cnt supports group-by columns via its new -g option.
  • aq_udb can shuffle the key bucket’s or table row’s order via its new -shf option.
  • aq_pp has new -d0 and -var0 options that initialize a column or variable to a blank or zero.
  • Extended aq_pp -sub to support multiple output (substitute-to) columns of arbitrary types.
  • Added a new aq_rst command that restructures the input dataset by converting it from a flattened layout to a structured one.
  • Added a -c LabName … option for aq_rst to handle predefined column labels.
  • Removed column name restriction on the label names when they are output in the title line.
  • aq_ord and aq_udb support outputting a sub-range of records with -top and -last.
  • Column names can contain any characters, even dots.
  • Support \uHHHH[\uHHHH] escape sequence for UTF8 characters in string constants.
  • A new AQ_TOOL_RX environment can be used to set the default regular expression attributes.
  • -filt following -if or -elif in aq_pp and aq_udb becomes optional.
  • Various sort features support mixed descending/ascending order in multi-columns sort.
  • Support unary !, ~, + and – operators in evaluation expressions.

Future Changes

Changes to be enforced in the next release:

  • None at this time. But consider making the changes listed below now.

Changes to be enforced in the next next release:

  • Common:
    • Function GmDateToTime(DateStr, “DateFmt”) –> DateToTime(DateStr, “DateFmt”, GMT)
  • aq_pp:
    • -alias will be removed.
    • -udb … -imp … –> -imp[,AtrLst] … (-udb not needed)
    • -ddef … -imp … –> -imp,ddef …
    • -sub will no longer treat “to” as a special column name in the lookup-table column spec. Use the actual column name of the substitute-to target column in the main dataset instead.
  • aq_udb:
    • -pp … -end_of_scan DestSpec … –> -pp,post=DestSpec …
    • -lim_usr –> -lim_key
    • next_bucket and proc_bucket –> next_key and proc_key

Changes to be enforced later:

  • Common:
    • Function KDec() –> KeyDec()
    • Function QryParmExt() –> QryDec()
  • aq_pp:
    • -map/-mapf and -mapc combination –> PatMap() function
    • -map,rx/-mapf,rx and -mapc combination –> RxMap() function
    • -kenc and -kdec –> KeyEnc() and KeyDec() functions

Common

Cf: aq_cat, aq_cnt, aq_ord, aq_pp, aq_udb, etc.

  • [Bug] 2.0.1-0:
    Input columns that are implicitly excluded may not be parsed correctly. Specifically, if the column spec is -d s@3:FirstCol, then columns 1 to 3 will be parsed correctly, but column 4 and beyond may not.
  • [Bug] 2.0.1-1:
    A bug in the Udb spec file parser caused the -f servers@DbName feature to fail if the server spec does not contain a |.
  • [Bug] 2.0.1-2:
    aq_rst may crash when the input dataset contains duplicated labels under the same key.
  • [Bug] 2.0.1-2:
    There was a bug in the evaluation/filter expressions column name parser. It reject any column names containing the 9, z and Z characters.
  • [Bug] 2.0.1-3:
    There was a bug in the evaluation/filter expressions column name parser. It cannot recognize column names that start with a number or IP address.
  • [Bug] 2.0.1-3:
    There was a filter expression parser bug that would print an error message even when there was no problem. This happened when the unary ! operator and the () group were used together in certain ways. For example, -filt ‘!0 was fine, but -filt ‘(!0) would trigger an unintended error message.
  • [Bug] 2.0.1-3:
    There was a filter evaluation bug that would produce unpredictable true/false results when there was a failure during the evaluation. For example, -filt ‘toi(“x”)>0 and -filt ‘toi(“x”)>0 yielded true, but -filt ‘toi(“x”)>=0 and -filt ‘toi(“x”)>=0 yielded false. Any failure during the evaluation now causes the evaluation to abort.
  • [New] 2.0.1-0:
    Updated/new evaluation functions:
    • Extended DateToTime() to handle %Z in the format spec. It can also take an optional UTC time offset argument.
    • New DateDiff() and DateDiff2() compare dates and return the differences (usually in seconds).
    • Extended RxMap() to handle multiple map-from inputs.
    • New PatMap() works like RxMap(), but uses the RT-style mapping.
    • New IPCmp() compares an IP value with one or more constant IP/mask.
  • [New] 2.0.1-0:
    New built-in evaluation variables:
    • $CurSec, $CurMSec and $CurUSec represent the current time in seconds, milliseconds and microseconds respectively.
  • [New] 2.0.1-0:
    Automatically detect and strip BOM mark at the beginning of each input. This behavior can be suppressed with a nobom attribute in the input option (e.g., -f,nobom).
  • [New] 2.0.1-0:
    Support fixed output column width with -c … ColName,n=[-]Width …. The column value is normally left adjusted. If Width has a – in front, then the value will be right adjusted. This can be done on a per column basis. For cases where all the columns have fixed widths without any column separators, use the new fix output type.
  • [New] 2.0.1-0:
    New output attributes:
    • bom for adding a BOM to the beginning of output.
    • cr for using PC-style <CR><LF>.
    • pc to enable both bom and cr.
  • [New] 2.0.1-0:
    aq_ord and aq_udb support outputting a sub-range of records with -top|-last [Start:]Num. Start is 1 by default, meaning the first or the last record.
    • -top – output records Start to Start + Num – 1 from the top.
    • -last – output records Start + Num – 1 to Start from the end.
  • [New] 2.0.1-1:
    Column name can contain any characters, even dots. Use this feature with care since commands containing multi-byte characters are more difficult to compose and debug. A dot can cause conflict with the TabName.ColName convention.
  • [New] 2.0.1-1:
    Support \uHHHH[\uHHHH] escape sequence for UTF8 characters in string constants.
  • [New] 2.0.1-1:
    A new AQ_TOOL_RX environment can be used to set the default regular expression attributes.
  • [New] 2.0.1-1:
    The -filt option is now optional following -if or -elif in aq_pp and aq_udb; i.e., aq_pp … -if -filt ‘Condition’ … -endif can simply be specified as aq_pp … -if ‘Condition’ … -endif.
  • [New] 2.0.1-3:
    Raised the internal name lengths (column name, table name, etc) limit from 31 to 127 bytes so that the 31 characters limit mentioned in our documents is more correct.
  • [New] 2.0.1-3:
    Updated/new evaluation functions:
    • Enhanced ToIP(), ToF() and ToI() to allow spaces around string type input values.
    • New Trim() function to trim spaces around a string.
    • New StrV*() functions to handle pseudo string arrays. The pseudo array is encoded as a string.
  • [New] 2.0.1-3:
    Support unary !, ~, + and – operators in evaluation expressions.

aq_ord

Cf: aq_ord

  • See also common changes.
  • [New] 2.0.1-3:
    Then dec attribute of -sort can now be assigned on a per-column basis. For example, -sort Col1,dec Col2,dec Col2,dec (the same meaning as -sort,dec Col1 Col2 Col2), -sort Col1,dec Col2 Col2,dec (mixed-order multi-column sort, not supported before).

aq_pp

Cf: aq_pp

  • See also common changes.
  • [New] 2.0.1-0:
    A new -d0 option can be used to add new columns that are automatically initialized to a blank or zero. A similar -var0 option does the same thing for new variables.
  • [New] 2.0.1-0:
    The -sub option now supports substituting into arbitrary target columns with arbitrary types.
  • [New] 2.0.1-0:
    Extended -sub to support multiple output (substitute-to) columns of arbitrary types. The lookup table’s column spec must be specified to use this feature since the output columns are deduced from it.

aq_cnt

Cf: aq_cnt

  • See also common changes.
  • [New] 2.0.1-0:
    A new -g option that sets the group-by columns for all the keys to be counted.

aq_rst

Cf: aq_rst

  • [New] 2.0.1-0:
    This is a new command. It restructures the input dataset by converting it from a flattened layout to a structured one.
  • [New] 2.0.1-0:
    Added -c LabName … option to support predefined label extraction.
  • [New] 2.0.1-1:
    No column name restriction on the label names when they are output in the title line.

aq_udb/udb server

Cf: aq_udb, udbd

  • See also common changes.
  • [New] 2.0.1-0:
    A new -shf option that tells the Udb server to shuffle the order of the key buckets or the order of the table rows in each key bucket stored in the server’s memory.
  • [New] 2.0.1-3:
    Then dec attribute of -ord and -sort can now be assigned on a per-column basis. For example, -sort Col1,dec Col2,dec Col2,dec (the same meaning as -sort,dec Col1 Col2 Col2), -sort Col1,dec Col2 Col2,dec (mixed-order multi-column sort, not supported before). The same for the -ord option.

loginf

Cf: loginf

  • See also common changes.
  • [Bug] 2.0.1-0:
    loginf did not enforce column name uniqueness when determining (or suggesting) column names. As a result, duplicated columns could appear in its outputs. The fix is to replace duplicated names with col_Num where Num is the column number (one-based).
  • [New] 2.0.1-0:
    New –utf8ok can be used to output column names containing multi-byte UTF8 characters.

mcc.pmod and mcc.umod

Cf: mcc.pmod, mcc.umod

  • [New] 2.0.1-3:
    Support column names that cannot be used as program variable names with the new DECL_COLUMN_STATIC(ProgColName, Type, “RealColName”) declaration.

Last Updated On