Replaces: | V1.2.3 |
---|---|
Revision: | V1.2.4-0 2016-03-10 |
Revision: | V1.2.4-1 2016-06-28 |
Revision: | V1.2.4-2 2016-08-09 |
Revision: | V1.2.4-3 2016-09-06 |
Revision: | V1.2.4-4 2016-09-29 |
Revision: | V1.2.4-5 2016-10-26 |
Revision: | V1.2.4-6 2016-12-06 |
Revision: | V1.2.4-7 2017-04-17 |
Revision: | V1.2.4-8 2017-05-23 |
Revision: | V1.2.4-9 2018-05-11 |
The following sections list the category/command specific changes. There are 4 kind of changes:
New features:
aq_pp
can take data from Udb as its input as in
aq_pp -exp Udb_export_options -- ...
where the standard -f
option
has been replaced by a set of Udb export options.-f,jsn
, -f,xml
and -f,div
respectively.-o,jsn
.aq_command ... -o,aq - -c c3 c9 c1 | aq_command -f,aq - ..
,
no column spec is needed in the second command.-c
can be negated. For example,
-c ~c5 ~c8
will exclude c5
and c8
.-c
can have a format spec as in
-d ... i:c1 ... -c c1+0x%x
. For numeric columns only.RxReplace()
is designed to perform generic substring replacement.aq_udb -crt ...
without importing.-del_row
and -del_key
during an aq_udb
export/count/scan operation.Incompatible changes:
-f File1 File2 ... -f File3 ...
–> -f File1 File2 File3 ... ...
-o File -notitle
–> -o,notitle File
-evlc
–> -eval
-f,noq
–> -f,sep=,
(add esc
as needed)-o,noq
–> -o,sep=,
(add esc
as needed)-f ... -d s:c1 s,noq:c2 ...
–> -f,div
with custom separator specs-maprx ...
–> -map,rx ...
-mapfrx ...
–> -mapf,rx ...
-imp ... -seg N1[-N2]/N
–> -imp,seg=N1[-N2]/N ...
-imp ... -nobnk
–> -imp,nobnk ...
-imp ... -nonew
–> -imp,nonew ...
-exp_usr
–> -exp DbName
-cnt_usr
–> -cnt DbName
-scn_usr
–> -scn DbName
-ord_all
–> -ord DbName
-clr_all
–> -clr DbName
-dec
–> -ord,dec
or -sort,dec
+add
attribute cannot be used on a string column.aq_sess
, logcnv
and objcnv
have been removed from the package.Changes to be enforced in a future release:
-udb ... -imp ...
–> -imp[,AtrLst] ...
(-udb
not needed)-ddef ... -imp ...
–> -imp,ddef ...
-map
, -mapf
and -mapc
–> RxMap()
function-kenc
and -kdec
–> KeyEnc()
and KeyDec()
functionsKDec()
–> KeyDec()
QryParmExt()
–> QryDec()
-pp ... -end_of_scan DestSpec ...
–> -pp,post=DestSpec ...
-lim_usr
–> -lim_key
next_bucket
and proc_bucket
–> next_key
and proc_key
Cf: aq_cat, aq_cnt, aq_ord, aq_pp, aq_udb, etc.
[Bug] 1.2.4-9:
The key exchange mechanism during client/server connection initialization
would failed if key could not be sent/received in a single message. This
led to occassional “auth failed” problems. Revised code to handle broken up
keys.
[Bug] 1.2.4-8:
There was a bug in the “-filt” expression parser. It affects aq_pp and Udb
when the negation group !()
is used in these ways:
!( anything )
- e.g., !(col==5)
(!( anything ))
- e.g., (!(col==5))
(!( anything )) operator anything
- e.g., (!(col==5))&&1
anything operator (!( anything ))
- e.g,, 1&&(!(col==5))
These constructions can be used to circumvent the problem if necessary:
!( anything ) operator anything
- e.g, !(col==5)&&1
, !(col==5)||0
anything operator !( anything )
- e.g, 1&&!(col==5)
, 0||!(col==5)
[Critical] 1.2.4-1/3:
Some depreciated options no longer supported. Use the newer specs:
-o File -notitle
–> -o,notitle File
-evlc
–> -eval
[Critical] 1.2.4-1:
The -bz
option has been replaced by an bz=BufSize
attribute of the
-f
option. In fact, each input related option can have its own
bz
attribute. This option is rarely needed since the default is usually
sufficient.
-bz 100 -f File
–> -f,bz=100 File
[Critical] 1.2.4-1:
Some aspects of the -f
(input source) option have changed. They were
changed to accommodate several new features.
-f
no longer supported. All inputs must be specifed under the
same -f
option. That is,-f File1 File2 ... -f File3 ...
–> -f File1 File2 File3 ... ...
-f
should be specified before any -d
(column spec) options.
This is because column spec interpretation may depend on the data format
chosen in the -f
attributes. For example, JSON and XML formats
require extended column specs.div
attribute is required to process data in the old logcnv
‘s
format. That is,-f ... -d ... SEP:"sep1" ...
–> -f,div ... -d ... SEP:"sep1" ...
[Critical] 1.2.4-1:
The noq
attribute of -f
, -o
and individual column spec has been
removed because it produces the wrong result on CSV data. Use one of these
approaches instead:
-f,noq
–> -f,sep=,
(add esc
as needed)-o,noq
–> -o,sep=,
(add esc
as needed)-f ... -d s:c1 s,noq:c2 ...
–> -f,div
with custom separator specs[Warn] 1.2.4-1:
The error messages of most commandline option/parameter specification errors
have changed.
[New] 1.2.4-1/2:
New input option attributes (apply to options like -f
, -cat
,
-sub
and so on):
jsn
- Input is in JSON format.xml
- Input is in XML format.aq
- The input is generated by another aq_tool outputting in aq
format. This is a special format that contains an embedded column spec.
For this reason, no column spec is needed (nor accepted).div
- Select a format that used to be handled by logcnv
.bz=BufSize
- Replaces the old -bz
option.nox
- Reject records with more fields than the column spec.
For separator delimited format and HTML table format only.eok[=Num[/Rows]]
- New optional parameter Num
or Num/Rows
.
Num
sets the number of errors per file to allow.
Num/Rows
allows Num
errors every Rows
rows.qui[=Num]
- New optional parameter Num
. It sets the number of
error messages to print for each input file before becoming quiet.[New] 1.2.4-1:
New output option attributes (apply to options like -o
, -ovar
and so on):
aq
- Output using an internal binary output format.jsn
- Output each record as an JSON object.nodelay
- Output each record as soon as possible instead of
waiting until the output buffer is full.[New] 1.2.4-1/7:
New evaluation functions:
RxCmp()
works like PatCmp()
with a RegEx attribute.RxMap()
works like the -map,rx
and -mapf,rx
options.RxReplace()
and RxRep()
are pattern replacement functions.StrIndex()
finds the position of a substring in another string.QryDec()
works like the the old QryParmExt()
function
but with a revised argument schematics.KeyEnc()
and KeyDec()
works like the -kenc
and
-kdec
options.UrlDec()
and Base64Dec()
are Web related decoding functions.ToUpper()
and ToLower()
are simple ASCII test conversion functions.Set()
sets a column’s value. Unlike a standard -eval
, the target
column here is determined at runtime during each evaluation (it is taken
from a string argument).AgentToUId()
converts an agent string to an RTmetrics user ID.UNameHash()
converts a string (usually an user name) to an RTmetrics
name hash.[New] 1.2.4-4:
Output column selection (-c
) can accept columns not to include.
For example, -c ~c5 ~c8
will remove c5
and c8
from the default
output column set.
[New] 1.2.4-4:
Output columns specified under -c
can have a format spec as in
-d ... i:c1 ... -c c1+0x%x
. For numeric columns only. Everything
following the +
(plus sign) is a format string that will be passed to
the C library’s printf
function. The user must specify the right format;
otherwise, the program may crash.
Note: This is an experimental option. It will not be documented until its design and usability have been confirmed.
[New] 1.2.4-7/9:
Updated/new evaluation functions:
RxReplace()
now updates the output column even when no replacement
is done. The old behavior was to leave the output column unchanged.RxRep()
is similar to RxReplace()
, except that it returns the
result directly.AgentToUId()
converts an agent string to an RTmetrics user ID.UNameHash()
converts a string (usually an user name) to an RTmetrics
name hash.IConv()
can choose the best conversion to return. In this way, eok
can output a blank. A FromCode
of -
(a dash) can explicitly output
a blank as fallback.Cf: aq_pp
See also common changes.
[Critical] 1.2.4-1:
The -maprx
and -mapfrx
options are no longer supported. Use the
rx
attribute instead:
-maprx ...
–> -map,rx ...
-mapfrx ...
–> -mapf,rx ...
Furthermore, the mapping related options will be depreciated soon.
Use the new RxMap()
function instead.
[Critical] 1.2.4-1:
Several Udb import related commandline options have changed:
-spec SpecFile -imp TabName
–> -imp,spec=SpecFile DbName:TabName
-db DbName -imp TabName
–> -imp DbName:TabName
-imp ... -seg N1[-N2]/N
–> -imp,seg=N1[-N2]/N ...
-imp ... -nobnk
–> -imp,nobnk ...
-imp ... -nonew
–> -imp,nonew ...
[Warn] 1.2.4-1:
The -udb
option is depreciated. The extended -imp
option alone
is sufficient.
-udb ... -imp ...
–> -imp[,AtrLst] ...
[Warn] 1.2.4-1:
The -ddef
option is depreciated. Use the new ddef
attribute on
-imp
instead:
-ddef ... -imp ...
–> -imp,ddef ...
[New] 1.2.4-1/3:
The -imp
(Udb import) option can now have attributes, as in
-imp[,AtrLst]
:
spec=SpecFile
, seg=N1[-N2]/N
, nobnk
, nonew
, ddef
-
They replace the old -spec
, -seg
, -nobnk
, -nonew
and
-ddef
options respectively.noold
- Only import to new keys. It is the opposite of nonew
.nodelay
- Sent each record to Udb as soon as possible instead of
waiting until the output buffer is full.[New] 1.2.4-1:
aq_pp
can now obtain its main input from an Udb export. To do this,
use the new -exp Udb_export_options --
spec instead of a -f
.
Udb_export_options
represents any of the export related options of
aq_udb (other than the -o
output option). The --
indicates the end of the export spec.
[New] 1.2.4-1:
Additonal information is now available at the end of an Udb import:
aq_pp
will show the combined server
memory usage if -stat
is enabled. Per server usages can be obtained
with -verb
.aq_pp
will show an error message from each failed server.[New] 1.2.4-2:
Added license file checks:
/opt/essentia/essentia.license
/opt/essentia/essentia.sign
Cf: aq_ord
-dec
–> -sort,dec
See also common changes.
[Bug] 1.2.4-1:
udbd
(script) has a hardcoded limit on the number of Udb servers it
would handle. This limit was set too low (32). Extended it yto 70.
Note that this is only a soft limit. To manage more servers,
run udbd
several times, each time on a different range of ports.
[Bug] 1.2.4-1:
Any Udb action immediately following a broken pipe may produce
unpredictable result. An example is:
$ aq_udb -exp mydb:mytable | head -1 ; aq_udb -exp mydb:mytable
The head
command will cause a broken pipe; for this reason, the second
export may not produce the correct result.
[Bug] 1.2.4-2:
When a Udb module source is supplied with the aq_udb
or aq_pp
command, the resulting module will get truncated if it is greater than 64K
byte in size (the truncated size can be less than 64K).
[Bug] 1.2.4-6:
The server normally writes log/error messages to a log file. If the server
cannot open the file (e.g., due to file permission problem), the old
behavior was to ignore the error and log to stdout/stderr implicitly.
But this caused a problem when the server was being started by a
ssh
command - the command will not exit until the server closes
stdout/stderr. The new behavior is to print an error message and quit.
[Critical] 1.2.4-1:
The action specification option - -exp
, -cnt
, -scn
, -ord
,
-clr
and -probe
- must be specified before options that depend
on it. For example, -sort
is only valid for export,
so an -exp
must be given first. If in doubt, follow the
aq_udb synopsis.
[Critical] 1.2.4-1:
Some depreciated options no longer supported. Use the newer specs:
-spec SpecFile -Action TabName
–> -Action,spec=SpecFile DbName:TabName
-db DbName -Action TabName
–> -Action DbName:TabName
-exp_usr
–> -exp DbName
or -exp,spec=SpecFile DbName
-cnt_usr
–> -cnt DbName
or -cnt,spec=SpecFile DbName
-scn_usr
–> -scn DbName
or -scn,spec=SpecFile DbName
-ord_all
–> -ord DbName
or -ord,spec=SpecFile DbName
-clr_all
–> -clr DbName
or -clr,spec=SpecFile DbName
-dec
–> -ord,dec
or -sort,dec
[Critical] 1.2.4-1:
The -probe
(server check) option now requires a parameter:
-spec SpecFile -probe
–> -probe,spec=SpecFile DbName
[Critical] 1.2.4-2:
The -sort
(output sorting) option now requires a sort column spec.
[Critical] 1.2.4-5:
Removed +add
attribute support on string columns.
[Warn] 1.2.4-1:
Output column labels in the title line have changed. This only happens when
the columns come from more than one source. For example, if table colums are
exported along with vector columns and var columns, the labels will appear
like this:
"col1","col2","vectorX.col1","vectorX.col2","var.col1","var.col2"
In older versions, the labels would be indistinguishable:
"col1","col2","col1","col2","col1","col2"
[Warn] 1.2.4-3:
The -end_of_scan DestSpec
option for -pp
is depreciated.
Use the new post
attribute on -pp
instead:
-pp ... -end_of_scan DestSpec ...
–> -pp,post=DestSpec ...
[New] 1.2.4-1:
The -probe
(server check) option will show the combined server
memory usage if -stat
is enabled. Per server usages can be obtained
with -verb
.
[New] 1.2.4-2:
Per key row count of a table can be obtained from the new
RowCount(TabName)
evaluation function.
The count is stored as part of the key specific data, no row scan is involved.
For vectors where the row count is always 1, the function returns 1
if the row has been initialized, 0 otherwise.
[New] 1.2.4-2:
The Udb primary key has been generalized:
[New] 1.2.4-2:
Each Udb server can now handle more than one databases at a time.
Databases are identified by their names. Their data are
stored independently. But they share a common string hash for efficiency.
The database name is obtained from the commandline - e.g., mydb
will
be the database name in these commands:
$ aq_pp ... -imp mydb:mytable ...
$ aq_udb -exp mydb:mytable ...
[New] 1.2.4-2:
udbd
will apply a default memory limit of (SystemTotal - 500M)
when
starting Udb servers. This can be overriden with the -mem
or -memx
option of udbd
.
[New] 1.2.4-3:
Support explicit DB creation with the new -crt
aq_udb
option.
[New] 1.2.4-3:
Support key-only import, no table/vector data needed.
This is useful for a DB that has no table or vector.
Example usage:
$ aq_pp ... -imp mydb ...
$ aq_udb -exp mydb ...
[New] 1.2.4-3:
Delete is now supported during an export/scan/count operation.
-del_row
will delete the current row.-del_key
will delete the current key and its associated data.aq_sess
has been retired.logcnv
has been retired. Its functionality is now supported by all
aq_*
commands. Use the div
input attribute to enable it.
For example,logcnv -f[,AtrLst] ...
–> aq_pp -f,div[,AtrLst] ...
loginf
crash that happens when any record from the log is greater
than 64KB long.objcnv
has been retired. Its functionality is now supported by all
aq_*
commands. Use the jsn
or xml
input attribute to enable it.
For example,objcnv -jsn -f[,AtrLst] ...
or objcnv -f,jsn[,AtrLst] ...
–>
aq_pp -f,jsn[,AtrLst] ...