Essentia Commands Overview

Reference Manuals

• ess
• ess select
• ess summary
• ess repository
• ess probe
• ess purge
• ess ls
• ess category
• ess file
• ess cat
• ess lsa
• ess cluster
• ess query
• ess server
• ess create
• ess drop
• ess use
• ess stream
• ess exec
• ess udbd
• ess redshift

Working With Categories

To learn more about creating and using categories, see Category Rules.

Overview

Command                   Arguments                                Description

ess version                                                        Print out the current version of Essentia and the AQ tools.

ess select                source                                   Select a data source / repository
                          [--region REGION]
                          [--credentials file]
                          [--aws_access_key publickey]
                          [--account_name AzureAccount]
                          [--aws_secret_access_key secretkey]
                          [--account_key AzureKey]
                          [--label name]
                          [--overwrite]

ess repository                                                     Summarize currently defined data sources / repositories

ess summary               [category]                               Provide a summary of the categories in the current 
                          [pkey|columnspec|schema|delimiter|       datastore, a deep summary of a single category, or a 
                                                                   single parameter of a single category
                      
                          compression|preprocess]
                          [--label name]
                          [--scan]
                          [--short]


ess purge                 label                                    Delete the reference to the datastore (not the 
                                                                   datastore itself)

ess ls                    [pattern]                                List the contents of the source datastore
                          [--exclude subpattern]
                          [--cat category]
                          [--label name]
                          [-r]
                          [--dateregex regex_pattern]
                          [--limit number]
                          [--nameonly]
                          [--nosize]
                          [--nodate]

ess probe                 category                                  Scans a file to determine its compression, file format, etc.
                          [--file filename]
                          [--label name]
                          [--pcmd ProbeCommand]
                          [--size Bytes]

ess category add          name pattern                              Add a category to the datastore
                          [--exclude subpattern]
                          [--dateregex regex_pattern|none]
                          [--dateformat auto|none|custom]
                          [--archive pattern]
                          [--compression type|none]
                          [--delimiter delimiter]
                          [--columnspec NewColumnSpec]
                          [--preprocess command]
                          [--allx]
                          [--alls]
                          [--overwrite]
                          [--label name]
                          [--comment comment]
                          [--noprobe]
                          [--usecache]
                          [--pkey PKEY]

ess category delete       ExistingCategory                           Remove the reference to this category
                          [--label name]

ess category copy         ExistingCategory NewCategory               Create a copy of an existing category reference
                          [--label name]
                          [--credentials file]
                          [--aws_access_key publickey]
                          [--aws_secret_access_key secretkey]

ess category change       columnspec|dateformat|dateregex|           Modify or override details about a category
                          usecache|comment
                          NewSpec|NewFormat|NewRegex|
                          NewCache|NewComment

ess cluster create        [--type NodeType]                          Create worker nodes
                          [--number Number]
                          [--add]
                          [--credentials CREDENTIALS]
                          [--aws_access_key AWS_ACCESS_KEY]
                          [--aws_secret_access_key 
                          AWS_SECRET_ACCESS_KEY]
                          [--error_return_code]

ess cluster add           ReservationIDs                             Add worker nodes to cluster by Reservation ID

ess cluster remove        ReservationIDs                             Remove worker nodes from cluster by Reservation ID

ess cluster terminate     [--all]                                    Shutdown all worker nodes, optionally deleting all
                          [-y]                                       pem files and security groups with or without
                          [--credentials CREDENTIALS]                confirmation
                          [--aws_access_key AWS_ACCESS_KEY]
                          [--aws_secret_access_key 
                          AWS_SECRET_ACCESS_KEY]

ess cluster status                                                   Summarizes the state of all workers and shows their 
                                                                     Reservations IDs

ess cluster stop         [--credentials CREDENTIALS]                 Suspend worker nodes (i.e. stop the EC2 instances)
                         [--aws_access_key AWS_ACCESS_KEY]
                         [--aws_secret_access_key 
                         AWS_SECRET_ACCESS_KEY]

ess cluster start        [--credentials CREDENTIALS]                 Restart suspended worker nodes
                         [--aws_access_key AWS_ACCESS_KEY]
                         [--aws_secret_access_key 
                         AWS_SECRET_ACCESS_KEY]

ess cluster set          local|cloud|custom                          Force essentia into master node only (local) or not.

ess cluster reset                                                    Clear and reset cluster configuration

ess cluster iplist       [--private]                                 List the public and private IP’s of cluster
                         [--public]

ess query                command                                     SQL style query on raw logs
                         [--label name]
                         [--check]

ess stream               category start stop command                 Stream data in given range to given command (cat
                         [--exclude pattern]                         being default)
                         [--master]
                         [--debug]
                         [--bulk]
                         [--threads number]
                         [--archive pattern]
                         [--s3out label:path]
                         [--label name]
                         [--progress]
                         [--limit number]
                         [--quitonerror]

ess exec                 command                                     Execute given command on all worker nodes
                         [--master]
                         [--debug]
                         [--s3out label:path]

ess server reset                                                     Delete all database definitions

ess server restart                                                   Clear database contents, but maintain schemas

ess server commit                                                    Upload database spec files to workers

ess server summary       [--name database]                           Summarize the databases and objects available for 
                                                                     processing

ess create               database|table|vector|variable name spec    Create an object. Name not required for variable

ess drop                 database|table|vector|variable name         Delete the object

ess use                  database                                    Select the given database for queries

ess cat                  filename                                    Read file and print to stdout
                         [--label name]
                         [--decompress]

ess lsa                  filename                                    List the files within an archive file with optional 
                         [--label name]                              filtering.
                         [--pattern]

ess file push            files                                       Send a local file to worker nodes
                         [--dest directory]

ess file get             file|folder                                 Fetch a file from workers to the master

ess file mkdir           name                                        Create a directory on worker nodes

ess redshift register    redshift_cluster_name                       Look up your redshift cluster and find its connection 
                         redshift_database_name                      information
                         username password

ess redshift stream      category startdate stopdate command         Import data from specified category between (and
                         redshift_table_name                         including) specified dates using the specified filter.
                         [--options RedshiftOptions]                 Default is to execute on worker nodes. Data is
                         [--threads number]                          streamed, one file at a time, to the given command
                         [--label name]                              and then into your redshift table
    
ess redshift sql         command                                     Query data in a redshift database using an SQL
                                                                     command

ess redshift gentable    category                                    Create a table in redshift using the column
                         redshift_table_name                         specification derived from an Essentia category
                         --key 'column = distkey'
 
ess redshift list                                                    Get information about all Redshift clusters accessible 
                                                                     by Essentia

ess redshift deregister                                              Unlink redshift cluster from Essentia

ess udbd                 start|stop|restart|status                   Direct manipulation of UDB

ess udbd                 ckmem|cklog                                 Check memory or logs of UDB daemons on workers

Advanced Options

Environment 
Variable                 Default                                     Description

ESS_CACHE_DIR            GUI and RStudio:                            Directory path where repository settings files
                         /var/www/html/mydmp/aws/1/.ess              will be stored. This is used to manage the
                         Command Line: ~/.ess                        local and cloud repositories associated with
                                                                     the instance you are on. The first default is for 
                                                                     files created by the Graphical User Interface 
                                                                     or RStudio and the second default is for files 
                                                                     created via the command line.

ESS_WORK_DIR             All Access Methods: ./.conf                 Directory path where udb settings files will
                                                                     be stored. This is used to manage the UDB 
                                                                     databases and ports associated with the 
                                                                     instance you are on. The default is always a 
                                                                     .conf folder in your current directory.

ESS_AWS_DIR             GUI and RStudio:                             Directory path where pem files will be stored.
                        /var/www/html/mydmp/aws/1/.aws               This is used to manage the computer clusters
                        Command Line: ~/.aws                         associated with the instance you are on. The
                                                                     first default is for files created by the 
                                                                     Graphical User Interface or RStudio and the 
                                                                     second default is for files created via the 
                                                                     command line.