Essentia Commands Reference Manuals

You are here:
← All Topics

ess

usage: ess [-h] [-v] 
           {select,summary,repository,probe,purge,ls,category,file,cat,lsa,cluster,query,server,create,drop,use,stream,exec,udbd,redshift}
           ...

The Essentia ETL Engine

optional arguments:
  -h, --help              show this help message and exit
  -v, --version           show program's version number and exit

Subcommands:
  {select,summary,repository,probe,purge,ls,category,file,cat,lsa,cluster,query,server,create,drop,use,stream,exec,udbd,redshift}
  select                  Choose a datastore
  summary                 Summarize datastore
  repository              Summarize repository
  probe                   Probe category
  purge                   Delete datastore
  ls                      File commands
  category                Category commands
  file                    File transfer with workers
  cat                     dump file contents to stdout
  lsa                     list contents of an archive
  cluster                 manage a cluster of worker nodes
  query                   SQL-like query command
  server                  Modify server files for udbd
  create                  Adds fields to config file
  drop                    Removes fields from config file
  use                     Change active database
  stream                  Import data
  exec                    Execute command
  udbd                    udbd commands
  redshift                Link essentia to redshift
 

ess select

usage: ess select [-h] [--region REGION]
                  [--credentials CREDENTIALS | --aws_access_key AWS_ACCESS_KEY | --account_name ACCOUNT_NAME]
                  [--aws_secret_access_key AWS_SECRET_ACCESS_KEY | --account_key ACCOUNT_KEY]
                  [--label LABEL] [--overwrite]
                  source

Choose a datastore

positional arguments:
source                    Source location

optional arguments:
  -h, --help              show this help message and exit
  --region                REGION Region of datastore
  --credentials CREDENTIALS
                          Credentials file
  --aws_access_key AWS_ACCESS_KEY
                          S3 access key
  --account_name ACCOUNT_NAME
                          Azure account name
  --aws_secret_access_key AWS_SECRET_ACCESS_KEY
                          S3 secret access key
  --account_key ACCOUNT_KEY
                          Azure account key
  --label LABEL           Assign a label to the datastore
  --overwrite             Overwrite credential information
 

ess summary

ess summary pkey

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
                   [category]
                   [{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
  category                category
  {pkey,columnspec,schema,delimiter,compression,preprocess}
                          category parameter to output

optional arguments:
  -h, --help              show this help message and exit
  --label [LABEL]         select a datastore
  --scan                  scan and update stats on datastore
  --short                 return total file count and size only. used with
                          --scan

ess summary columnspec

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
                   [category]
                   [{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
  category                category
  {pkey,columnspec,schema,delimiter,compression,preprocess}
                          category parameter to output

optional arguments:
  -h, --help              show this help message and exit
  --label [LABEL]         select a datastore
  --scan                  scan and update stats on datastore
  --short                 return total file count and size only. used with
                          --scan

ess summary schema

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
                   [category]
                   [{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
  category                category
  {pkey,columnspec,schema,delimiter,compression,preprocess}
                          category parameter to output

optional arguments:
  -h, --help              show this help message and exit
  --label [LABEL]         select a datastore
  --scan                  scan and update stats on datastore
  --short                 return total file count and size only. used with
                          --scan

ess summary delimiter

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
                   [category]
                   [{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
  category                category
  {pkey,columnspec,schema,delimiter,compression,preprocess}
                          category parameter to output

optional arguments:
  -h, --help              show this help message and exit
  --label [LABEL]         select a datastore
  --scan                  scan and update stats on datastore
  --short                 return total file count and size only. used with
                          --scan

ess summary compression

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
                   [category]
                   [{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
  category                category
  {pkey,columnspec,schema,delimiter,compression,preprocess}
                          category parameter to output

optional arguments:
  -h, --help              show this help message and exit
  --label [LABEL]         select a datastore
  --scan                  scan and update stats on datastore
  --short                 return total file count and size only. used with
                          --scan

ess summary preprocess

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
                   [category]
                   [{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
  category                category
  {pkey,columnspec,schema,delimiter,compression,preprocess}
                          category parameter to output

optional arguments:
  -h, --help              show this help message and exit
  --label [LABEL]         select a datastore
  --scan                  scan and update stats on datastore
  --short                 return total file count and size only. used with
                          --scan
 

ess repository

usage: ess repository [-h]

Summarize repository

optional arguments:
  -h, --help show this help message and exit
 

ess probe

usage: ess probe [-h] [--file FILE] [--label [LABEL]] [--pcmd [PCMD]]
                 [--size [SIZE]]
                 category

Probe a file within the category

positional arguments:
  category                category

optional arguments:
  -h, --help              show this help message and exit
  --file FILE             probe a specific file
  --label [LABEL]         select a datastore
  --pcmd [PCMD]           command to replace loginf
  --size [SIZE]           number of Bytes of file to fetch
 

ess purge

usage: ess purge [-h] label

Delete datastore

positional arguments:
  label                   datastore to delete

optional arguments:
  -h, --help              show this help message and exit
 

ess ls

usage: ess ls [-h] [--exclude [EXCLUDE]] [--cat CAT] [--label LABEL] [-r]
              [--dateregex DATEREGEX] [--limit [LIMIT]]
              [--nameonly | --nosize | --nodate]
              [pattern]

list files based on an expression

positional arguments:
  pattern                 Glob patterns to match for

optional arguments:
  -h, --help              show this help message and exit
  --exclude [EXCLUDE]     Glob patterns to exclude files within pattern
  --cat CAT               Name of category to show files for
  --label LABEL           select a datastore
  -r, --recursive         Ascend through sub paths
  --dateregex DATEREGEX
                          regex style pattern used to get date from filename.
                          Option: [auto|none|custom]
  --limit [LIMIT]         number of file to fetch
  --nameonly              return file names only
  --nosize                return file names and dates
  --nodate                return file names and sizes
 

ess category

ess category add

usage: ess category add [-h] [--exclude [EXCLUDE]] [--dateregex DATEREGEX]
                        [--dateformat DATEFORMAT] [--archive ARCHIVE]
                        [--compression COMPRESSION] [--delimiter DELIMITER]
                        [--columnspec COLUMNSPEC] [--preprocess PREPROCESS]
                        [--alls | --allx] [--overwrite] [--label LABEL]
                        [--comment COMMENT] [--noprobe] [--usecache]
                        [--pkey PKEY]
                        name pattern

positional arguments:
  name                    Name of Category to call these files
  pattern                 GLOB patterns to match files

optional arguments:
  -h, --help              show this help message and exit
  --exclude [EXCLUDE]     GLOB patterns to exclude files within pattern
  --dateregex DATEREGEX
                          regex style pattern used to get date from filename.
                          Option: [auto|none|custom]
  --dateformat DATEFORMAT
                          Format of the date encoded in the filename. Will be
                          deprecated soon, please use dateregex
  --archive               ARCHIVE GLOB patterns to match members within an archive (e.g.
                          zip,tar,tgz,tar.bz2)
  --compression COMPRESSION
                          Force file to be treated as compressed in given format
  --delimiter DELIMITER
                          Overwrite delimiter
  --columnspec COLUMNSPEC
                          Overwrite columnspec
  --preprocess PREPROCESS
                          preprocess command
  --alls                  Force column server to be all strings
  --allx                  Force column server to ignore all columns
  --overwrite             Overwrite if category exists
  --label LABEL           select a datastore
  --comment COMMENT       Comment for the category
  --noprobe               Skip file probe. Just index.
  --usecache              Use cached file list if possible.
  --pkey PKEY             pkey column name

ess category change

usage: ess category change [-h]
                           {columnspec,dateformat,dateregex,usecache,comment}
                           ...

Modify parameters of a category

optional arguments:
  -h, --help            show this help message and exit

Category change commands:
  {columnspec,dateformat,dateregex,usecache,comment}
    columnspec          Modify the columnspec
    dateformat          Modify the dateformat. Will be deprecated soon, please
                        use dateregex
    dateregex           Modify the dateregex
    usecache            Modify the usecache
    comment             Modify the comment

ess category delete

usage: ess category delete [-h] [--label LABEL] category

positional arguments:
  category                category name

optional arguments:
  -h, --help              show this help message and exit
  --label LABEL           select a datastore

ess category copy

usage: ess category copy [-h] [--label LABEL] src dest

positional arguments:
  src                     source name
  dest                    copy name

optional arguments:
  -h, --help              show this help message and exit
  --label LABEL           select a datastore
 

ess file

ess file push

usage: ess file push [-h] [--dest DEST] [files [files ...]]

positional arguments:
  files                   Files to push

optional arguments:
  -h, --help              show this help message and exit
  --dest DEST             destination directory on worker

ess file get

usage: ess file get [-h] [name [name ...]]

positional arguments:
  name                    name of files/folders to get

optional arguments:
  -h, --help              show this help message and exit

ess file mkdir

usage: ess file mkdir [-h] name

positional arguments:
  name                    Directory to create

optional arguments:
  -h, --help              show this help message and exit
 

ess cat

usage: ess cat [-h] [--label LABEL] [--decompress] filename

positional arguments:
  filename                 Filename to dump contents of

optional arguments:
  -h, --help               show this help message and exit
  --label LABEL            Select a datastore
  --decompress             decompress file if supported
 

ess lsa

usage: ess lsa [-h] [--pattern PATTERN] [--label LABEL] filename

positional arguments:
  filename                Name of the archive file

optional arguments:
  -h, --help              show this help message and exit
  --pattern PATTERN       GLOB patterns to match files
  --label LABEL           Select a datastore
 

ess cluster

ess cluster set

usage: ess cluster set [-h] {local,cloud,custom}

positional arguments:
  {local,cloud,custom}

optional arguments:
  -h, --help              show this help message and exit

See Also: Advanced Options

ess cluster create

usage: ess cluster create [-h] [--number NumberOfWorkers] [--type TYPE]
                          [--add]
                          [--credentials CREDENTIALS | --aws_access_key AWS_ACCESS_KEY]
                          [--aws_secret_access_key AWS_SECRET_ACCESS_KEY]
                          [--error_return_code]

optional arguments:
  -h, --help             show this help message and exit
  --number NumberOfWorkers
                         Number of worker nodes
  --type TYPE            Type of worker nodes
  --add                  create additional worker nodes
  --credentials CREDENTIALS
                         Credentials file
  --aws_access_key AWS_ACCESS_KEY
                         EC2 access key
  --aws_secret_access_key AWS_SECRET_ACCESS_KEY
                         EC2 secret access key
  --error_return_code return error code when some of worker nodes failed

See Also: Advanced Options

ess cluster terminate

usage: ess cluster terminate [-h] [--all] [-y]
                             [--credentials CREDENTIALS | --aws_access_key AWS_ACCESS_KEY]
                             [--aws_secret_access_key AWS_SECRET_ACCESS_KEY]

optional arguments:
  -h, --help             show this help message and exit
  --all                  delete all worker nodes, security group, keys
  -y                     confirm to terminate all
  --credentials CREDENTIALS
                         Credentials file
  --aws_access_key AWS_ACCESS_KEY
                         EC2 access key
  --aws_secret_access_key AWS_SECRET_ACCESS_KEY
                         EC2 secret access key

See Also: Advanced Options

ess cluster stop

usage: ess cluster stop [-h]
                        [--credentials CREDENTIALS | --aws_access_key AWS_ACCESS_KEY]
                        [--aws_secret_access_key AWS_SECRET_ACCESS_KEY]

optional arguments:
  -h, --help             show this help message and exit
  --credentials CREDENTIALS
                         Credentials file
  --aws_access_key AWS_ACCESS_KEY
                         EC2 access key
  --aws_secret_access_key AWS_SECRET_ACCESS_KEY
                         EC2 secret access key

See Also: Advanced Options

ess cluster start

usage: ess cluster start [-h]
                         [--credentials CREDENTIALS | --aws_access_key AWS_ACCESS_KEY]
                         [--aws_secret_access_key AWS_SECRET_ACCESS_KEY]

optional arguments:
  -h, --help             show this help message and exit
  --credentials CREDENTIALS
                         Credentials file
  --aws_access_key AWS_ACCESS_KEY
                         EC2 access key
  --aws_secret_access_key AWS_SECRET_ACCESS_KEY
                         EC2 secret access key

See Also: Advanced Options

ess cluster status

usage: ess cluster status [-h]

optional arguments:
  -h, --help             show this help message and exit

See Also: Advanced Options

ess cluster remove

usage: ess cluster remove [-h] reservation [reservation ...]

positional arguments:
  reservation reservation ids to remove

optional arguments:
  -h, --help             show this help message and exit

See Also: Advanced Options

ess cluster add

usage: ess cluster add [-h] reservation [reservation ...]

positional arguments:
  reservation reservation ids to add

optional arguments:
  -h, --help             show this help message and exit

See Also: Advanced Options

ess cluster reset

usage: ess cluster reset [-h]

optional arguments:
  -h, --help             show this help message and exit

See Also: Advanced Options

ess cluster iplist

usage: ess cluster iplist [-h] [--private | --public]

optional arguments:
  -h, --help             show this help message and exit
  --private              Private IPs only
  --public               Public IPs only

See Also: Advanced Options

 

ess query

usage: ess query [-h] [--label LABEL] [--check] command [command ...]

SQL-like command.

positional arguments:
  command                 SQL command

optional arguments:
  -h, --help              show this help message and exit
  --label LABEL           Specify the datastore to use
  --check                 check SQL syntax only
 

ess server

ess server reset

usage: ess server reset [-h]

Terminate all daemons and delete server files

optional arguments:
  -h, --help             show this help message and exit

See Also: Advanced Options

ess server restart

usage: ess server restart [-h]

Flush all memory by stopping and starting daemons

optional arguments:
  -h, --help             show this help message and exit

See Also: Advanced Options

ess server commit

usage: ess server commit [-h]

Upload server files to workers

optional arguments:
  -h, --help             show this help message and exit

See Also: Advanced Options

ess server summary

usage: ess server summary [-h] [--name [NAME]]

optional arguments:
  -h, --help             show this help message and exit
  --name [NAME]          Select database to show

See Also: Advanced Options

 

ess create

ess create database

usage: ess create database [-h] [--ports PORTS [PORTS ...]] dbname

positional arguments:
  dbname                  Specify database name

optional arguments:
  -h, --help              show this help message and exit
  --ports PORTS [PORTS ...]
                          Number of ports

ess create table

usage: ess create table [-h] name ...

positional arguments:
  name                    Specify table name
  columns                 Specify column server

optional arguments:
  -h, --help              show this help message and exit

ess create vector

usage: ess create vector [-h] name ...

positional arguments:
  name                    Specify vector name
  columns                 Specify column server

optional arguments:
  -h, --help              show this help message and exit

ess create variable

usage: ess create variable [-h] ...

positional arguments:
  columns                 Specify column server

optional arguments:
  -h, --help              show this help message and exit
 

ess drop

ess drop database

usage: ess drop database [-h] dbname

positional arguments:
  dbname                  Specify database name

optional arguments:
  -h, --help              show this help message and exit

ess drop table

usage: ess drop table [-h] name

positional arguments:
  name                    Specify table name

optional arguments:
  -h, --help              show this help message and exit

ess drop vector

usage: ess drop vector [-h] name

positional arguments:
  name                    Specify vector name

optional arguments:
  -h, --help              show this help message and exit

ess drop variable

usage: ess drop variable [-h]

optional arguments:
  -h, --help             show this help message and exit
 

ess use

usage: ess use [-h] dbname

Change active database

positional arguments:
  dbname                  Name of database to switch to

optional arguments:
  -h, --help              show this help message and exit
 

ess stream

usage: ess stream [-h] [--exclude EXCLUDE] [--master] [--debug] [--bulk]
                  [--threads THREADS] [--archive ARCHIVE] [--label LABEL]
                  [--progress] [--limit LIMIT] [--quitonerror] [--s3out S3OUT]
                  category lower upper [command]

Import data

positional arguments:
  category                Which category to use
  lower                   start
  upper                   stop
  command                 Command to stream data to

optional arguments:
  -h, --help              show this help message and exit
  --exclude EXCLUDE       exclude files that match pattern
  --master                where to run
  --debug                 debug mode
  --bulk                  bulk mode
  --threads THREADS       Number of threads
  --archive ARCHIVE       glob pattern to id file within archive
  --label LABEL           Assign a label to the datastore
  --progress              Show a progress bar
  --limit LIMIT           Limit # of files streamed
  --quitonerror           Stop stream when error occurs
  --s3out S3OUT           send output to an s3 bucket

See Also: Attributes

 

ess exec

usage: ess exec [-h] [--master] [--debug] [--s3out S3OUT] command

Execute arbitrary command

positional arguments:
  command                 Filter to use

optional arguments:
  -h, --help              show this help message and exit
  --master                where to run
  --debug                 debug mode
  --s3out S3OUT           send output to an s3 bucket

See Also: Attributes

 

ess udbd

ess udbd start
ess udbd stop
ess udbd status
ess udbd restart
ess udbd ckmem
ess udbd cklog

 

ess redshift

ess redshift list

usage: ess redshift list [-h]

optional arguments:
  -h, --help            show this help message and exit

ess redshift register

usage: ess redshift register [-h] clusterid dbname user password

positional arguments:
  clusterid               Cluster-id
  dbname                  Name of database on redshift
  user                    Username on redshift
  password                user password on redshift

optional arguments:
  -h, --help              show this help message and exit

ess redshift deregister

usage: ess redshift deregister [-h]

optional arguments:
  -h, --help              show this help message and exit

ess redshift sql

usage: ess redshift sql [-h] [command]

Run a command on the redshift cluster

positional arguments:
  command                 Command to stream data to

optional arguments:
  -h, --help              show this help message and exit

ess redshift gentable

usage: ess redshift gentable [-h] [--key KEY] [--label LABEL] table category

Create SQL table based off essentia colspec

positional arguments:
  table                   Name of table on redshift to create.
  category                Which category to use

optional arguments:
  -h, --help              show this help message and exit
  --key KEY               Set addtional options on a single column
  --label LABEL           Choose category from labelled datastore

ess redshift stream

usage: ess redshift stream [-h] [--label LABEL] [--threads THREADS]
                           [--options [OPTIONS [OPTIONS ...]]]
                           category lower upper [command] table

Import data

positional arguments:
  category                Which category to use
  lower                   start
  upper                   stop
  command                 Command to stream data to
  table                   Name of table on redshift to dump data.

optional arguments:
  -h, --help             show this help message and exit
  --label LABEL          Choose category from labelled datastore
  --threads THREADS      Number of threads
  --options [OPTIONS [OPTIONS ...]]
                         Reshift specific arguments

Last Updated On