Essentia Commands Reference Manuals

You are here:
← All Topics

ess

usage: ess [-h] [-v] 
{select,summary,repository,probe,purge,ls,category,file,cat,lsa,cluster,query,server,create,drop,use,stream,exec,udbd,redshift} ...

The Essentia ETL Engine

optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit

Subcommands:
{select,summary,repository,probe,purge,ls,category,file,cat,lsa,cluster,query,server,create,drop,use,stream,exec,udbd,redshift}
select Choose a datastore
summary Summarize datastore
repository Summarize repository
probe Probe category
purge Delete datastore
ls File commands
category Category commands
file File transfer with workers
cat dump file contents to stdout
lsa list contents of an archive
cluster manage a cluster of worker nodes
query SQL-like query command
server Modify server files for udbd
create Adds fields to config file
drop Removes fields from config file
use Change active database
stream Import data
exec Execute command
udbd udbd commands
redshift Link essentia to redshift
 

ess select

usage: ess select [-h] [--region REGION]
[--credentials CREDENTIALS | --aws_access_key AWS_ACCESS_KEY | --account_name ACCOUNT_NAME]
[--aws_secret_access_key AWS_SECRET_ACCESS_KEY | --account_key ACCOUNT_KEY]
[--label LABEL] [--overwrite]
source

Choose a datastore

positional arguments:
source Source location

optional arguments:
-h, --help show this help message and exit
--region REGION Region of datastore
--credentials CREDENTIALS
Credentials file
--aws_access_key AWS_ACCESS_KEY
S3 access key
--account_name ACCOUNT_NAME
Azure account name
--aws_secret_access_key AWS_SECRET_ACCESS_KEY
S3 secret access key
--account_key ACCOUNT_KEY
Azure account key
--label LABEL Assign a label to the datastore
--overwrite Overwrite credential information
 

ess summary

ess summary pkey

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
[category]
[{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
category category
{pkey,columnspec,schema,delimiter,compression,preprocess}
category parameter to output

optional arguments:
-h, --help show this help message and exit
--label [LABEL] select a datastore
--scan scan and update stats on datastore
--short return total file count and size only. used with
--scan

ess summary columnspec

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
[category]
[{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
category category
{pkey,columnspec,schema,delimiter,compression,preprocess}
category parameter to output

optional arguments:
-h, --help show this help message and exit
--label [LABEL] select a datastore
--scan scan and update stats on datastore
--short return total file count and size only. used with
--scan

ess summary schema

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
[category]
[{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
category category
{pkey,columnspec,schema,delimiter,compression,preprocess}
category parameter to output

optional arguments:
-h, --help show this help message and exit
--label [LABEL] select a datastore
--scan scan and update stats on datastore
--short return total file count and size only. used with
--scan

ess summary delimiter

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
[category]
[{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
category category
{pkey,columnspec,schema,delimiter,compression,preprocess}
category parameter to output

optional arguments:
-h, --help show this help message and exit
--label [LABEL] select a datastore
--scan scan and update stats on datastore
--short return total file count and size only. used with
--scan

ess summary compression

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
[category]
[{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
category category
{pkey,columnspec,schema,delimiter,compression,preprocess}
category parameter to output

optional arguments:
-h, --help show this help message and exit
--label [LABEL] select a datastore
--scan scan and update stats on datastore
--short return total file count and size only. used with
--scan

ess summary preprocess

usage: ess summary [-h] [--label [LABEL]] [--scan] [--short]
[category]
[{pkey,columnspec,schema,delimiter,compression,preprocess}]

Summarize datastore

positional arguments:
category category
{pkey,columnspec,schema,delimiter,compression,preprocess}
category parameter to output

optional arguments:
-h, --help show this help message and exit
--label [LABEL] select a datastore
--scan scan and update stats on datastore
--short return total file count and size only. used with
--scan
 

ess repository

usage: ess repository [-h]

Summarize repository

optional arguments:
-h, --help show this help message and exit
 

ess probe

usage: ess probe [-h] [--file FILE] [--label [LABEL]] [--pcmd [PCMD]]
[--size [SIZE]]
category

Probe a file within the category

positional arguments:
category category

optional arguments:
-h, --help show this help message and exit
--file FILE probe a specific file
--label [LABEL] select a datastore
--pcmd [PCMD] command to replace loginf
--size [SIZE] number of Bytes of file to fetch
 

ess purge

usage: ess purge [-h] label

Delete datastore

positional arguments:
label datastore to delete

optional arguments:
-h, --help show this help message and exit
 

ess ls

usage: ess ls [-h] [--exclude [EXCLUDE]] [--cat CAT] [--label LABEL] [-r]
[--dateregex DATEREGEX] [--limit [LIMIT]]
[--nameonly | --nosize | --nodate]
[pattern]

list files based on an expression

positional arguments:
pattern Glob patterns to match for

optional arguments:
-h, --help show this help message and exit
--exclude [EXCLUDE] Glob patterns to exclude files within pattern
--cat CAT Name of category to show files for
--label LABEL select a datastore
-r, --recursive Ascend through sub paths
--dateregex DATEREGEX
regex style pattern used to get date from filename.
Option: [auto|none|custom]
--limit [LIMIT] number of file to fetch
--nameonly return file names only
--nosize return file names and dates
--nodate return file names and sizes
 

ess category

ess category add

usage: ess category add [-h] [--exclude [EXCLUDE]] [--dateregex DATEREGEX]
[--dateformat DATEFORMAT] [--archive ARCHIVE]
[--compression COMPRESSION] [--delimiter DELIMITER]
[--columnspec COLUMNSPEC] [--preprocess PREPROCESS]
[--alls | --allx] [--overwrite] [--label LABEL]
[--comment COMMENT] [--noprobe] [--usecache]
[--pkey PKEY]
name pattern

positional arguments:
name Name of Category to call these files
pattern GLOB patterns to match files

optional arguments:
-h, --help show this help message and exit
--exclude [EXCLUDE] GLOB patterns to exclude files within pattern
--dateregex DATEREGEX
regex style pattern used to get date from filename.
Option: [auto|none|custom]
--dateformat DATEFORMAT
Format of the date encoded in the filename. Will be
deprecated soon, please use dateregex
--archive ARCHIVE GLOB patterns to match members within an archive (e.g.
zip,tar,tgz,tar.bz2)
--compression COMPRESSION
Force file to be treated as compressed in given format
--delimiter DELIMITER
Overwrite delimiter
--columnspec COLUMNSPEC
Overwrite columnspec
--preprocess PREPROCESS
preprocess command
--alls Force column server to be all strings
--allx Force column server to ignore all columns
--overwrite Overwrite if category exists
--label LABEL select a datastore
--comment COMMENT Comment for the category
--noprobe Skip file probe. Just index.
--usecache Use cached file list if possible.
--pkey PKEY pkey column name

ess category change

usage: ess category change [-h]
{columnspec,dateformat,dateregex,usecache,comment}
...

Modify parameters of a category

optional arguments:
-h, --help show this help message and exit

Category change commands:
{columnspec,dateformat,dateregex,usecache,comment}
columnspec Modify the columnspec
dateformat Modify the dateformat. Will be deprecated soon, please
use dateregex
dateregex Modify the dateregex
usecache Modify the usecache
comment Modify the comment

ess category delete

usage: ess category delete [-h] [--label LABEL] category

positional arguments:
category category name

optional arguments:
-h, --help show this help message and exit
--label LABEL select a datastore

ess category copy

usage: ess category copy [-h] [--label LABEL] src dest

positional arguments:
src source name
dest copy name

optional arguments:
-h, --help show this help message and exit
--label LABEL select a datastore
 

ess file

ess file push

usage: ess file push [-h] [--dest DEST] [files [files ...]]

positional arguments:
files Files to push

optional arguments:
-h, --help show this help message and exit
--dest DEST destination directory on worker

ess file get

usage: ess file get [-h] [name [name ...]]

positional arguments:
name name of files/folders to get

optional arguments:
-h, --help show this help message and exit

ess file mkdir

usage: ess file mkdir [-h] name

positional arguments:
name Directory to create

optional arguments:
-h, --help show this help message and exit
 

ess cat

usage: ess cat [-h] [--label LABEL] [--decompress] filename

positional arguments:
filename Filename to dump contents of

optional arguments:
-h, --help show this help message and exit
--label LABEL Select a datastore
--decompress decompress file if supported
 

ess lsa

usage: ess lsa [-h] [--pattern PATTERN] [--label LABEL] filename

positional arguments:
filename Name of the archive file

optional arguments:
-h, --help show this help message and exit
--pattern PATTERN GLOB patterns to match files
--label LABEL Select a datastore
 

ess cluster

ess cluster set

usage: ess cluster set [-h] {local,cloud,custom}

positional arguments:
{local,cloud,custom}

optional arguments:
-h, --help show this help message and exit

See Also: Advanced Options

ess cluster create

usage: ess cluster create [-h] [--number NumberOfWorkers] [--type TYPE]
[--add]
[--credentials CREDENTIALS | --aws_access_key AWS_ACCESS_KEY]
[--aws_secret_access_key AWS_SECRET_ACCESS_KEY]
[--error_return_code]

optional arguments:
-h, --help show this help message and exit
--number NumberOfWorkers
Number of worker nodes
--type TYPE Type of worker nodes
--add create additional worker nodes
--credentials CREDENTIALS
Credentials file
--aws_access_key AWS_ACCESS_KEY
EC2 access key
--aws_secret_access_key AWS_SECRET_ACCESS_KEY
EC2 secret access key
--error_return_code return error code when some of worker nodes failed

See Also: Advanced Options

ess cluster terminate

usage: ess cluster terminate [-h] [--all] [-y]
[--credentials CREDENTIALS | --aws_access_key AWS_ACCESS_KEY]
[--aws_secret_access_key AWS_SECRET_ACCESS_KEY]

optional arguments:
-h, --help show this help message and exit
--all delete all worker nodes, security group, keys
-y confirm to terminate all
--credentials CREDENTIALS
Credentials file
--aws_access_key AWS_ACCESS_KEY
EC2 access key
--aws_secret_access_key AWS_SECRET_ACCESS_KEY
EC2 secret access key

See Also: Advanced Options

ess cluster stop

usage: ess cluster stop [-h]
[--credentials CREDENTIALS | --aws_access_key AWS_ACCESS_KEY]
[--aws_secret_access_key AWS_SECRET_ACCESS_KEY]

optional arguments:
-h, --help show this help message and exit
--credentials CREDENTIALS
Credentials file
--aws_access_key AWS_ACCESS_KEY
EC2 access key
--aws_secret_access_key AWS_SECRET_ACCESS_KEY
EC2 secret access key

See Also: Advanced Options

ess cluster start

usage: ess cluster start [-h]
[--credentials CREDENTIALS | --aws_access_key AWS_ACCESS_KEY]
[--aws_secret_access_key AWS_SECRET_ACCESS_KEY]

optional arguments:
-h, --help show this help message and exit
--credentials CREDENTIALS
Credentials file
--aws_access_key AWS_ACCESS_KEY
EC2 access key
--aws_secret_access_key AWS_SECRET_ACCESS_KEY
EC2 secret access key

See Also: Advanced Options

ess cluster status

usage: ess cluster status [-h]

optional arguments:
-h, --help show this help message and exit

See Also: Advanced Options

ess cluster remove

usage: ess cluster remove [-h] reservation [reservation ...]

positional arguments:
reservation reservation ids to remove

optional arguments:
-h, --help show this help message and exit

See Also: Advanced Options

ess cluster add

usage: ess cluster add [-h] reservation [reservation ...]

positional arguments:
reservation reservation ids to add

optional arguments:
-h, --help show this help message and exit

See Also: Advanced Options

ess cluster reset

usage: ess cluster reset [-h]

optional arguments:
-h, --help show this help message and exit

See Also: Advanced Options

ess cluster iplist

usage: ess cluster iplist [-h] [--private | --public]

optional arguments:
-h, --help show this help message and exit
--private Private IPs only
--public Public IPs only

See Also: Advanced Options

 

ess query

usage: ess query [-h] [--label LABEL] [--check] command [command ...]

SQL-like command.

positional arguments:
command SQL command

optional arguments:
-h, --help show this help message and exit
--label LABEL Specify the datastore to use
--check check SQL syntax only
 

ess server

ess server reset

usage: ess server reset [-h]

Terminate all daemons and delete server files

optional arguments:
-h, --help show this help message and exit

See Also: Advanced Options

ess server restart

usage: ess server restart [-h]

Flush all memory by stopping and starting daemons

optional arguments:
-h, --help show this help message and exit

See Also: Advanced Options

ess server commit

usage: ess server commit [-h]

Upload server files to workers

optional arguments:
-h, --help show this help message and exit

See Also: Advanced Options

ess server summary

usage: ess server summary [-h] [--name [NAME]]

optional arguments:
-h, --help show this help message and exit
--name [NAME] Select database to show

See Also: Advanced Options

 

ess create

ess create database

usage: ess create database [-h] [--ports PORTS [PORTS ...]] dbname

positional arguments:
dbname Specify database name

optional arguments:
-h, --help show this help message and exit
--ports PORTS [PORTS ...]
Number of ports

ess create table

usage: ess create table [-h] name ...

positional arguments:
name Specify table name
columns Specify column server

optional arguments:
-h, --help show this help message and exit

ess create vector

usage: ess create vector [-h] name ...

positional arguments:
name Specify vector name
columns Specify column server

optional arguments:
-h, --help show this help message and exit

ess create variable

usage: ess create variable [-h] ...

positional arguments:
columns Specify column server

optional arguments:
-h, --help show this help message and exit
 

ess drop

ess drop database

usage: ess drop database [-h] dbname

positional arguments:
dbname Specify database name

optional arguments:
-h, --help show this help message and exit

ess drop table

usage: ess drop table [-h] name

positional arguments:
name Specify table name

optional arguments:
-h, --help show this help message and exit

ess drop vector

usage: ess drop vector [-h] name

positional arguments:
name Specify vector name

optional arguments:
-h, --help show this help message and exit

ess drop variable

usage: ess drop variable [-h]

optional arguments:
-h, --help show this help message and exit
 

ess use

usage: ess use [-h] dbname

Change active database

positional arguments:
dbname Name of database to switch to

optional arguments:
-h, --help show this help message and exit
 

ess stream

usage: ess stream [-h] [--exclude EXCLUDE] [--master] [--debug] [--bulk]
[--threads THREADS] [--archive ARCHIVE] [--label LABEL]
[--progress] [--limit LIMIT] [--quitonerror] [--s3out S3OUT]
category lower upper [command]

Import data

positional arguments:
category Which category to use
lower start
upper stop
command Command to stream data to

optional arguments:
-h, --help show this help message and exit
--exclude EXCLUDE exclude files that match pattern
--master where to run
--debug debug mode
--bulk bulk mode
--threads THREADS Number of threads
--archive ARCHIVE glob pattern to id file within archive
--label LABEL Assign a label to the datastore
--progress Show a progress bar
--limit LIMIT Limit # of files streamed
--quitonerror Stop stream when error occurs
--s3out S3OUT send output to an s3 bucket

See Also: Attributes

 

ess exec

usage: ess exec [-h] [--master] [--debug] [--s3out S3OUT] command

Execute arbitrary command

positional arguments:
command Filter to use

optional arguments:
-h, --help show this help message and exit
--master where to run
--debug debug mode
--s3out S3OUT send output to an s3 bucket

See Also: Attributes

 

ess udbd

ess udbd start
ess udbd stop
ess udbd status
ess udbd restart
ess udbd ckmem
ess udbd cklog

 

ess redshift

ess redshift list

usage: ess redshift list [-h]

optional arguments:
-h, --help show this help message and exit

ess redshift register

usage: ess redshift register [-h] clusterid dbname user password

positional arguments:
clusterid Cluster-id
dbname Name of database on redshift
user Username on redshift
password user password on redshift

optional arguments:
-h, --help show this help message and exit

ess redshift deregister

usage: ess redshift deregister [-h]

optional arguments:
-h, --help show this help message and exit

ess redshift sql

usage: ess redshift sql [-h] [command]

Run a command on the redshift cluster

positional arguments:
command Command to stream data to

optional arguments:
-h, --help show this help message and exit

ess redshift gentable

usage: ess redshift gentable [-h] [--key KEY] [--label LABEL] table category

Create SQL table based off essentia colspec

positional arguments:
table Name of table on redshift to create.
category Which category to use

optional arguments:
-h, --help show this help message and exit
--key KEY Set addtional options on a single column
--label LABEL Choose category from labelled datastore

ess redshift stream

usage: ess redshift stream [-h] [--label LABEL] [--threads THREADS]
[--options [OPTIONS [OPTIONS ...]]]
category lower upper [command] table

Import data

positional arguments:
category Which category to use
lower start
upper stop
command Command to stream data to
table Name of table on redshift to dump data.

optional arguments:
-h, --help show this help message and exit
--label LABEL Choose category from labelled datastore
--threads THREADS Number of threads
--options [OPTIONS [OPTIONS ...]]
Reshift specific arguments