Essentia Commands

Working With Categories

To learn more about creating and using categories, see Category Rules.

Overview

Command	Arguments	Description
ess version		Print out the current version of Essentia and the AQ tools.
ess select	source `[--region REGION]` `[--credentials file]` `[--aws_access_key publickey]` `[--account_name AzureAccount]` `[--aws_secret_access_key secretkey]` `[--account_key AzureKey]` `[--label name]` `[--overwrite]`	Select a data source / repository
ess repository		Summarize currently defined data sources / repositories
ess summary	`[category]` `[pkey\|columnspec\|schema\|delimiter\|compression\|preprocess]` `[--label name]` `[--scan]` `[--short]`	Provide a summary of the categories in the current datastore, a deep summary of a single category, or a single parameter of a single category
ess purge	label	Delete the reference to the datastore (not the datastore itself)
ess ls	`[pattern]` `[--exclude subpattern]` `[--cat category]` `[--label name]` `[-r]` `[--dateregex regex_pattern]` `[--limit number]` `[--nameonly]` `[--nosize]` `[--nodate]`	List the contents of the source datastore
ess probe	category `[--file filename]` `[--label name]` `[--pcmd ProbeCommand]` `[--size Bytes]`	Scans a file to determine its compression, file format, etc.
ess category add	name pattern `[--exclude subpattern]` `[--dateregex regex_pattern\|none]` `[--dateformat auto\|none\|custom]` `[--archive pattern]` `[--compression type\|none]` `[--delimiter delimiter]` `[--columnspec NewColumnSpec]` `[--preprocess command]` `[--allx]` `[--alls]` `[--overwrite]` `[--label name]` `[--comment comment]` `[--noprobe]` `[--usecache]` `[--pkey PKEY]`	Add a category to the datastore
ess category delete	ExistingCategory `[--label name]`	Remove the reference to this category
ess category copy	ExistingCategory NewCategory `[--label name]` `[--credentials file]` `[--aws_access_key publickey]` `[--aws_secret_access_key secretkey]`	Create a copy of an existing category reference
ess category change	columnspec\|dateformat\|dateregex\|usecache\|comment NewSpec\|NewFormat\|NewRegex\|NewCache\|NewComment	Modify or override details about a category
ess cluster create	`[--type NodeType]` `[--number Number]` `[--add]` `[--credentials CREDENTIALS]` `[--aws_access_key AWS_ACCESS_KEY]` `[--aws_secret_access_key AWS_SECRET_ACCESS_KEY]` `[--error_return_code]`	Create worker nodes
ess cluster add	ReservationIDs	Add worker nodes to cluster by Reservation ID
ess cluster remove	ReservationIDs	Remove worker nodes from cluster by Reservation ID
ess cluster terminate	`[--all]` `[-y]` `[--credentials CREDENTIALS]` `[--aws_access_key AWS_ACCESS_KEY]` `[--aws_secret_access_key AWS_SECRET_ACCESS_KEY]`	Shutdown all worker nodes, optionally deleting all pem files and security groups with or without confirmation
ess cluster status		Summarizes the state of all workers and shows their Reservations IDs
ess cluster stop	`[--credentials CREDENTIALS]` `[--aws_access_key AWS_ACCESS_KEY]` `[--aws_secret_access_key AWS_SECRET_ACCESS_KEY]`	Suspend worker nodes (i.e. stop the EC2 instances)
ess cluster start	`[--credentials CREDENTIALS]` `[--aws_access_key AWS_ACCESS_KEY]` `[--aws_secret_access_key AWS_SECRET_ACCESS_KEY]`	Restart suspended worker nodes
ess cluster set	local\|cloud\|custom	Force essentia into master node only (local) or not.
ess cluster reset		Clear and reset cluster configuration
ess cluster iplist	`[--private]` `[--public]`	List the public and private IP’s of cluster
ess query	command `[--label name]` `[--check]`	SQL style query on raw logs
ess stream	category start stop command `[--exclude pattern]` `[--master]` `[--debug]` `[--bulk]` `[--threads number]` `[--archive pattern]` `[--s3out label:path]` `[--label name]` `[--progress]` `[--limit number]` `[--quitonerror]`	Stream data in given range to given command (cat being default)
ess exec	command `[--master]` `[--debug]` `[--s3out label:path]`	Execute given command on all worker nodes
ess server reset		Delete all database definitions
ess server restart		Clear database contents, but maintain schemas
ess server commit		Upload database spec files to workers
ess server summary	`[--name database]`	Summarize the databases and objects available for processing
ess create	database\|table\|vector\|variable name spec	Create an object. Name not required for variable
ess drop	database\|table\|vector\|variable name	Delete the object
ess use	database	Select the given database for queries
ess cat	filename `[--label name]` `[--decompress]`	Read file and print to stdout
ess lsa	filename `[--label name]` `[--pattern]`	List the files within an archive file with optional filtering.
ess file push	files `[--dest directory]`	Send a local file to worker nodes
ess file get	file\|folder	Fetch a file from workers to the master
ess file mkdir	name	Create a directory on worker nodes
ess redshift register	redshift_cluster_name redshift_database_name username password	Look up your redshift cluster and find its connection information
ess redshift stream	category startdate stopdate command redshift_table_name `[--options RedshiftOptions]` `[--threads number]` `[--label name]`	Import data from specified category between (and including) specified dates using the specified filter. Default is to execute on worker nodes. Data is streamed, one file at a time, to the given command and then into your redshift table
ess redshift sql	command	Query data in a redshift database using an SQL command
ess redshift gentable	category redshift_table_name `--key 'column = distkey'`	Create a table in redshift using the column specification derived from an Essentia category
ess redshift list		Get information about all Redshift clusters accessible by Essentia
ess redshift deregister		Unlink redshift cluster from Essentia
ess udbd	start\|stop\|restart\|status	Direct manipulation of UDB
ess udbd	ckmem\|cklog	Check memory or logs of UDB daemons on workers

Advanced Options

Environment Variable	Default	Description
ESS_CACHE_DIR	`GUI and RStudio:` /var/www/html/mydmp/aws/1/.ess `Command Line:` ~/.ess	Directory path where repository settings files will be stored. This is used to manage the local and cloud repositories associated with the instance you are on. The first default is for files created by the Graphical User Interface or RStudio and the second default is for files created via the command line.
ESS_WORK_DIR	`All Access Methods:` ./.conf	Directory path where udb settings files will be stored. This is used to manage the UDB databases and ports associated with the instance you are on. The default is always a .conf folder in your current directory.
ESS_AWS_DIR	`GUI and RStudio:` /var/www/html/mydmp/aws/1/.aws `Command Line:` ~/.aws	Directory path where pem files will be stored. This is used to manage the computer clusters associated with the instance you are on. The first default is for files created by the Graphical User Interface or RStudio and the second default is for files created via the command line.