The “opg” script

The opg script unifies OPGEE functionality into a single script with sub-commands. Common (built-in) sub-commands are implemented directly in the ogpee package. Project-specific features can be added via plugins.

Note

Quick links to sub-commands: config, collect, csv2xml gensim, genwor, graph, gui, merge, run, runsim,

Usage

usage: opg [-h] [--logLevel LOGLEVEL] [--set name=value] [--verbose]
           [--version] [--VERSION]
           {collect,compare,config,csv2xml,gensim,graph,gui,merge,run,runmany,runsim}
           ...

Named Arguments

--logLevel

Sets the log level for modules of the program. A default log level can be set for the entire program, or individual modules can have levels set using the syntax “module:level, module:level,…”, where the level names must be one of {debug,info,warning,error,fatal} (case insensitive).

Default: “INFO”

--set

Assign a value to override a configuration file parameter. For example, to temporarily use a different attributes file, use –set “OPGEE.UserAttributesFile=/some/path/attr.xml”. Enclose the argument in quotes if it contains spaces or other characters that would confuse the shell. Use multiple –set flags and arguments to set multiple variables.

Default: []

--verbose

Show diagnostic output

Default: False

--version

show program’s version number and exit

--VERSION

Default: False

Subcommands

For help on subcommands, use the “-h” flag after the subcommand name

subcommand

Possible choices: collect, compare, config, csv2xml, gensim, graph, gui, merge, run, runmany, runsim

Sub-commands

collect

Collect partial results from packets of trials in a Monte Carlo simulation.

opg collect [-h] [-d] [-f FIELDS] [-s SIMULATION_DIR]

Named Arguments

-d, --delete

Delete the partial result files after combining them into a single file.

Default: False

-f, --fields

Generate trial data for the specified field or fields only. Argument

may be a comma-delimited list of Field names. Otherwise trial data is generated for all fields defined in the analysis.

-s, --simulation-dir

The top-level directory to create for this simulation “package”.

If the simulation directory already exists and you must specify –-overwrite, or gensim will refuse to overwrite the directory.

compare

Compare CSV files with fields in columns and processes in row,

with the value either blank for disabled processes, or the total energy use per day by process, after running the model. Mainly used for testing against OPGEEv3. Note that CSV files for comparison can be generated using “opg run –comparisonCSV filename”.

opg compare [-h] [-n COUNT] [-m MAX_DIFF] [-v] file1 file2

Positional Arguments

file1

A CSV file containing results for comparison against “file2”

file2

A CSV file containing results for comparison against “file1”

Named Arguments

-n, --count

The number of fields to compare. Default is 0, which

means compare all fields.

Default: 0

-m, --max-diff

The maximum (fractional) difference in values for “file2”

relative to “file1” that are deemed “approximately the same”. Default is 0.01, i.e., (file2_value - file1_value) / file1_value <= 0.01

Default: 0.01

-v, --verbose

Report all value differences. By default, only field differences

are reported.

Default: False

config

The config command list the values of configuration variables from ~/opgee.cfg. With no arguments, it displays the values of all variables for the default project. Use the -d flag to show only values from the [DEFAULT] section.

If an argument name is provided, it is treated as a substring pattern, unless the -x flag is given (see below). All configuration variables containing the give name are displayed with their values. The match is case-insensitive.

If the -x or --exact flag is specified, the argument is treated as an exact variable name (case-sensitive) and only the value is printed. This is useful mainly for scripting. For general use the substring matching is more convenient.

Examples:

$ opg config log
[test]
              $LOGNAME = rjp
      OPGEE.LogConsole = True
OPGEE.LogConsoleFormat = %(levelname)s %(name)s: %(message)s
          OPGEE.LogDir = /Users/rjp/tmp
         OPGEE.LogFile = /Users/rjp/tmp/opgee.log
   OPGEE.LogFileFormat = %(asctime)s %(levelname)s %(name)s:%(lineno)d %(message)s
        OPGEE.LogLevel = INFO, .mcs.simulation:DEBUG, .field:INFO, .smart_defaults:DEBUG

$ opg config -x OPGEE.LogLevel
INFO, .mcs.simulation:DEBUG, .field:INFO, .smart_defaults:DEBUG

opg config [-h] [-d] [-e] [-x] [name]

Positional Arguments

name

Show the names and values of all parameters whose name contains the given value. The match is case-insensitive. If not specified, all variable values are shown.

Default: “”

Named Arguments

-d, --useDefault

Indicates to operate on the DEFAULT section rather than the project section.

Default: False

-e, --edit

Edit the configuration file. The command given by the value of config variable OPGEE.TextEditor is run with the opgee.cfg file as an argument.

Default: False

-x, --exact

Treat the text not as a substring to match, but as the name of a specific variable. Match is case-sensitive. Prints only the value.

Default: False

csv2xml

Convert various CSV files to their corresponding XML representation.

opg csv2xml [-h] [-a ANALYSIS] [-f {fields,attributes}] [-m MODIFIES] -i
            INPUTCSV [-n COUNT] [-o OUTPUTXML] [--overwrite] [-p]
            [-s SKIPFIELDS]

Named Arguments

-a, --analysis

The name to give the <Analysis> element. Default is the file basename

with the extension removed.

-f, --format

Possible choices: fields, attributes

Which type of conversion to perform. Default is “fields”.

Default: “fields”

-m, --modifies

The name to use in the <Field modifies=”NAME”> element.

Default is “template”

Default: “template”

-i, --inputCSV

The pathname of the file to import

-n, --count

The number of rows to import from the CSV file.

Default is 0, which means import all rows.

Default: 0

-o, --outputXML

The pathname of the XML file to create. Default is the same

name as the input CSV file, but with the extension changed to “xml”. Refuses to overwrite an existing file unless –overwrite is specified.

--overwrite

If set, allows existing XML file to be overwritten.

Default: False

-p, --fromPackage

If specified, the inputCSV argument is treated as relative to

the opgee package and loaded as an internal resource.

Default: False

-s, --skipFields

Comma-delimited list of field names to exclude from analysis

gensim

Generate the simulation directory and trial data for a Monte Carlo simulation.

opg gensim [-h] [-a ANALYSIS] [-d DISTRIBUTIONS] [-f FIELDS] [-m MODEL_FILE]
           [-n] [--overwrite] [-s SIMULATION_DIR] [-t TRIALS]

Named Arguments

-a, --analysis

The name of the analysis for which to generate a simulation

-d, --distributions

The path to a CSV file with distribution definitions. If omitted, the

built-in file etc/parameter_distributions.csv is used.

-f, --fields

Generate trial data for the specified field or fields only. Argument

may be a comma-delimited list of Field names. Otherwise trial data is generated for all fields defined in the analysis.

-m, --model-file

XML model definition files to load. If –no-default-model is not specified,

the built-in files etc/opgee.xml and etc/attributes.xml are loaded first, and the XML files specified here will be merged with these. If –no-default-model is specified, only the given files are loaded; they are merged in the order stated.

-n, --no-default-model

Don’t load the built-in opgee.xml model definition.

Default: False

--overwrite

DELETE and recreate the simulation directory.

Default: False

-s, --simulation-dir

The top-level directory to create for this simulation “package”.

If the simulation directory already exists and you must specify –-overwrite, or gensim will refuse to overwrite the directory.

-t, --trials

The number of trials to create for this simulation (REQUIRED).

Default: 0

graph

Create graphs of various aspects of OPGEE models.

opg graph [-h] [-c {all,core}] [-C CLASSES_OUTPUT] [-f FIELD]
          [-F FIELD_OUTPUT] [-l LEVELS] [-m] [-M HIERARCHY_OUTPUT] [-n]
          [-x XML_FILE]

Named Arguments

-c, --classes

Possible choices: all, core

Graph the class structure, either “all”, including all defined Process subclasses (of which there are dozens) or only the “core” classes excluding Process subclasses.

-C, --classes-output

The pathname of the image file to create for classes. If none is specified, and the code is running in a jupyter notebook, the image is displayed inline. (Implies –classes.)

-f, --field

Graph the process network for the named field.

-F, --field-output

The pathname of the image file to create with process connections for the field specified in the –field argument. If no file is specified, and the code is running in a jupyter notebook, the image is displayed inline.

-l, --levels

How many levels to descend when graphing the model hierarchy

Default: 0

-m, --model-hierarchy

Graph the model container hierarchy.

Default: False

-M, --hierarchy-output

The pathname of the image file to create for classes. If none is specified, and the code is running in a jupyter notebook, the image is displayed inline. (Implies –model_hierarchy.)

-n, --no-default-model

Don’t load the built-in opgee.xml model definition.

Default: False

-x, --xml_file

The path to the model XML file to load. By default, the built-in opgee.xml is loaded.

gui

Run the OPGEE Graphical User Interface

opg gui [-h] [-d] [-H HOST] [-P PORT] [-a ANALYSIS] [-f FIELD] [-m MODEL_FILE]
        [-n] [--add-stream-components] [--use-class-path]

Named Arguments

-d, --debug

Enable debug mode in the dash server

Default: False

-H, --host

Set the host address to serve the application on. Default is localhost (127.0.0.1).

Default: “127.0.0.1”

-P, --port

Set the port to serve the application on. Default is 8050.

Default: 8050

-a, --analysis

The analysis to run. Default (for testing) is “test”

Default: “test”

-f, --field

The field to display. Default (for testing) is “test”

Default: “test”

-m, --model-file

The OPGEE model XML file to read. By default it is merged with the built-in

model file, “etc/opgee.xml”. If no model file is specified, etc/opgee.xml is read. Use –no-default-model to avoid reading the default model file.

-n, --no-default-model

Don’t load the built-in opgee.xml model definition.

Default: False

--add-stream-components

Include additional stream components listed in config variable “OPGEE.StreamComponents”

Default: False

--use-class-path

Search for Process subclasses in Python files found in the path(s) listed in config variable “OPGEE.ClassPath”

Default: False

merge

Merge two or more OPGEE XML files.

opg merge [-h] [-o OUTPUTXML] [-n] [--overwrite] [pathnames ...]

Positional Arguments

pathnames

Pathnames of the XML input files to be merged, in the order specified. By default,

the built-in {opgee}/etc/opgee.xml is included as the base file to merge with. To override this, use the -n/–no-default-model option.

Named Arguments

-o, --outputXML

The pathname of the XML file to create. Default is the same

name as the input CSV file, but with the extension changed to “xml”. Refuses to overwrite an existing file unless –overwrite is specified. If an output XML file is not specified, the merged XML is written to stdout.

-n, --no-default-model

Don’t use the built-in {opgee}/etc/opgee.xml model file as the base

file to merge with.

Default: False

--overwrite

If set, allows existing XML file to be overwritten.

Default: False

run

Run the specified portion of an OPGEE LCA model

opg run [-h] [-a ANALYSES] [-c SAVE_COMPARISON] [-f FIELDS] [-i]
        [-m MODEL_FILE] [-n] [-o OUTPUT] [-p PROCESSES]

Named Arguments

-a, --analyses

Run only the specified analysis or analyses. Argument may be a

comma-delimited list of Analysis names.

-c, --save-comparison

The name of a CSV file to which to save results suitable for

use with the “compare” subcommand.

-f, --fields

Run only the specified field or fields. Argument may be a

comma-delimited list of Field names. To specify a field within a specific Analysis, use the syntax “analysis_name.field_name”. Otherwise the field will be run for each Analysis the field name occurs within (respecting the –analyses flag).

-i, --ignore-errors

Keep running even if some fields raise errors when run

Default: False

-m, --model-file

XML model definition files to load. If –no_default_model is not

specified, the built-in files etc/opgee.xml and etc/attributes.xml are loaded first, and the XML files specified here will be merged with these. If –no_default_model is specified, only the given files are loaded; they are merged in the order stated.

-n, --no-default-model

Don’t load the built-in opgee.xml model definition.

Default: False

-o, --output

Write CI output to specified CSV file for all top-level processes

and aggregators, for all fields run

-p, --processes

Write CI output to specified CSV file for all processes, for all fields

run, rather than by top-level processes and aggregators (as with –output)

runmany

Run thousands of fields in parallel using dask.

opg runmany [-h] [-a ANALYSIS] [-b BATCH_START] [-f FIELDS] -m MODEL
            [-N SAVE_AFTER] [-n COUNT] -o OUTPUT [-p] [-S START_WITH]
            [-s SKIP_FIELDS]

Named Arguments

-a, --analysis

The name to give the <Analysis> element. Default is the file basename

with the extension removed. Default is the first analyses found in the given model XML file.

-b, --batch-start

The value to use to start numbering batch result files.

Default is zero. Ignored unless -N/–save-after is specified.

Default: 0

-f, --fields

A comma-delimited list of fields to run. Default is to

run all fields indicated in the named analysis.

-m, --model

[Required] The pathname of a model XML file to process.

-N, --save-after

Write a results to a new file after the given number of results are

returned. Implies –parallel.

-n, --count

The number of fields to run from the named analysis.

Default is 0, which means run all fields.

Default: 0

-o, --output

[Required] The pathname of the CSV files to create containing energy and

emissions results for each field. This argument is used as a basename, with the suffix ‘.csv’ replaced by ‘-energy.csv’ and ‘-emissions.csv’ to store the results. Each file has fields in columns and processes in rows.

-p, --parallel

Run the fields in parallel locally using dask.

Default: False

-S, --start-with

The name of a field to start with. Use this to resume a run after a failure.

Can be combined with -n/–count to run a large number of fields in smaller batches.

-s, --skip-fields

Comma-delimited list of field names to exclude from analysis

runsim

Run a Monte Carlo simulation using the model file stored in the simulation folder.

opg runsim [-h] [-c {local,slurm}] [-f FIELDS | -N NUM_FIELDS] [-C]
           [-m MINUTES] [-n NTASKS] [-p PARTITION] [-P PACKET_SIZE]
           [-s SIMULATION_DIR] [-S] [-t TRIALS]

Named Arguments

-c, --cluster-type

Possible choices: local, slurm

The type of cluster to use. Defaults to value of config

variable ‘OPGEE.ClusterType’, currently “local”.

-f, --fields

Run only the specified field or fields. Argument may be a

comma-delimited list of Field names. Otherwise all fields defined in the analysis are run. (Mutually exclusive with -N/–nfields.)

-N, --num-fields

Run MCS simulations on the first “num-fields” only.

(Mutually exclusive with -f/–fields.)

-C, --collect

Whether to combine per-packet files into a single CSV when

simulation is complete. Note that the “collect” subcommand can do this later if needed.

Default: False

-m, --minutes

The amount of wall time to allocate for each task. Default is

10 minutes. Acceptable time formats include “minutes”, “minutes:seconds”, “hours:minutes:seconds”, and formats involving days, which we shouldn’t require.

Default: 10

-n, --ntasks

Number of worker tasks to create. Default is the number of fields, if

specified using -f/–fields, otherwise -n/–ntasks is required.

-p, --partition

The name of the partition to use for job submissions. Default is the

value of config variable “SLURM.Partition”, currently ‘normal’.

-P, --packet-size

Divide trials for a single field in to packets of this number of trials

to run serially on a single worker. Default is the value of configuration file parameter “OPGEE.TrialPacketSize”, currently 100.

Default: 100

-s, --simulation-dir

The top-level directory to use for this simulation “package”

-S, --serial

Run the simulation serially in the currently running process.

Default: False

-t, --trials

The trials to run. Can be expressed as a string containing

comma-delimited ranges and individual trail numbers, e.g. “1-20,22, 35, 42, 44-50”). The special string “all” (the default) runs all defined trials.

Default: “all”

Extending “opg” using plug-ins

The opg script will load any python files whose name ends in _plugin.py, found in any of the directories indicated in the config file variable OPGEE.PluginPath. The value of OPGEE.PluginPath must be a sequence of directory names separated by colons (:) on Unix-like systems or by semi-colons (;) on Windows.

See opgee.subcommand for documentation of the plug-in API.