The “opg” script¶

The opg script unifies OPGEE functionality into a single script with sub-commands. Common (built-in) sub-commands are implemented directly in the ogpee package. Project-specific features can be added via plugins.

Note

Quick links to sub-commands: config, collect, csv2xml gensim, genwor, graph, gui, merge, run, runsim,

Usage¶

usage: opg [-h] [--logLevel LOGLEVEL] [--set name=value] [--verbose]
           [--version] [--VERSION]
           {collect,compare,config,csv2xml,gensim,graph,gui,merge,run,update}
           ...

Named Arguments¶

--logLevel

Sets the log level for modules of the program. A default log level can be set for the entire program, or individual modules can have levels set using the syntax “module:level, module:level,…”, where the level names must be one of {debug,info,warning,error,fatal} (case insensitive).

Default: 'INFO'

--set

Assign a value to override a configuration file parameter. For example, to temporarily use a different attributes file, use –set “OPGEE.UserAttributesFile=/some/path/attr.xml”. Enclose the argument in quotes if it contains spaces or other characters that would confuse the shell. Use multiple –set flags and arguments to set multiple variables.

Default: []

--verbose

Show diagnostic output

Default: False

--version

show program’s version number and exit

--VERSION

Default: False

Subcommands¶

For help on subcommands, use the “-h” flag after the subcommand name

subcommand: Possible choices: collect, compare, config, csv2xml, gensim, graph, gui, merge, run, update

Sub-commands¶

collect¶

Collect partial results from packets of trials in a Monte Carlo simulation.

opg collect [-h] [-d] [-f FIELDS] [-o OUTPUT_DIR] [-s SIM_DIR]

Named Arguments¶

-d, --delete

Delete the partial result files after combining them into a single file.

Default: False

-f, --fields

The names of the field to operate on. If not provided, all partial: result files found will be combined.

-o, --output-dir

The directory containing partial result files. Use only for non-MCS results.: For MCS, use the -s/–sim-dir option.

-s, --sim-dir

The simulation directory. Use for Monte Carlo Simulations only. For: non-MCS results, use the -o/–output-dir option.

compare¶

Compare CSV files with fields in columns and processes in row,: with the value either blank for disabled processes, or the total energy use per day by process, after running the model. Mainly used for testing against OPGEEv3. Note that CSV files for comparison can be generated using “opg run –comparisonCSV filename”.

opg compare [-h] [-n COUNT] [-m MAX_DIFF] [-v] file1 file2

Positional Arguments¶

file1: A CSV file containing results for comparison against “file2”
file2: A CSV file containing results for comparison against “file1”

Named Arguments¶

-n, --count

The number of fields to compare. Default is 0, which: means compare all fields.

Default: 0

-m, --max-diff

The maximum (fractional) difference in values for “file2”: relative to “file1” that are deemed “approximately the same”. Default is 0.01, i.e., (file2_value - file1_value) / file1_value <= 0.01

Default: 0.01

-v, --verbose

Report all value differences. By default, only field differences: are reported.

Default: False

config¶

The config command list the values of configuration variables from ~/opgee.cfg. With no arguments, it displays the values of all variables for the default project. Use the -d flag to show only values from the [DEFAULT] section.

If an argument name is provided, it is treated as a substring pattern, unless the -x flag is given (see below). All configuration variables containing the give name are displayed with their values. The match is case-insensitive.

If the -x or --exact flag is specified, the argument is treated as an exact variable name (case-sensitive) and only the value is printed. This is useful mainly for scripting. For general use the substring matching is more convenient.

Examples:

$ opg config log
[test]
              $LOGNAME = rjp
      OPGEE.LogConsole = True
OPGEE.LogConsoleFormat = %(levelname)s %(name)s: %(message)s
          OPGEE.LogDir = /Users/rjp/tmp
         OPGEE.LogFile = /Users/rjp/tmp/opgee.log
   OPGEE.LogFileFormat = %(asctime)s %(levelname)s %(name)s:%(lineno)d %(message)s
        OPGEE.LogLevel = INFO, .mcs.simulation:DEBUG, .field:INFO, .smart_defaults:DEBUG

$ opg config -x OPGEE.LogLevel
INFO, .mcs.simulation:DEBUG, .field:INFO, .smart_defaults:DEBUG

opg config [-h] [-d] [-e] [-x] [name]

Positional Arguments¶

name

Show the names and values of all parameters whose name contains the given value. The match is case-insensitive. If not specified, all variable values are shown.

Default: ''

Named Arguments¶

-d, --useDefault

Indicates to operate on the DEFAULT section rather than the project section.

Default: False

-e, --edit

Edit the configuration file. The command given by the value of config variable OPGEE.TextEditor is run with the opgee.cfg file as an argument.

Default: False

-x, --exact

Treat the text not as a substring to match, but as the name of a specific variable. Match is case-sensitive. Prints only the value.

Default: False

csv2xml¶

Convert various CSV files to their corresponding XML representation.

opg csv2xml [-h] [-a ANALYSIS] [-f {fields,attributes}] [-m MODIFIES] -i
            INPUTCSV [-n COUNT] [-o OUTPUTXML] [--overwrite] [-p]
            [-s SKIPFIELDS]

Named Arguments¶

-a, --analysis

The name to give the <Analysis> element. Default is the file basename: with the extension removed.

-f, --format

Possible choices: fields, attributes

Which type of conversion to perform. Default is “fields”.

Default: 'fields'

-m, --modifies

The name to use in the <Field modifies=”NAME”> element.: Default is “template”

Default: 'template'

-i, --inputCSV

The pathname of the file to import

-n, --count

The number of rows to import from the CSV file.: Default is 0, which means import all rows.

Default: 0

-o, --outputXML

The pathname of the XML file to create. Default is the same: name as the input CSV file, but with the extension changed to “xml”. Refuses to overwrite an existing file unless –overwrite is specified.

--overwrite

If set, allows existing XML file to be overwritten.

Default: False

-p, --fromPackage

If specified, the inputCSV argument is treated as relative to: the opgee package and loaded as an internal resource.

Default: False

-s, --skipFields

Comma-delimited list of field names to exclude from analysis

gensim¶

Generate the simulation directory and trial data for a Monte Carlo simulation.

opg gensim [-h] [-a ANALYSIS] [-d DISTRIBUTIONS] [-f FIELDS] [-m MODEL_FILE]
           [-n] [--overwrite] -s SIMULATION_DIR [-t TRIALS]

Named Arguments¶

-a, --analysis

The name of the analysis for which to generate a simulation

-d, --distributions

The path to a CSV file with distribution definitions. If omitted, the: built-in file etc/parameter_distributions.csv is used.

-f, --fields

Generate trial data for the specified field or fields only. Argument: may be a comma-delimited list of Field names. Otherwise trial data is generated for all fields defined in the analysis.

-m, --model-file

XML model definition files to load. If –no-default-model is not specified,: the built-in files etc/opgee.xml and etc/attributes.xml are loaded first, and the XML files specified here will be merged with these. If –no-default-model is specified, only the given files are loaded; they are merged in the order stated.

-n, --no-default-model

Don’t load the built-in opgee.xml model definition.

Default: False

--overwrite

DELETE and recreate the simulation directory.

Default: False

-s, --simulation-dir

The top-level directory to create for this simulation “package”.: If the simulation directory already exists and you must specify –-overwrite, or gensim will refuse to overwrite the directory.

-t, --trials

The number of trials to create for this simulation (REQUIRED).

Default: 0

graph¶

Create graphs of various aspects of OPGEE models.

opg graph [-h] [-c {all,core}] [-C CLASSES_OUTPUT] [-f FIELD]
          [-F FIELD_OUTPUT] [-l LEVELS] [-m] [-M HIERARCHY_OUTPUT] [-n]
          [-x XML_FILE]

Named Arguments¶

-c, --classes

Possible choices: all, core

Graph the class structure, either “all”, including all defined Process subclasses (of which there are dozens) or only the “core” classes excluding Process subclasses.

-C, --classes-output

The pathname of the image file to create for classes. If none is specified, and the code is running in a jupyter notebook, the image is displayed inline. (Implies –classes.)

-f, --field

Graph the process network for the named field.

-F, --field-output

The pathname of the image file to create with process connections for the field specified in the –field argument. If no file is specified, and the code is running in a jupyter notebook, the image is displayed inline.

-l, --levels

How many levels to descend when graphing the model hierarchy

Default: 0

-m, --model-hierarchy

Graph the model container hierarchy.

Default: False

-M, --hierarchy-output

The pathname of the image file to create for classes. If none is specified, and the code is running in a jupyter notebook, the image is displayed inline. (Implies –model_hierarchy.)

-n, --no-default-model

Don’t load the built-in opgee.xml model definition.

Default: False

-x, --xml_file

The path to the model XML file to load. By default, the built-in opgee.xml is loaded.

gui¶

Run the OPGEE Graphical User Interface

opg gui [-h] [-d] [-H HOST] [-P PORT] [-a ANALYSIS] [-f FIELD] [-m MODEL_FILE]
        [-n] [--add-stream-components] [--use-class-path]

Named Arguments¶

-d, --debug

Enable debug mode in the dash server

Default: False

-H, --host

Set the host address to serve the application on. Default is localhost (127.0.0.1).

Default: '127.0.0.1'

-P, --port

Set the port to serve the application on. Default is 8050.

Default: 8050

-a, --analysis

The analysis to run. Default (for testing) is “test”

Default: 'test'

-f, --field

The field to display. Default (for testing) is “test”

Default: 'test'

-m, --model-file

The OPGEE model XML file to read. By default it is merged with the built-in: model file, “etc/opgee.xml”. If no model file is specified, etc/opgee.xml is read. Use –no-default-model to avoid reading the default model file.

-n, --no-default-model

Don’t load the built-in opgee.xml model definition.

Default: False

--add-stream-components

Include additional stream components listed in config variable “OPGEE.StreamComponents”

Default: False

--use-class-path

Search for Process subclasses in Python files found in the path(s) listed in config variable “OPGEE.ClassPath”

Default: False

merge¶

Merge two or more OPGEE XML files.

opg merge [-h] [-o OUTPUTXML] [-n] [--overwrite] [pathnames ...]

Positional Arguments¶

pathnames

Pathnames of the XML input files to be merged, in the order specified. By default,: the built-in {opgee}/etc/opgee.xml is included as the base file to merge with. To override this, use the -n/–no-default-model option.

Named Arguments¶

-o, --outputXML

The pathname of the XML file to create. Default is the same: name as the input CSV file, but with the extension changed to “xml”. Refuses to overwrite an existing file unless –overwrite is specified. If an output XML file is not specified, the merged XML is written to stdout.

-n, --no-default-model

Don’t use the built-in {opgee}/etc/opgee.xml model file as the base: file to merge with.

Default: False

--overwrite

If set, allows existing XML file to be overwritten.

Default: False

run¶

Run the specified portion of an OPGEE LCA model

opg run [-h] [-a ANALYSES] [-b BATCH_SIZE] [-B BATCH_START]
        [-c {serial,slurm,local}] [-C] [-f FIELDS] [-F SKIP_FIELDS] [-g PATH]
        [-G] [-i] [-m MODEL_FILE] [-M MINUTES] [-n] [-N NUM_FIELDS]
        [-o OUTPUT_DIR] [-p PARTITION] [-P PACKET_SIZE] [-r {simple,detailed}]
        [-s SIMULATION_DIR] [-S START_WITH] [-t TRIALS] [-T NUM_TASKS]
        [-v SAVE_COMPARISON]

Named Arguments¶

-a, --analyses

Run only the specified analysis or analyses. Argument may be a: comma-delimited list of Analysis names.

-b, --batch-size

Write a results to a new file after the results for the given: number of packets are returned. If not specified, results are written out only after all trials have completed.

-B, --batch-start

The value to use to start numbering batch result files.: Default is zero. Ignored unless -S/–save-after is also specified.

Default: 0

-c, --cluster-type

Possible choices: serial, slurm, local

The type of cluster to use. Defaults to value of config: variable ‘OPGEE.ClusterType’, currently “serial”.

-C, --collect

Whether to combine per-packet files into a single CSV when: simulation is complete. Note that the “collect” subcommand can do this later if needed.

Default: False

-f, --fields

Run only the specified field or fields. Argument may be a: comma-delimited list of Field names. To specify a field within a specific Analysis, use the syntax “analysis_name.field_name”. Otherwise the field will be run for each Analysis the field name occurs within (respecting the –analyses flag).

-F, --skip-fields

Comma-delimited list of field names to exclude from analysis

-g, --post-proc-plugin

Loads the post-processing plugin from the given path. The file: must contain a single, valid subclass of opgee.post_process.PostProcess. This arg can be specified multiple times to load multiple post-processing classes. They will be invoked in the order given on the command-line.

Default: []

-G, --no-post-proc-plugin-path

Override the automatic loading of post-processing plugins from the path(s): specified in config variable OPGEE.PostProcPluginPath. Specific plugins can still be loaded using the –post-plugin argument.

Default: False

-i, --ignore-errors

Keep running even if some fields raise errors when run

Default: False

-m, --model-file

XML model definition files to load. If –no_default_model is not: specified, the built-in files etc/opgee.xml and etc/attributes.xml are loaded first, and the XML files specified here will be merged with these. If –no_default_model is specified, only the given files are loaded; they are merged in the order stated.

-M, --minutes

The amount of wall time to allocate for each task.: Default is 10 minutes. Acceptable time formats include “minutes”, “minutes:seconds”, “hours:minutes:seconds”, and formats involving days, which we shouldn’t require. Ignored if –cluster-type=slurm is not specified.

Default: 10

-n, --no-default-model

Don’t load the built-in opgee.xml model definition.

Default: False

-N, --num-fields

Run MCS simulations on the first “num-fields” only.: (Mutually exclusive with -f/–fields.)

-o, --output-dir

Write output to the specified directory.

-p, --partition

The name of the partition to use for job submissions. Default is the: value of config variable “SLURM.Partition”, currently ‘normal’. Ignored if –cluster-type=slurm is not specified.

-P, --packet-size

Divide runs for a single field into groups of this size: to run serially on a single worker. Default is the value of configuration file parameter “OPGEE.MaxTrialsPerPacket”, currently 10.

Default: 10

-r, --result-type

Possible choices: simple, detailed

The type of result to return from each field. Default is “simple”.: For “simple” results, the following values are saved per trial in a separate file for each field: trial_num, CI, total GHGs, and emissions from combustion, land use, venting/flaring, other. For “detailed” results, per-process emissions and energy use are stored.

-s, --simulation-dir

The top-level directory to use for this simulation “package”

-S, --start-with

The name of a field to start with. Use this to resume a run after a failure.: Can be combined with -n/–num-fields to run a large number of fields in smaller batches.

-t, --trials

The trials to run. Can be expressed as a string containing: comma-delimited ranges and individual trail numbers, e.g. “1-20,22, 35, 42, 44-50”). The special string “all” (the default) runs all defined trials. Ignored unless -s/–simulation-dir is specified.

Default: 'all'

-T, --num-tasks

Number of worker tasks to create. Default is the number of fields, if: specified using -f/–fields, otherwise -T/–num_tasks is required.

-v, --save-comparison

The name of a CSV file to which to save results suitable for: use with the “compare” subcommand.

update¶

Update model XML files with Fields inside Analyses to use FieldRef instead.

opg update [-h] [-o OUTPUT] [--overwrite] [--ignore-duplicates] model_file

Positional Arguments¶

model_file: The model XML file to operate on.

Named Arguments¶

-o, --output

The path of the updated model XML file. If not specified,: the output name will be the input name with “-updated” added before the “.xml” extension.

--overwrite

Force overwrite of an existing model XML. Cannot be the samefile as the input.

Default: False

--ignore-duplicates

Ignore duplicate field names in the input file, saving only the final one.

Default: False

Extending “opg” using plug-ins¶

The opg script will load any python files whose name ends in _plugin.py, found in any of the directories indicated in the config file variable OPGEE.PluginPath. The value of OPGEE.PluginPath must be a sequence of directory names separated by colons (:) on Unix-like systems or by semi-colons (;) on Windows.

See opgee.subcommand for documentation of the plug-in API.