Configuration System¶

The opg script relies on a configuration file to:

define the location of essential and optional files,

allow the user to set defaults for key command-line arguments to scripts, and

define both global default and user-specific values for all parameters.

The configuration file and variables are described below.

Configuration file sections¶

The configuration file is divided into sections indicated by a name within square brackets. All variable declarations following a section declaration, until the next section declaration (if any) appear in the declared section.

Default section¶

Default values are defined in the [DEFAULT] section. When opgee requests the value of a variable from a project section (see below), the default value is returned if the variable is not defined in the project section. Variables whose values apply to multiple projects can be defined conveniently in the [DEFAULT] section.

All pre-defined opgee variables are defined in the [DEFAULT] section, allowing them to be overridden on a project-by-project basis.

Project sections¶

Each project must have its own section. For example, to setup a project called, say, “myproj”, I would create the section [myproj]. Following this, I would define variables particular to this project.

The configuration files¶

There are up to 4 configuration files read, two of which are user-modifiable:

First, opgee/etc/system.cfg is read from within the opgee package. This defines all known config variables and provides their default values as described below. The values in this file are the appropriate values for Linux and similar systems. This file should not be modified by the user.

Next, a platform-specific file is read, if it exists. Currently, the only such files are opgee/etc/Windows.cfg and opgee/etc/Darwin.cfg, read on Windows and Macintosh systems, respectively. (N.B. “Darwin” is the official platform name for the Macintosh operating system.) These files should not be modified by the user.

Next, if the environment variable OPGEE_SITE_CONFIG is defined, it should refer to a configuration file that defines site-specific settings. This file is optional; it allows an administrator to consolidate site-specific values to simplify configuration for users.

Finally, the user’s configuration file is read if it exists; otherwise the file is created with the initial contents being a commented-out version of opgee/etc/system.cfg. This provides a handy reference to the available parameters and their default values.

On Linux and OS X, the user’s configuration file is found in $HOME/opgee.cfg

On Windows, the file opgee.cfg will be stored in the directory identified by the first of the following environment variables defined to have a non-empty value: OPGEE_HOME, HOMESHARE, and HOMEPATH. The first variable, OPGEE_HOME is known only to opgee, while at least one of the other two should be set by Windows.

In all cases, the directory in which the configuration file is located is assigned to the opgee configuration variable Home.

The values in each successive configuration file override default values for variables of the same name that are set in files read earlier. Values can also be set in project-specific section. Thus when a user specifies a project to operate on, either on the command-line to The “opg” script or as the value of OPGEE.DefaultProject in $HOME/opgee.cfg, the project-specific values override any values set in [DEFAULT] sections.

For example, consider the following values in $HOME/opgee.cfg:

[DEFAULT]
OPGEE.DefaultProject = test
OPGEE.LogConsole = True
OPGEE.ShowStackTrace = False

[test]
# example of setting different logging levels for individual modules
OPGEE.LogLevel = INFO, .core:DEBUG, .processes.reservoir_well_interface:DEBUG

# example of identifying user-defined Process subclasses
OPGEE.ClassPath = %(Home)s/tmp/opgee_procs.py

# example of adding new elements to stream components
OPGEE.StreamComponents = foo, bar ,baz

# show full stack trace on error (useful mostly for developers)
OPGEE.ShowStackTrace = True

The default value for OPGEE.ShowStackTrace is False, meaning that if errors are raised in the Python code, a full function call stack trace will not be displayed. The project test overrides this with the value True.

The available parameters and their default values are described below.

Editing the user configuration file¶

You can edit the configuration file, $HOME/opgee.cfg, with any editor capable of working with plain text, i.e., not a word-processor such as Word. Use the command opg config -e to invoke an editor on the configuration file.

The command invoked by opg config -e to edit the config file is the value of the configuration parameter OPGEE.TextEditor, which defaults to a system-appropriate value shown in the table below. Set this value in the configuration file to invoke your preferred editor.

For example, if you prefer the emacs editor on a Mac, you can add this line to ~/opgee.cfg to cause the Finder to open the file using the emacs application:

OPGEE.TextEditor = open -a emacs

Or, to edit the config file using the PyCharm app, use this:

OPGEE.TextEditor = open -a PyCharm

If the editor command is not found on your execution PATH, you can specify the full pathname. Use quotes around the path if it includes spaces, as in the examples below.

To use Notepad++ on Windows, use the following (adjusted as necessary for your installation location):

OPGEE.TextEditor = "C:/Program Files/Notepad++/notepad++.exe"

To use PyCharm, use the following – again, adjusted to match your installation location:

OPGEE.TextEditor  = "C:/Program Files/JetBrains/PyCharm 2018.1.4/bin/pycharm64.exe"

Invoking the command:

opg config -e

will cause the editor to be invoked on your configuration file.

Referencing configuration variables¶

A powerful feature of the configuration system is that variables can be defined in terms of other variables. The syntax for referencing the value of a variable is to precede the variable name with %( and follow it with )s. Thus to reference variable OPGEE.QueryDir, you would write %(OPGEE.QueryDir)s.

Note

When referencing a variable in the config file, you must include the trailing s after the closing parenthesis, or a Python exception will be raised.

Also note that variable names are case-sensitive.

Variable values are substituted when a variable’s value is requested, not when the configuration file is read. The difference is that if variable A is defined in terms of variable B, (e.g., A = %(B)s/something/else), you can subsequently change B and the value of A will reflect this when A is accessed by opgee.

All known variables are given default values in the opgee system files. Users can create variables in any of the user controlled config files, if desired.

Environment variables¶

All defined environmental variables are loaded into the config parameter space before reading any configuration files, and are accessible with a prefix of $, as in a UNIX shell. For example, to reference the environment variable SCRATCH, you can use %($SCRATCH)s.

Default configuration variables and values¶

The system default values are provided in the opgee package in the file opgee/etc/system.cfg, which is listed below. In addition to these values, several values are read from platform-specific files, as noted above. These values are shown below.

For Windows:¶

[DEFAULT]
OPGEE.TempDir = %($TEMP)s

For MacOS:¶

[DEFAULT]

The system defaults file¶

[DEFAULT]

# This project is used if '-P' flag not given to the 'gt' command
OPGEE.DefaultProject =

# This defines the variable for documentation purposes. The value
# is set automatically in each project section to the name of that
# section -- unless a non-blank value already exists in the section.
OPGEE.ProjectName =

# Root directory for where the user keeps project folders
OPGEE.ProjectRoot = %(Home)s/projects
OPGEE.ProjectDir  = %(OPGEE.ProjectRoot)s/%(OPGEE.ProjectName)s

# Where to find plug-ins. Internal plugin directory is added
# automatically. Use this to add custom plug-ins outside the opgee
# source tree. The value is a semicolon-delimited (on Windows) or
# colon-delimited (on Unix) string of directories to search for files
# matching the pattern '*_plugin.py'
OPGEE.PluginPath = %(OPGEE.ProjectDir)s/plugins

# Where to find user-defined LCA model classes. The value is one or more
# Python files or directories, delimited by semicolon (on Windows) or colon
# (on Unix). For directories, all '.py' files found in the directory are
# loaded.
OPGEE.ClassPath =

# Ordinarily, an error is raised if encounter a redefinition of a Process
# subclass. To allow the option of overriding built-in subclasses, the user
# can set this flag to True, which merely emits a warning rather than raising
# an exception.
OPGEE.AllowProcessRedefinition = False

# Where to find user's customizations
OPGEE.UserXmlDirectory = %(Home)s/opgee/%(OPGEE.ProjectName)s

# Default model file. Paths not starting with drive specifier or "/"
# are assumed to be within the opgee package.
OPGEE.ModelFile = etc/opgee.xml

# Default attributes file. Paths not starting with drive specifier or "/"
# are assumed to be within the opgee package.
OPGEE.AttributesFile = etc/attributes.xml

# Where to find user's attributes.xml files (optional)
OPGEE.UserAttributesFile =

# Allow user to provide comma-delimited list of additional components
# that should be allowed in Streams.
OPGEE.StreamComponents =

# If set to a filename, the final expanded, merged XML is save there
OPGEE.XmlSavePathname =

# Change this if desired to increase or decrease diagnostic messages.
# A default value can be set here, and a project-specific value can
# be set in the project's config file section.
# Possible values (from most to least verbose) are:
# DEBUG, INFO, WARNING, ERROR, CRITICAL
OPGEE.LogLevel = INFO

OPGEE.LogConsole = True

# If set, application logger messages are written here.
OPGEE.LogDir = %(Home)s/tmp
OPGEE.LogFile = %(OPGEE.LogDir)s/opgee.log

# Format strings for log files and console messages. Note doubled
# '%%' required here around logging parameters to avoid attempted
# variable substitution within the config system.
OPGEE.LogFileFormat    = %%(asctime)s %%(levelname)s %%(name)s:%%(lineno)d %%(message)s
OPGEE.LogConsoleFormat = %%(levelname)s %%(name)s: %%(message)s

# Where to create temporary files
OPGEE.TempDir = /tmp

# For debugging purposes: tool.py can show a stack trace on error
OPGEE.ShowStackTrace = False

# TextEditor to open via the --edit option to the 'config' sub-command
OPGEE.TextEditor = vi

# This is the two subprocesses excluded to calculate energy consumption for the NG reinjection option
# energy for all other subprocesses is summarized and converted to the amount of NG consumed
OPGEE.ExcludeFromReinjectionEnergySummary = GasReinjectionCompressor, GasReinjectionWell

OPGEE.FunctionalUnits = oil, gas
OPGEE.Boundaries      = Production, Transportation, Distribution

# Options are "local" or "slurm"
OPGEE.ClusterType = local

# The target number of trials to run serially on a worker task
OPGEE.MaxTrialsPerPacket = 100

# The name of an account to use on SLURM, if any
SLURM.Account =

# Where to chdir to before running sbatch script
SLURM.RunDir = %(Home)s/slurm

SLURM.MinutesPerTask = 10

# On Sherlock, there are 20-32 cores on "normal", 24-128 on "serc",
# 20-128 on "owners", with a min of 128 GB per node, and default mem
# of at least 4 GB per CPU
SLURM.CoresPerNode = 20

# By default we use one process per core, avoiding threading issues
SLURM.ProcessesPerCore = 1

# The default partition (other options on Sherlock are: normal, owners)
SLURM.Partition = normal

# Default assumes use of anaconda, but user can redefine SLURM.LoadEnvironment
# to change this
SLURM.CondaEnvironment = opgee

# The name to give the job when running on SLURM
SLURM.JobName = opgee-mcs

# Total amount of memory per job
SLURM.MemPerJob = 20GiB

# Whether to run "nanny" processes on dask
SLURM.UseNanny = True

# Location of log file for use on SLURM
SLURM.LogFile = %(OPGEE.LogDir)s/opgee-slurm.log

# Local (per node) temporary storage
SLURM.TempDir = /tmp/opgee

# Network interface like ‘eth0’ or ‘ib0’. This will be used both for the
# Dask scheduler and the Dask workers interface.
SLURM.Interface =

# Which shell to use for the sbatch script
SLURM.Shell = /usr/bin/bash

Configuring the logging system¶

Setting logging verbosity¶

When the The “opg” script runs, or when opgee functions are called from your own code, diagnostic and informational messages are printed. You can control the level of output by setting the OPGEE.LogLevel in your opgee.cfg file. (See logging for further details.)

The simplest setting is just one of the following values, in order of decreasing verbosity: DEBUG, INFO, WARNING, ERROR, and FATAL. This will apply to all opgee modules.

You can also specify verbosity by module, by specifying a module name and the level for that module as a comma-separated list of “module:level” strings, e.g.,:

OPGEE.LogLevel = WARN, .utils:INFO, .core:DEBUG

In this example, the default level is set to WARN, and two opgee modules have their levels set: opgee.utils is set to INFO, and opgee.core is set to DEBUG. A user’s plugin can also use the logging system.

Console / file logs and message formatting¶

Note that the module name is shown in the console log messages. Setting OPGEE.LogLevel to DEBUG produces the maximum number of log messages; setting it to FATAL minimizes message verbocity.

Other relevant variables are shown here with their default values:

# If set, application logger messages are written here.
OPGEE.LogFile = %(Home)s/log/opgee.log

# Show log messages on the console (terminal)
OPGEE.LogConsole = True

# Format strings for log files and console messages. Note doubled
# '%%' required here around logging parameters to avoid attempted
# variable substitution within the config system.
OPGEE.LogFileFormat    = %%(asctime)s %%(levelname)s %%(name)s:%%(lineno)d %%(message)s
OPGEE.LogConsoleFormat = %%(levelname)s %%(name)s: %%(message)s