Configuration¶

ProgridPy reads configuration from environment variables at runtime. The recommended approach is to define these in a .env file at the project root; the scripts load it automatically via python-dotenv.

cp .env.example .env   # then fill in your values

Environment Variables¶

AWS / S3¶

These variables control how S3Handler authenticates and which buckets are used for raw and processed data storage.

Variable	Description	Default
`AWS_PROFILE`	Named AWS CLI profile to use for sessions	(none)
`AWS_DEFAULT_PROFILE`	Fallback if `AWS_PROFILE` is unset	(none)
`AWS_REGION`	AWS region for the S3 client	`us-west-2`
`AWS_ACCESS_KEY_ID`	Explicit access key (if not using a profile)	(none)
`AWS_SECRET_ACCESS_KEY`	Explicit secret key (if not using a profile)	(none)
`PROGRID_S3_RAW_BUCKET`	S3 bucket for raw ISO data	`progrid-rawdata`
`PROGRID_S3_PROCESSED_BUCKET`	S3 bucket for processed hive-partitioned data	`progrid-processed-data`

Authentication precedence

S3Handler creates a boto3.Session using AWS_PROFILE (or AWS_DEFAULT_PROFILE) if set; otherwise it falls back to the standard boto3 credential chain (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, instance profile, etc.).

ERCOT API¶

The ERCOT Public API requires account credentials and a subscription key. Register at apiexplorer.ercot.com and follow the ERCOT Public API documentation to obtain these values.

Variable	Description	Default
`ERCOT_API_USERNAME`	ERCOT account username	(required)
`ERCOT_API_PASSWORD`	ERCOT account password	(required)
`ERCOT_API_SUBSCRIPTION_KEY`	Subscription key from the ERCOT developer portal	(required)
`ERCOT_CLIENT_ID`	OAuth2 client ID	`fec253ea-0d06-4272-a5e6-b478baeecd70`
`ERCOT_TOKEN_URL`	OAuth2 token endpoint	`https://ercotb2c.b2clogin.com/.../token`

Required for ERCOT downloads

All three credential variables (ERCOT_API_USERNAME, ERCOT_API_PASSWORD, ERCOT_API_SUBSCRIPTION_KEY) must be set before calling ERCOT.download_raw_data(). The client ID and token URL have sensible defaults and typically do not need to be changed.

Enverus¶

The Enverus API client authenticates with a user ID only.

Variable	Description	Default
`ENVERUS_USERID`	Enverus account user ID	(required)

Data Directories¶

Helper scripts resolve local data directories through environment variables. Each ISO has a raw and processed directory variable. When unset, scripts fall back to hard-coded defaults.

Variable	Description	Script Default
`MISO_RAW_DIR`	Local directory for raw MISO files	`/home/ubuntu/MISO-operational-data/raw`
`MISO_PROCESSED_DIR`	Local directory for processed MISO data	`/home/ubuntu/MISO-operational-data/processed`
`ERCOT_RAW_DIR`	Local directory for raw ERCOT files	`/mnt/Data/progrid/progrid-rawdata/ercot`
`ERCOT_PROCESSED_DIR`	Local directory for processed ERCOT data	`/mnt/Data/progrid/progrid-processed-data`
`SPP_RAW_DIR`	Local directory for raw SPP files	`/home/ubuntu/SPP-operational-data/raw`
`SPP_PROCESSED_DIR`	Local directory for processed SPP data	`/home/ubuntu/SPP-operational-data/processed`

Override for local development

Set these variables to point at a convenient local path when developing outside the production environment. For example:

export MISO_RAW_DIR=~/data/miso/raw
export MISO_PROCESSED_DIR=~/data/miso/processed

Example `.env` File¶

A minimal .env for ERCOT and Enverus access:

# ERCOT API
ERCOT_API_USERNAME=your_username
ERCOT_API_PASSWORD=your_password
ERCOT_API_SUBSCRIPTION_KEY=your_key

# Enverus
ENVERUS_USERID=your_userid

# AWS (optional -- uses CLI profile by default)
AWS_PROFILE=progrid

# Local data directories (optional)
MISO_RAW_DIR=~/data/miso/raw
MISO_PROCESSED_DIR=~/data/miso/processed