Progress Data Download Process Module

class progress.data_download_process.DataProcess(input_file)

Bases: object

A class used to process wind and solar data from configuration and create relevant input files for further modeling.

This class reads a configuration file, and uses it to download, prepare, and organize wind and solar data. It delegates certain tasks (e.g., data download, transformation, K-means clustering) to specialized modules:

  • Wind for wind data processing

  • Solar for solar data processing

  • KMeans_Pipeline for clustering the resultant data

Parameters:

input_file (str) – Path to the YAML configuration file containing user settings and file/directory paths.

ProcessSolarData()

Downloads, processes, and prepares solar data for modeling.

This method performs the following sequence of operations:

  1. Initialize Solar Module: Creates a Solar instance with user-specified site data paths and directory locations.

  2. Download & Compute Solar Generation: Invokes solar.SolarGen() to fetch weather data from the user’s API and compute solar generation profiles across the specified date range.

  3. Gather Computed Profiles: Executes solar.SolarGenGather() to combine yearly generation results into a single dataset, suitable for clustering or probability analysis.

  4. Perform K-Means Clustering: Creates and runs a KMeans_Pipeline instance. It clusters the processed solar generation data into a specified number of clusters, calculates probabilities, and saves the results to file.

  5. Obtain Solar Profiles & Probabilities: Uses solar.GetSolarProfiles() to retrieve final site-level data, maximum possible generation, and the probability distribution for solar capacity factors.

In the end, this prepares solar data for further simulation or scenario analysis.

Returns:

None

Return type:

None

Raises:

  • FileNotFoundError – If required CSV files for solar sites or probabilities are not present.

  • ValueError – If solar data files contain invalid values or structures.

ProcessWindData()

Downloads, processes, and prepares wind data for modeling.

This method performs the following steps:

  1. Set Up Paths: Derives file paths for wind sites, power curves, windspeed data, and transition rate Excel sheets from the loaded configuration.

  2. Download Wind Data: Invokes wind.DownloadWindData() to fetch raw wind speed data using the user’s API credentials and configuration parameters.

  3. Prepare Wind Farms Data: Calls wind.WindFarmsData() to aggregate site information, farm names, zone numbers, wind classes, and power curves.

  4. Calculate Transition Rates: Uses wind.CalWindTrRates() to compute wind transition rates (wind speed state transitions) from the downloaded datasets.

  5. Read Transition Matrices: Loads transition matrices from an Excel file and organizes them into a NumPy array.

This step ultimately collects and prepares wind-related data so that it can be integrated into larger studies (e.g., generating time-series or capacity factor profiles for wind generation).

Returns:

None

Return type:

None

Raises:

  • FileNotFoundError – If expected CSV/Excel files for wind data are missing.

  • ValueError – If wind data files contain invalid values or structures.