Note
Go to the end to download the full example code.
RobustWorkflow¶
This tutorial follows the robust_workflow notebook and demonstrates the
RobustWorkflow class. The workflow can process data streams
that are not cleanly stationary: with operate_safe=False it will, when
needed, drop an initial fraction of the stream to find a stationary tail and
return “ball-park” statistics (flagged in the result metadata) rather than
aborting.
For each signal the procedure is the same as in the notebook:
build / load the signal into a
DataStream,set up
RobustWorkflow(operate_safe=False, verbosity=2)(high verbosity so the intermediate steps are plotted),plot the raw signal with
plot_signal_basic_stats(),call
process_data_streamto get the statistics, andre-plot the signal with its mean, confidence interval, and SSS start.
For clean single traces the DataStream Class guide is simpler; for many runs
see Ensemble Analysis. RobustWorkflow assumes equally spaced time points.
Setup¶
RobustWorkflow displays its plots (it calls plt.show() and then
closes the figure). The small keep_figures helper keeps those figures open
just long enough for the documentation gallery to capture the workflow’s own
output – it is not needed in a notebook or script.
import contextlib
import pprint
from pathlib import Path
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import quends as qnds
plotter = qnds.Plotter()
def example_data_dir() -> Path:
"""Find the shared example data directory during script or gallery runs."""
starts = []
if "__file__" in globals():
starts.append(Path(__file__).resolve())
starts.append(Path.cwd().resolve())
for start in starts:
for parent in [start, *start.parents]:
for candidate in (parent / "examples" / "data", parent / "data"):
if candidate.is_dir():
return candidate
raise FileNotFoundError("Could not locate examples/data")
DATA_DIR = example_data_dir()
@contextlib.contextmanager
def keep_figures():
_close = plt.close
plt.close = lambda *a, **k: None
try:
yield
finally:
plt.close = _close
Synthetic data: linear transient to a plateau with some noise¶
A linear ramp into a flat signal, plus noise. With verbosity=2 the
workflow shows its intermediate steps: the raw signal, the autocorrelation
used to set the averaging window, and the smoothed signal with the detected
start of statistical steady state (SSS).
arr_time = np.linspace(start=0, stop=1000, num=1001)
arr_signal = np.zeros_like(arr_time)
n_pts = arr_signal.shape[0]
for i_arr in range(100):
arr_signal[i_arr] = 0.05 * arr_time[i_arr]
for i_arr in range(100, n_pts):
arr_signal[i_arr] = arr_signal[99]
np.random.seed(42)
arr_signal += np.random.normal(loc=0.0, scale=0.1, size=arr_signal.shape)
my_label = "Linear Transient with Noise"
ds_flat = qnds.DataStream(pd.DataFrame({"time": arr_time, "flux": arr_signal}))
col = "flux"
my_wrkflw = qnds.RobustWorkflow(operate_safe=False, verbosity=2)
with keep_figures():
# Plot raw signal
plotter.plot_signal_basic_stats(ds_flat, col, label=my_label)
# Get statistics
my_stats = my_wrkflw.process_data_stream(ds_flat, col)
# Plot trace with mean and start of steady state
if not my_stats[col]["metadata"]["mitigation"] == "Drop":
plotter.plot_signal_basic_stats(ds_flat, col, stats=my_stats, label=my_label)
Original size of data stream: 1001 points.
After enforcing start time there are 1001 points left.
/home/runner/work/quends/quends/src/quends/base/data_stream.py:562: FutureWarning: adfuller currently returns a plain tuple whose length depends on the store and autolag arguments. In release 0.16 or after July 2027, whichever is later, the default behavior will switch to always returning an ADFullerResult. Set result_object=True to switch now, or result_object=False to keep the current behavior and silence this warning.
p_value = adfuller(self.data[column].dropna(), autolag="AIC")[1]
stats decorrelation time 73.85 gives smoothing window of 443 points.
Getting start of SSS based on smoothed signal:
Index where criterion is met: 474
Rolling window: 443
time where criterion is met: 474.00
time at start of SSS (adjusted for rolling window): 252.00
The dictionary my_stats holds the numerical results; metadata reveals
how the data was processed. Here mitigation: None / status: Regular
means no special handling was needed.
pprint.pprint(my_stats)
StatsResult({'flux': {'mean': 4.952894547302614, 'mean_uncertainty': 0.0036575729502063693, 'variance': 0.00968975840513791, 'confidence_interval': (4.945725704320209, 4.960063390285018), 'standard_deviation': 0.09843657046615303, 'pm_std': (4.949236974352408, 4.9565521202528195), 'effective_sample_size': 725, 'window_size': 1, 'n_short_averages': 749, 'ess_blocks': 724.3141260211527, 'se_effective_n': 724.3141260211527, 'se_method': 'ess_blocks', 'independence_status': 'tau_window', 'independent': True, 'ljungbox_lags': [5, 10], 'ljungbox_pvalues': [0.9154812760718971, 0.8902154368162933], 'ljungbox_pvalue': 0.8902154368162933, 'ci_method': 'normal', 'confidence_level': 0.95, 'sss_start': np.float64(252.0), 'metadata': {'status': 'Regular', 'mitigation': 'None'}, 'start_time': 0.0}}, metadata={'estimator': 'single', 'columns': ['flux'], 'total_samples': 749, 'schema_version': '1.0', 'warnings': []})
Regular signals: CGYRO¶
A well-behaved CGYRO run. The observable is the third column (after the index
and time), exactly as in the notebook.
data_paths = [DATA_DIR / "cgyro" / "output_nu0_02.csv"]
col = pd.read_csv(data_paths[0]).columns[2] # 3rd column (after index, time)
data_stream = qnds.from_csv(data_paths[0], col)
print("The data stream contains the following variables:")
for column, name in enumerate(data_stream.variables()):
print(f"{column}: {name}")
my_wrkflw = qnds.RobustWorkflow(operate_safe=False, verbosity=2)
with keep_figures():
plotter.plot_signal_basic_stats(data_stream, col, label=data_paths[0])
my_stats = my_wrkflw.process_data_stream(data_stream, col)
if not my_stats[col]["metadata"]["mitigation"] == "Drop":
plotter.plot_signal_basic_stats(
data_stream, col, stats=my_stats, label=data_paths[0]
)
The data stream contains the following variables:
0: time
1: Q_D/Q_GBD
Original size of data stream: 1523 points.
After enforcing start time there are 1523 points left.
/home/runner/work/quends/quends/src/quends/base/data_stream.py:562: FutureWarning: adfuller currently returns a plain tuple whose length depends on the store and autolag arguments. In release 0.16 or after July 2027, whichever is later, the default behavior will switch to always returning an ADFullerResult. Set result_object=True to switch now, or result_object=False to keep the current behavior and silence this warning.
p_value = adfuller(self.data[column].dropna(), autolag="AIC")[1]
stats decorrelation time 27.57 gives smoothing window of 165 points.
Getting start of SSS based on smoothed signal:
Index where criterion is met: 164
Rolling window: 165
time where criterion is met: 82.50
time at start of SSS (adjusted for rolling window): 41.00
Most of this signal is in SSS: the mean is steady and its high standard deviation gives the steady-state criterion plenty of wiggle room.
pprint.pprint(my_stats)
StatsResult({'Q_D/Q_GBD': {'mean': 37.83144165486063, 'mean_uncertainty': 1.1171091055746663, 'variance': 74.94522325094795, 'confidence_interval': (35.64190780793429, 40.02097550178698), 'standard_deviation': 8.657090923107367, 'pm_std': (36.71433254928597, 38.948550760435296), 'effective_sample_size': 69, 'window_size': 21, 'n_short_averages': 68, 'ess_blocks': 60.055498203144, 'se_effective_n': 60.055498203144, 'se_method': 'ess_blocks', 'independence_status': 'tau_window', 'independent': False, 'ljungbox_lags': [5, 10], 'ljungbox_pvalues': [0.03984594199389475, 0.07841061956251077], 'ljungbox_pvalue': 0.03984594199389475, 'ci_method': 'normal', 'confidence_level': 0.95, 'sss_start': np.float64(41.0), 'metadata': {'status': 'Regular', 'mitigation': 'None'}, 'start_time': 0.0}}, metadata={'estimator': 'single', 'columns': ['Q_D/Q_GBD'], 'total_samples': 1442, 'schema_version': '1.0', 'warnings': []})
Regular signals: GX¶
A GX heat-flux trace (HeatFlux_st).
data_paths = [DATA_DIR / "gx" / "tprim_2_4.out.csv"]
data_stream = qnds.from_csv(data_paths[0], "HeatFlux_st")
print("The data stream contains the following variables:")
for column, name in enumerate(data_stream.variables()):
print(f"{column}: {name}")
col = "HeatFlux_st"
my_wrkflw = qnds.RobustWorkflow(operate_safe=False, verbosity=2)
with keep_figures():
plotter.plot_signal_basic_stats(data_stream, col, label=data_paths[0])
my_stats = my_wrkflw.process_data_stream(data_stream, col)
if not my_stats[col]["metadata"]["mitigation"] == "Drop":
plotter.plot_signal_basic_stats(
data_stream, col, stats=my_stats, label=data_paths[0]
)
The data stream contains the following variables:
0: time
1: HeatFlux_st
Original size of data stream: 947 points.
After enforcing start time there are 947 points left.
/home/runner/work/quends/quends/src/quends/base/data_stream.py:562: FutureWarning: adfuller currently returns a plain tuple whose length depends on the store and autolag arguments. In release 0.16 or after July 2027, whichever is later, the default behavior will switch to always returning an ADFullerResult. Set result_object=True to switch now, or result_object=False to keep the current behavior and silence this warning.
p_value = adfuller(self.data[column].dropna(), autolag="AIC")[1]
stats decorrelation time 10.96 gives smoothing window of 65 points.
Getting start of SSS based on smoothed signal:
Index where criterion is met: 574
Rolling window: 65
time where criterion is met: 728.15
time at start of SSS (adjusted for rolling window): 686.29
Here the decorrelation length is short, so the smoothed signal is more variable and about half the trace is identified as steady state.
pprint.pprint(my_stats)
StatsResult({'HeatFlux_st': {'mean': 6.867678944197531, 'mean_uncertainty': 0.2119063711223739, 'variance': 0.5735871460822006, 'confidence_interval': (6.452342456797679, 7.283015431597384), 'standard_deviation': 0.7573553631434854, 'pm_std': (6.655772573075158, 7.079585315319905), 'effective_sample_size': 16, 'window_size': 27, 'n_short_averages': 15, 'ess_blocks': 12.773543219361203, 'se_effective_n': 12.773543219361203, 'se_method': 'ess_blocks', 'independence_status': 'tau_window', 'independent': True, 'ljungbox_lags': [5, 10], 'ljungbox_pvalues': [0.6740959783989092, 0.7806562742031611], 'ljungbox_pvalue': 0.6740959783989092, 'ci_method': 'normal', 'confidence_level': 0.95, 'sss_start': np.float64(686.2853296659573), 'metadata': {'status': 'Regular', 'mitigation': 'None'}, 'start_time': 0.0}}, metadata={'estimator': 'single', 'columns': ['HeatFlux_st'], 'total_samples': 406, 'schema_version': '1.0', 'warnings': []})
Total running time of the script: (0 minutes 1.244 seconds)














