ALI Performance Tests on Blake

Introduction

Currently testing the Greenland Ice Sheet (GIS) and Antartica Ice Sheet (AIS) in Albany Land Ice (ALI) using Intel Skylake CPUs on blake.

Architectures:

Name	Blake (SKX)
CPU	Dual-socket Intel Xeon Platinum 8160 (Skylake)
Cores/Node	48
Threads/Core	2
Memory/Node	187 GB
Interconnect	Intel OmniPath Gen-1 (100 GB/s)
Compiler	Intel 18.1.163
MPI	openmpi 2.1.2

Cases:

Case Name	Number of Processes (np)	Description
ant-2-20km_ml_ls	384	Unstructured 2-20km AIS, ML w/ line smoothing
ant-2-20km_mu_ls	384	Unstructured 2-20km AIS, MueLu w/ line smoothing
ant-2-20km_mu_dls	384	Unstructured 2-20km AIS, MueLu w/ decoupled line smoothing
green-1-10km_ent_fea_1ws_tet	384	Unstructured 1-10km GIS, Enthalpy problem, finite element assembly only, single workset, tetrahedron
green-1-10km_ent_fea_1ws_wdg	384	Unstructured 1-10km GIS, Enthalpy problem, finite element assembly only, single workset, wedge
green-1-10km_ent_fea_mem_tet	384	Unstructured 1-10km GIS, Enthalpy problem, finite element assembly only, memoization, tetrahedron
green-1-10km_ent_fea_mem_wdg	384	Unstructured 1-10km GIS, Enthalpy problem, finite element assembly only, memoization, wedge
green-1-10km_ent_mu_wdg	384	Unstructured 1-10km GIS, Enthalpy problem, MueLu, wedge
green-1-7km_fea_1ws	384	Unstructured 1-7km GIS, finite element assembly only, single workset
green-1-7km_mu_dls_1ws	384	Unstructured 1-7km GIS, MueLu w/ decoupled line smoothing, single workset
green-1-7km_fea_mem	384	Unstructured 1-7km GIS, finite element assembly only, memoization
green-1-7km_ml_ls_mem	384	Unstructured 1-7km GIS, ML w/ line smoothing, memoization
green-1-7km_mu_ls_mem	384	Unstructured 1-7km GIS, MueLu w/ line smoothing, memoization
green-1-7km_mu_dls_mem	384	Unstructured 1-7km GIS, MueLu w/ decoupled line smoothing, memoization
green-1-7km_muk_ls_mem	384	Unstructured 1-7km GIS, MueLu w/ kokkos and line smoothing, memoization
green-3-20km_vel_fea_mem_tet	384	Unstructured 3-20km GIS, Velocity problem, finite element assembly only, memoization, tetrahedron
green-3-20km_vel_fea_mem_wdg	384	Unstructured 3-20km GIS, Velocity problem, finite element assembly only, memoization, wedge
green-3-20km_ent_fea_mem_tet	384	Unstructured 3-20km GIS, Enthalpy problem, finite element assembly only, memoization, tetrahedron
green-3-20km_ent_fea_mem_wdg	384	Unstructured 3-20km GIS, Enthalpy problem, finite element assembly only, memoization, wedge
green-3-20km_beta_1ws	384	Unstructured 3-20km GIS, basal friction initialization, single workset
green-3-20km_beta_mem	384	Unstructured 3-20km GIS, basal friction initialization, memoization
green-3-20km_beta_memp	384	Unstructured 3-20km GIS, basal friction initialization, memoization for parameters (beta)

Timers:

Timer Name	Level	Description
Albany Total Time	0	Total wall-clock time of simulation
Albany: Setup Time	1	Preprocess
Albany: Total Fill Time	1	Finite element assembly
Albany Fill: Residual	2	Residual assembly
Albany Residual Fill: Evaluate	3	Compute the residual, local/global assembly
Albany Residual Fill: Export	3	Update global residual across MPI ranks
Albany Fill: Jacobian	2	Jacobian assembly
Albany Jacobian Fill: Evaluate	3	Compute the Jacobian, local/global assembly
Albany Jacobian Fill: Export	3	Update global Jacobian across MPI ranks
NOX Total Preconditioner Construction	1	Construct Preconditioner
NOX Total Linear Solve	1	Linear Solve

Specifications

Test cases:
  ant-2-20km_ml_ls_np384
  ant-2-20km_mu_ls_np384
  ant-2-20km_mu_dls_np384
  green-1-10km_vel_mu_tet_np384
  green-1-10km_vel_muk_tet_np384
  green-1-10km_vel_mu_wdg_np384
  green-1-10km_ent_fea_1ws_tet_np384
  green-1-10km_ent_fea_1ws_wdg_np384
  green-1-10km_ent_fea_mem_tet_np384
  green-1-10km_ent_fea_mem_wdg_np384
  green-1-10km_ent_mu_tet_np384
  green-1-10km_ent_muk_tet_np384
  green-1-10km_ent_mu_wdg_np384
  green-1-7km_fea_1ws_np384
  green-1-7km_fea_mem_np384
  green-1-7km_ml_ls_mem_np384
  green-1-7km_mu_dls_1ws_np384
  green-1-7km_mu_dls_mem_np384
  green-1-7km_mu_ls_mem_np384
  green-1-7km_muk_ls_mem_np384
  green-3-20km_vel_fea_mem_tet_np384
  green-3-20km_vel_fea_mem_wdg_np384
  green-3-20km_ent_fea_mem_tet_np384
  green-3-20km_ent_fea_mem_wdg_np384
  green-3-20km_ent_muk_wdg_np384
  green-3-20km_beta_1ws_np384
  green-3-20km_beta_mem_np384
  green-3-20km_beta_memp_np384
  humboldt-1-10km_cop_fro_wdg_np384
  humboldt-1-10km_cop_if2_wdg_np384
Metrics:
  Total Time
  Setup Time
  Total Fill Time
  Residual Fill
  Residual Fill Evaluate
  Residual Fill Export
  Jacobian Fill
  Jacobian Fill Evaluate
  Jacobian Fill Export
  NOX Total Preconditioner Construction
  NOX Total Linear Solve
  max host memory
  max kokkos memory
Metadata:
  Albany cxx compiler
  Albany git commit id
  Trilinos git commit id
Model threshold: 0.005000

Performance Timelines

Events in the most recent 10 days:

Plot of wall-clock times or memory for nightly runs

Changepoints are estimated using a generalized likelihood ratio method on each timer, and then merged over all timers for a given test case.

• Blue markers: recorded wall-clock time or memory
• Solid red line: average wall-clock time or memory between changepoints
• Dotted red lines: average wall-clock time or memory $\pm$ two standard deviations

Plot window controls

• Test case and timer can be selected from the drop-down menus (* denotes recent changes detected)
• Hovering over data points shows various metadata.
  • The mean with upper and lower bounds of a 99% confidence interval.
  • The relative performance (speedup,slowdown) between sets identified by changepoints with a 99% confidence interval for the ratio (difference between log mean of the two sets).
  • Note: The confidence interval is based on a t-statistic, so for very small amounts of data (<5), the interval may be very large. NaNs are converted to zero.
• Clicking on the legend will show/hide individual plot elements
• Click and drag to zoom in; double click to reset zoom

Pollak, Moshe; Siegmund, D. Sequential Detection of a Change in a Normal Mean when the Initial Value is Unknown. Ann. Statist. 19 (1991), no. 1, 394--416. doi:10.1214/aos/1176347990. https://projecteuclid.org/euclid.aos/1176347990

Siegmund, D.; Venkatraman, E. S. Using the Generalized Likelihood Ratio Statistic for Sequential Detection of a Change-Point. Ann. Statist. 23 (1995), no. 1, 255--271. doi:10.1214/aos/1176324466. https://projecteuclid.org/euclid.aos/1176324466

Hawkins, D. M., & Zamba, K. D. (2005). Statistical Process Control for Shifts in Mean or Variance using a Change Point Formulation. Technometrics, 47, 164-173.

Hawkins DM, Qiu P, Kang CW. The changepoint model for statistical process control. Journal of Quality Technology. 2003 Oct 1;35(4):355-366.