ALI Performance Tests on Blake

Introduction

Currently testing the Greenland Ice Sheet (GIS) and Antarctica Ice Sheet (AIS) in Albany Land Ice (ALI) using Intel Sapphire Rapids CPUs on blake.

Architectures:

Name	Blake
CPU	Dual-socket Intel Xeon Platinum 8468 (Sapphire Rapids)
Cores/Node	96
Threads/Core	2
Memory/Node	251 GB
Compiler	gcc/12.2.0
MPI	openmpi/4.1.5

Cases:

Case Name	Number of Processes (np)	Description
ant-4-20km_vel_cpu	96	Unstructured 4-20km AIS, Velocity problem, CPU preconditioner
ant-8-30km_veldep	96	Unstructured 8-30km AIS, Velocity depth integrated model
ant-8-30km_velopt	96	Unstructured 8-30km AIS, Velocity optimization
ant-8-30km_vel_lsgs	96	Unstructured 8-30km AIS, Velocity problem, Line smoothing Gauss-Seidel
green-1-10km_vel_cpu	96	Unstructured 1-10km GIS, Velocity problem, CPU preconditioner
green-3-20km_vel_cpu	96	Unstructured 3-20km GIS, Velocity problem, CPU preconditioner
humboldt-1-10km_vel_cpu	96	Unstructured 1-10km Humboldt Glacier, Velocity problem, CPU preconditioner
humboldt-1-10km_ent_cpu	96	Unstructured 1-10km Humboldt Glacier, Enthalpy problem, CPU preconditioner
humboldt-1-10km_cop_fro_slu	96	Unstructured 1-10km Humboldt Glacier, Coupled problem, FROSch + SuperLU
mali-ais4km_cpu	96	MALI, Unstructured 4-20km AIS, CPU preconditioner
mali-ais8km_cpu	96	MALI, Unstructured 8-30km AIS, CPU preconditioner
thwaites-1-10km_vel_cpu	96	Unstructured 1-10km Thwaites Glacier, Velocity problem, CPU preconditioner
thwaites-1-10km_ent_cpu	96	Unstructured 1-10km Thwaites Glacier, Enthalpy problem, CPU preconditioner
ant-16km_vel_cpu	96	Structured 16km AIS, Velocity problem, CPU preconditioner

Timers:

Timer Name	Level	Description
MALI Total Time	0	Total wall-clock time of MALI simulation
MALI Extrude Time	1	Setup Albany problem
Albany Total Time	0	Total wall-clock time of Albany simulation
Albany Setup Time	1	Setup Albany problem
Residual Fill	1	Residual assembly
Jacobian Fill	1	Jacobian assembly
DistParam Fill	1	Distributed parameter assembly
RespParamHesVec Fill	1	Response parameter Hessian-vector product assembly
RespDistParamHesVec Fill	1	Response distributed parameter Hessian-vector product assembly
ResDistParamHesVec Fill	1	Residual distributed parameter Hessian-vector product assembly
NOX Total Preconditioner Construction	1	Construct Preconditioner
NOX Total Linear Solve	1	Linear Solve

Specifications

Test cases:
  ant-4-20km_vel_cpu_np96
  ant-8-30km_veldep_np96
  ant-8-30km_velopt_np96
  ant-8-30km_vel_lsgs_np96
  green-1-10km_vel_cpu_np96
  green-3-20km_vel_cpu_np96
  humboldt-1-10km_vel_cpu_np96
  humboldt-1-10km_ent_cpu_np96
  humboldt-1-10km_cop_fro_slu_np96
  mali-ais4km_cpu_np96
  mali-ais8km_cpu_np96
  thwaites-1-10km_vel_cpu_np96
  thwaites-1-10km_ent_cpu_np96
  ant-16km_vel_cpu_np96
Metrics:
  MALI Total Time
  MALI Extrude Time
  Albany Total Time
  Albany Setup Time
  Residual Fill
  Jacobian Fill
  DistParam Fill
  RespParamHesVec Fill
  RespDistParamHesVec Fill
  ResDistParamHesVec Fill
  NOX Total Preconditioner Construction
  NOX Total Linear Solve
  max host memory
  max kokkos memory
Metadata:
  Albany cxx compiler
  Albany git commit id
  Trilinos git commit id
Model threshold: 0.005000

Performance Timelines

Events in the most recent 10 days:
11/16/2024:
    ant-4-20km_vel_cpu_np96: Albany Total Time
                             Albany Setup Time
                             Residual Fill
                             Jacobian Fill
                             NOX Total Linear Solve
                             max host memory
                             max kokkos memory
    ant-8-30km_veldep_np96: Albany Total Time
                            Albany Setup Time
                            Residual Fill
                            Jacobian Fill
                            NOX Total Linear Solve
                            max host memory
                            max kokkos memory
    ant-8-30km_velopt_np96: Albany Total Time
                            Albany Setup Time
                            Residual Fill
                            Jacobian Fill
                            RespParamHesVec Fill
                            RespDistParamHesVec Fill
                            ResDistParamHesVec Fill
                            NOX Total Linear Solve
                            max host memory
                            max kokkos memory
    ant-8-30km_vel_lsgs_np96: Albany Total Time
                              Albany Setup Time
                              Residual Fill
                              Jacobian Fill
                              NOX Total Linear Solve
                              max host memory
                              max kokkos memory
    green-1-10km_vel_cpu_np96: Albany Total Time
                               Albany Setup Time
                               Residual Fill
                               Jacobian Fill
                               NOX Total Linear Solve
                               max host memory
                               max kokkos memory
    green-3-20km_vel_cpu_np96: Albany Total Time
                               Albany Setup Time
                               Residual Fill
                               Jacobian Fill
                               NOX Total Linear Solve
                               max host memory
                               max kokkos memory
    humboldt-1-10km_vel_cpu_np96: Albany Total Time
                                  Albany Setup Time
                                  Residual Fill
                                  Jacobian Fill
                                  NOX Total Linear Solve
                                  max host memory
                                  max kokkos memory
    humboldt-1-10km_ent_cpu_np96: Albany Total Time
                                  Albany Setup Time
                                  Residual Fill
                                  Jacobian Fill
                                  NOX Total Linear Solve
                                  max host memory
                                  max kokkos memory
    humboldt-1-10km_cop_fro_slu_np96: Albany Total Time
                                      Albany Setup Time
                                      Residual Fill
                                      Jacobian Fill
                                      NOX Total Linear Solve
                                      max host memory
                                      max kokkos memory
    mali-ais4km_cpu_np96: MALI Total Time
                          MALI Extrude Time
                          Residual Fill
                          Jacobian Fill
                          NOX Total Linear Solve
                          max host memory
                          max kokkos memory
    mali-ais8km_cpu_np96: MALI Total Time
                          MALI Extrude Time
                          Residual Fill
                          Jacobian Fill
                          NOX Total Linear Solve
                          max host memory
                          max kokkos memory
    thwaites-1-10km_vel_cpu_np96: Albany Total Time
                                  Albany Setup Time
                                  Residual Fill
                                  Jacobian Fill
                                  NOX Total Linear Solve
                                  max host memory
                                  max kokkos memory
    thwaites-1-10km_ent_cpu_np96: Albany Total Time
                                  Albany Setup Time
                                  Residual Fill
                                  Jacobian Fill
                                  NOX Total Linear Solve
                                  max host memory
                                  max kokkos memory
    ant-16km_vel_cpu_np96: Albany Total Time
                           Albany Setup Time
                           Residual Fill
                           Jacobian Fill
                           NOX Total Linear Solve
                           max host memory
                           max kokkos memory
11/21/2024:
    mali-ais4km_cpu_np96: max kokkos memory
    mali-ais8km_cpu_np96: max kokkos memory

Plot of wall-clock times or memory for nightly runs

Changepoints are estimated using a generalized likelihood ratio method on each timer, and then merged over all timers for a given test case.

• Blue markers: recorded wall-clock time or memory
• Solid red line: average wall-clock time or memory between changepoints
• Dotted red lines: average wall-clock time or memory $\pm$ two standard deviations

Plot window controls

• Test case and timer can be selected from the drop-down menus (* denotes recent changes detected)
• Hovering over data points shows various metadata.
  • The mean with upper and lower bounds of a 99% confidence interval.
  • The relative performance (speedup,slowdown) between sets identified by changepoints with a 99% confidence interval for the ratio (difference between log mean of the two sets).
  • Note: The confidence interval is based on a t-statistic, so for very small amounts of data (<5), the interval may be very large. NaNs are converted to zero.
• Clicking on the legend will show/hide individual plot elements
• Click and drag to zoom in; double click to reset zoom

Pollak, Moshe; Siegmund, D. Sequential Detection of a Change in a Normal Mean when the Initial Value is Unknown. Ann. Statist. 19 (1991), no. 1, 394--416. doi:10.1214/aos/1176347990. https://projecteuclid.org/euclid.aos/1176347990

Siegmund, D.; Venkatraman, E. S. Using the Generalized Likelihood Ratio Statistic for Sequential Detection of a Change-Point. Ann. Statist. 23 (1995), no. 1, 255--271. doi:10.1214/aos/1176324466. https://projecteuclid.org/euclid.aos/1176324466

Hawkins, D. M., & Zamba, K. D. (2005). Statistical Process Control for Shifts in Mean or Variance using a Change Point Formulation. Technometrics, 47, 164-173.

Hawkins DM, Qiu P, Kang CW. The changepoint model for statistical process control. Journal of Quality Technology. 2003 Oct 1;35(4):355-366.