ALI Performance Tests on Weaver

Introduction

Currently testing the Greenland Ice Sheet (GIS) and Antarctica Ice Sheet (AIS) in Albany Land Ice (ALI) using Nvidia Tesla V100 GPUs on weaver. Note: Waterman was moved to a different network and renamed weaver. The last waterman test was executed on 6/16/2020.

Architectures:

Name	Weaver
CPU	Dual-socket IBM POWER9
GPU	Nvidia Tesla V100
Cores/Node	40
Threads/Core	8
GPUs/Node	4
Memory/Node	319 GB
Compiler	gcc/8.3.1
GPU Compiler	cuda/11.2.2
MPI	openmpi/4.1.1

Cases:

Case Name	Number of Processes (np)	Description
ant-8-30km_veldep	4	Unstructured 8-30km AIS, Velocity depth integrated model
ant-8-30km_velopt	4	Unstructured 8-30km AIS, Velocity optimization
ant-8-30km_vel_cheby_fixt	4	Unstructured 8-30km AIS, Velocity problem, Chebyshev smoothers, Fixed Newton step
green-3-20km_vel_gpu	4	Unstructured 3-20km GIS, Velocity problem, GPU preconditioner
humboldt-1-10km_vel_gpu	4	Unstructured 1-10km Humboldt Glacier, Velocity problem, GPU preconditioner
humboldt-1-10km_ent_gpu	4	Unstructured 1-10km Humboldt Glacier, Enthalpy problem, GPU preconditioner
humboldt-1-10km_cop_fea	4	Unstructured 1-10km Humboldt Glacier, Coupled problem, Finite element assembly only
mali-ais8km_gpu	4	MALI, Unstructured 8-30km AIS, GPU preconditioner

Timers:

Timer Name	Level	Description
MALI Total Time	0	Total wall-clock time of MALI simulation
MALI Extrude Time	1	Setup Albany problem
Albany Total Time	0	Total wall-clock time of Albany simulation
Albany Setup Time	1	Setup Albany problem
Residual Fill	1	Residual assembly
Jacobian Fill	1	Jacobian assembly
DistParam Fill	1	Distributed parameter assembly
RespParamHesVec Fill	1	Response parameter Hessian-vector product assembly
RespDistParamHesVec Fill	1	Response distributed parameter Hessian-vector product assembly
ResDistParamHesVec Fill	1	Residual distributed parameter Hessian-vector product assembly
NOX Total Preconditioner Construction	1	Construct Preconditioner
NOX Total Linear Solve	1	Linear Solve

Specifications

Test cases:
  ant-8-30km_veldep_np4
  ant-8-30km_velopt_np4
  ant-8-30km_vel_cheby_fixt_np4
  green-3-20km_vel_gpu_np4
  humboldt-1-10km_vel_gpu_np4
  humboldt-1-10km_ent_gpu_np4
  humboldt-1-10km_cop_fea_np4
  mali-ais8km_gpu_np4
Metrics:
  MALI Total Time
  MALI Extrude Time
  Albany Total Time
  Albany Setup Time
  Residual Fill
  Jacobian Fill
  DistParam Fill
  RespParamHesVec Fill
  RespDistParamHesVec Fill
  ResDistParamHesVec Fill
  NOX Total Preconditioner Construction
  NOX Total Linear Solve
  max host memory
  max kokkos memory
Metadata:
  Albany cxx compiler
  Albany git commit id
  Trilinos git commit id
Model threshold: 0.005000

Performance Timelines

Events in the most recent 10 days:
01/04/2025:
    ant-8-30km_vel_cheby_fixt_np4: max kokkos memory
01/05/2025:
    humboldt-1-10km_vel_gpu_np4: Jacobian Fill
01/07/2025:
    mali-ais8km_gpu_np4: Residual Fill
01/09/2025:
    ant-8-30km_veldep_np4: NOX Total Preconditioner Construction
    ant-8-30km_velopt_np4: DistParam Fill
                           NOX Total Preconditioner Construction
    ant-8-30km_vel_cheby_fixt_np4: NOX Total Preconditioner Construction
    green-3-20km_vel_gpu_np4: NOX Total Preconditioner Construction
    humboldt-1-10km_vel_gpu_np4: NOX Total Preconditioner Construction
    humboldt-1-10km_ent_gpu_np4: NOX Total Preconditioner Construction
    humboldt-1-10km_cop_fea_np4: NOX Total Preconditioner Construction
    mali-ais8km_gpu_np4: NOX Total Preconditioner Construction
01/10/2025:
    ant-8-30km_velopt_np4: Albany Setup Time
    ant-8-30km_vel_cheby_fixt_np4: Albany Total Time
01/12/2025:
    green-3-20km_vel_gpu_np4: Residual Fill

Plot of wall-clock times or memory for nightly runs

Changepoints are estimated using a generalized likelihood ratio method on each timer, and then merged over all timers for a given test case.

• Blue markers: recorded wall-clock time or memory
• Solid red line: average wall-clock time or memory between changepoints
• Dotted red lines: average wall-clock time or memory $\pm$ two standard deviations

Plot window controls

• Test case and timer can be selected from the drop-down menus (* denotes recent changes detected)
• Hovering over data points shows various metadata.
  • The mean with upper and lower bounds of a 99% confidence interval.
  • The relative performance (speedup,slowdown) between sets identified by changepoints with a 99% confidence interval for the ratio (difference between log mean of the two sets).
  • Note: The confidence interval is based on a t-statistic, so for very small amounts of data (<5), the interval may be very large. NaNs are converted to zero.
• Clicking on the legend will show/hide individual plot elements
• Click and drag to zoom in; double click to reset zoom

Pollak, Moshe; Siegmund, D. Sequential Detection of a Change in a Normal Mean when the Initial Value is Unknown. Ann. Statist. 19 (1991), no. 1, 394--416. doi:10.1214/aos/1176347990. https://projecteuclid.org/euclid.aos/1176347990

Siegmund, D.; Venkatraman, E. S. Using the Generalized Likelihood Ratio Statistic for Sequential Detection of a Change-Point. Ann. Statist. 23 (1995), no. 1, 255--271. doi:10.1214/aos/1176324466. https://projecteuclid.org/euclid.aos/1176324466

Hawkins, D. M., & Zamba, K. D. (2005). Statistical Process Control for Shifts in Mean or Variance using a Change Point Formulation. Technometrics, 47, 164-173.

Hawkins DM, Qiu P, Kang CW. The changepoint model for statistical process control. Journal of Quality Technology. 2003 Oct 1;35(4):355-366.