OPENSEES DAYS PORTUGAL 2014 UNCERTAINTY AND SENSITIVITY ANALYSIS USING HPC AND HTC André R. Barbosa (1) Andre.Barbosa@oregonstate.edu (1) Assistant Professor, School of Civil and Construction Engineering, Oregon State University Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Introduction Design Alternatives Hazard Analysis P [IM | X, D] Structural Analysis P [EDP | IM] Damage Analysis P [DM | EDP] Loss Analysis P [DV | DM] L,D L: Location D: Design Decision Making Select ν [IM] ν [EDP] ν [DM] ν [DV] Intensity Measure Engineering Demand Par. Damage Measure Decision Variable L,D q Parametric sensitivity studies / optimization / design (Luis Celorrio-‐Barragué) q Probabilistic seismic demand analysis Ø Cloud Method Ø Incremental dynamic analysis (Filipe Ribeiro) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 2 Introduction Design Alternatives Hazard Analysis P [IM | X, D] Structural Analysis P [EDP | IM] Damage Analysis P [DM | EDP] Loss Analysis P [DV | DM] L,D L: Location D: Design Decision Making Select ν [IM] ν [EDP] ν [DM] ν [DV] Intensity Measure Engineering Demand Par. Damage Measure Decision Variable L,D q Parametric sensitivity studies q Probabilistic seismic demand analysis Ø Cloud Method Ø Incremental dynamic analysis Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 3 Probabilistic Seismic Hazard Analysis i =1 Ri M i Fault km0 fIM R (r) Site i = m, Ri = r ⎤⎦ f M i ( m ) f Ri ( r ) dm dr AAenua8on rela8ons Fault j Fault i fM(m) ∫ P ⎡⎣ IM > im M RR M mu Seismic hazard curve Magnitude Source-‐to-‐site distance fR(r) IM ν IM ( im ) = ∑ν i ∫ fM(m) N flt m0 M mu RR M-‐R deaggrega8on ν IM (im ) IM = Sa ( T1 ) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Response estimation accounting for modeling uncertainty q PSDA equa9on accoun9ng for model parameter uncertainty: ν EDP ( edp ) = ∫ P [ EDP > edp | IM , Θ] f ( Θ ) d Θ ⋅ dν (im ) Θ IM IM q Response es9ma9on: P ⎡⎣ EDP > edp | IM = im, Θ = {θ1,k ,...,θl ,k }⎤⎦ INPUT NLTH ANALYSIS OUPUT µθ + aσ θ XLB XM XUB EDPLB EDPM EDPUB Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 5 Parameter uncertainty progagation OUTPUT INPUT Probability Distribution of EDPj Probability Distribution of RV X XLB XM EDP(XLB) XUB Uncertainty in ground motion Intensity Measure (IM) Ground motion profile (GM) Uncertainty in structural properties Mass Viscous damping Strength Stiffness 3D NL FE MODEL TIME HISTORY ANALYSIS EDP(XM) EDP(XUB) Global EDPs U : Max Roof Displacement A : Max Floor Acceleration. IDR : Max Interstory Drift Ratio Local EDPs Member: Curvature Strains: Reinforcing Steel Concrete Faggella , Barbosa, Conte, Spacone, Restrepo, 2013 Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Parameter uncertainty progagation OUTPUT INPUT Probability Distribu9on of EDP j Probability Distribu9on of Variable X X LB X M EDP(X LB ) X UB TORNADO 3D NL FE MODEL TIME HISTORY ANALYSIS EDP(X M ) EDP(X TORNADO (swing) x10 , x50 , x90 EDP(x10) – EDP( x90) FOSM FOSM (First Order Second Moments) mEDP , sEDP xm-as , xm , xm+as MEAN and STD Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto UB ) Tornado sensitivity analysis TORNADO x10 , x50 , x90 1 0.9 0.8 XLB 3D NL FE MODEL TIME HISTORY ANALYSIS XM XUB Procedure 1. Perform Monte Carlo Simulation using all ground motions (GM), fixing all other variables at their best estimates (median values) (e.g. GM = 20) Swing = EDP(x10) – EDP(x90) 2. For each EDP, determine Median GM, and perturbe all other variables one at a time about their median value Empirical CDF 0.7 Median GM 11th value 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.5 EDP 1 1.5 2 2.5 3 Sa GM Damping Mass Fy Fc Es Ec Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto First Order Second Moment (FOSM) sensitivity analysis q Mean values q Variance-covariance matrix µθ = [µ1 , µ2 , K , µn ] T Σθ = ⎡⎣ ρijσ iσ j ⎤⎦ ; i, j = 1, 2,K , n µθ + aσ θ q Taylor series expansion of the response EDP r (θ ) ≈ rlin (θ ) = r (µθ ) + ∇θ r θ =µ ⋅ (θ − µθ ) θ Ø Sensitivity ∂r (θ ) r ( µi + Δθi ) − r ( µi − Δθi ) = ∂θi 2Δθi XLB XM XUB Δθi = a σ θi Ø Covariance matrix of the response Σ 2r 2 n ⎛ ∂r ⎞ 2 = ⎜ ⎟ ⋅ σ θi + 2 i =1 ⎝ ∂θi ⎠ i =1 n ∑ ⎛ ∂r ρθiθ j ⎜ j =1 ⎝ ∂θi i −1 ∑∑ ⎞ ⎛ ∂r ⎟ ⎜⎜ ⎠ ⎝ ∂θ j ⎞ ⎟ ⋅ σ θi ⋅ σ θ j ⎟ ⎠ EDPLB EDPM EDPUB Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 9 Number of FE runs for TORNADO or FOSM analyses 1 Number of FE runs: 0.9 0.8 n runs = GM + 2 ⋅ RV ⋅ EDP Median GM 0.6 11th value (e.g., n runs = 20 + 2 × 7 ×10 = 160) 0.5 0.4 EDP 1 1.5 2 2.5 Sa 9 PF_temb PF_cs08 PF_cs05 MH_hall MH_clyd 8 MH_andd 7 LV_mgnp 6 LV_fgnr 5 CL_gil6 4 CL_clyd 3 10 11 12 13 14 15 16 17 18 19 20 MONTE CARLO TORNADO 3 dLB TORNADO 4 mLB TORNADO Damping 5 fyLB TORNADO Mass 6 fcLB TORNADO 7 EsLB TORNADO 8 EcLB TORNADO 9 IMUB TORNADO 10 dUB TORNADO 11 mUB TORNADO Es 12 fyUB TORNADO Ec 13 fcUB TORNADO 14 EsUB TORNADO 15 EcUB TORNADO GM Fy Fc TORNADO 1 2 TO_ttrh02 2 3 EDP 2 TO_ttr007 0.5 LP_srtg 0 LP_lgpc 0 LP_lex1 KB_kobj GM 1 med IMLB 0.1 LP_gilb EZ_erzi 0.2 LP_gav 0.3 EDP 1 LP_cor Empirical CDF 0.7 10 Swing = EDP(x10) – EDP(x90) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Parallelization of the analyses using XSEDE Acceleration (g) 0.4 0.2 0 -0.2 -0.4 0 5 10 Time (sec ) 15 20 10 Time (sec ) 15 20 Parallel Computer GM 1, Par j Acceleration (g) 0.4 0.2 0 -0.2 -0.4 0 5 GM 2, Par j … Acceleration (g) … 0.4 0.2 0 -0.2 -0.4 0 5 10 Time (sec ) 15 20 SUPERCOMPUTERS GM N, Par j OpenSees Mul9ple Parallel Interpreter (McKenna and Fenves 2007) hVp://opensees.berkeley.edu/OpenSees/parallel/TNParallelProcessing.pdf Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Case study: Bonefro 4 story Example 1: Bonefro Italybuilding Molise 2002 earthquake, Italy Faggella et al. 2008 Severe damage to first story infills and columns Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Variation of the response under different modeling Model (class) uncertainty assumptions Bare Frame NL Infills Stairs NL Inf. Bare 1st story Diaphragms (2x2) NL Shear columns Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Variation of the response under different modeling Model uncertainty assumptions 4 infilled 1500 part. infilled 3 shell 2x2 Floor Base Shear (KN) 2000 stairs 1000 bare frame 500 1 0 0 50 100 150 Top floor displacement (mm) TC 1 0.7 0.4 infilled part. infilled 3 stairs 8 . 0 .89 0 shell 2x2 9 1.0 5 1.2 bare frame 0.2 0 0.05 0.1 Sd e (m) 0.15 50 100 150 200 TH Average 3 2 1 2 0 0 4 Floor * Se/g , F /gm 0.6 Bare Frame Diaph.2x2 Stairs NL Inf. Bare1 NL Infills NLshear col. Displacements (mm) * 0.8 200 ADRS Demand Spectrum Capacity Spectra 0.4 1 0.15 0 TH Average 2 0.2 0 0 0.5 1 Drift % 1.5 2 12 Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Ground motion and structural random variables Parameter uncertainty Uncertainty in structural properties Uncertainty in ground motion • Mass • Viscous damping • Strength • Stiffness • Intensity Measure (IM) • Ground motion profile (GM) GM IM=Sa(T1) (g) Damping (%) Mass (ton/m2) Fy (MPa) Fc (MPa) Es (GPa) Ec (GPa) MCS Logn. Norm. Norm. Logn. Norm. Norm. Norm. XM On EDP 0.2931 0.03 0.87 451 25 210 28 COV % // 84 40 10 10 6.4 3.3 8 Distrib. Probability Functions based on • Seismic hazard • Values adopted in the literature • Experimental samples (material testing) 5 Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 3D Response Engineering OutputsDemand (EDPs) Parameters (EDPs) GLOBAL V Y U : Max Roof Displacement X Rz A : Max Floor Acceleration. G IDR : Max Interstory Drift Ratio 4001 4008 R σ,ε Steel Concrete core Concrete unconf. 3001 121 2001 122 1001 Μ, Χ 2008 Member Sections Moment 1008 Member Sections Curvature 3008 LOCAL 25 Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Results of MCS and(EDPs) TORNADO analysis Outputs Monte Carlo using 20 ground motions all other variables at medians Median MGM (11° value) V Y X Rz G 3D EDPs Floor DOFs R Tornado for MGM, all other variables perturbed one at a time about the median 26 Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto X Rz G A : Max Floor Acceleration. Outputs (EDPs) IDR : Max Interstory Drift Ratio 4001 4008 R σ,ε Steel Concrete core Concrete unconf. 3001 121 2001 122 1001 Μ, Χ 2008 Member Sections Moment 1008 Member Sections Curvature 3008 LOCAL 25 ! Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Outputs (EDPs) 3D Response Engineering Demand Param GLOBAL V Y U : Max Roof Displacement X Rz A : Max Floor Acceleration. G IDR : Max Interstory Drift Ratio 4001 4008 R σ,ε Steel Concrete core Concrete unconf. 3001 121 2001 122 1001 Μ, Χ 2008 Member Sections Moment 1008 Member Sections Curvature 3008 LOCAL Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto PEER PBEE Methodology Design Alternatives Hazard Analysis P [IM | X, D] Structural Analysis P [EDP | IM] Damage Analysis P [DM | EDP] Loss Analysis P [DV | DM] L,D L: Location D: Design Decision Making Select ν [IM] ν [EDP] ν [DM] ν [DV] Intensity Measure Engineering Demand Par. Damage Measure Decision Variable L,D q Parametric sensitivity studies q Probabilistic seismic demand analysis Ø Cloud Method Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 20 Example 2: NEHRP Building Modeling Approach q Comprehensive/signiﬁcant valida8on at system level ? … q Comprehensive/signiﬁcant valida8on at component level Ø Walls: Nonlinear truss modeling approach Ø Columns and beams: Force-based beam-column elements Ø Diaphragms: Flexible diaphragms allowing for plastic hinge elongation u&&g q Rigid-end zone modeling for beam-column joints (ASCE41-06) NL REZ NL NL NL NL 21 Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Observed computational building behavior EW: 0.44 % NS: 2.93 % N (%) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 22 “Cloud method”: Selection of earthquake records q NGA database (total 3551 records) Ø Mechanism: Strike-slip (1004 records) Ø Magnitude range: 5.5 to 8 (772 records) Ø Distance: 0 – 40 kms (203 records) Ø Vs30: C/D range (90 records) Source-to-site distance Rrup 40 35 30 25 20 Non-pulse 15 Pulse 10 Non-pulse 5 Pulse 0 5.5 6.0 6.0 6.5 6.5 7.0 7.5 8.0 q 90 ground mo8on records selected from 14 earthquakes Magnitude Mw 7.0 7.5 8.0 WorkshopMagnitude on Multi-Hazard M Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 23 OpenSees and Large Number of Runs q Motivation Ø Perform parametric studies that involve large-scale nonlinear models of structure or soil-structure systems with OpenSees runs. q Application Example/Production campaign 1 (1) Probabilistic seismic demand hazard analysis using the “cloud method” q Some numbers for this application example Number of NLTH analyses 180 Average duration of NLTH analysis 12 hours Average size of output data (compressed) 1.4 GB Estimated clock time on a desktop computer (180x12) Estimated size of output data GM1 (180x1.4) 2,160 hours 90 days 250 GB GM2 1. OpenSeesMP + Xsede? 2. Local Cluster? 3. Other options? .. . GM180 Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 24 Possible Parallelization Options q OpenSeesMP + MPICH2 – useful for Domain Decomposition + Parameter Studies (addressed by other talks in this meeting) q Condor + OpenSees Sequential – Parameter Studies Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto HTCondor q HTCondor (http://research.cs.wisc.edu/htcondor/) is a specialized workload management system for computational-intensive jobs. Ø Project started in 1988, directed at users with large computing needs and environments with heterogeneous distributed resources. (3) Worker Node Ø HTCondor is composed of 3 parts: Startd (1) Submit Node Submit job (2) Central Manager Collector Worker Node Schedd Get results GM1 Negotiator Startd … GM180 Worker Node Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Oregon State University: HTCondor + OpenSees q “Opportunistic” computing resources: q Student computer labs (used by students mainly during the day, and during the term …) q Instruction computer labs (used during the term only during classes …) q College of Engineering at OSU: 16 computer labs (~1500 cores) http://monhost.engr.orst.edu/labs/ Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Implementation of HTCondor at Oregon State University (1) Submit Node • • • • • • (3) Worker Nodes 8 core Intel i7 Windows Server 16 GB RAM SSD drive 2 TB HDD 15K 20 TB NAS (2) Central Manager • Windows 7 Premium • 8 GB RAM • 2 x 1GB cards • 1 TB 7.2 K 1 … The good news: ~ 1500cores Communication w/ IT, Dealing w/ Job recovery, W/O speed, data transfers, …? Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto OpenSees and Large Number of Runs q Motivation Ø Perform parametric studies that involve large-scale nonlinear models of structure or soil-structure systems with OpenSees runs. q Application Example/Production campaign 1 (1) Probabilistic seismic demand hazard analysis using the “cloud method” q Some numbers for this application example Number of NLTH analyses 180 Average duration of NLTH analysis 12 hours Average size of output data 1.4 GB Estimated clock time on a desktop computer (180x12) Estimated size of output data (180x1.4) 2,160 hours 90 days 250 GB Clock time 36 hours !! Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 29 OpenSees and Parameters Studies (a) (b) Individual Ekqe (c) 2.5- and 97.5-perc Median PFD – peak floor displacement; PIDR – peak interstory drift ratio; PFA – peak floor absolute acceleration Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 30 HTCondor and Open Science Grid q HTCondor (hAp://research.cs.wisc.edu/htcondor/) is a specialized workload management system for computa9onal-‐intensive jobs. Ø Project started in 1988, directed at users with large compu9ng needs and environments with heterogeneous distributed resources. q Open Science Grid is a national, distributed computing grid for data-intensive research. Ø Consortium of approx. 80 national laboratories and universities. Ø Version of Condor for the grid Ø Opportunistic resource usage: resources are sized for peak needs of large experiments (Atlas, CMS, etc.), OSG allows for non-paying organizations to use their resources. q NEES and Open Science Grid have been active partners in creating the tools and infrastructures for making use of opportunistic resources Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 31 Response estimation accounting for parameter uncertainty GM Damping (%) Mass fy (ksi) *fc (ksi) Es (ksi) *Ec (ksi) XM MCS 0.02 µθ 68.7 6.84 29000 4714 COV % // 40 10 10 10 3.3 8 INPUT NLTH ANALYSIS OUPUT µθ + aσ θ XLB XM XUB EDPLB EDPM EDPUB Uncertainty in structural properties Engineering demand parameters • Mass • Viscous damping • Strength • Stiffness • Roof drift ratio • Peak floor accelerations • Shear demand in walls • Residual deformatios.. Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 32 Using Open Science Grid: Production Campaign 2 q Production campaign (1) Probabilistic seismic demand hazard analysis using the cloud method (2) Sensitivity of probabilistic seismic demand hazard to FE model parameters q Some numbers for production campaign 2 (99% complete) Number of NLTH analyses per parameter set realization 180 Average duration of NLTH analysis 12 hours Average size of output data 1.4 GB Parameters considered 6 Perturbations considered 4 Estimated clock time on a desktop computer (180x12x[(6x4x2)+1]) 105,840 hours Estimated size of output (compressed) data (180x1.4x[(6x4x2)+1]) 12 TB 12.1 years Clock time 30 days !! Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 33 Wall clock time in HTCondor / OSG 30,000 Wall Time (hours) 25,000 20,000 12 clusters of 180 jobs “Desktop”: 26,000 hours OSG: 60,000 hours 15,000 (job preemp9on) 10,000 5,000 0 OSG users: André R. Barbosa, Taylor Gugino (UCSD) OSG support: Gabriele Garzoglio, Marko Slyz (OSG) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 34 Wall clock time in HTCondor / OSG Wall Time (hours) 160,000 120,000 80,000 40,000 0 OSG users: André R. Barbosa, Taylor Gugino (UCSD) OSG support: Gabriele Garzoglio, Marko Slyz (OSG) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Comparison Between Parallelization Options OpenSeesMP HTCondor Straight forward implementation of No ready built solution for large problems, Domain Decomposition through OpenSees OpenSees sequential does not have framework with parallel solving algorithm parallel solvers for large problems like MUMPS MPICH2 networking setup is relatively easier Job management easier Condor pool setup requires some learning Condor requires maintenance and administration Very active user support through There is no specific user community as OpenSees user community, most attractive such. aspect of using OpenSeesMP Limited tests show 190 % Speed up from one processor to two processor Limited tests show 153 % Speed up from one processor to two processor Main complication is compilation of OpenSeesMP, really really tough!! Global implementation, if want to connect to other grid systems. But once over it OpenSeesMP is really powerfull!!! Steep learning curve , knowledge of networking (Computer science) Khaled Mashﬁq, MS – La Sapienza, Rome Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto Conclusions ü A workflow for running parametric studies that involve large-scale nonlinear models of structure or soil-structure systems with large number of parameters and OpenSees runs has been developed for using NEEShub, Xsede, and Open Science Grid. ü HTCondor ü Pegassus (see Frank Mckenna’s presentation) ü OpenSees + Condor q User interfaces for submitting jobs, receiving results q Data visualization ü Management and Analysis of Large Research Data Sets q Where and what to store? q Post-processing? Data compression algorithms? 37 Andre.Barbosa@oregonstate.edu Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 38

1/--pages