close

Enter

Log in using OpenID

embedDownload
OPENSEES DAYS PORTUGAL 2014
UNCERTAINTY AND SENSITIVITY ANALYSIS
USING HPC AND HTC
André R. Barbosa
(1) Andre.Barbosa@oregonstate.edu (1) Assistant Professor, School of Civil and Construction Engineering, Oregon State University
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Introduction
Design
Alternatives
Hazard
Analysis
P [IM | X, D]
Structural
Analysis
P [EDP | IM]
Damage
Analysis
P [DM | EDP]
Loss
Analysis
P [DV | DM]
L,D L: Location
D: Design
Decision
Making
Select
ν [IM]
ν [EDP]
ν [DM]
ν [DV]
Intensity
Measure
Engineering
Demand Par.
Damage
Measure
Decision
Variable
L,D q  Parametric sensitivity studies / optimization / design
(Luis Celorrio-­‐Barragué)
q  Probabilistic seismic demand analysis
Ø Cloud Method
Ø Incremental dynamic analysis (Filipe Ribeiro) Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
2
Introduction
Design
Alternatives
Hazard
Analysis
P [IM | X, D]
Structural
Analysis
P [EDP | IM]
Damage
Analysis
P [DM | EDP]
Loss
Analysis
P [DV | DM]
L,D L: Location
D: Design
Decision
Making
Select
ν [IM]
ν [EDP]
ν [DM]
ν [DV]
Intensity
Measure
Engineering
Demand Par.
Damage
Measure
Decision
Variable
L,D q  Parametric sensitivity studies
q  Probabilistic seismic demand analysis
Ø Cloud Method
Ø Incremental dynamic analysis
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
3
Probabilistic Seismic Hazard Analysis
i =1
Ri M i
Fault km0
fIM
R (r)
Site
i
= m, Ri = r ⎤⎦ f M i ( m ) f Ri ( r ) dm dr
AAenua8on rela8ons Fault j
Fault i
fM(m)
∫ P ⎡⎣ IM > im M
RR
M mu
Seismic hazard curve Magnitude Source-­‐to-­‐site distance fR(r)
IM
ν IM ( im ) = ∑ν i ∫
fM(m)
N flt
m0
M mu
RR
M-­‐R deaggrega8on ν IM (im )
IM = Sa ( T1 )
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Response estimation accounting for modeling uncertainty
q  PSDA equa9on accoun9ng for model parameter uncertainty: ν EDP ( edp ) =
∫ P [ EDP > edp | IM , Θ] f ( Θ ) d Θ ⋅ dν (im )
Θ
IM
IM
q  Response es9ma9on: P ⎡⎣ EDP > edp | IM = im, Θ = {θ1,k ,...,θl ,k }⎤⎦
INPUT
NLTH ANALYSIS
OUPUT
µθ + aσ θ
XLB XM XUB EDPLB EDPM EDPUB Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
5
Parameter uncertainty progagation
OUTPUT
INPUT
Probability Distribution of EDPj
Probability Distribution of RV X
XLB
XM
EDP(XLB)
XUB
Uncertainty in ground
motion
Intensity Measure (IM)
Ground motion profile (GM)
Uncertainty in structural
properties
Mass
Viscous damping
Strength
Stiffness
3D NL FE MODEL
TIME HISTORY ANALYSIS
EDP(XM)
EDP(XUB)
Global EDPs
U : Max Roof Displacement
A : Max Floor Acceleration.
IDR : Max Interstory Drift Ratio
Local EDPs
Member: Curvature
Strains: Reinforcing Steel
Concrete
Faggella , Barbosa, Conte, Spacone, Restrepo, 2013 Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Parameter uncertainty progagation
OUTPUT INPUT Probability Distribu9on of EDP j Probability Distribu9on of Variable X X LB X M EDP(X LB ) X UB TORNADO
3D NL FE MODEL TIME HISTORY ANALYSIS EDP(X M ) EDP(X TORNADO (swing)
x10 , x50 , x90
EDP(x10) – EDP( x90)
FOSM
FOSM
(First Order Second Moments)
mEDP , sEDP
xm-as , xm , xm+as
MEAN and STD
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
UB )
Tornado sensitivity analysis
TORNADO
x10 , x50 , x90
1
0.9
0.8
XLB
3D NL FE MODEL
TIME HISTORY ANALYSIS
XM
XUB
Procedure
1.  Perform Monte Carlo
Simulation using all ground
motions (GM), fixing all
other variables at their best
estimates (median values)
(e.g. GM = 20)
Swing =
EDP(x10) – EDP(x90)
2.  For each EDP, determine
Median GM, and perturbe
all other variables one at a
time about their median
value
Empirical CDF
0.7
Median GM
11th value
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.5
EDP
1
1.5
2
2.5
3
Sa
GM
Damping
Mass
Fy
Fc
Es
Ec
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
First Order Second Moment (FOSM) sensitivity analysis
q  Mean values
q  Variance-covariance matrix
µθ = [µ1 , µ2 , K , µn ]
T
Σθ = ⎡⎣ ρijσ iσ j ⎤⎦ ; i, j = 1, 2,K , n
µθ + aσ θ
q  Taylor series expansion of the response EDP
r (θ ) ≈ rlin (θ ) = r (µθ ) + ∇θ r θ =µ ⋅ (θ − µθ )
θ
Ø  Sensitivity
∂r (θ ) r ( µi + Δθi ) − r ( µi − Δθi )
=
∂θi
2Δθi
XLB XM XUB Δθi = a σ θi
Ø  Covariance matrix of the response
Σ 2r
2
n
⎛ ∂r ⎞
2
= ⎜
⎟ ⋅ σ θi + 2
i =1 ⎝ ∂θi ⎠
i =1
n
∑
⎛ ∂r
ρθiθ j ⎜
j =1
⎝ ∂θi
i −1
∑∑
⎞ ⎛ ∂r
⎟ ⎜⎜
⎠ ⎝ ∂θ j
⎞
⎟ ⋅ σ θi ⋅ σ θ j
⎟
⎠
EDPLB EDPM EDPUB Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
9
Number of FE runs for TORNADO or FOSM analyses
1
Number of FE runs:
0.9
0.8
n runs = GM + 2 ⋅ RV ⋅ EDP
Median GM
0.6
11th value
(e.g., n runs = 20 + 2 × 7 ×10 = 160)
0.5
0.4
EDP
1
1.5
2
2.5
Sa
9
PF_temb
PF_cs08
PF_cs05
MH_hall
MH_clyd
8
MH_andd
7
LV_mgnp
6
LV_fgnr
5
CL_gil6
4
CL_clyd
3
10 11 12 13 14 15 16 17 18 19 20
MONTE CARLO
TORNADO
3
dLB
TORNADO
4
mLB
TORNADO
Damping
5
fyLB
TORNADO
Mass
6
fcLB
TORNADO
7
EsLB
TORNADO
8
EcLB
TORNADO
9
IMUB
TORNADO
10
dUB
TORNADO
11
mUB
TORNADO
Es
12
fyUB
TORNADO
Ec
13
fcUB
TORNADO
14
EsUB
TORNADO
15
EcUB
TORNADO
GM
Fy
Fc
TORNADO
1
2
TO_ttrh02
2
3
EDP 2 TO_ttr007
0.5
LP_srtg
0
LP_lgpc
0
LP_lex1
KB_kobj
GM 1
med
IMLB
0.1
LP_gilb
EZ_erzi
0.2
LP_gav
0.3
EDP 1 LP_cor
Empirical CDF
0.7
10
Swing = EDP(x10) – EDP(x90)
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Parallelization of the analyses using XSEDE
Acceleration (g)
0.4
0.2
0
-0.2
-0.4
0
5
10
Time (sec )
15
20
10
Time (sec )
15
20
Parallel Computer
GM 1, Par j
Acceleration (g)
0.4
0.2
0
-0.2
-0.4
0
5
GM 2, Par j
…
Acceleration (g)
…
0.4
0.2
0
-0.2
-0.4
0
5
10
Time (sec )
15
20
SUPERCOMPUTERS
GM N, Par j
OpenSees Mul9ple Parallel Interpreter (McKenna and Fenves 2007) hVp://opensees.berkeley.edu/OpenSees/parallel/TNParallelProcessing.pdf Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Case study:
Bonefro
4 story
Example
1: Bonefro
Italybuilding
Molise 2002 earthquake, Italy
Faggella et al. 2008 Severe damage to first story
infills and columns
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Variation of the response under different modeling
Model (class) uncertainty
assumptions
Bare Frame
NL Infills
Stairs
NL Inf. Bare 1st story
Diaphragms (2x2)
NL Shear columns
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Variation of the response under different modeling
Model
uncertainty
assumptions
4
infilled
1500
part. infilled
3
shell 2x2
Floor
Base Shear (KN)
2000
stairs
1000
bare frame
500
1
0
0
50
100
150
Top floor displacement (mm)
TC
1
0.7
0.4
infilled
part. infilled
3 stairs
8
.
0 .89
0
shell 2x2
9
1.0
5
1.2 bare frame
0.2
0
0.05
0.1
Sd e (m)
0.15
50
100
150
200
TH Average
3
2
1
2
0
0
4
Floor
*
Se/g , F /gm
0.6
Bare Frame
Diaph.2x2
Stairs
NL Inf. Bare1
NL Infills
NLshear col.
Displacements (mm)
*
0.8
200
ADRS Demand Spectrum
Capacity Spectra
0.4
1
0.15
0
TH Average
2
0.2
0
0
0.5
1
Drift %
1.5
2
12
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Ground motion
and structural
random variables
Parameter
uncertainty
Uncertainty in structural properties
Uncertainty in ground motion
• Mass
• Viscous damping
• Strength
• Stiffness
• Intensity Measure (IM)
• Ground motion profile (GM)
GM
IM=Sa(T1)
(g)
Damping
(%)
Mass
(ton/m2)
Fy
(MPa)
Fc
(MPa)
Es
(GPa)
Ec
(GPa)
MCS
Logn.
Norm.
Norm.
Logn.
Norm.
Norm.
Norm.
XM
On EDP
0.2931
0.03
0.87
451
25
210
28
COV %
//
84
40
10
10
6.4
3.3
8
Distrib.
Probability Functions based on
• Seismic hazard
• Values adopted in the literature
• Experimental samples (material testing)
5
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
3D Response Engineering
OutputsDemand
(EDPs) Parameters (EDPs)
GLOBAL
V
Y
U : Max Roof Displacement
X
Rz
A : Max Floor Acceleration.
G
IDR : Max Interstory Drift Ratio
4001
4008
R
σ,ε
Steel
Concrete core
Concrete unconf.
3001
121
2001
122
1001
Μ, Χ
2008
Member Sections Moment
1008
Member Sections Curvature
3008
LOCAL
25
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Results of MCS
and(EDPs)
TORNADO analysis
Outputs
Monte Carlo using 20 ground motions
all other variables at medians
Median MGM
(11° value)
V
Y
X
Rz
G
3D EDPs
Floor DOFs
R
Tornado for MGM, all other variables perturbed one at a time about the median
26
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
X
Rz
G
A : Max Floor Acceleration.
Outputs
(EDPs)
IDR : Max
Interstory Drift Ratio
4001
4008
R
σ,ε
Steel
Concrete core
Concrete unconf.
3001
121
2001
122
1001
Μ, Χ
2008
Member Sections Moment
1008
Member Sections Curvature
3008
LOCAL
25
!
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Outputs (EDPs)
3D Response Engineering Demand Param
GLOBAL
V
Y
U : Max Roof Displacement
X
Rz
A : Max Floor Acceleration.
G
IDR : Max Interstory Drift Ratio
4001
4008
R
σ,ε
Steel
Concrete core
Concrete unconf.
3001
121
2001
122
1001
Μ, Χ
2008
Member Sections Moment
1008
Member Sections Curvature
3008
LOCAL
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
PEER PBEE Methodology
Design
Alternatives
Hazard
Analysis
P [IM | X, D]
Structural
Analysis
P [EDP | IM]
Damage
Analysis
P [DM | EDP]
Loss
Analysis
P [DV | DM]
L,D L: Location
D: Design
Decision
Making
Select
ν [IM]
ν [EDP]
ν [DM]
ν [DV]
Intensity
Measure
Engineering
Demand Par.
Damage
Measure
Decision
Variable
L,D q  Parametric sensitivity studies
q  Probabilistic seismic demand analysis
Ø Cloud Method
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 20
Example 2: NEHRP Building Modeling Approach
q Comprehensive/significant valida8on at system level ? … q Comprehensive/significant valida8on at component level Ø  Walls: Nonlinear truss modeling approach
Ø  Columns and beams: Force-based beam-column elements
Ø  Diaphragms: Flexible diaphragms allowing for plastic hinge
elongation
u&&g
q  Rigid-end zone
modeling for
beam-column
joints
(ASCE41-06)
NL REZ NL NL NL NL 21
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Observed computational building behavior
EW: 0.44 %
NS: 2.93 %
N
(%)
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 22
“Cloud method”: Selection of earthquake records
q  NGA database (total 3551 records)
Ø  Mechanism: Strike-slip (1004 records)
Ø  Magnitude range: 5.5 to 8 (772 records)
Ø  Distance: 0 – 40 kms (203 records)
Ø  Vs30: C/D range (90 records)
Source-to-site distance Rrup
40
35
30
25
20
Non-pulse
15
Pulse
10
Non-pulse
5
Pulse
0
5.5
6.0
6.0
6.5
6.5
7.0
7.5
8.0
q  90 ground mo8on records selected from 14 earthquakes Magnitude Mw
7.0
7.5
8.0
WorkshopMagnitude
on Multi-Hazard
M Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 23
OpenSees and Large Number of Runs
q  Motivation
Ø Perform parametric studies that involve large-scale nonlinear models of structure or
soil-structure systems with OpenSees runs. q  Application Example/Production campaign 1
(1)  Probabilistic seismic demand hazard analysis using the “cloud method”
q  Some numbers for this application example
Number of NLTH analyses
180
Average duration of NLTH analysis
12 hours
Average size of output data (compressed)
1.4 GB
Estimated clock time on a desktop computer
(180x12)
Estimated size of output data
GM1
(180x1.4)
2,160 hours
90 days
250 GB
GM2
1.  OpenSeesMP +
Xsede?
2.  Local Cluster?
3.  Other options?
..
.
GM180
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 24
Possible Parallelization Options
q OpenSeesMP + MPICH2 – useful for Domain
Decomposition + Parameter Studies (addressed by other
talks in this meeting)
q Condor + OpenSees Sequential – Parameter Studies
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
HTCondor
q  HTCondor (http://research.cs.wisc.edu/htcondor/) is a specialized workload management
system for computational-intensive jobs.
Ø  Project started in 1988, directed at users with large computing needs and environments
with heterogeneous distributed resources.
(3) Worker Node
Ø  HTCondor is composed of 3 parts:
Startd
(1) Submit Node
Submit job
(2) Central Manager
Collector
Worker Node
Schedd
Get results
GM1
Negotiator
Startd
…
GM180
Worker Node
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Oregon State University: HTCondor + OpenSees
q  “Opportunistic” computing resources:
q  Student computer labs (used by students mainly during the day, and during the
term …)
q  Instruction computer labs (used during the term only during classes …)
q  College of Engineering at OSU: 16 computer labs (~1500 cores)
http://monhost.engr.orst.edu/labs/
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Implementation of HTCondor at Oregon State University
(1) Submit Node
• 
• 
• 
• 
• 
• 
(3) Worker Nodes
8 core Intel i7
Windows Server
16 GB RAM
SSD drive
2 TB HDD 15K
20 TB NAS
(2) Central Manager
•  Windows 7
Premium
•  8 GB RAM
•  2 x 1GB cards
•  1 TB 7.2 K
1
… The good news: ~ 1500cores
Communication w/ IT, Dealing w/ Job
recovery, W/O speed, data transfers, …?
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
OpenSees and Large Number of Runs
q  Motivation
Ø Perform parametric studies that involve large-scale nonlinear models of structure or
soil-structure systems with OpenSees runs. q  Application Example/Production campaign 1
(1)  Probabilistic seismic demand hazard analysis using the “cloud method”
q  Some numbers for this application example
Number of NLTH analyses
180
Average duration of NLTH analysis
12 hours
Average size of output data
1.4 GB
Estimated clock time on a desktop computer
(180x12)
Estimated size of output data
(180x1.4)
2,160 hours
90 days
250 GB
Clock time
36 hours !!
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 29
OpenSees and Parameters Studies (a)
(b)
Individual Ekqe
(c)
2.5- and 97.5-perc
Median
PFD – peak floor displacement; PIDR – peak interstory drift ratio; PFA – peak floor absolute
acceleration
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 30
HTCondor and Open Science Grid
q  HTCondor (hAp://research.cs.wisc.edu/htcondor/) is a specialized workload management system for computa9onal-­‐intensive jobs. Ø  Project started in 1988, directed at users with large compu9ng needs and environments with heterogeneous distributed resources. q  Open Science Grid is a national, distributed computing grid for data-intensive research.
Ø  Consortium of approx. 80 national laboratories and universities.
Ø  Version of Condor for the grid
Ø  Opportunistic resource usage: resources are sized for peak needs of large experiments
(Atlas, CMS, etc.), OSG allows for non-paying organizations to use their resources.
q  NEES and Open Science Grid have been active partners in creating the tools and
infrastructures for making use of opportunistic resources
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 31
Response estimation accounting for parameter uncertainty
GM
Damping
(%)
Mass
fy
(ksi)
*fc
(ksi)
Es
(ksi)
*Ec
(ksi)
XM
MCS
0.02
µθ
68.7
6.84
29000
4714
COV %
//
40
10
10
10
3.3
8
INPUT
NLTH ANALYSIS
OUPUT
µθ + aσ θ
XLB XM XUB EDPLB EDPM EDPUB Uncertainty in structural properties
Engineering demand parameters
•  Mass
•  Viscous damping
•  Strength
•  Stiffness
•  Roof drift ratio
•  Peak floor accelerations
•  Shear demand in walls
•  Residual deformatios..
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 32
Using Open Science Grid: Production Campaign 2
q  Production campaign
(1)  Probabilistic seismic demand hazard analysis using the cloud method
(2)  Sensitivity of probabilistic seismic demand hazard to FE model parameters
q  Some numbers for production campaign 2 (99% complete)
Number of NLTH analyses per parameter
set realization
180
Average duration of NLTH analysis
12 hours
Average size of output data
1.4 GB
Parameters considered
6
Perturbations considered
4
Estimated clock time on a desktop computer
(180x12x[(6x4x2)+1])
105,840 hours
Estimated size of output (compressed) data
(180x1.4x[(6x4x2)+1])
12 TB
12.1 years
Clock time
30 days !!
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 33
Wall clock time in HTCondor / OSG
30,000
Wall Time (hours)
25,000
20,000
12 clusters of 180 jobs
“Desktop”: 26,000 hours
OSG:
60,000 hours
15,000
(job preemp9on) 10,000
5,000
0
OSG users: André R. Barbosa, Taylor Gugino (UCSD)
OSG support: Gabriele Garzoglio, Marko Slyz (OSG)
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 34
Wall clock time in HTCondor / OSG
Wall Time (hours)
160,000
120,000
80,000
40,000
0
OSG users: André R. Barbosa, Taylor Gugino (UCSD)
OSG support: Gabriele Garzoglio, Marko Slyz (OSG)
Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Comparison Between Parallelization Options
OpenSeesMP
HTCondor
Straight forward implementation of
No ready built solution for large problems,
Domain Decomposition through OpenSees OpenSees sequential does not have
framework with parallel solving algorithm parallel solvers for large problems
like MUMPS
MPICH2 networking setup is relatively
easier
Job management easier
Condor pool setup requires some learning
Condor requires maintenance and
administration
Very active user support through
There is no specific user community as
OpenSees user community, most attractive such.
aspect of using OpenSeesMP
Limited tests show 190 % Speed up from
one processor to two processor
Limited tests show 153 % Speed up from
one processor to two processor
Main complication is compilation of
OpenSeesMP, really really tough!!
Global implementation, if want to connect
to other grid systems.
But once over it OpenSeesMP is really
powerfull!!!
Steep learning curve , knowledge of
networking (Computer science)
Khaled Mashfiq, MS – La Sapienza, Rome Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto
Conclusions
ü  A workflow for running parametric studies that involve
large-scale nonlinear models of structure or soil-structure
systems with large number of parameters and OpenSees
runs has been developed for using NEEShub, Xsede, and
Open Science Grid.
ü  HTCondor
ü  Pegassus (see Frank Mckenna’s presentation)
ü  OpenSees + Condor
q  User interfaces for submitting jobs, receiving results
q  Data visualization
ü  Management and Analysis of Large Research Data Sets
q  Where and what to store?
q  Post-processing? Data compression algorithms?
37
Andre.Barbosa@oregonstate.edu Workshop on Multi-Hazard Analysis of Structures using OpenSees – Faculty of Engineering of the University of Porto 38
1/--pages
Report inappropriate content