WIXLIB

Proof: NN for 1D shape function approximationFor 1D linear basis function, take the reflection to construct the right part andthen combine these two parts.Step 3: 𝑅𝑒𝐿𝑈𝑅𝑒𝐿𝑈 𝑥Step 4:1Reflection: right part𝑏, 1𝑏, 1 ‐1𝑊 1, 1𝑏x𝑊𝑏,Unpublished results𝑏 0𝑊 1Input Layer（Layer 1）1Combination: basis function𝑊𝑊𝑅𝑒𝐿𝑈 𝑅𝑒𝐿𝑈 𝑥 111𝑅𝑒𝐿𝑈 𝑅𝑒𝐿𝑈 𝑥Layer 2,𝑇 1 1𝑊Layer 31, 1Output Layer(Layer 4) Northwestern Univ. 2019, 24

Effect of weights and biases on the output𝒃𝒍𝒋𝑊𝑾𝒍𝒊𝟑𝟏,𝒋 𝟏𝑏 1𝑊 �𝟑𝟐,𝒋 𝟏𝑏Layer 2𝑇 1𝑊 1Input Layer（Layer 1）,Layer 3𝑏,𝑿𝑻Data point 110Data point 20.50.3 Output Layer(Layer 4)Change in bias 𝒃𝒍𝒋 𝟐 , changes the locationChange in weights 𝑾𝒍𝒊𝟑𝟏,𝒋 ,changes the slopeZhang, L., Yang, Y., Li H., Gao J., Reno D., Tang S., Liu W.K. Neural network finite element method , in preparationUnpublished results Northwestern Univ. 2019, 25

Outline1.2.Motivation: source of data in mechanical science and engineeringMechanistic Machine Learning (MML) for mechanical science andengineering–––3.Interpretation of the dataRelevant concepts in data scienceIntroduction to different Machine Learning (ML) methodsa. Unsupervised learningb. Supervised learningApplications of ML methods1.Topology optimization1.2.2.Adolescent Idiopathic Scoliosis1.2.4.5.6.Feed Forward Neural Network (FFNN)FFNN Convolutional Neural Network (CNN)FFNNPhysics Guided Neural Network (PGNN)Why we need reduced order models/methods (ROM)Summary and conclusionsReferences Northwestern Univ. 2019, 26

A simple illustration on unsupervised learning for clusteringObjective: Group 4 data points (each having five features) into 2clustersFive FeaturesFour individual data points123424686824624Two clustersCluster 1 Cluster 2Clustering(e.g., K‐meansclustering)1.53.53747386570231Averaged two columnsClustering: Reduces the data dimensionality Northwestern Univ. 2019, 27

Concept of K-means ClusteringHow does K‐means clustering work ?Raw data𝑺 , k 1𝑁 dimensional𝑨𝐴 ,𝐴 𝐴𝑺 , k 2Data point 𝑨𝑺 , k 3 𝑺 , k 4𝑀 data points 𝑨𝑘: index of clusters𝑚: index of data points Cluster: Points with most similar values Has one average point: mean average of nearby datapoints Objective:Minimize total distance between each average point andthe data points within its �𝑧𝑒:𝑨Watt, J., Borhani, R., & Katsaggelos, A. K. (2016).Machine learning refined:foundations, algorithms, and applications. Cambridge University Press.𝑾𝑺 , k 5𝑺 , k 6𝐾clustersAverage points 𝑾𝑨 ：Data point in cluster 𝑺𝑾 ：Average point in cluster 𝑺：Euclidean distance 𝑺 Northwestern Univ. 2019, 28

K-means clustering for Unidirectional (UD) compositeGrouping local material points in the microstructure based on strainresponses (or other quantities, such as effective plastic strain)Strain distributionCluster distribution based on strain intensity3225201510512D microstructure with600 by 600 voxels2D microstructure with 32clusters The strain field, originally represented by 360,000 voxels, is now representedby 32 clusters The strain patterns are adequately captured by the clusters Northwestern Univ. 2019, 29

Self-Organizing Map (SOM) - ConceptHow does Self-Organizing Map work? [12,13] 𝑁 dimensional𝑨Raw data𝐾𝐴 ,𝐴 𝐴𝐾 clustersData point 𝑨3 2𝑀 data points 𝑨Average points 𝑾 Cluster: Points with most similar values Has one average point: weighted average of nearby data points Objective:Distribute all data points into a map of 𝑲𝟏 𝑲𝟐 clusters so that the dissimilarity within acluster is minimized, and the dissimilarity between clusters with nearby indexes is minimized,,ℎ 𝑘𝑚𝑖𝑛𝑖𝑚𝑖𝑧𝑒:,𝑨 𝑺,𝑘′ 𝑨𝑾 𝑘𝑘′ : Euclidean distance betweenclusters’ indexesℎ 𝑘 𝑘′ : Gaussian kernel function Northwestern Univ. 2019, 30

Sample clustering results from K-means and SOMUnidirectional UD fiber composite (plane strain condition)K 4 clusters (matrix)K‐meansK 8 clusters (matrix)k 1k 1k 2(Matrix phase)k 2k 3k 4k 3k 5k 6Self‐organizingmapk 4k 7k 8(Matrix phase)SOM provides orderly ranking of clusters, feature indicators, andphysical insights, e.g. strain distribution, damage Northwestern Univ. 2019, 31

Supervised learning Supervised learning establishes the hidden relationship between the inputand output data.DataTrainingPredictionQuick overview of Supervised LearningCourtesy: http://aiobserve.com/AI/ML/31.html Can predict the material law from input and output strain‐stress data.σ NN(ε)Sample [1]experimentTrainingPrediction Used for – Regression and Classification[1] Tang, S., Zhang, G., Yang, H., Guo, X., Li, Y., & Liu, W. K. (2019). MAP123: A Data-driven Approach to Use 1D Data for 3DNonlinear Elastic Materials Modeling, CMAME (Submitted) Northwestern Univ. 2019, 32

Regression Regression: prediction of response to an input based on a prioriknowledge of the relationship between input and output data.𝜎𝑣𝑠 𝜀Input Data1,2𝜺𝝈.Find a relationshipbetween stress and straine.g. 𝜎 2 𝜀 11. Gather data from experimentor high-fidelity microstructurenumerical simulation2. Assume plain strain. 𝜺 and 𝝈each contain three components𝝈𝑓 𝜺 (hypothesis)Hypothesis: linear ornonlinear relationshipLi, H, Kafka, OL, Gao, J, Yu, C, Nie, Y, Zhang, L, Tajdari, M, Tang, S, Guo, X, Li, G, Tang, S, Cheng, G & Liu, WK 2019, Clusteringdiscretization methods for generation of material performance databases in machine learning and designoptimization, Computational Mechanics. https://doi.org/10.1007/s00466-019-01716-0 Northwestern Univ. 2019, 33

Regression using Feed Forward Neural Network (FFNN) The relationship between input and output can be explored usingFFNN (an ANN with multiple hidden layers)Input Data1,2𝜺𝝈.FFNN training: solving the following optimization problemLi, H, Kafka, OL, Gao, J, Yu, C, Nie, Y, Zhang, L, Tajdari, M, Tang, S, Guo, X, Li, G, Tang, S, Cheng, G & Liu, WK 2019, Clusteringdiscretization methods for generation of material performance databases in machine learning and design optimization,Computational Mechanics. https://doi.org/10.1007/s00466-019-01716-0 Northwestern Univ. 2019, 34

Regression using Convolutional Neural Network (CNN) for inverse modelingFor a microstructure with given micro-stress distribution, can CNNpredict the macro-strain?Micro‐stress xx distributionInverse modeling𝜀𝛾Micro stress yy distribution𝜀What is the macrostrain ?Micro‐stress xy distributionUnidirectionalmicrostructure Northwestern Univ. 2019, 35

Regression using CNN for inverse modelingInverse modeling approach to obtain macroscopic strain from microscopic stressdistribution using regression through CNNFeed Forward Neural Networkfor regressionConvolutionlayers epeat if necessaryFlattenedm:CNN training: solving the following optimization problemmicroscopicM: macroscopicin Convolution layers Northwestern Univ. 2019, 36

Classification of damage Classification is a process of predicting the known class of given datapoints.E.g., classify the state of the microstructure as “no damage” or “damage” basedon local stress distribution Microscale material point damage is defined as: for any material point in themicroscale domain, if the micro-stress exceeds certain threshold, themicro material point is damaged Macroscale material point damage is defined as:1) PD PND, damage in the microstructure*P is probability2) PD PND, no damage in the microstructureMacroscaleMicroscaleMicroscaleNo damageDamageLi, H, Kafka, OL, Gao, J, Yu, C, Nie, Y, Zhang, L, Tajdari, M, Tang, S, Guo, X, Li, G, Tang, S, Cheng, G & Liu, WK 2019, Clustering discretization methods forgeneration of material performance databases in machine learning and design optimization, Computational Mechanics. https://doi.org/10.1007/s00466-01901716-0 Northwestern Univ. 2019, 37

Application of CNN for damage classificationConvolution layers forfeature extractionFeedforward Neural Network withSoftmax utionRepeat if necessaryFully connectedPD: probability of damagePND: probability of no damageMacroscale material point damage is defined:1) PD PND, damage in the microstructure2) PD PND, no damage in the microstructureLi, H, Kafka, OL, Gao, J, Yu, C, Nie, Y, Zhang, L, Tajdari, M, Tang, S, Guo, X, Li, G, Tang, S, Cheng, G & Liu, WK 2019, Clustering discretizationmethods for generation of material performance databases in machine learning and design optimization, ComputationalMechanics. https://doi.org/10.1007/s00466-019-01716-0 Northwestern Univ. 2019, 38

Outline1.2.Motivation: source of data in mechanical science and engineeringMechanistic Machine Learning (MML) for mechanical science andengineering–––3.Interpretation of the dataRelevant concepts in data scienceIntroduction to different Machine Learning (ML) methodsa. Unsupervised learningb. Supervised learningApplications of ML methods1.Topology optimization1.2.2.Adolescent Idiopathic Scoliosis1.2.4.5.6.Feed Forward Neural Network (FFNN)FFNN Convolutional Neural Network (CNN)FFNNPhysics Guided Neural Network (PGNN)Why we need reduced order models/methods (ROM)Summary and conclusionsReferences Northwestern Univ. 2019, 39

Multiple length scales composite systems design & Optimization6m30cm84 μm1 μm10.2 mmCar frame[1]Part[2]Nanoscalereinforcedpolymer matrixUDfiber compositeWovencompositeTopologyMultiscale Design:𝑊𝑜𝑣𝑒𝑛 𝑡ℎ𝑖𝑐𝑘𝑛𝑒𝑠𝑠: ℎ𝐿ayer angle: 𝛼𝐹𝑖𝑏𝑒𝑟 𝑟𝑎𝑑𝑖𝑢𝑠: 𝑟𝐹𝑖𝑏𝑒𝑟 𝑣𝑜𝑙𝑢𝑚𝑒 𝑓𝑟𝑎𝑐𝑡𝑖𝑜𝑛: 𝑉𝑀𝑜𝑆 𝑣𝑜𝑙𝑢𝑚𝑒 ���𝑖𝑡𝑦 � talLength scalecmmmμmμm-Numberof 001.296 ��Total calculationcost‐‐‐‐A ion‐with‐the‐density‐method/ Northwestern Univ. 2019, 40

Topology optimization (TopOpt)Single-scale topology optimizationMicrostructure-based topologyoptimization is a two-scale problemDesign region60 cmAppliedexternalloading30 cm30 cm60 cmDesign region𝜀𝜺Minimizing [1] Homogenous material assumedNo microstructureOnly elastic responses considered[1] Sigmund, O. (2001). A 99 line topology optimization code written inMatlab. Structural and multidisciplinary optimization, 21(2), 120-127.Two-scale TopOpt: Microstructures in all material points Design of microstructures and structuretopology- Evaluation of microstructure is time consumingduring design iterations- Can FFNN and CNN approximatemicrostructure responses efficiently andaccurately? Northwestern Univ. 2019, 41

Topology optimization with FFNNDesign ZoneDesign Zone𝐹𝜺𝐹𝝈𝜺𝝈TrainingRepresentative VolumeElement(RVE)Feedforward Neural NetworkFFNN approximates microstructure responses almost instantaneouslyTo be presented by Hengyang Li, 7/29/2019, 4:50‐5:10pm, Room 202Li, H, Kafka, OL, Gao, J, Yu, C, Nie, Y, Zhang, L, Tajdari, M, Tang, S, Guo, X, Li, G, Tang, S, Cheng, G & Liu, WK 2019, Clustering discretizationmethods for generation of material performance databases in machine learning and design optimization, ComputationalMechanics. https://doi.org/10.1007/s00466-019-01716-0 Northwestern Univ. 2019, 42

TopOpt with FFNN for nonlinear elastic materialsOptimization𝜺Design Zone𝜎𝝈𝐹𝐶𝜀Linear elastic response𝜺Design ZoneOptimization𝝈𝐹𝜎ℱ𝜀Nonlinear RVE responsepredicted by FFNNLinear materialNonlinear FEM-FEM two scaleNonlinear FEM-FFNN twoscaleInitial compliance 𝑁 · 𝑐𝑚295.0‐375.0Optimized compliance 𝑁 · 𝑐𝑚28.0‐38.0Optimization calculation time 𝑠338Factor of speed‐up over FE‐FE‐𝟏𝟎𝟔𝟐𝟐𝟎‐Li, H, Kafka, OL, Gao, J, Yu, C, Nie, Y, Zhang, L, Tajdari, M, Tang, S, Guo, X, Li, G, Tang, S, Cheng, G & Liu, WK 2019, Clusteringdiscretization methods for generation of material performance databases in machine learning and design optimization, ComputationalMechanics. 55FEM: finite element method Northwestern Univ. 2019, 43

TopOpt with FFNN CNN for nonlinear elastic materials with damage constraintsOptimization with linear elastic materialOptimization with FFNN CNNLinear materialNonlinear FEM‐FEM two scaleNonlinear FEM‐(FFNN CNN)two scaleInitial compliance 𝑁 · 𝑐𝑚295.0‐295Optimized compliance 𝑁 · 𝑐𝑚30‐31Optimization calculation time 𝑠12.6Factor of speed‐up over FE‐FE‐𝟏𝟎𝟔𝟔𝟔𝟎‐Li, H, Kafka, OL, Gao, J, Yu, C, Nie, Y, Zhang, L, Tajdari, M, Tang, S, Guo, X, Li, G, Tang, S, Cheng, G & Liu, WK 2019, Clusteringdiscretization methods for generation of material performance databases in machine learning and design optimization, ComputationalMechanics. 𝟓𝟏𝟎𝟔FEM: finite element method Northwestern Univ. 2019, 44

Outline1.2.Motivation: source of data in mechanical science and engineeringMechanistic Machine Learning (MML) for mechanical science andengineering–––3.Interpretation of the dataRelevant concepts in data scienceIntroduction to different Machine Learning (ML) methodsa. Unsupervised learningb. Supervised learningApplications of ML methods1.Topology optimization1.2.2.Adolescent Idiopathic Scoliosis1.2.4.5.6.Feed Forward Neural Network (FFNN)FFNN Convolutional Neural Network (CNN)FFNNPhysics Guided Neural Network (PGNN)Why we need reduced order models/methods (ROM)Summary and conclusionsReferences Northwestern Univ. 2019, 45

Data-driven approach in predicting Adolescent Idiopathic Scoliosis (AIS)6 Landmarks# vertebras2x6 x 176 LandmarksSpinalglobalanglesFeaturesXαtΔtX*Data points (l)1 2 3 . . Ns.X Vector of input coordinates of a landmark [𝑋 𝑋 𝑋 ]α Global angle vector [𝛼 𝛼 𝛼 𝛼 𝛼 ]t Age of the patient.Δt age variance between target age and current age (month).X* Vector of output co-ordinates of a landmark 𝑋 𝑋 𝑋 ].Ns Total number of landmarks 2x6x17 204Using all features totrain the NeuralNetwork for all I 1,.,NsTo be presented by Mahsa Tajdari, 7/31/2019, 2:40-3:00pm, Room 208 Northwestern Univ. 2019, 46

Physics Guided Neural Network (PGNN) to predict patient specific constantsAssume linear relationship between effective stress and growth rate 𝑋time m 𝑡 and n 𝑡m n A 1 B𝑋 𝑡 : unit of time (month)𝐀 𝑚𝑜𝑛𝑡ℎand 𝑩 𝑀𝑃𝑎between𝜎𝜎: effective stress at time n 𝑡are patient specific constants that are calibrated inside the NNLandmarkX‐ray 2D imagesGenerate patient specific modelMean Square Error (MSE)corresponding ℱMSE𝑀𝑆𝐸𝑿 ,MSE 𝑿 ,MSEMSE 𝑋mn 𝑨𝑻𝑯 1MSEMSE 𝑋mn 𝑨𝐋𝐔 1and 𝑩𝑻𝑯 𝑀𝑃𝑎and 𝑩𝑳𝑼 𝑀𝑃𝑎𝑀𝑆𝐸𝜎𝑋 ,𝜎𝑋 ,𝜎𝑋 ,𝑿 ,MSE𝐀𝑻𝑯 𝑚𝑜𝑛𝑡ℎ𝐀𝑳𝑼 𝑚𝑜𝑛𝑡ℎ𝑀𝑆𝐸Spine FE ModelCalculating Hyper Parameters in NN𝑩𝑻𝑯𝑩𝐋𝐔𝜎ℱ𝜎Hyper parameters (𝑨𝑻𝑯 , 𝑩𝑻𝑯 , 𝑨𝐋𝐔 , 𝑩𝐋𝐔 ) willpatient specific constants for Thoracic vertebrae be calculate inside NNpatient specific constants for Lumbar vertebrae W.K. Liu Group, Northwestern University 47

Composite NN using Multi-fidelity dataSpine FE ModelPhysical data based FFNNData basedFFNNRaw DataFFNNPhysical Equationmn 𝑨𝑻𝑯 1mn 𝑨𝐋𝐔 1𝑩𝑻𝑯𝜎𝑋0𝑩𝐋𝐔𝜎𝑋0α Global angle vector [𝛼 𝛼 𝛼 𝛼 𝛼 ]t Age of the patient.PGNN𝑿 : position of data point in 𝑚 month𝑿 : position of data point in 𝑛 monthRelative ErrorData based FFNN: 18.5%PGNN: 4.63%Δt age variance between target age andcurrent age (month).X* Vector of output co-ordinates of alandmark 𝑋 𝑋 𝑋 ] Northwestern Univ. 2019, 48

Outline1.2.Motivation: source of data in mechanical science and engineeringMechanistic Machine Learning (MML) for mechanical science andengineering–––3.Interpretation of the dataRelevant concepts in data scienceIntroduction to different Machine Learning (ML) methodsa. Unsupervised learningb. Supervised learningApplications of ML methods1.Topology optimization1.2.2.Adolescent Idiopathic Scoliosis1.2.4.5.6.Feed Forward Neural Network (FFNN)FFNN Convolutional Neural Network (CNN)FFNNPhysics Guided Neural Network (PGNN)Why we need reduced order models/methods (ROM)Summary and conclusionsReferences Northwestern Univ. 2019, 49

Rich database of mechanical response information arenecessary for training various Neural Networks Multiscale design and optimization is not feasible with direct microstructureresponses calculation with Finite Element Method (FEM) Well-trained NNs accelerates microstructure and structure designprocess, e.g. Topology Optimization Material microstructure responses database is required for the trainingprocess. The database includes: Macro-strain and macro-stress pairs Micro-stress distribution and macro-strain pairs Other microstructure quantities of interestThe gap: Microstructure response simulation can be expensiveusing FEM Rich database requires a lot of runs of microstructuresimulation Northwestern Univ. 2019, 50

Cost of microstructure responses database generation1,000 load cases for training a 2D hyper elastic problem:External loading statesMicroscopic stress𝜎𝜀Averaged stress𝛾𝜀𝜺𝑛𝜎𝜏𝜀 ,𝜀 ,𝛾1,2 , 1000𝜏𝜎𝝈𝑛𝜎𝜎 ,𝜎 ,𝜏1,2 , 1000600 x 600 x 3 x 1000Running 1,000 microstructure simulation is expensive:Microstructuresimulation methodTotal simulation time (s)FFT3.01 x 105FEM2.04 x 107HPC is neededLi, H, Kafka, OL, Gao, J, Yu, C, Nie, Y, Zhang, L, Tajdari, M, Tang, S, Guo, X, Li, G, Tang, S, Cheng, G & Liu, WK 2019, 'Clustering discretization methodsfor generation of material performance databases in machine learning and design optimization', Computational Mechanics. https://doi.org/10.1007/s00466019-01716-0 Northwestern Univ. 2019, 51

Approximate material responses using FFNNExternal loading statesData pointAveraged stressData �𝟐 𝜺𝒏n𝜺𝑛𝜀 ,𝜀 ,𝛾1,2 , 1000FFNN with welltrained parameters 𝝈𝒏n𝝈𝑛𝜎 ,𝜎 ,𝜏1,2 , 1000How to generate microstructure responses efficiently?Li, H, Kafka, OL, Gao, J, Yu, C, Nie, Y, Zhang, L, Tajdari, M, Tang, S, Guo, X, Li, G, Tang, S, Cheng, G & Liu, WK 2019, 'Clusteringdiscretization methods for generation of material performance databases in machine learning and design optimization', ComputationalMechanics. https://doi.org/10.1007/s00466-019-01716-0 NorthwesternUniv. 2019, 52

Curse of dimensionality in complex material systemsMaterial Scale:Single lap bolt jointWoven compositeMoS2 reinforced polymerUD composite84 μm1 μm10.2 mmTypical number ofelements:Clusters used:Fiber diameter: 7 μm1x1062x10636x10664x10310 clusters96 clusters10 clusters10 clustersFEM1.38x1025𝐊𝐊𝐊𝐊 𝐊MCA 𝐊𝐊𝐊𝛿𝐝𝛿𝐝 𝛿𝐝 𝐊𝐫𝐫𝐌𝐌 𝐫𝐌𝐌 639420 𝐌𝐌1.38x1025𝐌𝐌𝛿𝛆𝛿𝛆 𝛿𝛆 𝐌𝐫𝐫 𝐫639420Strong interaction between scalesDoFs using FEM:𝑁 𝑁3 𝑁 DoFs using MCA: 𝑁𝑁𝑁𝑁Number of elements using FEM:1x10 2.6x10 36x10 64x10 6x10 𝟐𝟓 elements𝑁 : number of integr

2. Mechanistic Machine Learning (MML) for mechanical science and engineering - Interpretation of the data - Relevant concepts in data science - Introduction to different Machine Learning (ML) methods a. Unsupervised learning b. Supervised learning 3. Applications of ML methods 1. Topology optimization 1. Feed Forward Neural Network (FFNN) 2.