Introduction To Deep Learning With TensorFlow

1y ago

30 Views

1 Downloads

6.03 MB

69 Pages

Report/dmca

Download PDF

Transcription

Introduction to Deep Learningwith TensorFlowJian Taojtao@tamu.eduHPRC Short Course4/16/2021

Introduction to Deep Learning with TensorFlowPart III03Introduction toTensorFlow (60 mins)Part IQ&A(5 mins/part)Setting up a workingenvironment (15 mins)0102Part IIIntroduction to DeepLearning (60 mins)

Part I. Working EnvironmentHPRC Portal* VPN is required for off-campus users.

Terra Shell Access - I

Terra Shell Access - II

Python Virtual Environment (VENV)Load ModulesCreate a VENVActivate the VENVInstall PythonModulesDeactivate (Optional)# clean up and load Anacondacd SCRATCHmodule purgemodule load Python/3.7.4-GCCcore-8.3.0# create a Python virtual environmentpython -m venv mylab# activate the virtual environmentsource mylab/bin/activate# install required package to be used in the portalpip install --upgrade pip setuptoolspip install jupyterlab tensorflow sklearn matplotlib# deactivate the virtual environment# source deactivate

Check out Exercises# git clone (check out) the Jupyter notebooks for the short coursesgit clone https://github.com/jtao/shortcourses.git

Go to JupyterLab Page

Set Virtual Environment# enter the full path of the activate command of your te

Connect to JupyterLab

Create a Jupyter Notebook

Test JupyterLab

Part II. Introduction to DeepLearningDeep Learningby Ian Goodfellow, Yoshua Bengio, and Aaron Courvillehttp://www.deeplearningbook.org/Animation of Neutron Networksby Grant Sandersonhttps://www.3blue1brown.com/

Relationship of AI, ML, and DL Artificial Intelligence (AI)is anything aboutman-made intelligenceexhibited by machines. Machine Learning (ML) isan approach to achieve AI. Deep Learning (DL) is onetechnique to implementML.Artificial IntelligenceMachine LearningDeep Learning

Machine LearningTraditional ne Learning (Supervised uterPredictionModelData

Types of ML Algorithms Supervised Learning trained with labeled data;including regression andclassification problems Unsupervised Learning trained with unlabeled data;clustering and association rulelearning problems. Reinforcement Learning no training data; stochasticMarkov decision process; roboticsand self-driving cars.Machine LearningSupervised LearningUnsupervised LearningReinforcement Learning

Supervised LearningWhen both input variables - X and output variables - Y are known, one canapproximate the mapping function from X to Y.Training DataML AlgorithmStep 2: TestingModelStep 1: TrainingTest Data

Unsupervised LearningWhen only input variables - X are known and the training data is neitherclassified nor labeled. It is usually used for clustering problems.DataClass 1Class 2Class 3

Reinforcement LearningWhen the input variables are only available via interacting with theenvironment, reinforcement learning can be used to train an "agent".(Image Credit: Wikipedia.org)(Image Credit: deeplearning4j.org)

Why Deep Learning? Limitations of traditional machine learning algorithms not good at handling high dimensional data. difficult to do feature extraction and object recognition. Advantages of deep learning DL is computationally expensive, but it is capable ofhandling high dimensional data. feature extraction is done automatically.

What is Deep Learning?Deep learning is a class of machine learning algorithms that: use a cascade of multiple layers of nonlinear processing unitsfor feature extraction and transformation. Each successivelayer uses the output from the previous layer as input. learn in supervised (e.g., classification) and/or unsupervised(e.g., pattern analysis) manners. learn multiple levels of representations that correspond todifferent levels of abstraction; the levels form a hierarchy ofconcepts.(Source: Wikipedia)

Artificial Neural NetworkInput(Image Credit: Wikipedia)Hidden LayersOutput

Inputs and Outputs256 X 256MatrixDL model4-Element VectorX123456YACTGMFWith deep learning, we are searching for a surjective(or onto) function f from a set X to a set Y.

Learning Principle - IDatasetOutput/PredictionTarget Outputxx12 .xnError:- 5Credit: nvidia.com25

Learning Principle - IIOutput/PredictionTarget Outputxx12 .xn 15- PrincipleError:LearningCredit: nvidia.com26

Learning Principle - IIIOutput/PredictionTarget Outputxx12 .xn 2.5- PrincipleError:LearningCredit: nvidia.com27

Deep Neural Network as a Nonlinear FunctionInputMapping FunctionOutputBackward PropagationX1X2X3Forward Propagation Training: given input and output, find best-fit F Inference: given input and F, predict output

Supervised Deep Learning with Neural NetworksInputHidden LayersFrom one layer to the nextX1f is the activation function,Wi is the weight, and bi isthe bias.X2W1W2Y3X3W3Output

Training - Minimizing the LossThe loss function with regard to weightsand biases can be defined asInputW1, b1OutputX1Y2The weight update is computed by movinga step to the opposite direction of the costgradient.W2, b2LW3, b3Iterate until L stops decreasing.X2X3

Convolution in 2D(Image Credit: Applied Deep Learning Arden Dertat)

Convolution Kernel(Image Credit: Applied Deep Learning Arden Dertat)

Convolution on ImageImage Credit: Deep Learning Methods for Vision CVPR 2012 Tutorial

Activation FunctionsImage Credit: towardsdatascience.com

Introducing Non Linearity (ReLU)Image Credit: Deep Learning Methods for Vision CVPR 2012 Tutorial

Max Pooling(Image Credit: Applied Deep Learning Arden Dertat)

Pooling - Max-Pooling and Sum-PoolingImage Credit: Deep Learning Methods for Vision CVPR 2012 Tutorial

CNN Implementation - Drop OutDropout is used to prevent overfitting. A neuron is temporarily“dropped” or disabled with probability P during training.(Image Credit: Applied Deep Learning Arden Dertat)

CNN Implementation - Data Augmentation (DA)DA helps to popularartificial traininginstances from theexisting train data sets.(Image Credit: Applied Deep Learning Arden Dertat)

Convolutional Neural NetworksA convolutional neural network (CNN, or ConvNet) is a class of deep, feed-forwardartificial neural networks that explicitly assumes that the inputs are images, which allowsus to encode certain properties into the architecture.LeNet-5 Architecture (image Credit: https://becominghuman.ai)

Deep Learning for Facial Recognition(Image Credit: www.edureka.co)

Best Practice Guide for Training ML/DL ModelsModel Capacity (what can the model learn?) Overtain on a small data set Synthetic data (with known features and properties)Optimization Issues (can we make the model learn?) Look at the learning curves (testing vs training errors) Monitor gradient update ratios Hand-pick parameters for synthetic dataOther Model "Bugs" (is the model doing what I want it to do?) Generate samples from your model (if you can) Visualize learned representations (e.g., embeddings, nearest neighbors) Error analysis (examples where the model is failing, most "confident" errors) Simplify the problem/model Increase capacity, sweep hyperparametershttps://youtu.be/zCEYiCxrL 0

MNIST - Introduction MNIST (Mixed NationalInstitute of Standards andTechnology) is a database forhandwritten digits, distributedby Yann Lecun. 60,000 examples, and a testset of 10,000 examples. 28x28 pixels each. Widely used for research andeducational purposes.(Image Credit: Wikipedia)

MNIST - CNN Visualization(Image Credit: http://scs.ryerson.ca/ aharley/vis/)

Neural Network Playground(Image Credit: http://playground.tensorflow.org/)

Part III. Introduction to TensorFlowTensorFlow Official Websitehttp://www.tensorflow.org46

A Brief History of TensorFlowTensorFlow is an end-to-end FOSS (free and open source software)library for dataflow, differentiable programming. TensorFlow is one ofthe most popular program frameworks for building machine learningapplications. Google Brain built DistBelief in 2011 for internal usage. TensorFlow 1.0.0 was released on Feb 11, 2017 TensorFlow 2.0 was released in Jan 2018. The latest stable version of TensorFlow is 2.3.0 as of Nov 2020.

TensorFlow, Keras, and PyTorchTensorFlow is anend-to-end opensource platform formachine learning. Ithas a comprehensive,flexible ecosystem tobuild and deploy MLpowered applications.Keras is a high-levelneural networks API,written in Python andcapable of running ontop of TensorFlow,CNTK, or Theano. Itwas developed with afocus on enabling fastexperimentation.PyTorch is an opensource machinelearning frameworkthat accelerates thepath from researchprototyping toproductiondeployment.

Google Trends for Popular ML FrameworksKerasreleased inMar 2015Caffe paperpublished inJun 2014PyTorchreleased inSep 2016Tensorflowreleased inNov 2015(Image Credit: https://trends.google.com/)

TensorFlow 2.0 Toolkits(Image Credit: tensorflow.org)

Architecture of TF 2.0(Image Credit: tensorflow.org)

What is a Tensor in TensorFlow? TensorFlow uses a tensordata structure to represent alldata. A TensorFlow tensor asan n-dimensional array orlist. A tensor has a static type,a rank, and a shape.NameRankTensorScalar0[5]Vector1[1 2 3]Matrix2[[1 2 3 4],[5 6 7 8]]Tensor3.

Computational Graph in TF 2.0x tf.random.normal(shape (10,10))w tf.Variable(tf.random.normal(shape (10,5)))b tf.Variable(tf.random.normal(shape (5,)))linear model w * x bxMultiplyAddwb

A Connected Pipeline for the Flow of Tensors(Image Credit: Plumber Game by Mobiloids)

TensorFlow Data TypesBasic TensorFlow data types include: int[8 16 32 64], float[16 32 64], double bool stringWith tf.cast(), the data types of variables could beconverted.

Hello World with TensorFlowimport tensorflow as tfv tf.constant("Hello World!")tf.print(v)

TensorFlow ConstantsTensorFlow provides several operations to generate constant tensors.import tensorflow as tfx tf.constant(1, tf.int32)zeros tf.zeros([2, 3], tf.int32)ones tf.ones([2, 3], tf.int32)y x *(zeros ones ones)tf.print(y)

TensorFlow VariablesTensorFlow variables can represent shared, persistent state manipulated byyour program. Weights and biases are usually stored in variables.import tensorflow as tfW tf.Variable(tf.random.normal([2,2], stddev 0.1),name "W")b tf.Variable(tf.zeros(shape (2)), name "b")

GPU AccelerationTensorFlow automatically decides if to use the CPU or GPU. One canexplicitly pick a device to use. The string ends with CPU/GPU: N if thetensor is placed on the N-th CPU/GPU on the host.# Force execution on CPUwith tf.device("CPU:0"):do something()# Force execution on GPU #0/1/2/. if availableif tf.config.experimental.list physical devices("GPU"):with tf.device("GPU:0"):do something else()

Machine Learning Workflow with tf.kerasStep 1Step 2Step 3Step 4Prepare Train DataDeﬁne ModelTraining ConﬁgurationTrain ModelThe preprocessed data set needsto be shuﬄed and splitted intotraining and testing data.A model could be deﬁned withtf.keras Sequential model for alinear stack of layers or tf.kerasfunctional API for complexnetwork.The conﬁguration of the trainingprocess requires thespeciﬁcation of an optimizer, aloss function, and a list ofmetrics.The training begins by calling theﬁt function. The number ofepochs and batch size need to beset. The measurement metricsneed to be evaluated.

tf.keras Built-in Datasets tf.keras provides many popular reference datasets that could be usedfor demonstrating and testing deep neural network models. To name afew, Boston Housing (regression) CIFAR100 (classification of 100 image labels) MNIST (classification of 10 digits) Fashion-MNIST (classification of 10 fashion categories) Reuters News (multiclass text classification) The built-in datasets could be easily read in for training purpose. E.g.,from tensorflow.keras.datasets import boston housing(x train, y train), (x test, y test) boston housing.load data()

Prepare Datasets for tf.kerasIn order to train a deep neural network model withKeras, the input data sets needs to be cleaned,balanced, transformed, scaled, and splitted. Balance the classes. Unbalanced classes willinterfere with training. Transform the categorical variables intoone-hot encoded variables. Extract the X (variables) and y (targets) valuesfor the training and testing datasets. Scale/normalize the variables. Shuffle and split the dataset into training andtesting datasetsOne-hot encodingDog100Cat010Horse001Numerical encodingDog1Cat2Horse3

Create a tf.keras Model Layers are the fundamentalbuilding blocks of tf.kerasmodels. The Sequential model is alinear stack of layers. A Sequential model can becreated with a list of layerinstances to the constructor oradded with the .add() method. The input shape/dimension ofthe first layer need to be set.from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Dense,Activationmodel Sequential([Dense(64, activation 'relu', input dim 20),Dense(10, activation 'softmax')])InputHidden LayersOutput

Compile a tf.keras ModelThe compile method of a Keras model configures the learningprocess before the model is trained. The following 3 arguments needto be set (the optimizer and loss function are required). An optimizer: Adam, AdaGrad, SGD, RMSprop, etc. A loss function: mean squared error, mean absolute error,mean squared logarithmic error, categorical crossentropy,kullback leibler divergence, etc. A list of measurement metrics: accuracy, binary accuracy,categorical accuracy, etc.

Train and Evaluate a tf.keras Modeltf.keras is trained on NumPy arrays of inputdata and labels. The training is done with the fit() function of the model class. In the fitfunction, the following twohyperparameters can be set: number of epochs batch size evaluate() function returns the loss value& metrics values for the model in testmode. summary() function prints out thenetwork architecture.Model: "sequential 1"Layer (type)Output ShapeParam # dense 11 (Dense)(None, 64)1344dense 12 (Dense)(None, 10)650 Total params: 1,994Trainable params: 1,994Non-trainable params: 0None

Make Predictions and MoreAfter the model is trained, predict() function of the model class could be used togenerate output predictions for the input samples. get weights() function returns a list of all weight tensors inthe model, as Numpy arrays. to json() returns a representation of the model as a JSONstring. Note that the representation does not include theweights, only the architecture. save weights(filepath) saves the weights of the model as aHDF5 file.

Monitoring Training with Tensorboard TensorBoard is a UserInterface (UI) toolsdesigned for TensorFlow.More details onTensorBoard can be foundat TensorBoard.Once you’ve installedTensorBoard, these utilitieslet you log TensorFlowmodels and metrics into adirectory for visualizationwithin the TensorBoard UI.

Hands-on Session #1Getting Started with TensorFlow

Hands-on Session #2Classify Handwritten Digits withTensorFlow