WIXLIB

Machine Learning & Deep LearningSoluzioni di grafica 3D in applicazioni biometricheLeonardo TanziPh.D. Studentleonardo.tanzi@polito.it

Machine LearningTableofContents Definition Types of learning Regression ClassificationDeep Learning Definition Neural Network Convolutional Neural NetworkCase Study Coding a CNN

MachineLearning

DefinitionApplication of Artificial Intelligence that provides a system with the ability to learn automaticallyfrom experience without being explicitly programmed for the given task.Arthur Samuel - 1959A computer program is said to learn from experience E with respect to some task T and someperformance measure P, if its performance on T, as measured by P, improves with experience E.Tom Mitchell - 1998InputDataAnalyzeDataFindPatternsPredictLearn fromFeedback

Types of forcementLearningThe machine learns fromlabeled dataThe machine learns withoutlabeled dataThe machine learns on itsown

Supervised LearningModelDataTrainingIt’s anApple!PredictionLabelsNew Data

Unsupervised LearningSupervised learningModelTrainingUnlabeled DataSearch for apattern

Reinforcement LearningSupervised learningIt’s a banana!Wrong!It’s anApple!NotedReinforcedpredictionUnknown DataResponseFeedbackLearningNew Data

onUsed when the output iscontinuous (ex. price of a house)Used when the output iscategorical (ex. Yes or No)Linear RegressionLogistic RegressionKNNDecision TreesSVM

rning

Regression: DefinitionRegression ModelSize (m2)Price ( )100120.0005380.000220340.00078110.000145.000 Input variablesNew Size131m2ContinuosOutput Variable

Regression: Linear RegressionHouse Price Prediction: estimate the price of a house given its sizeWe want to fit a line that best approximates the behaviorof the training data.PriceWe call this function hypothesis h(x) and it is a functionof 𝒘𝟏 (slope) and 𝒘𝟎 (y-intercept)h(x)ℎ 𝑥 𝑤0 𝑤1 𝑥xSize How can we fit this line?Find the values of 𝑤0 and 𝑤1 which minimizes thedifference between the prediction and the actual values

Regression: Loss and Gradient DescentGradient DescentLoss𝐽(𝑤0 , 𝑤1 )𝑤0ℎ𝑤 𝑥 𝑤0 𝑤1 𝑥Changing 𝑤0 move the line up and downChanging 𝑤1 change the slopeSquared Error: J(𝑤0 , 𝑤1 ) 12σ(ℎ𝑤 𝑥 𝑦)Should be minimized!2𝑤1In the beginning, the line has random 𝑤0 and 𝑤1 .Gradient descent moves the point (by updating thevalues of 𝑤0 and 𝑤1 ) in the direction where theLoss function decrease.This process is repeated until a minimum isreached.

Classification: DefinitionClassificationModelFruitsLabeled inputvariablesAnimalsDiscreteOutput

Classification: Important TerminologiesKeywordDefinitionClassifierAn algorithm used to map the input data to a specific categoryClassification modelThe model that predicts a class given an input dataFeatureAn individual measurable property of the observed dataTrainThe process where the model learn to distinguish between classesPredictThe process where the model make a prediction on unseen dataBinary classificationCondition with two outcomes which are either true or falseMulti-class classificationCondition with more than two classes where each sample is assigned to one andonly labelMulti-label classificationCondition where each sample is assigned to a set of labels or targets

Classification: MetricsTraining a classifier to predict whether a person is diabetic or healthy (binary output). Four possible outcomes:These are goodTrue PositiveTPPred: diabeticCorrect: diabeticTrue NegativeTNPred: healthyCorrect: healthyFalse PositiveFPPred: diabeticCorrect: healthyFalse NegativeFNPred: healthyCorrect: diabeticThese are bad (especially FN)How many people werecorrectly labeled among allthe people?How many of thoselabeled as diabetic areactually diabetic?𝑇𝑃 𝑇𝑁𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 𝑇𝑃 𝐹𝑃 𝐹𝑁 𝑇𝑁𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 𝑇𝑃𝑇𝑃 𝐹𝑃Of all the diabetic people,how many were ��𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 Of all the healthy people,how many were �𝑖𝑡𝑦 𝑇𝑃𝑇𝑃 𝐹𝑁𝑇𝑁𝐹𝑃 𝑇𝑁

Classification: Logistic RegressionIn linear regression, we computed a continuous value (the price of a house given its size).In logistic regression, we want to output a discrete value 𝑦.ො For example, in a binary problem, given the size ofa house, we want to build an algorithm that returns 0 if we can afford it and 1 if we can’t.SigmoidWe can’tafford itWe need an S-shapedfunction to fit this dataWe canafford itSize𝑦 11 𝑒 𝑥

Classification: Logistic RegressionStarting from the linear regression function ℎ𝑤 𝑥 𝑤0 𝑤1 𝑥, we add a Sigmoid function that squeezes theoutput between 0 and 1, in this way, the system will predict values between 0 and 1 (values 0.5 will beassigned to one class while values 0.5 to the other): 𝒉𝒘 𝒙 𝝈(𝒘𝟎 𝒘𝟏 𝒙)-log(x)Loss function has to be minimized!𝐽 𝑦,ො 𝑦 (𝑦log 𝑦ො 1 𝑦 log 1 𝑦ො )xNB: in our case, x has a valuebetween 0 and 1, for this reason,the minimum value is obtainedfor x 1 and not for x 1If y 0 𝐽 𝑦,ො 𝑦 log 1 𝑦ොWhen the input y is 0 we want 𝑦ො close as0 as possible. The minimum value of log 1 𝑦ො is obtained when 𝑦ො 0,because log 1 0 log 1 0If y 1 𝐽 𝑦,ො 𝑦 log 𝑦ොWhen the input y is 1, we want 𝑦ො close as1 as possible. The minimum value of log 𝑦ො is obtained when 𝑦ො 1,because log 1 0

Classification: KNN IntuitionK Nearest NeighborsWe have a distribution, when a new sample arrives, we assign it to a class looking at theclasses of the k nearest samplesK 1K 3New sampleWe assign it to thebanana classWe assign it to theapple class

Classification: Decision Trees IntuitionIf we have a distribution like this,we can not fit a S-shaped function as weshowed for logistic sed on this data, wecan build a DecisionTree that is able toreturn the predictionfor a new personYesNoDoes notlike cinemasAge 12.5?YesDoes notlike cinemasNoDoes likecinemas

Classification: SVM IntuitionWe have just seen that we cannot fit anS-shaped function to this specific distributionSupport Vector MachineWe represent data in the lowest dimension possible, and with SVM we can add a new axis to thedata and move the points in a way that makes it relatively easy to draw a straight line that correctlyclassifies peopleAge2Support Vector Machines usesomething called KernelFunctions that we can be usedto systematically find higherdimensions’ axisAdd y-axisAge42Age

DeepLearning

IntroductionMany features / Big dataset Classic ML algorithm become inefficient Alternative: Deep LearningDeep Learning is a branch of Machine Learning which utilizes neural network for computationMain ideaNeural networks are algorithms inspired bythe human brain.The brain uses just a single learningalgorithm that can do anything, can we findan approximation of what the brain does andtry to implement it?Auditory Cortex

NeuronA biological neuron in the brain receives inputsfrom the dendrites, performs some computation,and outputs the result through the axonsDendriteAxonAn artificial neuron tries to mimic its behavior:it receives some inputs (x1, x2, x3), computes a linearfunction operation using weights (w1 , w2 , w3)𝑧 𝑤1 𝑥1 𝑤2 𝑥2 𝑤3 𝑥3and applies a non-linear activation function toformulate the hypothesisℎ𝑤 𝑥 earactivation function

Neural NetworkA neural network is a group of these neurons.w11(1)Each neuron has a linear combination of inputsand weights and then applies non-linear activation,for example, for the first layer:x1w12(1)a2(1)x2hw(x)a1(1) 𝑔(𝑤11 (1)𝑥1 𝑤21 (1)𝑥2 𝑤31(1) 𝑥3 )a2(1) 𝑔(𝑤12 (1)𝑥1 𝑤22 (1)𝑥2 𝑤32 (1)𝑥3 )a3(1)x3w14(1) Notationan(k) n indicates the neuron and k thelayerwij(k) i is the index of the neuron of theinput layer, j the index of thehidden layer, and k is the ly Connected Network (FCN)

Neural NetworkA neural network is a group of these neurons.w11(1)Each neuron has a linear combination of inputsand weights and then applies non-linear activation,for example, for the first layer:x1w12(1)a2(1)x2hw(x)a1(1) 𝑔(𝑤11 (1)𝑥1 𝑤21 (1)𝑥2 𝑤31(1) 𝑥3 )a2(1) 𝑔(𝑤12 (1)𝑥1 𝑤22 (1)𝑥2 𝑤32 (1)𝑥3 ) Notationan(k) n indicates the neuron and k thelayerwij(k) i is the index of the neuron of theinput layer, j the index of thehidden layer, and k is the layera1(1)a3(1)x3w14(1)a4(1)InputNon-linear Hiddenactivation OutputLayerLayerfunctions makeneural Layernetworks flexible andable to fit just about anyFully Connected Network (FCN)data

Neural NetworkFor the second layer:a1And finally:w11(2)a1(2)x1 a1(2) 𝑔(𝑤11 (2)a1(1) 𝑤21 (2)a2(1) 𝑤31(2) a3(1) 𝑤41(2) a4(1))(1)a2(2)a2(1)x2h 𝑔(𝑤11 (3)a1(2) 𝑤21 (3)a2(2) 𝑤31(3) a3(2) 𝑤41(3) a4(2))hw(x)a3(1)a3(2)a4(1)a4(2)x3Each w(k) is a matrix of weights that controls themapping from layer k to layer k 1w44(2)

Neural Network Classification28x28 784 neuronsMNIST10 handwrittendigits28 286SoftmaxInput layern hidden layerOutputSoftmax is a function used for multiclass classification that turns avector of K real values into a vector of K real values that sum to 1[2.33, -1.46, 0.56]Softmax[0.83, 0.01, 0.14]

Neural Network Training28 28LossSoftmaxInput layern hidden layerOutputLabelOne of the most used functions for multi-class classification is cross-entropy loss, defined as:𝐶𝐸 𝑦𝑖 log(𝑦ො𝑖 )𝑖𝑦𝑖 is the label vector with one-hot encoding, meaning that the number “6” is coded as [0 0 0 0 0 0 1 0 0 0].𝑦ො𝑖 is the (wrong) prediction after the Softmax, let’s say [0.01, 0.02, 0.11, 0.70, 0.01, 0.08, 0.03, 0.01, 0.02, 0.01],which is actually a “3”.Thus, CE will be equal to -log(0.03) as all the other values are multiplied by 0. Given the minus sign, inminimizing this function, we are trying to maximize the 6th value in the output vector.

Gradient Descent & BackpropagationAs we already said, the loss function represents how much the prediction is similar to the actual values.For this reason, the more we minimize the loss function, the more the prediction will be accurate. To dothis, we need two powerful tools: gradient descent and backpropagation.Optimization: we want to find the value of w which minimizesJ(w). To do this, we will update w at each step by the quantity:Loss J(w)𝑤 𝑤 𝛼21𝑑𝐽(𝑤)𝑑𝑤𝛼 is the learning rate, which is a parameter that determines how bigthe steps are in the direction of the minimum.minwBackpropagation is the techniquethat allows to compute thederivative of the loss with respect tothe parameters.𝑑𝐽(𝑤)𝑑𝑤is the derivative of J(w) with respect to w, i.e., the slope. Thus, if𝑑𝐽 𝑤we are in 1, the slope is negative, so 𝛼 𝑑𝑤 will be a positive value,meaning that we are increasing w in the direction of the minimum.𝑑𝐽 𝑤On the contrary, if we are in 2, the slope is positive, so 𝛼 𝑑𝑤 willbe a negative value, meaning that we are decreasing w in thedirection of the minimum.

Neural Network Overview28 28LossSoftmaxInput layern hidden layerOutputLabelIn summary, during training, we take an image (or more than one, i.e., a batch), pass it through the network,and make a prediction. We use the prediction and the actual label to compute the loss. We then usebackpropagation to compute the derivative of the loss with respect to all the weights. Finally, we applygradient descent to adjust the value of the weights in the direction of the minimum. This process is repeateduntil the minimum (or a local minimum) of the loss is reached, meaning that the network has (hopefully)learned to distinguish the different classes.Reminder: all this works with 28 28 1 images, which can be flattened to just 784 neurons. But if we use high-resimages, such as 1920 1080 3, the input layer should have around 6 million neurons, becoming computationallyinfeasible