WIXLIB

Some ConvNet training « tricks » Importance of input normalization(zero mean, unit variance) Importance of weights initializationrandom but SMALL and prop. to 1/sqrt(nbInputs) Decreasing (or adaptive) learning rate Importance of training set sizeConvNets often have a LARGE number of free parameterstrain them with a sufficiently large training-set ! Avoid overfitting by:– Use of L1 or L2 regularization (after some epochs)– Use « Dropout » regularization (esp. on large FC layers)Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201731Avoid overfitting usingL1/L2 regularization (« weight decay »)Trying to fit too manyfree parameters withnot enough informationcan lead to overfittingRegularization penalizing too complex modelsOften done by adding a special term to cost functionFor neural network, the regularization term is justnorm L2 or L1 of vector of all weights:K Σm(loss(Ym,Dm)) β Σij Wij pname “Weight decay”with p 2 (L2) or p 1 (L1)Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201732

DropOut Regularization forconvNet trainingAt each training stage, individual nodes can be temporarily"dropped out" of the net with probability p (usually 0.5),or re-installed with last values of weightsDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201733Outline Introduction to Deep Learning Convolutional Neural Networks (CNN or ConvNets)– Intro Short reminder on Neural Nets– Convolution layers & Pooling layers global architecture– Training algorithm Dropout Regularization Useful pre-trained convNets and coding frameworksTransfer LearningDeep Belief Networks (DBN)AutoencodersRecurrent Neural Networks (RNN)Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201734

Examples of successful ConvNets LeNet: 1st successful applications of ConvNets, by Yann LeCun in 1990’s.Used to read zip codes, digits, etc. AlexNet: Beginning of ConvNet “buzz”: largely outperformed competitorsin ImageNet ILSVRC2012 challenge. Developped by Alex Krizhevsky etal., architecture similar to LeNet (but deeper larger, and some chainedConvLayers before Pooling). 60 M parameters ! ZF Net: ILSVRC 2013 winner. Developped by Zeiler&Fergus, by modif ofAlexNet on some architecture hyperparameters. GoogLeNet: ILSVRC 2014 winner, developed by Google. Introducedan Inception Module, AveragePooling instead of FullyConnected layer atoutput. Dramatic reduction of number of parameters (4M, compared toAlexNet with 60M). VGGNet: Runner-up in ILSVRC 2014. Very deep (16 CONV/FC layers)140M parameters !! ResNet: ILSVRC 2015, “Residual Network” introducing “skip” connections.Currently SoA in convNet. Very long training but fast execution.Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201735LeNet, for digits/letters recognition[LeCun et al., 1998]Input: 32x32 imageDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201736

AlexNet, for image categorisation[Krizhevsky et al. 2012]Input: 224x224x3 image60 million parameters !.Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201737ZFnet[Zeiler & Fergu, 2013]Input: 224x224x3 imageAlexNet but:CONV1: change from (11x11 stride 4) to (7x7 stride 2)CONV3,4,5: instead of 384, 384, 256 filters use 512, 1024, 512Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201738

GoogleNet[Szegedy et al., 2014]Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201739ResNet (Residual Net), by Microsoft[He et al., 2015] ILSVRC 2015 large winner in 5 main tracks(3.6% top 5 error) 152 layers!!! But novelty "skip" connectionsDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201740

ResNet global architectureBasic block 2-3 weeks of training on 8 GPU machine !! However, at runtime faster than a VGGNet!(even though it has 8x more layers)Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201741Summary of recent ConvNet historyBut most important is the choice ofARCHITECTURAL STRUCTUREDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201742

Programming environments forDeep-Learning Caffehttp://caffe.berkeleyvision.org/C library, hooks from Pythonnotebooks Torch http://torch.ch/TensorFlow https://www.tensorflow.orgTheano e http://lasagne.readthedocs.iolightweight library to build train neural nets in Theano KERAS https://keras.ioPython front-end APIs mapped eitheron Tensor-Flow or Theano back-endAll of them handle transparent use of GPU,and most of them are used in Python code/notebookDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201743Example of convNet code in Kerasmodel Sequential()# Convolution Pooling layers, with Dropoutmodel.add(Convolution2D(conv depth 1, kernel size, kernel size,border mode 'valid', input shape (depth, height, width)))model.add( MaxPooling2D(pool size (pooling size, pooling size)) op prob))# Now flatten to 1D, and apply 1 Fully Connected layermodel.add(Flatten())model.add(Dense(hidden size1, init 'lecun uniform'))model.add(Activation('sigmoid'))# Finally add a Softmax output layer, with 1 neuron per classmodel.add(Dense(num classes, init 'lecun uniform'))model.add(Activation('softmax'))# Training "sessionsgd SGD(lr learning rate, momentum 0.8) # Optimizermodel.compile(loss 'categorical crossentropy', optimizer sgd)model.fit(X train, Y train, batch size 32, nb epoch 2, verbose 1,validation split valid proportion)# Evaluate the trained model on the test setmodel.evaluate(X test, Y test, verbose 1)Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201744

Outline Introduction to Deep Learning Convolutional Neural Networks (CNN or ConvNets)– Intro Short reminder on Neural Nets– Convolution layers & Pooling layers global architecture– Training algorithm Dropout Regularization Useful pre-trained convNets and coding frameworksTransfer LearningDeep Belief Networks (DBN)AutoencodersRecurrent Neural Networks (RNN)Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201745Generality of learnt representation Transfer learningBy removing last layer(s) (those for classification) of a convNettrained on ImageNet, one obtains a transformation of anyinput image into a semi-abstract representation, which can beused for learning SOMETHING ELSE (« transfer learning »):– either by just using learnt representation as features– or by creating new convNet output and perform learningof new output layers fine-tuning of re-used layersDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201746

Transfer Learning and fine-tuning Using a CNN pre-trained on a large dataset,possible to adapt it to another task, using onlya SMALL training set!Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201747Examples of transfer-learningapplications Recognition/classification for OTHER categoriesor classes Direct control of driving wheel! (DeepDrive) Precise localisation (position bearing) PoseNet or even 3D informations from monovision!Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201748

Learning to drivewith transfer-learningnVidia approachDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201749Transfer-Learning for6-DOF Camera Relocalization[A. Kendall, M. Grimes & R. Cipolla, "PoseNet: A Convolutional Networkfor Real-Time 6-DOF Camera Relocalization« , ICCV’2015, pp. 2938-2946]Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201750

Summary and perspectives onConvNets & Deep-Learning Proven advantage of learning features empiricallyfrom data Large ConvNets require huge amounts oflabelled examples data for training Current research/progresses finding efficientglobal architecture of ConvNets Enormous potential of transfer learning on smalldatasets for restricted/specialized problems Next frontier: methods for combining UNsuperviseddeep-learning on unlabelled data with supervisedtraining on smaller labelled datasetDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201751Outline Introduction to Deep Learning Convolutional Neural Networks (CNN or ConvNets)– Intro Short reminder on Neural Nets– Convolution layers & Pooling layers global architecture– Training algorithm Dropout Regularization Useful pre-trained convNets and coding frameworksTransfer LearningDeep Belief Networks (DBN)AutoencodersRecurrent Neural Networks (RNN)Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201752

Deep vs Shallow Learningtechniques overviewDEEPSHALLOWDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201753Deep Belief Networks (DBN) One of first Deep-Learning models Proposed by G. Hinton in 2006 Generative probabilistic model (mostly UNSUPERVISED)For capturing high-order correlations ofobserved/visible data ( pattern analysis,or synthesis); and/or characterizingjoint statistical distributions of visible dataGreedy successive UNSUPERVISED learning of layersof Restricted Boltzmann Machine (RBM)Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201754

Restricted Boltzmann Machine (RBM)h, hidden( latent variables)v, observedNB: connections areBI-DIRECTIONAL(with same weight)Modelling probability distribution as:with « Energy » E given byDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201755Use of trained RBM Input data "completion" : set some vi thencompute h, and generate compatible full samples Generating representative samples Classification if trainedwith inputs data labelDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201756

Training RBMFinding θ (W,a,b) maximizing likelihood minimize NegLogLikelihood In binary input case: logℎ) 1Independance within layersSo objective find ℎ 1 ℎ) ℎ 1) ℎ ) 1ℎ( )(, ( )and argMin ' ' log (v) of dataset S:,ℎ,:with/0. 0/ 1Algo: Contrastive Divergence Gibbs sampling used inside a gradient descent procedureDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201757Contrastive Divergence algoRepeat:1. Take a training sample v, compute 23( 4) 5 6( 7(,:and sample a vector h from this probability distribution2. Compute positive gradient as outer product 89 3 3;3. From h, compute 2 ′ 4 3) 5 7:, 3 and sample reconstructed v',then resample h' using 2 3′( 4 ′) 5 6( 7(,: ′[Gibbs sampling single step; should theoretically be repeated until convergence]4. Compute negative gradient as outer product 8? ′ 3′ ′3′;5. Update weight matrix by @7 A 89 8? A6. Update biases a and b analogously: @ A3; B 3′; ′ and @6 A 3 3′Gibbs samplingDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201758

Modeling of input data distributionobtained by trained RBMInitial data is in blue, reconstructed in red(and green line connects each data point withreconstructed one).Learnt energy function:minima created where data points areDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201759Interpretation of trained RBMhidden layer Look at weights of hidden nodeslow-level featuresDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201760

Why go deeper with DBN ?DBN: upper layersmore « abstract » featuresDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201761Learning of DBNGreedy learning of successive layersDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201762

Using low-dim final featuresfor clusteringMuch better results than clustering in input spaceor using other dimension reduction (PCA, etc )Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201763Example application of DBN:Clustering of documents in databaseDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201764

Image Retrievalapplication example of DBNDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201765DBN supervised tuningUNSUPERVISEDSUPERVISEDDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201766

Outline Introduction to Deep Learning Convolutional Neural Networks (CNN or ConvNets)– Intro Short reminder on Neural Nets– Convolution layers & Pooling layers global architecture– Training algorithm Dropout Regularization Useful pre-trained convNets and coding frameworksTransfer LearningDeep Belief Networks (DBN)AutoencodersRecurrent Neural Networks (RNN)Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201767AutoencodersLearn W and W’ to minimize:Σk zk – xk 2Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201768

Variants of autoencoders Denoising autoencoders Sparse autoencoders Stochastic autoencoders Contractive autoencoders Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201769Deep Stacked AutoencodersProposed by Yoshua Bengio in 2007Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201770

Training of Stacked Autoencoersetc Greedy layerwise training:for each layer k, use backpropagation to minimize Ak(h(k))-h(k) 2 ( regularization cost λ Σij Wij 2)possibly additional term for "sparsity"Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201771Use of Deep Autoencoders Data compression / dimension reduction Learn a compact "code "Information Retrieval Noise removal Manifold learningDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201772

Summary and perspectives onDBN/RBM/DBM / AE Intrinsicly UNSUPERVISEDcan be used on UNLABELLED DATA Impressive results in Image Retrieval DBN/RBM/DBM Generative probabilistic models Strong potential for enhancement of datasets Interest for "creative« /artistic computing?Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201773Outline Introduction to Deep Learning Convolutional Neural Networks (CNN or ConvNets)– Intro Short reminder on Neural Nets– Convolution layers & Pooling layers global architecture– Training algorithm Dropout Regularization Useful pre-trained convNets and coding frameworksTransfer LearningDeep Belief Networks (DBN)AutoencodersRecurrent Neural Networks (RNN)Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201774

Recurrent Neural Networks tTime-delayfor each connectionEquivalent formDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201775Canonical form of RNNOutput Y(t)X(t)State variables.Non-recurrent network.U(t)External input1. 11.X(t-1)State variablesDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201776

Time unfolding of RNNOutput at time t :State variablesX(t)at time tY(t)Non-recurrent networkExternal input at time t :Output at time t-1 :U(t)State variables X(t-1)at time t-1Y(t-1)Non-recurrent networkExternal input at time t-1 :U(t-1)Output at time t-2 :State variablesat time t-2 X(t-2)Y(t-2)Non-recurrent networkExternal input at time t-2 :U(t-2)State variablesX(t-3)at time t-3Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201777RNN trainingHorizon Nt 4W(t)W(t)W(t)W(t)e4e3Temporal sequencett 1W(t 4)t 2t 3t 4 BackPropagation Through Time (BPTT)gradients update for a whole sequence Real Time Recurrent Learning (RTRL)gradients update for each frame in a sequenceDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201778

BackPropagation THROUGH TIME(BPTT)U(t)U(t 1)bloc 1W(t)Xd(t-1)D(t 1)U(t 2)bloc 2W(t)D(t 2)bloc 3W(t)X(t 1)X(t)X(t 2)dE/dXnδW1dE/dXn 1δW3δW2δW δW1 δW2 δW3Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201779BPTT delayW(t Nt) W(t) - λ gradW(E) avec E KEFEFG 'EEandGLJG?JEIG EIG?MEIG 'EIG?M EEMLJ D,Στ (Yτ-Dτ)2EFG EFG EHG EIG?J EEHG EIG?J E(chain rule)GEIEIG 1EI ?JEIG?MLJJacobian matrix of theFeedforward netDeep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.201780

Vanishing/exploding gradientproblem If eigenvalues of Jacobian matrix 1,then gradients tend to explodeLearning will never converge. C

Deep-Learning: principles convNets DBN RNN, Pr. Fabien Moutarde, Robotics Lab, MINES ParisTech March.2017 3 Outline Introduction to Deep Learning Convolutional Neural Networks (CNN or ConvNets) - Intro Short reminder on Neural Nets - Convolution & Pooling layers global architecture - Training algorithm Dropout Regularization Useful pre-trained convNets and coding frameworks