WIXLIB

MatConvNetDeep learning research in MATLABDr Andrea VedaldiUniversity of OxfordMATLAB Expo, October 2017

Deep learning: a magic box2Pixels & labels in, model parameters outc1c2c3c4c5f6f7f8w1w2w3w4w5w6w7w8bike

Text spotting3Confounding turesSizes

Visual searchFast retrieval, learn concepts on the fly4

Object detectionSingle shot (feed forward) detector5

Pose recognitionDense part and keypoint labelling6

Neural artReal-time visual style transfer7

8Demos

Dark magic9Big Data GPU Compute OptimisationA few millionlabelled imagesc1c2c3c4c5f6f7f8w1w2w3w4w5w6w7w8A few hundredteraflops of computecapabilitybikeA few dozengrad students

Convolutional neural networks10A composition of parametric linear and non-linear r dataRGB Tensor height width channels

Operators (aka layers)Linear convolutionFilter bankseveral filtersF1F2ΣΣeach generating an output channelTensor input-outputbig filtersmulti-dimensionalNon-linear activationSimple non-linear functionsmax{0, x}ReLUReLU11

How deep is deep enough?AlexNet (2012)5 convolutional layers3 fully-connected layers12

How deep is deep enough?AlexNet (2012)VGG-M (2013)13VGG-VD-16 (2014)16 conv layers

How deep is deep enough?AlexNet (2012)VGG-M (2013)VGG-VD-16 (2014)14GoogLeNet (2014)

How deep is deep enough?AlexNet (2012)VGG-M (2013)VGG-VD-16 (2014)15GoogLeNet (2014)

How deep is deep enough?16GoogLeNet (2014)ResNet 50 (2015)VGG-VD-16 (2014)ResNet 152 (2015)VGG-M (2013)AlexNet (2012)16 convolutional layers50 convolutional layers152 convolutional layersKrizhevsky, I. Sutskever, and G. E. Hinton.ImageNet classification with deepconvolutional neural networks. In Proc.NIPS, 2012.C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S.Reed, D. Anguelov, D. Erhan, V.Vanhoucke, and A. Rabinovich. Goingdeeper with convolutions. In Proc. CVPR,2015.K. Simonyan and A. Zisserman. Very deepconvolutional networks for large-scaleimage recognition. In Proc. ICLR, 2015.K. He, X. Zhang, S. Ren, and J. Sun.Deep residual learning for imagerecognition. In Proc. CVPR, 2016.

The need for gradients17Learning optimise the parameters w to minimise a fitting errorbikec1c2c3c4c5f6f7f8w1w2w3w4w5w6w7w8lossThe error function is optimised using (stochastic) gradient descentWe require the error function derivativeserror(w1, ,w8)

Backpropagation18Efficient computation of the rwarderror(w1, ,w8)ℝxbackwardderror derror derror derror derrordw1dw2dw3dw4dw5derrorderrorderrordw6dw7dw8

Deep learning software19RequirementsFlexible and usable APIExtensibleEfficientConcise & powerfulKeep up with researchGPUAutomatic differentiationTest new ideasOptimised compute graphMATLABSimple yet powerful languageHistorically, widely adopted in computer vision and roboticsGreat GPU supportGreat documentationRecently, native support for deep nets

MatConvNet20The first modern deep learning toolbox in MATLABWhy?Fully MATLAB-hackableAs efficient as other tools(Caffe, TensorFlow, Torch, )Real-world state-of-the-art applicationsSee demosMany moreCutting-edge research900 citations in academic papersEducationSeveral international courses use itPedigreeSpawn of VLFeat (Mark Everingham Award)Has been around since the “beginning” ( 2012)

MatConvNet21Deep learning sandwichMatConvNet SimpleNNMatConvNet DagNNMatConvNet AutoNNVery basic network abstractionExplicit compute graph abstractionImplicit compute graphMatConvNet Primitivesvl nnconv, vl nnpool, (MEX/M files)MATLABParallel Computing Toolbox (GPU)MatConvNet KernelGPU/CPU implementation of lowlevel opsPlatform (Win, macOS, Linux)NVIDIA CUDA (GPU)NVIDIA CuDNN (Deep Learning Primitives; optional)Pure MATLAB codeApplications

MatConvNet22What can you do with itUse a pre-trained modelVGG-VD, ResNet,ResNext, SSD, R-CNN, Hack the compute graphVisualisation, debugging,optimisationsLearn a new modelArbitrary compute graphsSGD on multi GPUsHack autodiffDefine a new APICreate new layer typesNative MATLAB(gpuArrays)Hack everythingEverything is open

MatConvNet23Deep learning sandwichApplicationsMatConvNet SimpleNNMatConvNet DagNNMatConvNet AutoNNVery basic network abstractionExplicit compute graph abstractionImplicit compute graphMatConvNet Primitivesvl nnconv, vl nnpool, (MEX/M files)MATLABParallel Computing Toolbox (GPU)MatConvNet KernelGPU/CPU implementation of lowlevel opsPlatform (Win, macOS, Linux)NVIDIA CUDA (GPU)NVIDIA CuDNN (Deep Learning Primitives; optional)

MatConvNet24Deep learning sandwichApplicationsMatConvNet SimpleNNMatConvNet DagNNMatConvNet AutoNNVery basic network abstractionExplicit compute graph abstractionImplicit compute graphMatConvNet Primitivesvl nnconv, vl nnpool, (MEX/M files)MATLABParallel Computing Toolbox (GPU)MatConvNet KernelGPU/CPU implementation of lowlevel opsPlatform (Win, macOS, Linux)NVIDIA CUDA (GPU)NVIDIA CuDNN (Deep Learning Primitives; optional)

Primitive: convolutionforward (eval)xvl nnconvW, by vl nnconv(x, W, b)y25

Primitive: convolution26forward (eval)yxvl nnconvW, by vl nnconv(x, W, b)z()z𝜖ℝ

Primitive: convolution27forward (eval)yvl nnconvxz()W, by vl nnconv(x, W, b)backward (backprop)dzdxdzdyz()vl nnconvdzdWdzdbdzdx vl nnconv(x, W, b, dzdy)z𝜖ℝ

Primitive: convolution28forward (eval)yvl nnconvxz()W, by vl nnconv(x, W, b)backward (backprop)dzdxdzdyz()vl nnconvdzdWdzdbdzdx vl nnconv(x, W, b, dzdy)z𝜖ℝ

MatConvNet29Deep learning sandwichApplicationsMatConvNet SimpleNNMatConvNet DagNNMatConvNet AutoNNVery basic network abstractionExplicit compute graph abstractionImplicit compute graphMatConvNet Primitivesvl nnconv, vl nnpool, (MEX/M files)MATLABParallel Computing Toolbox (GPU)MatConvNet KernelGPU/CPU implementation of lowlevel opsPlatform (Win, macOS, Linux)NVIDIA CUDA (GPU)NVIDIA CuDNN (Deep Learning Primitives; optional)

MatConvNet30Deep learning sandwichApplicationsMatConvNet SimpleNNMatConvNet DagNNMatConvNet AutoNNVery basic network abstractionExplicit compute graph abstractionImplicit compute graphMatConvNet Primitivesvl nnconv, vl nnpool, (MEX/M files)MATLABParallel Computing Toolbox (GPU)MatConvNet KernelGPU/CPU implementation of lowlevel opsPlatform (Win, macOS, Linux)NVIDIA CUDA (GPU)NVIDIA CuDNN (Deep Learning Primitives; optional)

Network abstraction: AutoNN31Defining and evaluating a deep networkyx0vl nnconvx1,biasx1,filtx1vl nnpoolx2vl nnconvx3,biasx3,filt% Define network & lossx0 Input() ;y Input();x1 vl nnconv(x0, 'size', [5, 5, 1, 20]) ;x2 vl nnpool(x1, 2, 'stride', 2) ;x3 vl nnconv(x2, 'size', [5, 5, 20, 10]) ;loss vl nnloss(x3, y);x3vl nnlossloss

Network abstraction: AutoNN32Defining and evaluating a deep network%abcb.*t1 ½t3amax(., 0)t2 x4Define compute graph Input() ; Input(); sqrt(max(a,0) a.*b/2) ;sqrtc

Autodiff vs symbolic differentiation33Why this instead of Maple / Symbolic Toolboxb.*t1 ½t3Autodiff is not symbolic differentiationadbdamax(., 0)d(.*)dmax(., 0)t2dt1dt2 d( ½)d( )t4sqrtcUnder the hoodAutodiff appends a backward extension to thegraphdt3dt4Autodiffcomputes derivatives numericallyas efficiently as possibledsqrtdcexecuting the graph computes both function andits derivative

MatConvNet vs Neural Network ToolboxAn increasingly powerful alternativeMatConvNet pre-trained modelsApplicationsExamples, demos, tutorialsMatConvNet SimpleNNMatConvNet DagNNMatConvNet AutoNNVery basic network abstractionExplicit compute graph abstractionImplicit compute graphMatConvNet Primitivesvl nnconv, vl nnpool, (MEX/M files)MATLABParallel Computing Toolbox (GPU)MatConvNet KernelGPU/CPU implementation of lowlevel opsPlatform (Win, macOS, Linux)NVIDIA CUDA (GPU)NVIDIA CuDNN (Deep Learning Primitives; optional)34

MatConvNet vs Neural Network Toolbox35An increasingly powerful alternativeMatConvNet pre-trained modelsApplicationsExamples, demos, tutorialsMatConvNet SimpleNNMatConvNet DagNNMatConvNet AutoNNVery basic network abstractionExplicit compute graph abstractionImplicit compute graphMatConvNet Primitivesvl nnconv, vl nnpool, (MEX/M files)MATLABParallel Computing Toolbox (GPU)MatConvNet KernelGPU/CPU implementationof low-level opsPlatform (Win, macOS, Linux)NVIDIA CUDA (GPU)NVIDIA CuDNN (Deep Learning Primitives; optional)MATLABNeural NetworkToolbox

New in the Neural Network ToolboxData AccessNetworksTrain App for Ground Truth labeling Alexnet, VGG-16, VGG-19 Caffe model importer TensorFlow-Keras importer GoogLeNet model Label for semantic segmentation Resize & augment images CNN Regression Object detection using Fast RCNN and R-CNN Object detector evaluation Multi-GPUs in parallel Visual features usingactivations LSTM (time series, text) DAG Networks Create new layersNewProductDeploy / Share Validation Training plots Hyper-parameter optimization GPU Coder: convert MATLABmodels to NVIDIA CUDA code36

MatConvNet: Check it /vlfeat/matconvnetKarel LencSam AlbanieJoao Henriques37

Deep learning research in MATLAB Dr Andrea Vedaldi University of Oxford MATLAB Expo, October 2017. Pixels & labels in, model parameters out Deep learning: a magic box 2 c1 c2 c3 c4 c5 f6 f7 f8 w1 w2 w3 w4 w5 w6 w7 w8 bike. Text spotting Confounding factors Fonts Distortions Colors Blur Shadows Borders Textures