WIXLIB

Computer Vision System Design: Deep Learning, 3DVision, and Embedded VisionDr. Amod AnandkumarTabrez Khan 2016 The MathWorks, Inc.1

Outline 3D Vision and Point Cloud Processing Object Classification Using Machine Learning and Deep Neural Networks Embedded Vision System Development using Automatic Code Generation2

Creating Computer Vision and Machine Learning Algorithmsthat Can Analyze Works of ArtChallengeDevelop an algorithm that can classify a painting by style, genre, andartist as easily as a human beingSolutionUse MATLAB and machine learning techniques to extract features forclassification and train the algorithmResults Machine learning techniques quickly tested and applied Thousands of paintings classified with 60% accuracy Tools to enable art historians to navigate vast databases of imagesLink to articleThe algorithm readily identified the influence of Diego Velazquez's“Portrait of Pope Innocent X” (left) on Francis Bacon's “StudyAfter Velazquez's Portrait of Pope Innocent X” (right).“MATLAB is a tremendous advantagebecause it provides many ways toquickly and easily visualize results.These visualizations enable us tounderstand the results and use them toinform ongoing AI research.”Ahmed ElgammalRutgers University3

3D Vision and Point Cloud Processing

What are Point Clouds? Point clouds represent a set of data points in a 3-D coordinate system Typically used to measure physical world surfaces Used for navigation and perception in robotics and Advanced DriverAssistance Systems (ADAS)5

Common Sources of Point Cloud mera6

Point Cloud Processing I/O and visualization Live acquisition from Kinect (pcfromkinect) Read and write PLY file format (pcread, pcwrite) Static point cloud viewer ( pcshow) Streaming point cloud viewer ( pcplayer) Point cloud difference visualization ( pcshowpair) Point cloud handling Object to manage 3-D data (pointCloud) Methods to find closest points (findNearestNeighbors) points in ROI (findPointsInROI) points in radius (findNeighborsInRadius)7

Point Cloud Registration and Stitching Rigid transformation (pctransform) Rigid registration (pcregrigid)– Iterative closest point ( ICP) algorithm Downsampling (pcdownsample) Denoising (pcdenoise) Stitching (pcmerge)Link to example8

Point Cloud Fitting to Geometric Shapes Estimate normal (pcnormals) Fit plane (pcfitplance) Fit sphere (pcfitsphere) Fit cylinder (pcfitcylinder)Link to example10

Structure From Motion From Multiple ViewsCamera pose estimation and 3-D scene reconstruction from multiple views Manage view attributes (viewSet)– Keypoints and matches– Camera poses Perform 3-D reconstruction from multiple views(triangulateMultiview) Refine camera poses and 3-D world points(bundleAdjustment)Link to example11

Object Classification Using Machine Learning andDeep Neural Networks

Object ClassificationorCar?SUV?Goal: Recognize objects in video stream13

Overall Workflow for Object ClassificationCarCarSUVCar1. Traditional machine learning2. Deep learningImage / VideoFramePreprocessObject Detection Resize / crop Enhance / denoise ForegroundExtractionObjectClassification Car or SUV?14

Common Challenges with Object Classification WorkflowStepsChallengeImporting DataManaging large sets of labeled imagesPreprocessingResizing, Enhancement, Data augmentationChoosing a classifieror architectureBackground in machine learning and neuralnetworksTraining and ClassificationComputation-intensive task (program GPUs)15

Machine Learning Workflow for Object ClassificationTraining DataFeatureExtractionLearning orModellingTrainingClassifier / ModelInput ification 16

Object Classification with Bag of WordsCreate Bag of FeaturesEncode into Histogram of Visual WordsPrepare Training DataEvaluate the Classifier17

Many Options for Feature Extraction and Machine LearningFeature Extraction BRISK,FREAK, SURF Histogram of Oriented Gradients (HoG) Using box filters(integral images ) Bag of visual words Color-based features Frequency-domain featuresComputer Vision System ToolboxMachine Learning SVM Decision trees AdaBoost Bagged trees k-NN Discriminant analysis Bayes classifiersStatistics & Machine Learning ToolboxMany permutations and combinations to fit your needs18

Traditional Machine Learning vs Deep LearningTraditional Machine LearningFeature ExtractionClassificationMachineLearningHandcrafted FeaturesCar Truck Bicycle Deep LearningConvolutional Neural Network (CNN)Learned features𝟗𝟓%End-to-end learning 𝟑% Feature learning Classification 𝟐%Car Truck Bicycle Deep learning performs end-end learning by learning features, representationsand tasks directly from images, text and sound19

Why is Deep Learning so Popular ?Year Results: Achieved substantially betterresults on ImageNet large-scale recognitionchallenge– 95% accuracy on ImageNet 1000 classchallenge Computing Power: GPU’s and advances inprocessor technologies have enabled us totrain networks on massive sets of data. Data: Availability of storage and access tolarge sets of labeled data– E.g. ImageNet , PASCAL VoC , KagglePre-2012 (traditionalcomputer vision andmachine learningtechniques)Error Rate 25%2012 (Deep Learning ) 15%2015 ( Deep Learning) 5 %20

Neural Network ToolboxParallel Computing ToolboxConvolutional Neural Networks Train “deep” neural networks on structured data (e.g. images, signals, text)Implements Feature Learning: Eliminates need for “hand crafted” featuresTrained using GPUs for performancecartruckvan bicycleInputConvolution ReLUPoolingConvolution ReLUFeature ication21

Two Approaches for Deep Learning1. Train a deep neural network from scratchConvolutional Neural Network (CNN)Learned features𝟗𝟓% 𝟑% 𝟐%Lots of dataCar Truck Bicycle 2. Fine-tune a pre-trained model (transfer learning)Fine-tune network weights Truck CarPre-trained CNNMedium amountsof dataNew Task25

Two Deep Learning ApproachesApproach 1: Train a Deep Neural Network from ScratchConvolutional Neural Network (CNN)Learned features𝟗𝟓% 𝟑% 𝟐%Car Truck Bicycle Recommended only when:Training data1000s to millions of labeled imagesComputationCompute intensive (requires GPU)Training TimeDays to Weeks for real problemsModel accuracyHigh (can over fit to small datasets)26

Two Deep Learning ApproachesApproach 2: Fine-Tune a Pre-Trained Model (Transfer Learning)CNN trained on massive sets of data Learned robust representations of images from larger data set Can be fine-tuned for use with new data or task with small – medium size datasetsFine-tune network weights Truck CarPre-trained CNNNew TaskNew DataRecommended when:Training data100s to 1000s of labeled images (small)ComputationModerate computation (GPU optional)Training TimeSeconds to minutesModel accuracyGood, depends on the pre-trained CNN model27

Object Classification Using Convolutional Neural NetworksTransfer Learning: Fine-Tune a Pre-Trained ModelPre-trained CNN(AlexNet – 1000 Classes)CarSUVNew DataNew Task – 2 ClassClassification28

ConclusionChallengeSolutionManaging large sets of labeledimagesResizing, Enhancement, DataaugmentationBackground in machinelearning and neural networksimageSet or imageDataStore tohandle large sets of imagesimresize, imcrop, imadjust,imageInputLayer, etc.Computation intensive task(program GPUs)Training supported on GPUsNo GPU programming expertise requiredClassification Learner App, intuitive welldocumented interfaces and examples 29

Embedded Vision System Development usingAutomatic Code Generation

Typical Workflow for Embedded Vision System DevelopmentMany ny iterationsPrototypingArchitectureAlgorithm Development Is my idea new? What is required? Is it robust to all kinds of conditions?(lighting noise, etc.)PrototypingDetailedDesignImplementation Consideration of HW platform FPGA? CPU? DSP? GPU?Speed and resource requirement Resolution, Frame-rate constraint Memory constraintDevelopment of the algorithm and implementation are often done by different groups31

MATLAB Coder app with Integrated Editor and Simplified WorkflowNew user interface simplifies code generation workflow32

Embedded Coder for Optimized CodeEmbedded Coder extends MATLAB Coder with Processor-specific code generation Built-in support for select processors Open APIs for use with any processor Speed, memory, and code appearanceadvanced features33

MATLAB Language Support for Code Generationvisualizationvariable-sized datastructJavamallocfunctionsnumericglobalSystem objectsnested graphicssparse34

Supported MATLAB Language Features and FunctionsBroad set of language features and functions/system objects supported forcode generationMatrices andArrays Matrix operationsN-dimensional arraysSubscriptingFramesPersistent variablesGlobal variablesData Types Complex numbersInteger mathDouble/single-precisionFixed-point arithmeticCharactersStructuresCell arraysNumeric classVariable-sized dataMATLAB ClassSystem objectsProgramming Constructs Arithmetic, relational, and logicaloperators Program control(if, for, while, switch)Functions MATLAB functions and subfunctions Variable-length argument lists Function handlesSupported algorithms More than 1300 MATLAB operators,functions, and System objects for: Communications Computer vision Image processing Neural networks Phased Array signal processing Robotics Signal processing Statistics and machine learning35

Supported Functions1275 Aerospace ToolboxCommunications System ToolboxComputer Vision System ToolboxDSP System ToolboxImage Acquisition ToolboxImage Processing Toolbox Neural Networks ToolboxPhased Array System ToolboxRobotics System ToolboxSignal Processing ToolboxStatistics and Machine Learning ToolboxSystem Identification Toolbox36

Automatic Translation of MATLAB to Ci terate.c.exeAlgorithm Design andCode Generation inMATLAB.lib.dllverify/a ccelerateMEXWith MATLAB Coder, design engineers can: Maintain one design in MATLABDesign faster and get to C quicklyTest more systematically and frequentlySpend more time improving algorithms in MATLAB37

Vision HDL ToolboxDesign and prototype video image processing systems Modeling hardware behavior of the algorithms Pixel-based functions and blocks Conversion between frames and pixels Standard and custom frame sizes Prototyping algorithms on hardware(With HDL Coder) Efficient and readable HDL code(With HDL Verifier) FPGA-in-the-loop testing and acceleration38

Pixel Based Video Image Algorithms Analysis & Enhancement – Edge Detection, Median Filter – Demosaic Interpolator, Gamma Corrector,Look-up Table Filters– Image Filter, Median Filter – Histogram– Image StatisticsConversions– Chroma Resampling, Color-SpaceConverterStatistics I/O Interfaces– Frame to Pixels, Pixels to Frame, FILversions Utilities– Pixel Control Bus Creator– Pixel Control Bus SelectorMorphological Operations– Dilation, Erosion,– Opening, Closing39

Frame To Pixels and Pixels To Frame40

Shipping Examples41

A Complete Solution for Embedded ypingFrame basedArchitecturedesignChipdesignPixel basedComputer VisionSystem ToolboxImageProcessingToolboxPrototypingHDL CoderVision HDLToolboxMATLABCoderFixed PointDesignerHDL VerifierMATLAB / Simulink42

FLIR Accelerates Development of Thermal Imaging FPGAChallengeAccelerate the implementation of advanced thermal imaging filtersand algorithms on FPGA hardwareSolutionUse MATLAB to develop, simulate, and evaluate algorithms, anduse HDL Coder to implement the best algorithms on FPGAsRaw image (left) and image after applying filter developed with HDL Coder(right).“With MATLAB and HDL Coder we are much moreresponsive to marketplace needs. We nowembrace change, because we can take a new ideato a real-time-capable hardware prototype in just aResults Time from concept to field-testable prototype reduced by 60% Enhancements completed in hours, not weeks Code reuse increased from zero to 30%few weeks. There is more joy in engineering, sowe’ve increased job satisfaction as well ascustomer satisfaction.”Nicholas HogastenFLIR SystemsLink to user story43