WIXLIB

TENSORFLOWLITE DELEGATESON ARM-BASEDDEVICESPavel MacenauerSenior Software Engineer, NXPMARCH 2021PUBLICNXP, THE NXP LOGO AND NXP SECURE CONNECTIONS FOR A SMARTER W ORLD ARE TRADEMARKS OF NXP B.V.ALL OTHER PRODUCT OR SERVICE NAMES ARE THE PROPERTY OF THEIR RES PECTIVE OW NERS. 2021 NXP B.V.

TF Lite Model DeploymentDelegates OverviewPartitioning and Implementing a CustomDelegateBenchmarking and CorectnessDelegates More Thoroughly NNAPI Delegate ARM NN DelegatePython and C Code SamplesPUBLIC1

TF LITE MODEL DEPLOYMENTDefaultKernelsTensorFlow model(Keras, SavedModel).h5, .pbTF LiteConverter.tfliteInterpreterExisting Interface:NN APIXNN PACK DelegatesImplement yourown Develop a model in TensorFlow or load from *.h5 (Keras) / *.pb (SavedModel/checkpoint)TensorFlow is a popular machine learning opensource frameworkdeveloped by GoogleTF Lite is a module targeted mostly for inference on IoT / embeddeddevices(for Microcontrollers there is the new TF Micro, which we will not cover)PUBLIC2

TF LITE MODEL DEPLOYMENTDefaultKernels.h5, .pbTensorFlow model(Keras, SavedModel)TF LiteConverter.tfliteInterpreterExisting Interface:NN APIXNN PACK DelegatesHigh-level APILow-level APItf.Keras.*Implement yourowntf.* Convert to Flatbuffer (serialized) formatKeras Model (.h5, .hdf5)Saved Model (.pb)TF Lite ConverterConcretefunctions Can be converted to uint8 (quantized) orkept in float32 Enables different quantization methods Float16, uint8, hybrid uint8/uint16TF Lite Flatbuffer file(.tflite)PUBLIC3

TF LITE MODEL DEPLOYMENTDefaultKernelsTensorFlow model(Keras, SavedModel).h5, .pbTF LiteConverter.tfliteInterpreterExisting Interface:NN APIXNN PACK DelegatesImplement yourown Runs the model on the device (by default on the CPU) experimental delegates enables TfLiteDelegate API To implement a custom delegate see also SimpleDelegate API, which is a wrapperPUBLIC4

TF LITE MODEL DEPLOYMENTDefaultKernelsTensorFlow model(Keras, SavedModel).h5, .pbTF LiteConverter.tfliteInterpreterExisting Interface:NN APIXNN PACK DelegatesImplement yourown Default kernels run on the CPU and they are optimized for Arm NEON Delegates are able to offload execution to a different device (GPU, NPU, DSP, )PUBLIC5

TF LITE MODEL DEPLOYMENTDefaultKernelsTensorFlow model(Keras, SavedModel).h5, .pbTF LiteConverter.tfliteInterpreterExisting Interface:NN APIXNN PACK DelegatesImplement yourown Delegate Examples of existing delegates are NNAPI (Android), XNN Pack, GPU (OpenCL), Hexagon DSP, CoreML, NNAPI defines an interface, implementation is found on the device NXP i.MX8 microprocessors use NN API delegate to offload execution to the GPU or the NPU depending onwhat is available XNN Pack Delegate is an alternative to the default CPU kernels A custom delegate can be provided – examples are VX Delegate, Arm NN Delegate, https://github.com/ARM-software/armnn (under delegate) https://github.com/VeriSilicon/TIM-VX (in development)PUBLIC6

W H Y T O U S E A D E L E G AT E AN D W H AT AR E T H E D O W N S I D E S ?PROSCONS Unsupported ops run on the CPU and causeperformance degradation due to additionaltensor copies Memory transfer can become a bottleneckespecially when graph is partitioned a lotSpecialized hardware provides betterperformance and/or power consumption Hardwaresuch as a GPU/NPU/DSP ifavailable Alot of hardware fuses operations(activations, pooling layers, FC, ) Possibleto free CPU for other tasksPUBLIC7

D E L E G AT E PAR T I T I O N I N G Graph is partitioned based on op supportExecuted on the DConcatenationOnly Conv2D andConcatenationoperations aresupported by ourdelegateDELEGATEDPARTITIONExecuted by the delegatePUBLIC8

S W I T C H I N G B E T W E E N PAR T I T I O N SThe preferred method to implement a Delegate is using SimpleDelegate API:SimpleDelegateInterface Capabilities of the Delegate (options), op support, factory rface Logic for initializing, preparing and running the delegated partitionsCOMMON PARAMETERS (SimpleDelegateInterface)3 partitions?Might beexpensive e.g.due to memorytransfersnum threads: int (default 1)Conv2D The number of threads to use for running the inference on CPU.max delegated partitions: int (default 0, i.e. no limit)DepthwiseConv2D The maximum number of partitions that will be delegated.Currently supported by the GPU, Hexagon, CoreML and NNAPI delegate.Conv2Dmin nodes per partition: int (default delegate's own choice) DepthwiseConv2D The minimal number of TFLite graph nodes of a partition that needs to be reached tobe delegated. A negative value or 0 means to use the default choice of each delegate.This option is currently supported by the Hexagon and CoreML delegate. and other Delegate specific optionsPUBLIC9

BENCHM ARKINGbenchmark model Simple tool to evaluateperformance and memory averageinference latency initialization memoryoverheadfootprint./benchmark model \--graph mobilenet v1 1.0 224 quant.tflite \--use nnapi trueSTARTING! The input model file size (MB): 4.27635Initialized session in 3.84ms.Running benchmark for at least 1 iterations and at least 0.5 secondsbut terminate if exceeding 150 seconds.count 1 curr 6036807Running benchmark for at least 50 iterations and at least 1 seconds butterminate if exceeding 150 seconds.count 361 first 2795 curr 2704 min 2653 max 2849 avg 2693.12 std 21Inference timings in us: Init: 3840, First inference: 6036807, Warmup(avg): 6.03681e 06, Inference (avg): 2693.12Note: as the benchmark tool itself affects memory footprint, thefollowing is only APPROXIMATE to the actual memory footprint of themodel at runtime. Take the information at your discretion.Peak memory footprint (MB): init 2.75391 overall 29.0273PUBLIC10

CORRECTNESSinference diff Simple tool to evaluatecorrectnessnum runs: 50process metrics {inference profiler metrics { output errors {max value: 0.000999001min value: 0avg value: 1.54487801942406548e-05std deviation: 0.00029687365}}}PUBLIC11

N N AP I D E L E G AT E Android C API designed to run machinelearning operations on Android devices Limited to float16, float32, int8 and uint8 Supports acceleration on a GPU,an NPU or a DSP depending on thetarget deviceTF Lite CPU fallback ispreferred, but it can be disabledusing setUseNnapiCpu inNnApiDelegate.OptionsApplicationML framework/libraryCPU fallback(TensorFlow Lite)Android NN APICPU fallbackAndroid Neural Network RuntimeHardware accelerationAndroid NN HALDSP DriverDSPUnified NPU/GPU DriverGPUNPUi.MX8 microprocessorsPUBLIC12

W H AT I S AR M N N ? A middleware inference engine for machinelearning on the edge donated mid-2018 toLinaro AI initiative Single API integrating popular high-level MLframeworks (TensorFlow, TF Lite, Caffe, ONNX –MXNet, PyTorch) Connects high-level ML frameworks to computeengines, drivers, HW through Arm NN backends Optimized for ARM and NXP hardware Cortex-A CPUs, Mali GPUs, Ethos-N NPUs i.MX8 microprocessors(Cortex-A CPUs GPU/NPU for hub.com/ARM-software/armnnPUBLIC13

AR M N N D E L E G AT E Released originally in 20.11, but much more mature in es/tag/v21.02) For installation, build, integration see the docs: https://arm-software.github.io/armnn/21.02/ Allows to offload execution to all the backends supported by Arm NNArm Compute Library (Arm NEON – Cortex-A CPU, OpenCL – GPU/Mali) Arm Ethos-N NPU Custom backends such as NXP’s GPU/NPU (VSI NPU) Allows to replace Arm NN parser front-end with TF Lite Builds as a dynamic/shared library (libarmnnDelegate.so) which is linked by the target applicationBuilds standalone or together with Arm NN (CMake)Requires prebuilt TF Lite as the only dependency PUBLIC14

P Y T H O N AP I AR M N N E X AM P L Eimport numpy as npimport tflite runtime.interpreter as tflite Import any Python module such as numpy tflite runtime (Python wrapper) is required# Load TFLite model and allocate tensorsarmnn delegate tflite.load delegate(library "/usr/lib/libarmnnDelegate.so",options {"backends": "VsiNpu, CpuAcc, CpuRef","logging-severity": "info"})# Delegates/Executes all operations supported by ArmNN to/with ArmNNinterpreter tflite.Interpreter(model path "mobilenet v1 1.0 224 quant.tflite",experimental delegates [armnn delegate])# Now we may allocate input, output tensors, and run inferenceinterpreter.allocate tensors()input details interpreter.get input details()output details interpreter.get output details()input shape input details[0]['shape']input data np.array(np.random.random sample(input shape), dtype np.uint8)interpreter.set tensor(input details[0]['index'], input data)interpreter.invoke()# Print out resultoutput data interpreter.get tensor(output details[0]['index'])PUBLIC15

P Y T H O N AP I AR M N N E X AM P L Eimport numpy as npimport tflite runtime.interpreter as tflite Load the dynamic delegate Set delegate options – most importantly choose allthe backends# Load TFLite model and allocate tensorsarmnn delegate tflite.load delegate(library "/usr/lib/libarmnnDelegate.so",options {"backends": "VsiNpu, CpuAcc, CpuRef","logging-severity": "info"})# Delegates/Executes all operations supported by ArmNN to/with ArmNNinterpreter tflite.Interpreter(model path "mobilenet v1 1.0 224 quant.tflite",experimental delegates [armnn delegate])# Now we may allocate input, output tensors, and run inferenceinterpreter.allocate tensors()input details interpreter.get input details()output details interpreter.get output details()input shape input details[0]['shape']input data np.array(np.random.random sample(input shape), dtype np.uint8)interpreter.set tensor(input details[0]['index'], input data)interpreter.invoke()# Print out resultoutput data interpreter.get tensor(output details[0]['index'])PUBLIC16

P Y T H O N AP I AR M N N E X AM P L Eimport numpy as npimport tflite runtime.interpreter as tflite Load TF Lite model Link the Arm NN delegate instance# Load TFLite model and allocate tensorsarmnn delegate tflite.load delegate(library "/usr/lib/libarmnnDelegate.so",options {"backends": "VsiNpu, CpuAcc, CpuRef","logging-severity": "info"})# Delegates/Executes all operations supported by ArmNN to/with ArmNNinterpreter tflite.Interpreter(model path "mobilenet v1 1.0 224 quant.tflite",experimental delegates [armnn delegate])# Now we may allocate input, output tensors, and run inferenceinterpreter.allocate tensors()input details interpreter.get input details()output details interpreter.get output details()input shape input details[0]['shape']input data np.array(np.random.random sample(input shape), dtype np.uint8)interpreter.set tensor(input details[0]['index'], input data)interpreter.invoke()# Print out resultoutput data interpreter.get tensor(output details[0]['index'])PUBLIC17

P Y T H O N AP I AR M N N E X AM P L Eimport numpy as npimport tflite runtime.interpreter as tflite Set inputs, process outputs, run inference# Load TFLite model and allocate tensorsarmnn delegate tflite.load delegate(library "/usr/lib/libarmnnDelegate.so",options {"backends": "VsiNpu, CpuAcc, CpuRef","logging-severity": "info"})# Delegates/Executes all operations supported by ArmNN to/with ArmNNinterpreter tflite.Interpreter(model path "mobilenet v1 1.0 224 quant.tflite",experimental delegates [armnn delegate])# Allocate input, output tensors, and run inferenceinterpreter.allocate tensors()input details interpreter.get input details()output details interpreter.get output details()input shape input details[0]['shape']input data np.array(np.random.random sample(input shape), dtype np.uint8)interpreter.set tensor(input details[0]['index'], input data)interpreter.invoke()# Print out resultoutput data interpreter.get tensor(output details[0]['index'])PUBLIC18