Intro To TensorFlow 2.0 MBL, August 2019 PDF Free Download

1y ago

40 Views

1 Downloads

2.93 MB

100 Pages

Report/dmca

Download PDF

Transcription

Intro to TensorFlow 2.0MBL, August 2019Josh Gordon (@random forests)1

Agenda 1 of 2Exercises Fashion MNIST with dense layersCIFAR-10 with convolutional layersConcepts (as many as we can intro in this short time) Gradient descent, dense layers, loss, softmax, convolutionGames QuickDraw

Agenda 2 of 2Walkthroughs and new tutorials Deep Dream and Style TransferTime series forecastingGames Sketch RNNLearning more Book recommendations

Deep Learning is representation learning

Image link

Latest tutorials and guidestensorflow.org/betaNews and updatesmedium.com/tensorflowtwitter.com/tensorflow

DemoPoseNet and BodyPixbit.ly/pose-netbit.ly/body-pix

TensorFlow for JavaScript, Swift,Android, and .org/lite

Minimal MNIST in TF 2.0A linear model, neural network, and deepneural network - then a short exercise.bit.ly/mnist-seq

.Softmaxmodel Sequential()model.add(Dense(256, activation 'relu',input shape (784,)))model.add(Dense(128, activation 'relu'))model.add(Dense(10, activation 'softmax'))Linear modelNeural networkDeep neural network

.Softmax activationAfter training, select all theweights connected to thisoutput.model.layers[0].get weights()# Your code here# Select the weights for a single output# .img weights.reshape(28,28)plt.imshow(img, cmap plt.get cmap('seismic'))

.After training, select all theweights connected to thisoutput.Softmax activation

Exercise 1 (option #1)Exercise: orials/keras/basic classiﬁcationTODO:Add a validation set. Add code to plot loss vs epochs (next slide).

Exercise 1 (option #2)bit.ly/ijcav advAnswers: next slide.

import matplotlib.pyplot as plt# Add a validation sethistory model.fit(x train, y train, validation data (x test, y test) .)# Get stats from the history objectacc history.history['accuracy']val acc history.history['val accuracy']epochs range(len(acc))# Plot accuracy vs epochsplt.title('Training and validation accuracy')plt.plot(epochs, acc, color 'blue', label 'Train')plt.plot(epochs, val acc, color 'orange', label .legend()

Exercise 1 (option #2)bit.ly/ijcav advAnswers: next slide.bit.ly/ijcai adv answer

About TensorFlow 2.019

Install# GPU!pip install tensorflow-gpu 2.0.0-beta1# CPU!pip install tensorflow 2.0.0-beta1In either case, check your installation (in Colab, you may need to use runtime - restart after installing).import tensorflow as tfprint(tf. version ) # 2.0.0-beta1Nightly is available too, but best bet: stick with a named release for stability.

TF2 is imperative by defaultimport tensorflow as tfprint(tf. version ) # 2.0.0-beta1x tf.constant(1)y tf.constant(2)z x yprint(z) # tf.Tensor(3, shape (), dtype int32)

You can interactive explore layersfrom tensorflow.keras.layers import Denselayer Dense(units 1, kernel initializer 'ones', use bias False)data tf.constant([[1.0, 2.0, 3.0]]) # Note: a batch of dataprint(data) # tf.Tensor([[1. 2. 3.]], shape (1, 3), dtype float32)# Call the layer on our dataresult layer(data)print(result) # tf.Tensor([[6.]], shape (1, 1), dtype float32)print(result.numpy()) # tf.Tensors have a handy .numpy() method

TF1: Build a graph, then run it.import tensorflow as tf # 1.14.0print(tf. version )x tf.constant(1)y tf.constant(2)z tf.add(x, y)print(z)

TF1: Build a graph, then run it.import tensorflow as tf # 1.14.0print(tf. version )x tf.constant(1)y tf.constant(2)z tf.add(x, y)print(z) # Tensor("Add:0", shape (), dtype int32)with tf.Session() as sess:print(sess.run(x)) # 3

Keras is built-in to TF2

How to import tf.kerasIf you want to use tf.keras and see the message “Using TensorFlow Backend”, you have accidentallyimported Keras (which is installed by default on Colab) from outside of TensorFlow.Example# !pip install tensorflow 2.0.0-beta1, then from tensorflow.keras import layers # Right from keras import layers # OopsUsing TensorFlow backend. # You shouldn’t see thisWhen in doubt, copy the imports from one of the tutorials on tensorﬂow.org/beta

NotesA superset of the reference implementation. Built-in to TensorFlow 2.0 (no need to install Kerasseparately).Documentation and examples Tutorials: tensorﬂow.org/betaGuide: tensorﬂow.org/beta/guide/keras/!pip install tensorflow 2.0.0-beta1from tensorflow import kerasI’d recommend the examples you ﬁnd on tensorﬂow.org/beta over otherresources (they are better maintained and most of them are carefullyreviewed).tf.keras adds a bunch of stuff, including model subclassing (Chainer / PyTorchstyle model building), custom trainingloops using a GradientTape, a collectionof distributed training strategies, supportfor TensorFlow.js, Android, iOS, etc.

More notesTF 2.0 is similar to NumPy, with: GPU supportAutodiffDistributed trainingJIT compilationA portable format (train in Python on Mac, deploy on iOS using Swift, or in a browser usingJavaScript)Write models in Python, JavaScript or Swift (and run anywhere).API doc: tensorﬂow.org/versions/r2.0/api docs/python/tfNote: make sure you’re looking at version 2.0 (the website still defaults to 1.x)

Three model building stylesSequential, Functional, Subclassing29

Sequential modelsmodel n(),tf.keras.layers.Dense(512, activation rs.Dense(10, activation 'softmax')])model.compile(optimizer 'adam',loss 'sparse categorical crossentropy',metrics ['accuracy'])model.fit(x train, y train, epochs 5)model.evaluate(x test, y test)

TF 1.xmodel n(),tf.keras.layers.Dense(512, activation rs.Dense(10, activation 'softmax')])model.compile(optimizer 'adam',loss 'sparse categorical crossentropy',metrics ['accuracy'])model.fit(x train, y train, epochs 5)model.evaluate(x test, y test)

TF 2.0model n(),tf.keras.layers.Dense(512, activation rs.Dense(10, activation 'softmax')])model.compile(optimizer 'adam',loss 'sparse categorical crossentropy',metrics ['accuracy'])model.fit(x train, y train, epochs 5)model.evaluate(x test, y test)

Functional modelsinputs keras.Input(shape (32, 32, 3))y layers.Conv2D(3, (3, 3),activation 'relu',padding 'same')(inputs)outputs layers.add([inputs, y])model keras.Model(inputs, outputs)keras.utils.plot model(model, 'skip connection.png', show shapes True)

Subclassed modelsclass MyModel(tf.keras.Model):def init (self, num classes 10):super(MyModel, self). init (name 'my model')self.dense 1 layers.Dense(32, activation 'relu')self.dense 2 layers.Dense(num classes,activation 'sigmoid')def call(self, inputs):# Define your forward pass herex self.dense 1(inputs)return self.dense 2(x)

Two training stylesBuilt-in and custom35

Use a built-in training loopmodel.fit(x train, y train, epochs 5)

Or, define your ownmodel MyModel()with tf.GradientTape() as tape:logits model(images)loss value loss(logits, labels)grads tape.gradient(loss value, model.trainable variables)optimizer.apply gradients(zip(grads, model.trainable variables))

A few concepts38

A vector of partial derivatives.LossGradient points indirection of steepestascent, so we step inreverse direction.Gradient descentCalculate the gradient.Take a step.Repeat.Step size (learning rate).t 1t 2t 3Parameter

Loss (w0, w1)With more than one variableThe gradient is a vector of partial derivatives (thederivative of a function w.r.t. each variable whilethe others are held constant).ww1The gradient points 0in the direction of steepest ascent. We usually want tominimize a function (like loss), so we take a step in the opposite direction.40

Training models with gradient descentForward pass Linear regression: y mx bNeural network: f(x) softmax(W2(g(W1x)))Calculate loss Regression: squared error.Classification: cross entropy.Backward pass Backprop: efficient method to calculate gradientsGradient descent: nudge parameters a bit in the opposite direction

Try it: Linear regressionbit.ly/tf-ws1Bonus: Deep Dream training loop will besimilar.

A neuronx0x1Linear combination ofinputs and weights𝜃0𝜃1𝜃2 gŷŷ g ( x i 𝜃i )x2Can rewrite as a dotproductInputsweightssumactivationoutputBias not drawn (you could set x1 to be a constant input of 1).ŷ g ( x T𝜃 )

One image and one classMultiple inputs; one output1296481.40.50.71.2120.5489618 130.1Plane 18wxbOutput

One image and two classesMultiple inputs; multiple outputs12481.40.71.2120.548-2.0960.50.1-0.79618W is now a matrix0.2 1.2 130.1Plane-11.4Car18WxbOutput46

Two images and two classes12964818418296 Image 1Image 6bOutputWx0.5

.After training, select all theweights connected to thisoutput.Softmax activation

A neural network

Truckg(12.8)TruckOutput ?Plane?Car?Truck

Applied r12.8Truckg(12.8)TruckOutput 130.1Plane0Car12.8Truck

Activation functions introduce non-linearitiesNotes- You can make similar plots (and more) with this example. Note: from an older version of TF, but should work out of the box in Colab.- Each of our convolutional layers used an activation as well (not shown in previous slides).- You can make a demo of this in TensorFlow Playground by setting activation Linear (or none)

Without activation, many layers are equivalent to one# If you replace 'relu' with 'None', this model .model Sequential([Dense(256, activation 'relu', input shape (2,)),Dense(256, activation 'relu'),Dense(256, activation 'relu'),Dense(1, activation 'sigmoid')])# . has the same representation power as this onemodel Sequential([Dense(1, activation 'sigmoid', input shape (2,))])

Softmax converts scores to max([130.1, -11.4, 12.8]) 0.999, 0.001, 0.001ProbabilitiesNote: these are ‘probability like’ numbers (do not go to vegas and bet in this ratio).

Cross entropy compares two distributionsEach example has a label in a one-hotformatCross entropy loss for a batch ofexamplesTrue prob (either 1 or 0)in our case!This is a bird012345678001000000900.1 0.2 0.6 0.2 0.0 0.0 0.0 0.0 0.0 0.0Rounded! Softmax output is always 0 x 1Sum over all examplesTrue probabilitiesPredicted probabilitiesPredicted prob(between 0-1)

Exercisebit.ly/ijcai 1-aComplete the notebook for Fashion MNISTAnswers: next slide.

Exercisebit.ly/ijcai 1-aComplete the notebook for Fashion MNISTAnswers: bit.ly/ijcai 1-a answers

TensorFlow RFPjbgordon@google.comgoo.gle/tensorﬂow-rfp

Convolution60

Not a Deep Learning conceptimport scipyfrom skimage import color, dataimport matplotlib.pyplot as pltimg data.astronaut()img color.rgb2gray(img)plt.axis('off')plt.imshow(img, cmap plt.cm.gray)

Convolution example-1-1-1-18-1-1-1-1NotesEdge detection intuition: dotproduct of the ﬁlter with aregion of the image will bezero if all the pixels aroundthe border have the samevalue as the center.Does anyone know who this is?

A simple edge detectorkernel np.array([[-1,-1,-1],[-1,8,-1],[-1,-1,-1]])result scipy.signal.convolve2d(img, kernel, 'same')plt.axis('off')plt.imshow(result, cmap plt.cm.gray)

Easier to see with seismic-1-1-1-18-1-1-1-1NotesEdge detection intuition: dotproduct of the ﬁlter with aregion of the image will bezero if all the pixels aroundthe border have the samevalue as the center.Eileen Collins

Example200110101000010000100300An input image(no padding)1A ﬁlter(3x3)Output image(after convolving with stride 1)

Example200110101000010000100300An input image(no padding)1A ﬁlter(3x3)2*1 0*0 1*1 0*0 1*0 0*0 0*0 0*1 1*03Output image(after convolving with stride 1)

Example200110101000010000100300An input image(no padding)1A ﬁlter(3x3)32Output image(after convolving with stride 1)

Example200001031010An input image(no padding)1101000010003230A ﬁlter(3x3)Output image(after convolving with stride 1)

Example200001031010An input image(no padding)11010000100032310A ﬁlter(3x3)Output image(after convolving with stride 1)

In 3dmodel Sequential()model.add(Conv2D(filters 4,kernel size (4,4),input shape (10,10,3))

A RGB image as a 3d volume.Each color (or channel) is alayer.

weights443In 3d, our ﬁlters have width,height, and depth.

weights443

weights4.43Applied in the same way as 2d(sum of weight * pixel value asthey slide across the image).

weights443Applying the convolution overthe rest of the input image.

weights44More ﬁlters, more output channels.3

Going deepermodel Sequential()model.add(Conv2D(filters 4,kernel size (4,4),input shape (10,10,3))model.add(Conv2D(filters 8,kernel size (3,3))

weights334

Edges?Shapes.Textures.

Exercisebit.ly/ijcai 1 bWrite a CNN from scratch for CIFAR-10.Answers: next slide.Ref: tensorﬂow.org/beta/tutorials/images/intro to cnns

Exercisebit.ly/ijcai 1bWrite a CNN from scratch for CIFAR-10.Answers: bit.ly/ijcai 1 b answers

Game 1Would you like to volunteer?quickdraw.withgoogle.com

Example: transfer learningbit.ly/ijcai 2Transfer learning using a pretrained MobileNet and a Denselayer.Ref: tensorﬂow.org/beta/tutorials/images/transfer learningRef: tensorﬂow.org/beta/tutorials/images/hub with keras

Example: transfer learningbit.ly/ijcai 2Transfer learning using a pretrainedMobileNet and a Dense layer.Answers: bit.ly/ijcai 2 answers

Deep DreamNew tutorialbit.ly/dream-wip

Image segmentationRecent tutorialbit.ly/im-seg

Timeseries forecastingRecent tutorial

Game 2Who would like to volunteer?magenta.tensorﬂow.org/assets/sketch rnndemo/index.html

CycleGANRecent tutorial

Under the hood93

Let’s make this fasterlstm cell tf.keras.layers.LSTMCell(10)def fn(input, state):return lstm cell(input, state)input tf.zeros([10, 10]); state [tf.zeros([10, 10])] * 2lstm cell(input, state); fn(input, state) # warm up# benchmarktimeit.timeit(lambda: lstm cell(input, state), number 10) # 0.03

Let’s make this fasterlstm cell tf.keras.layers.LSTMCell(10)@tf.functiondef fn(input, state):return lstm cell(input, state)input tf.zeros([10, 10]); state [tf.zeros([10, 10])] * 2lstm cell(input, state); fn(input, state) # warm up# benchmarktimeit.timeit(lambda: lstm cell(input, state), number 10) # 0.03timeit.timeit(lambda: fn(input, state), number 10) # 0.004

AutoGraph makes this possible@tf.functiondef f(x):while tf.reduce sum(x) 1:x tf.tanh(x)return x# you never need to run this (unless curious)print(tf.autograph.to code(f))

Generated codedef tf f(x):def loop test(x 1):with ag .function scope('loop test'):return ag .gt(tf.reduce sum(x 1), 1)def loop body(x 1):with ag .function scope('loop body'):with ag .utils.control dependency on returns(tf.print(x 1)):tf 1, x ag .utils.alias tensors(tf, x 1)x tf 1.tanh(x)return x,x ag .while stmt(loop test, loop body, (x,), (tf,))return x

Going big: tf.distribute.Strategymodel 64, input shape [10]),tf.keras.layers.Dense(64, activation 'relu'),tf.keras.layers.Dense(10, activation 'softmax')])model.compile(optimizer 'adam',loss 'categorical crossentropy',metrics ['accuracy'])

Going big: Multi-GPUstrategy tf.distribute.MirroredStrategy()with strategy.scope():model 64, input shape [10]),tf.keras.layers.Dense(64, activation 'relu'),tf.keras.layers.Dense(10, activation 'softmax')])model.compile(optimizer 'adam', loss 'categorical crossentropy',metrics ['accuracy'])

Learning moreLatest tutorials and guides tensorflow.org/betaBooks Hands-on ML with Scikit-Learn, Keras and TensorFlow (2nd edition)Deep Learning with PythonFor details deeplearningbook.org100

If you want to use tf.keras and see the message “Using TensorFlow Backend”, you have accidentally imported Keras (which is installed by default on Colab) from outside of TensorFlow. Example.