Transcription
CS6670: Computer VisionNoah SnavelyLecture 2: Image filteringHybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm
CS6670: Computer VisionNoah SnavelyLecture 2: Image filteringHybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm
CS6670: Computer VisionNoah SnavelyLecture 2: Image filteringHybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm
CS6670: Computer VisionNoah SnavelyLecture 2: Image filteringHybrid Images, Oliva et al., http://cvcl.mit.edu/hybridimage.htm
Reading Szeliski, Chapter 3.1-3.2
What is an image?
What is an image?Digital CameraWe’ll focus on these in this class(More on this process later)The EyeSource: A. Efros
What is an image? A grid (matrix) of intensity values255 255 255 255 255 255 255 255 255 255 255 255255 255 255 255 255 255 255 255 255 255 255 255 255 255 255200 255 255 255 255 255 255 255255 255 255757575 255 255 255 255 255 255255 25575959575 255 255 255 255 255 255255 25596 127 145 175 255 255 255 255 255 255255 255 127 145 175 175 175 255 255 255 255 255255 255 127 145 200 200 175 17595 255 255 255255 255 127 145 200 200 175 1759547 255 255255 255 127 145 145 175 127 1279547 255 255255 25547 255 25574 127 127 127255 255 255747474959595747474 255 255 255255 255 255 255 255 255 255 255 255 255 255 255255 255 255 255 255 255 255 255 255 255 255 255(common to use one byte per value: 0 black, 255 white)
What is an image? We can think of a (grayscale) image as afunction, f, from R2 to R (or a 2D signal):– f (x,y) gives the intensity at position (x,y)f (x, y)xysnoop3D view– A digital image is a discrete (sampled, quantized)version of this function
Image transformations As with any function, we can apply operatorsto an imageg (x,y) f (x,y) 20g (x,y) f (-x,y) We’ll talk about a special kind of operator,convolution (linear filtering)
Question: Noise reduction Given a camera and a still scene, how canyou reduce noise?Take lots of images and average them!What’s the next best thing?Source: S. Seitz
Image filtering Modify the pixels in an image based on somefunction of a local neighborhood of each pixel1053451117Local image dataSome function7Modified image dataSource: L. Zhang
Linear filtering One simple version: linear filtering(cross-correlation, convolution)– Replace each pixel by a linear combination of itsneighbors The prescription for the linear combination iscalled the “kernel” (or “mask”, “filter”)1053046101180Local image data000.5 081 0.5kernelModified image dataSource: L. Zhang
Cross-correlationLet be the image, be the kernel (ofsize 2k 1 x 2k 1), and be the outputimageThis is called a cross-correlation operation:
Convolution Same as cross-correlation, except that thekernel is “flipped” (horizontally and vertically)This is called a convolution operation: Convolution is commutative and associative
ConvolutionAdapted from F. Durand
Mean 00000000000
Linear filters: examples*Original000010000 Identical imageSource: D. Lowe
Linear filters: examples*Original000100000 Shifted leftBy 1 pixelSource: D. Lowe
Linear filters: examples*Original111111111 Blur (with a mean filter)Source: D. Lowe
Linear filters: examples*Original000020000-111111111 Sharpening filter(accentuates edges)Source: D. Lowe
SharpeningSource: D. Lowe
Smoothing with box filter revisitedSource: D. Forsyth
Gaussian KernelSource: C. Rasmussen
Gaussian filters 1 pixel 5 pixels 10 pixels 30 pixels
Gaussian filter Removes “high-frequency” components fromthe image (low-pass filter) Convolution with self is another Gaussian* – Convolving two times with Gaussian kernel ofwidth convolving once with kernel of widthSource: K. Grauman
SharpeningSource: D. Lowe
Sharpening revisited What does blurring take away? –detailsmoothed (5x5)originalLet’s add it back: αoriginaldetailsharpenedSource: S. Lazebnik
Sharpen filterimageblurredimagescaled impulseunit impulse(identity)GaussianLaplacian of Gaussian
Sharpen filterunfilteredfiltered
Convolution in the real worldCamera shake *Source: Fergus, et al. “Removing Camera Shake from a Single Photograph”, SIGGRAPH 2006Bokeh: Blur in out-of-focus regions of an image.Source: http://lullaby.homepage.dk/diy-camera/bokeh.html
Questions?
Edge detection Convert a 2D image into a set of curves– Extracts salient features of the scene– More compact than pixels
Origin of Edgessurface normal discontinuitydepth discontinuitysurface color discontinuityillumination discontinuity Edges are caused by a variety of factors
Characterizing edges An edge is a place of rapid change in theimage intensity functionimageSource: L. Lazebnikintensity function(along horizontal scanline)first derivativeedges correspond toextrema of derivative
Image derivatives How can we differentiate a digital image F[x,y]?– Option 1: reconstruct a continuous image, f, thencompute the derivative– Option 2: take discrete derivative (finite difference)How would you implement this as a linear filter?:1-1:-11Source: S. Seitz
Image gradient The gradient of an image:The gradient points in the direction of most rapid increase in intensityThe edge strength is given by the gradient magnitude:The gradient direction is given by: how does this relate to the direction of the edge?Source: Steve Seitz
Image gradient
Effects of noiseNoisy input imageWhere is the edge?Source: S. Seitz
Solution: smooth firstfhf*hTo find edges, look for peaks inSource: S. Seitz
Associative property of convolution Differentiation is convolution, and convolutionis associative: This saves us one operation:fSource: S. Seitz
2D edge detection filtersGaussianderivative of Gaussian (x)
Derivative of Gaussian filterx-directiony-direction
Side note: How would youcompute a directional derivative?(From vector calculus) ? Directional deriv. is alinear combination ofpartial derivatives
Derivative of Gaussian filtery-directionx-direction
The Sobel operator Common approximation of derivative of Gaussian-101121-202000-101-1 -2 -1 The standard defn. of the Sobel operator omits the 1/8 term– doesn’t make a difference for edge detection– the 1/8 term is needed to get the right gradient value
Sobel operator: exampleSource: Wikipedia
Example original image (Lena)
Finding edgesgradient magnitude
Finding edgeswhere is the edge?thresholding
Non-maximum supression Check if pixel is local maximum along gradient direction– requires interpolating pixels p and r
Finding edgesthresholding
Finding edgesthinning(non-maximum suppression)
Canny edge detectorMATLAB: edge(image,‘canny’)1. Filter image with derivative of Gaussian2. Find magnitude and orientation of gradient3. Non-maximum suppression4. Linking and thresholding (hysteresis):– Define two thresholds: low and high– Use the high threshold to start edge curves andthe low threshold to continue themSource: D. Lowe, L. Fei-Fei
Canny edge detector Still one of the most widely used edgedetectors in computer visionJ. Canny, A Computational Approach To Edge Detection, IEEE Trans. PatternAnalysis and Machine Intelligence, 8:679-714, 1986. Depends on several parameters:: width of the Gaussian blurhigh thresholdlow threshold
Canny edge detectororiginalCanny withCanny with The choice ofdepends on desired behavior– largedetects “large-scale” edges– smalldetects fine edgesSource: S. Seitz
Scale space (Witkin 83)first derivative peakslargerGaussian filtered signal Properties of scale space (w/ Gaussian smoothing)– edge position may shift with increasing scale ( )– two edges may merge with increasing scale– an edge may not split into two with increasing scale
Questions? 3-minute break
Images as vectors Very important idea!102D imageScanline (1D signal)(A 2D, n x m image can be represented by a vectorof length nm formed by concatenating the rows)Vector
Multiplying row and column vectors ?
Filtering as matrix multiplicationWhat kind of filter is this?
Filtering as matrix multiplication
Another matrix transformation2D DCT basis1D Discrete cosine transform (DCT) basis
Another matrix transformation2D DCT basis1D Discrete cosine transform (DCT) basis
What is an image? A grid (matrix) of intensity values (common to use one byte per value: 0 black, 255 white) 255 255 255 255 255 255 255 255 255 255 255 255
