WIXLIB

Lecture 2Camera ModelsProfessor Silvio SavareseStanford Vision and Learning LabSilvio Savarese & Jeanette BohgLecture 2 -13-Jan-21

Announcements- P0 was out on Monday and due on Jan 17th- P1 is also released (today!) and due on 1/29- We a CA session this Friday:- Python Introduction- Linear Algebra ReviewSilvio Savarese & Jeanette BohgLecture 2 -13-Jan-21

Lecture 2Camera Models Pinhole cameras Cameras & lenses The geometry of pinhole camerasReading:[FP] Chapter 1, “Geometric Camera Models”[HZ] Chapter 6 “Camera Models”Some slides in this lecture are courtesy to Profs. J. Ponce, S. Seitz, F-F LiSilvio Savarese & Jeanette BohgLecture 2 -13-Jan-21

How do we see the world? Let’s design a camera– Idea 1: put a piece of film in front of an object– Do we get a reasonable image?

Pinhole cameraAperture Idea 2: Add a barrier to block off most ofthe rays– This reduces blurring– The opening known as the aperture

Some history Milestones: Leonardo da Vinci (1452-1519):first record of camera obscura (1502)

Some history Milestones: Leonardo da Vinci (1452-1519):first record of camera obscura Johann Zahn (1685): firstportable camera

Some history Milestones: Leonardo da Vinci (1452-1519):first record of camera obscura Johann Zahn (1685): firstportable camera Joseph Nicéphore Niépce(1822): first photo - birth ofphotographyPhotography (Niépce, “LaTable Servie,” 1822)

Some history Milestones: Leonardo da Vinci (1452-1519):first record of camera obscura Johann Zahn (1685): firstportable camera Joseph Nicéphore Niépce(1822): first photo - birth ofphotography Daguerréotypes (1839) Photographic Film (Eastman, 1889) Cinema (Lumière Brothers, 1895) Color Photography (Lumière Brothers,1908)Photography (Niépce, “LaTable Servie,” 1822)

Let’s also not forget Motzu(468-376 BC)Oldest existentbook ongeometry inAristotleAl-Kindi (c. 801–873)(384-322 BC)Ibn al-HaithamAlso: Plato, Euclid(965-1040)

PinholeprojectioncameraPinhole perspectivefof focal lengtho aperture pinhole center of the camera

Pinhole cameraféx ùé x ùêúP ê y ú P ê úy ûëêë z úûìx' fïïíï y' fïîxz[Eq. 1]yzDerived using similar triangles

Pinhole camerafkP’ [x’, f ]fiOP [x, z][Eq. 2]x x fz

Pinhole cameraIs the size of the aperture important?Kate lazuka

Shrinkingaperturesize-What happens if the aperture is too small?-Less light passes throughAdding lenses!

Cameras & LensesPimageP’ A lens focuses light onto the film

Cameras & LensesPimageOut of focusP’ A lens focuses light onto the film– There is a specific distance at which objects are “infocus”– Related to the concept of depth of field

Cameras & Lenses A lens focuses light onto the film– There is a specific distance at which objects are “infocus”– Related to the concept of depth of field

Cameras & Lensesfocal pointf– A lens focuses light onto the film– All rays parallel to the optical (or principal) axis converge toone point (the focal point) on a plane located at the focallength f from the center of the lens.– Rays passing through the center are not deviated

Paraxial refraction modelZ’p’ [x’,y’]P [x,y,z]-zxìïïx ' z' zFrom Snell’s law:í[Eq. 3] ï y' z' yïîzfzoìx' fïïíï y' fïîxzyz[Eq. 1]

Paraxial refraction modelZ’p’ [x’,y’]P [x,y,z]-zxìïïx ' z' zFrom Snell’s law:í[Eq. 3] ï y' z' yïîz[FP] sec 1.1, page 8fzo[Eq. 4]z' f z oRf 2( n - 1)

Issues with lenses: Radial Distortion– Deviations are most noticeable for rays that pass throughthe edge of the lensNo distortionPin cushionBarrel (fisheye lens)Image magnification decreaseswith distance from the optical axis

Lecture 2Camera Models Pinhole cameras Cameras & lenses The geometry of pinhole cameras Intrinsic ExtrinsicSilvio Savarese & Jeanette BohgLecture 2 -13-Jan-21

Pinhole cameraPinhole perspective projectionféx ùé x ùêúP ê y ú P ê úy ûëêë z úû[Eq. 1]ìx' fïïíï y' fïîxzyzEÂ Â32f focal lengtho center of the camera

From retina plane to imagesRetina planefDigital imagePixels, bottom-left coordinate systems

Coordinate systemsfy1. Off setycxcC’’ [cx, cy]xy( x , y, z ) ( f c x , f c y )zz[Eq. 5]x

Converting to pixelsfy1. Off set2. From metric to pixelsycxcC [cx, cy]xxy( x , y, z ) ( f k c x , f l c y )zzab[Eq. 6]Units: k,l : pixel/mf :mNon-square pixelsa, b: pixel

Is this projective transformation linear?fyycC [cx, cy]xyP (x, y, z) P ' (α cx , β cy )zz[Eq. 7]xc Is this a linear transformation?x No — division by z is nonlinearCan we express it in a matrix form?

Homogeneous coordinatesEàHhomogeneous imagecoordinateshomogeneous scenecoordinates Converting back from homogeneous coordinatesHàE

Projective transformation in thehomogenous coordinate system!! x ##&P # y & ### z &#"%"x &y &z &&1 %Ph

Projective transformation in thehomogenous coordinate system! α x c zx#Ph ' # β y cy z#z#"Homogenous ! α& #& # 0& #&% #" 00cxβ cy01EuclidianxyPh ' P ' (α cx , β cy )zz! 0 #&0 &#� &% #"x &y &z &&1 %! α#M # 0##" 0Ph[Eq.8]0cxβ cy010 &0 &&0 &%

The Camera MatrixfCameramatrix K[Eq.9]P' M P!# K" I 0 P! α#P' # 0##" 00cxβ cy01! 0 #&0 &#� &%#"x &y &z &&1 %

Camera SkewnessfyycxcθC [cx, cy]# α α cot θ%β%P! % 0sin θ%0% 0cxcy10 &# x(%(% y0 (% z(%0 ( 1'&(((('xsec. 1.2.2 [FP] or sec. 6.2.4 [HZ]

Degrees of freedom of KfyycxcθC [cx, cy]x# α α cot θ%β%P! % 0sin θ%0% 0cxcy10 &# x(%(% y0 (% z(%0 ( 1'&(((('How many degrees of freedom does K have?5 degrees of freedom!

Canonical ProjectiveTransformation!! x ! #1000#& #&#P' # y & # 0 1 0 0 &#& # 0 0 1 0 &#%##" z &% ""MPi ' !####"xzyz &&&&%xyz1 &&&&%[Eq.10]P' M PHÂ Â43

Lecture 2Camera Models Pinhole cameras Cameras & lenses The geometry of pinhole cameras Intrinsic Extrinsic Other camera modelsSilvio Savarese & Jeanette BohgLecture 2 -13-Jan-21

World reference systemR,TjwfkwOwiw The mapping so far is defined within the camera referencesystem What if an object is represented in the world referencesystem? Need to introduce an additional mapping from world refsystem to camera ref system

2D TranslationP'PtPlease refer to CA sessionon transformations formore details

2D Translation EquationP’tyPyxtP ( x, y )t (t x , t y )txP ' P t (x t x , y t y )

2D Translation using Homogeneous CoordinatesP’tyPyxP (x, y) (x, y,1)ttxé x t x ù é1 0 t x ù é x ùP ' êê y t y úú êê0 1? t y úú êê y úúêë 1 úû êë0 0 1 úû êë 1 úû! x ! x ! I t #&#& #& # y & T # y &" 0 1 %#&# 1 &1"%"%

ScalingP'P

Scaling EquationP’sy yyP ( x , y ) P ' (s x x , s y y )PP (x, y) (x, y,1)xé sx x ù ésxP ' êê s y y úú êê 0êë 1 úû êë 0sx x0sy0S! x ! x 0ù é x ù! S' 0 #&#&úêú0ú ê y ú #& # y & S # y &0 1 %#"&# 1 &1 úû êë 1 úû1"%"%

RotationP'P

Rotation Equations Counter-clockwise rotation by an angle 𝜃x ' cos q x - sin q yy’y ' cos q y sin q xP’qPyx’xé x ' ù é cos qê y 'ú ê sin që û ëHow many degrees of freedom? 1- sin q ù é x ùcos q úû êë y úû# cosθ%P' % sinθ% 0 sinθcosθ0P' R P001&# x &(%((% y ((% 1 (''

Scale Rotation Translation" 1 P' 0 # 0é cos q êê sinqêë 0! R #" 0010t x %" cosθ' t y ' sinθ' 1 '&# 0- sinqt ! S
%" 0cos q0t x ù é sxt y úú êê 01 úû êë 0 sinθcosθ00sy0! x !0 # y & # R S&#&1 %#& #" 01"%" s%0 x'0 ' 0 1 ' 0�ù é x ù0 úú êê y úú1 úû êë 1 úû! x &t #&# y &1 &%#&1"%0sy00 %" x' 0 ' y' 1 '&# 1%'''&If sx sy, this is asimilaritytransformation

3D Translation of PointsTyyP’PTxzTx! T# xT # Ty##" Tz &&&&%"" 0I T % P' ' # 0 1 &4 4 #x %'y 'z ''1 &A translation vector in 3D has 3 degrees of freedom

3D Rotation of PointsRotation around thecoordinate axes,counter-clockwise:P’y’yzP0é1Rx (a ) êê0 cos aêë0 sin aù- sin a úúcos a úûé cos b 0 sin b ùR y ( b ) êê 010 úúêë- sin b 0 cos b úûécos g - sin g 0ùRz (g ) êê sin g cos g 0úúêë 001úû0"x’ x" R 0 % P' R Rx (α ) Ry (β ) Rz (γ )' # 0 1 &4 4 #A rotation matrix in 3D has 3 degrees of freedomx %'y 'z ''1 &

3D Translation and RotationR Rx (α ) Ry (β ) Rz (γ )! T# xT # Ty##" Tz"" R T % P' '# 0 1 &4 4 #x %'y 'z ''1 & &&&&%

World reference systemR,TfjwPkwOwiwP’! R T In 4D homogeneous coordinates: P #& Pw" 0 1 %4 4[Eq.9]P ' K !" IInternal parameters!#####"xw &yw &&zw &1 &%External parameters! R T ##!!#0 P K " I 0 %& Pw K R"" 0 1 4 4MT # Pw[Eq.11]

The projective transformationR,TfjwPkwOwiwP’P '3 1 M 3x 4 Pw K 3 3 "# R T % Pw 4 13 4[Eq.11]How many degrees of freedom does M have?5 3 3 11!

The projective transformationR,TfjwPkwOwiwP’P '3 1 M Pw K 3 3 "# R T % Pw 4 13 4! mP! m # 1 W# 1 & # m 2 & PW # m 2 PW##&#" m3 &%#" m3 PW &&&&%Eé m1 ùêúM êm 2 úêëm 3 úûm1Pw m2 Pw (,) [Eq.12]m3 Pw m3 Pw

Properties of projective transformations Points project to points Lines project to lines Distant objects look smaller

Properties of Projection Angles are not preserved Parallel lines meet!Parallel lines in the worldintersect in the image at a“vanishing point”

Horizon line (vanishing line)

One-point perspective Masaccio, Trinity,Santa MariaNovella, Florence,1425-28Credit slide S. Lazebnik

Next lecture How to calibrate a camera?

Supplemental material

Thin Lenses[FP] sec 1.1, page 8.zoz' f z oRf 2( n - 1)Focal lengthSnell’s law:n1 sin a1 n2 sin a2Small angles:n 1 a1 » n 2 a2n1 n (lens)n1 1 (air)xìïïx ' z' zíï y' z ' yïîz

Camera Models. Silvio Savarese & Jeanette Bohg Lecture 2 - 13-Jan-21-P0 was out on Monday and due on Jan 17th-P1 is also released (today!) and due on 1/29-We a CA session this Friday: - Python Introduction - Linear Algebra Review Announcements. Silvio Savarese & Jeanette Bohg Lecture 2 - 13-Jan-21 Pinhole cameras Cameras & lenses The geometry of pinhole cameras Lecture 2 Camera