Transcription
GDC 2007 Demo Team Secrets:Advanced Skin RenderingEugene d’Eon
OutlineDemos: Adrianne and FroggyQuick Overview of where we’re goingReflectance properties of real skinReview of current techniquesWays to improve real-time skin rendering NVIDIA Corporation 2007
Demos NVIDIA Corporation 2007
Quick OverviewHow are these images generated? NVIDIA Corporation 2007
Overview DiagramRender texturespace lightStart NVIDIA Corporation 2007
Overview DiagramRender texturespace lightStart NVIDIA Corporation 2007blur
Overview DiagramRender texturespace lightStart NVIDIA Corporation 2007blurblur blur
Overview DiagramRender texturespace lightStartblurblurLinearcombination NVIDIA Corporation 2007 blurDifferent weights for R, G, and B
Overview DiagramRender texturespace lightStartblurblur Linearcombinationtexture mappingFinal pass: combine blurs NVIDIA Corporation 2007blur
Overview DiagramRender texturespace lightStartblurblur blurLinearcombinationtexture mappingFinal pass: combine blurs specular NVIDIA Corporation 2007
Pre-computed MaterialAmbient occlusion map(static)Hand painted mapsDiffuse ColorBumpSpecular Brightness NVIDIA Corporation 2007
Pre-computed ambient occlusion NVIDIA Corporation 2007
Diffuse Color map NVIDIA Corporation 2007
Bump map NVIDIA Corporation 2007
Specular Brightness map(brightened here by 16X for display purposes) NVIDIA Corporation 2007
Computed each frameSubsurface IrradianceSeveral blurred versions of SubsurfaceIrradianceBlend: Final Subsurface RadianceSpecularRim light (optional) NVIDIA Corporation 2007
Start with incident light (irradiance)(Here we show just a single point light) NVIDIA Corporation 2007
Compute Subsurface Irradiance(explained later – actually computed in an off-screen texture) NVIDIA Corporation 2007
Blur Subsurface Irradiance NVIDIA Corporation 2007
Blur Subsurface Irradiance Again NVIDIA Corporation 2007
Blur Subsurface Irradiance Again NVIDIA Corporation 2007
Blur Subsurface Irradiance Again NVIDIA Corporation 2007
Blur Subsurface Irradiance Again NVIDIA Corporation 2007
Final render passRender mesh in 3DCombine blurry texturesAdd specular and rim lighting NVIDIA Corporation 2007
Overview DiagramRender texturespace lightStartblurblur blurLinearcombinationtexture mappingFinal pass: combine blurs specular NVIDIA Corporation 2007
Final pass NVIDIA Corporation 2007
Ummm What?Why blurring?Why keep around several blurs?Why different recombination weights for R, G,and B?Why isn’t specular blurred? NVIDIA Corporation 2007
Reflectance properties of skin NVIDIA Corporation 2007
Reflectance properties of skinDominated by subsurface scattering 6% direct reflection, 94% subsurface **[KRISHNASWAMY, A., AND BARONOSKI, G. V. G. 2004] NVIDIA Corporation 2007
Reflectance properties of skinDominated by subsurface scattering 6% direct reflection, 94% subsurface *Reflectance and Scattering are different inred, green and blue wavelengths*[KRISHNASWAMY, A., AND BARONOSKI, G. V. G. 2004] NVIDIA Corporation 2007
Reflectance properties of skinDominated by subsurface scattering 6% direct reflection, 94% subsurface *Reflectance and Scattering are different inred, green and blue wavelengthsScattering is poorly modeled by assuming asingle layer of roughly uniform material*[KRISHNASWAMY, A., AND BARONOSKI, G. V. G. 2004] NVIDIA Corporation 2007
Why is this?Thin oily layer 0.25 mmEpidermisBloody dermisLayers of skin NVIDIA Corporation 2007
Direct reflectionHow much light reflects off the oily layer?Thin oily layer 0.25 mmEpidermisBloody dermisLayers of skin NVIDIA Corporation 2007
Direct Reflection 6% of the light *Change of index of refraction between air andskinFresnel reflection & refraction takes place*[KRISHNASWAMY, A., AND BARONOSKI, G. V. G. 2004] NVIDIA Corporation 2007
SpecularThis is typically treated as a specular termHighly dependant on both view direction andincoming light directionNot colored by the surface in any way NVIDIA Corporation 2007
Specular - RoughnessSurface is not a mirrorSkin has very small scale roughnessUse a physically based BRDF*Oily layerEpidermisExtremely small scale*Bidirectional Reflectance Distribution Function NVIDIA Corporation 2007
Specular - RoughnessMost specular BRDFs haveRoughness parameter “m”Index of refraction (use 1.4)*Phong and Blinn-Phong aren’t ideal for skin*[Donner and Jensen 2005] NVIDIA Corporation 2007
Phong vs. physically-based BRDFPhongKS BRDF *PhongKS BRDF*[Kelemen and Szirmay-Kalos 2001] NVIDIA Corporation 2007
Phong vs. physically-based BRDFA physically basedBRDF capturesincreased specularity atgrazing anglesHere we use KelemenSzirmay-Kalos 2001 NVIDIA Corporation 2007
Roughness parametersHow do we set roughness parameter m?0.3 is a good average value for the face*Look to current research (Weyrich et al 2006)*[Donner and Jensen 2005] NVIDIA Corporation 2007
Roughness parametersAverage measured values from many realfaces – Weyrich et al. 2006 (SIGGRAPH)Specular amount NVIDIA Corporation 2007Roughness, m
Debevec et al 2000 trick*Linearly polarized lightLinearly polarizedcameraSome magicPictures of JUSTspecular!Check out the paper(SIGGRAPH 2000)http://www.debevec.org/Research/LS/*Image courtesy of Paul Debevec NVIDIA Corporation 2007
Overview DiagramRender texturespace lightStartblurblur blurLinearcombinationtexture mappingFinal pass: combine blurs specular NVIDIA Corporation 2007
What about the rest of the light?How do we handle the subsurface light?Thin oily layerEpidermis?Bloody dermisLayers of skin NVIDIA Corporation 2007 0.25 mm
Subsurface scatteringunderneath layers dominated by subsurface scatteringThin oily layer 0.25 mmEpidermisBloody dermisLayers of skin NVIDIA Corporation 2007
Subsurface ScatteringLight comes out somewhere elseLight gets colored depending on where it wentLayers of skin NVIDIA Corporation 2007
How do we deal with this?Seemingly impossible task:infinite number of possible pathsWhat about directional effects?Layers of skin NVIDIA Corporation 2007
Transport TheoryScattering models from physicsHow far between scattering events?How far between absorption events?Many scattering events make light diffuseHow long until that happens? NVIDIA Corporation 2007
Transport TheoryFor skin this distance is quite short:0.02083mm (red wavelength) *Thin oily layer 0.25 mmEpidermisBloody dermisLayers of skin*Computed using three-layer skin parameters from [Donner and Jensen 2005] NVIDIA Corporation 2007
Transport Theory1/10th through first layer – We’re diffuse!Track total light – ignore directionAny light reflection back is diffuse (equal in alldirections) NVIDIA Corporation 2007
Sweet, we know how to do thisDo an (N L) for each light and multiply by thediffuse colorThen add specularEasy, right? NVIDIA Corporation 2007
WrongThe skinlooks dryand hard NVIDIA Corporation 2007
Why didn’t this work?We forgot two thingsWe didn’t handle how light enters in one locationand exits in another (Important!)The oily rough surface has non-uniformtransmittance (subtle)Thin oily layerEpidermisBloody dermis NVIDIA Corporation 2007 0.25 mm
So what now?We need to employ more complicatedmethodsRely heavily on a diffusion approximation NVIDIA Corporation 2007
Previous techniquesOffline rendering:Monte-carlo rendering of multi-layered materialsReal-timeTexture space diffusion NVIDIA Corporation 2007
Multi-layer 2005*Image Courtesy of Craig Donner and Henrik Wann Jensen NVIDIA Corporation 2007
Multi-layer 20052005 SIGGRAPH paper byCraig Donner and HenrikWann Jensen3 layer skin modelScattering parametersmeasured in the medical/opticscommunityHigh quality head scan(XYZRGB) 5 minutes rendering time*Image Courtesy of Craig Donner and Henrik Wann Jensen NVIDIA Corporation 2007
Three-Layer vs. dipoleIn particular they showed the importance of a multilayered skin model3 layer skin model*Images Courtesy of Craig Donner and Henrik Wann Jensen NVIDIA Corporation 20071 layer skin model
What’s going on here?One-layer skin model looks waxyEpidermisNarrow scatteringLower layersWide scatteringMostly redAssuming one layer doesn’t work NVIDIA Corporation 2007
Approximating multi-layer diffusionWe can approximate thisStart with incident lightBlur over surfaceBlur some moreMix (linear combination) NVIDIA Corporation 2007
Why a blur?Consider a narrow focused white beam oflight hitting a flat, highly scattering surfaceIncoming focused beam NVIDIA Corporation 2007
Why a blur?“Highly scattering” means diffuseIncoming focused beamDiffuse output(ignoring roughSurface interface)scattering NVIDIA Corporation 2007
Diffusion profilesHow much light at distance r?Name: diffusion profile of the materialIncoming focused beamHow much diffuselight at distance r?r NVIDIA Corporation 2007
Diffusion profilesWe could compute thisWhat matters for skin?Distinctly different in R, G, BShape of these profiles means waxy skin vs. real NVIDIA Corporation 2007
Main scattering ideaFor every surface pointCollect incoming light (color value)Diffusion approximation: sum ignoring directionScatter into neighboring surface points NVIDIA Corporation 2007
Main scattering idea – 2D flat surface2D flat surfaceEach pointGather initial lightScatter into neighboring pixelsDepends on separation, rThis is a blur! NVIDIA Corporation 2007
Diffusion over curved surfacesWe actually require more complicated surfacesVertical laser line reflecting off an ear NVIDIA Corporation 2007
Texture space diffusionUnfold 3D surface – texture coordinatesUse this layout to perform blursMap back after using texture mappingWe call this texture space diffusion **Invented by George Borshukov and J.P. Lewis 2003 NVIDIA Corporation 2007
Texture space diffusionThis has been done in real-time before (SimonGreen: GPU Gems 1) *We improve upon this technique in severalwaysCorrect distortionKeep several blurs and linearly combineHow to use diffuse color*Invented by George Borshukov and J.P. Lewis 2003 NVIDIA Corporation 2007
Overview DiagramRender texturespace lightStartblurblur blurLinearcombinationtexture mappingFinal pass: combine blurs specular NVIDIA Corporation 2007
Rendering using texture coordinatesModify the projCoord in the vertex shader using thetexture coordinatesPass the true worldCoord and worldNormal to thefragment shader for lightingv2f.projCoord float4( texCoord.x * 2.0 - 1.0,texCoord.y * 2.0 - 1.0, 0.0, 1.0 ); NVIDIA Corporation 2007
Rendering using texture coordinatesThe fragment shader computes subsurface irradiance(explained shortly)Store in an offscreen texture NVIDIA Corporation 2007
Rendering using texture coordinatesNow it can be efficiently blurred NVIDIA Corporation 2007
Overview DiagramRender texturespace lightStartblurblur blurLinearcombinationtexture mappingFinal pass: combine blurs specular NVIDIA Corporation 2007
Each blur operation is 2 passesBlur weights.006.061.242.383.242.061Recurse several times building a setof different blursHorizontalblurVertical blurTemporary buffer NVIDIA Corporation 2007.006Texture used infinal pass
Distortion problemsAvoid uniform blur in texture spaceOne pixel in texture space ! constant distance inreal worldUniform blur NVIDIA Corporation 2007Distortioncorrected blur
Distortion problemsThis can make the ears look like candle waxVery hard dramatic shadows will have an ugly edgeUniform blurs NVIDIA Corporation 2007Distortion correction
Accurate Distortion CorrectionWe can easily estimate distortionCompute a map and inversely stretch our blursfloat3 derivu ddx( v2f.worldCoord );float3 derivv ddy( v2f.worldCoord );// 0.001 scales the values to map into [0,1]// this depends on the modelfloat stretchU 0.001 * 1.0 / length( derivu );float stretchV 0.001 * 1.0 / length( derivv ); NVIDIA Corporation 2007
Accurate Distortion CorrectionUse these values to scale the directional blursPaint the eyebrows black to eliminate blur thereBlur radius in X NVIDIA Corporation 2007Blur Radius in Y
Accurate Distortion CorrectionIn general these distortions are different in X and YSeparable blurs advantageous for 2 reasonsCan correct distortions in each direction separatelyMore efficient to compute than a 7 x 7 blurBuild a hierarchy of blurred versions of irradianceGood for mixing to get any diffusion profile you wantMore efficient than computing a 100x100 blur (evenseparably) NVIDIA Corporation 2007
Dealing with SeamsDepending on the UVlayout, there might beseamsBlack regions surroundseam edgesBlack regions will blurinto nearby pixels,creating dark seams NVIDIA Corporation 2007
Dealing with Seams NVIDIA Corporation 2007
Dealing with SeamsWays around it:Change Clear colorEdit distortion mapObject / no-object alpha mapMultiple UV sets (slow) NVIDIA Corporation 2007
Dealing with SeamsChangeClear color NVIDIA Corporation 2007Edge pixels:Can still filterportions ofempty space
Computing subsurface IrradianceRender texturespace lightStartblurblur blurLinearcombinationtexture mappingFinal pass: combine blurs specular NVIDIA Corporation 2007
Computing subsurface IrradianceWhat color do we write here?How should diffuse color get used?First guess:dot( N, L ) * diffuseColAfterBlur, blur, blurLinearly Combine blursAdd specular NVIDIA Corporation 2007
Computing subsurface IrradianceFirst guess: NVIDIA Corporation 2007dot( N, L ) * diffuseCol
Computing subsurface IrradianceThis can look okCan lose high frequenciesSecond guess:Store only N L termsBlur, blur, blurLinearly Combine blursMultiply DiffuseColAdd specular NVIDIA Corporation 2007
Computing subsurface IrradianceSecond guess:by diffuseCol NVIDIA Corporation 2007blur dot(N,L), mix, then multiply
Computing subsurface IrradianceThis can look okProbably ok for photo-based mapsNo Color bleeding – Too much high frequencycontent leftHigh frequency detail “sits” above underneathlayersThird guess: can we somehow do both? NVIDIA Corporation 2007
Computing subsurface IrradianceWant some coloring before and after blurNeed a multiplication each timeCan’t multiply DiffuseCol twice NVIDIA Corporation 2007
Computing subsurface IrradianceWant some coloring before and after blurNeed a multiplication each timeCan’t multiply DiffuseCol twiceMultiply sqrt( DiffuseCol ) twiceOnce before blurringOnce after blending NVIDIA Corporation 2007
Computing subsurface IrradianceHalf and half: apply sqrt(diffuseCol), blur, combine,then apply sqrt(diffuseCol) NVIDIA Corporation 2007
Computing subsurface IrradianceThis simulates an infinitely thin absorption layer onthe surface which doesn’t scatter lightThin oil layerDetail absorptionlayerEpidermisBloody DermisSkin layers NVIDIA Corporation 2007
Computing subsurface IrradianceThis is realistic in thatLight must interact with this detail layer twiceOnce going inOnce coming outAssumptionsVariation in skin lies in a top thin layer NVIDIA Corporation 2007
Computing subsurface IrradianceArtistic libertyGeneralizeInstead of* sqrt( diffuseCol ) before* sqrt( diffuseCol ) afterUse* pow( diffuseCol, mix ) before* pow( diffuseCol, 1.0 – mix ) afterIn Adrianne we used mix 0.82 NVIDIA Corporation 2007
Combining blursRender texturespace lightStartblurblur blurLinearcombinationtexture mappingFinal pass: combine blurs specular NVIDIA Corporation 2007
Combining blursFor human skin we recommendBlurWidth(mm)redBlur Weightsgreenblue0.042 00.7530.114001.4120.364002.7220.08000*Smallest blur level is lighting re-computed directly in the final render pass (it has not been blurred) NVIDIA Corporation 2007
Combining blurs – Choosing weightsWeights should sum to 1.0Somewhat artistic processDo it one color at a timereturn finalCol.xxxreturn finalCol.yyyEtcCompare to reference photos (red, then green, then blue)Avoid:Heavy blur specularSingle gaussian specular NVIDIA Corporation 2007
Combining blursFor human skin:Broadest scatter in redSmall scatter in greenBlue has very little at allSingle Gaussian is badEven for one-layered materials NVIDIA Corporation 2007
Combining blursfloat3 diffuseLight nonBlur * E1 * pow( diffuseCol, 0.5 );float3float3float3float3float3blur2tap f3tex2D( blur2Tex, v2f.c texCoord.xy );blur4tap f3tex2D( blur4Tex, v2f.c texCoord.xy );blur8tap f3tex2D( blur8Tex, v2f.c texCoord.xy );blur16tap f3tex2D( blur16Tex, v2f.c texCoord.xy );blur32tap f3tex2D( blur32Tex, v2f.c texCoord.xy diffuseLight blur2 * blur2tap.xyz;blur4 * blur4tap.xyz;blur8 * blur8tap.xyz;blur16 * blur16tap.xyz;blur32 * blur32tap.xyz;// renormalize weights so they sum to 1.0float3 norm2 nonBlur blur2 blur4 blur8 blur16 blur32;diffuseLight / norm2;diffuseLight * pow( diffuseCol, 0.5 ); NVIDIA Corporation 2007
Combining blursHighly scalable processUse as many as you needTry at least two blurs NVIDIA Corporation 2007
Combining blurs6 levels of blurphysically based specular NVIDIA Corporation 20073 levels of blurBlinn-Phong specular
Energy ConservationProblems with using only N L?We didn’t conserve energyIgnores directional effects of rough, oily surfaceFor skin the difference is quite subtledot(N,L) NVIDIA Corporation 2007Careful conservation of energy
ExtrasGamma correctionTry several specularsMany tileable bump maps NVIDIA Corporation 2007
Gamma correctionYour monitor is lying to you!displayed brightness pixelValue 2.2All our displays do thisDigital Cameras know thisThey correct for itWe should too NVIDIA Corporation 2007
Gamma correction comparisonNo Gamma correction NVIDIA Corporation 2007Gamma correction
Gamma correctionAffects image quality greatlyScrews up lighting and shadingWe need two fixes:1) Correct textures2) Correct framebuffer pixels NVIDIA Corporation 2007
Gamma correction – Correct texturesPictures & painted maps will have non-linear pixelsTo convert them to linear:diffuseCol pow( f3tex2d( diffTex, v2f.tex ), 2.2 );//Or (cheaper)diffuseCol f3tex2d( diffTex, v2f.tex );diffuseCol diffuseCol * diffuseCol; NVIDIA Corporation 2007
Gamma correction – Correct ImagesDo this once or automate it upon export/build(avoids the pow() instructions)Use sRGB texture formatDirectXDXGI FORMAT R8G8B8A8 UNORM SRGBDXGI FORMAT BC{1,2,3} UNORM SRGBOpenGLGL EXT texture sRGBAvoid correctingambient occlusionNormal mapsAlpha channels NVIDIA Corporation 2007
Gamma correct the final color valueOur displays warp pixel valuesWhen we write to the framebuffer:Invert the warpingfloat3 finalCol do all lighting and shading();float pixelAlpha compute pixel alpha();return float4(pow(finalCol, 1.0 / 2.2), pixelAlpha);// or (cheaper)return float4( sqrt( finalCol ), pixelAlpha ); NVIDIA Corporation 2007
Gamma correction: sRGBsRGB framebuffersYou write linear pixelsHardware does correction for youBlending is done linearlyEXT framebuffer sRGB NVIDIA Corporation 2007
Try more than one specular at onceOne specular roughness might not be enoughSkin is a complex materialTry mixing a few specular calculations togetherLinear combinationMany of the intermediate calculations can be reusedIn Adrianne we used four NVIDIA Corporation 2007
Try more than one specular at oncefloat4 specWeights float4( 0.1, 0.2, 0.3, 0.5 );float4 specRough float4( 0.05, 0.1, 0.2, 0.3 );float spec QuadSpec( N, V, L, specRough, specWeights );specLight spec * lightcolor * shadow; NVIDIA Corporation 2007
Try more than one specular at onceLinear combinationof 4 speculars NVIDIA Corporation 2007
Tileable detail bumpsBump detail on skin is importantCreating one large high resolution normal maprequires a lot of memoryTry creating several small tileable bump mapsDon’t average normalsAdd several bump height values and compute anormalCompute normal in shader using bump heights NVIDIA Corporation 2007
Don’t average normalsBump map 1 Bump map 2 Normals don’t add or average when two bump maps are combined NVIDIA Corporation 2007
Tileable detail bumpsFragment shader:Find tex coords for pixel neighbors (use ddx, ddy)Compute total bump height for 3 locationsCompute tangent space normalupOverCenter bump height NVIDIA Corporation 2007
Tileable detail bumpsEstimate tex coord for one pixel to the right, and one pixel upfloat2float2float2float2uv offset over ddx( v2f.c texCoord );uv offset up ddy( v2f.c texCoord );texCoordOver v2f.c texCoord uv offset over;texCoordUp v2f.c texCoord uv offset up;Use texCoordOver and texCoordUp to evaluate all bump mapsupOverCenter bump height NVIDIA Corporation 2007
Tileable detail bumpsfloat2 uv offset over ddx( v2f.c texCoord );float2 uv offset up ddy( v2f.c texCoord );float2 texCoordOver v2f.c texCoord uv offset over;float2 texCoordUp v2f.c texCoord uv offset up; // compute finalHeightCenter, finalHeightOver, finalHeight up float yscalefactor length(uv offset over) /length(uv offset up);float3 tanNormal normalize(float3(finalHeightCenter finalHeightOver, yscalefactor * (finalHeightCenter finalHeightUp), length(uv offset over) * 4048.0)); NVIDIA Corporation 2007
Tileable detail bumpsFour bump tiles used on Adrianne NVIDIA Corporation 2007
SummaryImproved texture space diffusionSeparable blurs, separable distortion correctionBuild one blur from the lastBlend several blurs for more realistic diffusionBlend blurs differently in red, green, and blueGamma Correct!!!Layered bump tilesDon’t layer normal mapsLayer bump heights and compute an accurate normalMany speculars NVIDIA Corporation 2007
Play around to get new effects*Any kind of fleshy surface requires subtlescattering NVIDIA Corporation 2007*Texture maps courtesy of XYZRGB
Future workSkin shading research hasn’t stoppedWatch for upcoming demos with even morerealistic skin shadingWatch for GPU Gems 3 NVIDIA Corporation 2007
ReferencesBORSHUKOV G., AND LEWIS, J. 2003. Realistic human face rendering for “the matrixreloaded”. In ACM SIGGRAPH 2003 Conference Abstracts and Applications (TechnicalSketch).DEBEVEC, P., HAWKINS, T., TCHOU, C., DUIKER, H.-P., SAROKIN, W., AND SAGAR,M. 2000. Acquiring the reflectance field of a human face. In Computer Graphics,SIGGRAPH 2000 Proceedings, 145–156.DONNER C., AND JENSEN, H.W. 2005. Light diffusion in multi-layered translucentmaterials. In Proceedings of SIGGRAPH 2005, 1032-1039.GREEN, S. 2004. Real-time approximations to subsurface scattering. In GPU Gems, R.Fernando, Ed. Addison Wesley, Mar., ch. 16, 263–278.JENSEN, H. W., AND BUHLER, J. 2002. A rapid hierarchical rendering technique fortranslucent materials. ACM Trans. Graph. 21, 3, 576–581.JENSEN, H. W., MARSCHNER, S. R., LEVOY, M., AND HANRAHAN, P. 2001. A practicalmodel for subsurface light transport. In Proceedings of SIGGRAPH 2001, 511–518.KRISHNASWAMY, A., AND BARONOSKI, G. V. G. 2004. A biophysically-based spectralmodel of light interaction with human skin. In Proceedings of EUROGRAPHICS 2004, vol.23.PHARR, M., AND HUMPHREYS, G. 2004. Physically Based Rendering: From Theory toImplementation. Morgan Kaufmann Publishers Inc.WEYRICH, T., MATUSIK, W., PFISTER, H., BICKEL, B., DONNER, C., TU, C.,MCANDLESS, J., LEE, J., NGAN, A., JENSEN, H. W., AND GROSS, M. 2006. Analysis ofhuman faces using a measurement-based skin reflectance model. ACM Transactions onGraphics 25, 3 (July), 1013–1024. NVIDIA Corporation 2007
New Developer Tools at GDC 02007PerfKit 5FX Composer 2SDK 10GPU-AcceleratedTexture Tools NVIDIA Corporation 2007ShaderPerf 2Shader Library
Questions? NVIDIA Corporation 2007
GDC 200
