Transcription
Prasanna KrishnaswamyIntelPlatform ArchitectImaging Systems Design forMixed Reality Scenarios
Agenda Mixed Reality (MR) – IntroductionKey Use CasesMR System Design OverviewImaging Sensors for MRUse Case DecompositionOpportunitiesIntel 2017 MIPI Alliance, Inc.2
Mixed Reality - IntroductionExperiences that overlay graphics on video streamsof physical world are Augmented Reality Experiences that occlude your view to present adigital experience are Virtual Reality Experiences enabled between these two extremesare Mixed Reality, which is blending of the physicalworld and digital world Intel 2017 MIPI Alliance, Inc.Diagrams from Microsoft MixedReality s/mixed-reality/academy3
Key Use CasesExperience VectorsOptical See Through vs Video See Through HMDAll Day vs Limited TimeWired vs WirelessIntel 2017 MIPI Alliance, Inc.4
System Design OverviewTrackingCamerasDepthCamerasIMUVideo SeeThrough CameraInside Out6DOFI2C6DOF andDense PI/USBGPU/CVACPU/CVAImage PreProcessing nFace/Person/Object Detectionand TrackingDecode RGBDewarp DepthStitchingGPU/CVAGPU /ISPCVAMIPI/USB2MP-13MPMR CameraSystem DesignMonocular vsBinocular SLAMCompressionGPU/ISP/CVAAccelerationHolo APIUnity PluginsGPU/CVAMR Workload Distribuitonand RepresentationGPUGPU/CVAAccelerationIntel 2017 MIPI Alliance, ialUnderstanding3D ReconstructionReal timereconstructionOS Services orSDK PluginsWorkloadPartitioningMetadataContainers5
Image Sensors for MRType of SensorApplicationTypical Specification for MIPI CSI-2 Imaging SensorsSystem DesignConsiderationsTracking SensorInside out 6DOFWFOV, Fish Eye, Global Shutter,Monochrome, 120fps, 720pWider FOV than display totrack in peripheryDepth Sensor3D Reconstruction,Semantic, GesturesActive vs Passive, Upto1080p/30fps,0.3m-5m, few mm errorIndependent of trackingsensorRGB SensorTexture, See ThroughModeUpto 13MP/30fps for high textureMatch Display Resolution & fps for SeeThrough Mode (trending 4k/120fps)Configurable ISP processingfor simultaneous highquality texture and videosee throughEvent SensorInside out Tracking,Object TrackingQVGA @ 2k fps, FOV matching depthcameraAugmented with RGB andDepth cameraHyper Spectral orThermal SensorsFast ClassificationRGB-NIR/LWIR Sensor, QVGA @ 30fpsLow power ROI or eventgenerationIntel 2017 MIPI Alliance, Inc.6
MIPI CSI-2SM Advantages in MR Allows flexible ISP, Vision Accelerator and Host SOC combinationsin MR enabling segmentation and mobilitySOCCSI-2CSI-2SOCwith ISPMIPI RxCSI-2MIPI RxISP &VPUMIPI TxCSI-2MIPI RxCSI-2CSI-2CSI-2CSI-2CSI-2MIPI RxCSI-2MIPI ToHostConverterUSB3orTBTHMDAll in One MR HMD with vision processingdistributed between a host and a visionprocessing unitAll in One MR HMD withall vision processing onhostIntel 2017 MIPI Alliance, Inc.SOCHostMR HMD tethered to a host PCVPU - Vision Processing Unit, typically consists of CV andNeural Network accelerators7
MIPI CSI-2 Advantages in MR Total power consumption on HMD, especially for optical seethrough or AIO HMD with MIPI CSI-2300mW - 1.5WRGB Camera300mw - 1WTracking Camera300 - 500 mwCSI-2CSI-2ISP &VPUMIPI RxDepth CameraCSI-2MIPI TxTypical PowerMIPI RxCSI-2 Camera TypeSOCTotal power dissipated on a AIO HMD can be anywhere from 10-25W. Camera power starts to play a significantportion of the sameIntel 2017 MIPI Alliance, Inc.8
MIPI CSI-2 Advantages in MR Easier synchronization between sensorsIMUI2C/MIPI I3CHW SyncMasterSPI/MIPI I3CCSI-2CSI-2CSI-2SOCSOCSlaveSlaveSW Synchronization between cameras and I3CSensor HubI2C/I3C HW SyncHW SynchronizationIntel 2017 MIPI Alliance, Inc.Time Stamping between MIPI framessynchronized with host or SOC HW time andsensor hub9
MIPI CSI-2 Future Opportunities for MR Custom camera modules for MR– Different combinations that can be integrated into MR HMD’s Metadata Containers– Standardized Metadata for MR vision analytics from on board HMD for consumption by host MIPI to USB/TBT/Wifi Reference Designs for MR Headsets– Enables users to buy off market lower cost HMD’s to be used with PC’s increasing adoption Low power event sensing for MR– Standardized event triggers from event sensors to feed into other sensors for better powermanagement Adoption of Beyond Visible Cameras– Adoption of MIPI based Thermal, Hyperspectral sensors for MR use casesIntel 2017 MIPI Alliance, Inc.10
NOTICES AND DISCLAIMERS Intel, the Intel logo, Intel are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.*Other names and brands may be claimed as the property of others. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or serviceactivation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with yoursystem manufacturer or retailer or learn more at intel.com.Intel technologies may require enabled hardware, specific software, or services activation. Check with your system manufacturer orretailer.No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel 2017 MIPI Alliance, Inc.11
System Design Overview 5 Intel 2MP-13MP FE Global Shutter FE Global Shutter Active/Pa ssive Stereo Data Coalescing and/or Compression 6DOF and Dense SLAM Decode RGB Dewarp Depth Stitching GPU / ISP CVA GPU/ CVA Gesture Detection Face/Person/Obj ect Detection and Tracking Activity Semantic Segmentation Material Understanding CPU/ CVA GPU/ CVA .
