WIXLIB

A HUMAN ARM MOTION TRACKINGSYSTEM DESIGN USING AN IMUAND A CAMERA SENSORSStudents:Nigina BaymuhamedovaDias AlimbayProject supervisor: Almas Shintemirov

PROJECTOBJECTIVESPrecise orientationof the hand in 3DDESIGN ATRACKINGSYSTEM1Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVAExact path of the bodyDevelopment of a multisensor fusion frameworkCreate a userfriendly interface

KEY PROJECT COMPONENTSVSLAMBVSEKFRTVISVisual SimultaneousLocalization and MappingCamera as Body-VelocitySensor augmented by IMUExtended Kalman Filter SensorData FusionReal Time Visualization2Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVA

ROSROBOTIC OPERATION SYSTEMIs a collection of software frameworks for robot software development, providingoperating system-like functionality on a heterogeneous computer clusterPACKAGEPACKAGE MIGHT CONTAIN ROS NODES,ROS-INDEPENDENT LIBRARY, A DATASET,CONFIGURATION FILES OR ANYTHING THATCONSTITUTES LOGICALLY USEFUL MODULENODE IS AN EXECUTABLE FILE WITHIN A ROSPACKAGE. CAN PUBLISH AND SUBSCRIBE TOA TOPICROSTOPICNODESource goes here3Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVATOPIC IS A NAMED BUS OVER WHICH NODESEXCHANGE MESSAGE

CAMERA AS POSE SENSOR VSLAMTRACKING:DATAASSOCIATION store map (calibration); predict new pose; map features projected to predictedpose; search for corresponding features.PTAM OF Klein and Murray, 2007, P.254Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVAMAPPING:BUNDLEADJUSTMENT refine orientation and position: total errois min; select key-frames to build a 3Dmap; batch optimization to refine mappoints/key-frame poses;

CAMERA AS A MOTION SENSOR BVS Feature management: ensures enough features equally spread in the image Feature extraction and matching establishes correspondence in consecutive frames; Visual velocity calculations: 2D Continuous “8-point” Algorithm.Weiss, 2012, P.475Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVA

SENSOR DATA FUSION EKFKalman filter(KF) effectiverecursive filter,which evaluatesthe state ofdynamic systemReal case(non-linear dynamics and unpredictable noise)EKF - isthe nonlinear version ofthe Kalman filter whichlinearizes about anestimate of the currentmean and covarianceIn the extended Kalman filter, the state transition andobservation models don't need to be linear functions ofthe state but may instead be differentiable functions.Weiss, 2014, p616Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVA

HARDWARE//COMPONENTSPOLOLU UM-LT7POLOLU WIXELorientation sensorgeneral-purpose programmablemodule7Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVALOGITECH C9203D CASEdesigned in solidworks

OVERALL ALGORITHM ialization8Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVAPathVisualization

ROS NODES LAYOUTImage proc removescamera distortion from theraw image streamptam-remote provides withptam modified version ofSLAM framework,simplified and robustvisual preview during worldframe initializationgscam makes ROScompatible our cameraum7-driver streams data fromthe sensor in the form offiltered orientation,accelerations and angularrotation9Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVAptam-visualizer manage theinformation to be displayed inRViz and to store the data in textfiles

FeaturesPositionVSLAMVSLAM,EKF10Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVAOrientationBVS

SYSTEM DEMONSTRATIONhttps://youtu.be/hH3-NnsnYU011Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVA

THIS SYSTEM CAN BE USEFUL IN DEVELOPMENT OF :MEDICINESurgery, RehabilitationENTERTAINMENTVideo games12Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVAADVANCE GPSWhich will remember the path.NAVIGATIONSYSTEMRobots

ONCE YOU STOP LEARNING YOU START DYINGMULTISENSORALIGNMENTSOLIDWORKS, C,C , BASHTROUBLESHOOTINGROSLearningoutcomes13Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVABeagle bone,ARDUINO

14Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVATHANK YOU

REFERENCE:1.Weiss, S., Scaramuzza, D. and Siegwart, R. (2011). Monocular-SLAMbased navigation for autonomous micro helicopters in GPS-deniedenvironments. Journal of Field Robotics, 28(6), pp.854-874.2.Weiss, S. (2012). Vision based navigation for micro helicopters. [Zürich].3.Klein, G. and Murray, D. (2010). Simulating Low-Cost Cameras forAugmented Reality Compositing. IEEE Trans. Visual. Comput. Graphics,16(3), pp.369-380.4.Fernandez, E. (n.d.). Learning ROS for robotics programming.15Michael Smith www.slideheroes.comDIAS ALIMBAY NIGINABAIMUKHAMEDOVA

dynamic system Weiss, 2014, p61 SENSOR DATA FUSION EKF DIAS ALIMBAY NIGINA BAIMUKHAMEDOVA Real case (non-linear dynamics and unpredictable noise) EKF - is the nonlinear version of the Kalman filter which linearizes about an estimate of the current mean and covariance In the extended Kalman filter, the state transition and