WIXLIB

Segmentation, Classification&Tracking of Humans for SmartAirbag ApplicationsDr. Michael E. FarmerDept. of Computer Science, EngineeringScience, and PhysicsUniversity of Michigan-FlintImportance of Topic Between 1986 and 2001 airbags caused thefollowing deaths, mostly during low speedcrashes:– 19 infants in RFIS– 85 children– 7 adults due to proximity at time of deployment Unfortunately, as recently as 1998:– over 50% of a NHTSA survey respondents had placedtheir children in the front seat of their vehicle over thelast 30 days Deaths may continue until a system toautomatically disable the airbag is developed.1

Automotive Occupant Classification&Tracking Problem National Highway Transportation SafetyAdministration (NHTSA) mandate (2004) Requires two suppression conditions: Suppress the airbag if child or infant Detect change within 10 seconds Suppress the airbag if occupant is withinAutomatic Suppression Zone (ASZ) Disable bag within 20 msecsDefinition of Automatic SuppressionZone (ASZ) Oriented along Instrument Panel– Distance from Airbag where risk of injury in minimizedASZASZ2

NHTSA Approved Methods forPerforming Airbag Suppression1-year old infantProtected byClassificationSuppress ifPresent3 to 6-year oldchildAdultSuppress ifPresentORProtected byTrackingSuppress ifin ASZSuppress ifin ASZ If decide to protect children by classification then get4-class problem (infant, child, adult, empty) If decide to protect children by tracking then get2-class problem (infant versus all else)4-class Classification ProblemInfantAdultChildEmpty3

2-class Classification ProblemVs.AdultsInfantsDifficulties with Occupant ClassificationLarge Intra-class VariabilityInfant ClassAdult Class4

Difficulties with Occupant Classification IILow Inter-class Variability for 4-Class Classification Problem6 Year-old on Booster5th % Adult FemaleNote that the 6 YO is actually taller in these imageswhen seat is in forward position for child and rear-mostposition for the adult.Difficulties with Occupant Classification- System experiences extreme variations in illumination5

Difficulties with Occupant TrackingExtensive occupant deformation during movementOccupant occlusionVariability in size of occupantSummary of Difficulties with OccupantAirbag Suppression Problem Large intra-class variability of the variousoccupant types Low inter-class variability for 4-classclassification problem Camouflaged classes (e.g., blanketed RFIS) Large variation in illumination Severe automotive environmental conditions Low cost Extremely high reliability and performance6

Summary of Image Data UsedTraining images:Occupant TypeClassificationNumber of ImagesRFIS FFISInfant2657ChildChild620AdultAdult983Empty SeatTotal number ofimages:Empty724332Test images:Occupant TypeClassificationNumber of ImagesInfant (RFIS FFIS)Infant1807ChildChild236AdultAdult210Empty SeatTotal number ofimages:Empty82261System Algorithm Architecture Standard components of pattern recognition system:1. data acquisition and pre-processing2. data representation3. decision makingEvery mageEvery ssifierHead/TorsoTracker&Predictor7

Approach for Classifier menterOccupantClassifier Pal and Pal state: “hundreds of segmentation techniques in theliterature, but there is no single method which can be considered goodfor all images” Pal and Pal also state:“semantics and prior information about the typeof images are critical to the solution of the segmentation problem” In light of these, we will utilize all the information we have regardingthe interior of the vehicle The approach we will take is one of background removal.Segmentation Processing For Background Removal tested 2 methods– De-correlation Processing– Eigen-images– Results: De-correlation processing outperforms Eigenimages Also developed 2 post-processing methods– Hole filling using binary morphology() Use closing: A Bˆ ΘB– Further background reduction with Watershed8

Eigen-image ProcessingL ΦCΦTp new ΦTMb (I new µnew )Used sequence of722 images tocompute covariancematrixΦ MbI background Φ Mbp new µ new I new I background thresholdDe-correlation Processing[g (i, j ) grad x , grad y]T g ( x, y ) g ( x, y )C A B2 g ( x, y ) g ( x, y )A2B9

Eigen-image Processing ResultsInput ImageTransformedImageDifferenceImageHistogram ofDifference ImageThresholdedDifference ImageSegmentation Processing ResultsInput ImageDe-correlationImageThresholdedDe-correl. ImageReference ImagePostprocessingInput ImageHole FillingWatershed Image10

Approach for Feature tionOccupantClassifier Devijer and Kittler define Feature Extraction as:– “extracting from the raw data the information which is mostrelevant for classification”.– “feature extraction is probably the single most important factor inachieving high recognition performance”Feature Extraction TaxonomyContent ain11

Feature Extraction- Child on seat has same boundary as empty seat so cannotuse boundary methodsRecall wesegment theoccupant &the seat- Child and infant have common color/grayscale & texturedistribution so cannot use color/grayscale or texture- We will use region methods to characterize the occupantLegendre Moments of Edge Image forFeature ExtractionSegmentedImagesEdgeImages12

Legendre Moment ReconstructionInfant ImageAdult Image25 Order(351 features)35 Order(666 features)45 Order(1081 features)Will use thehighest orderto capture themost internalfeaturesResults for Feature Extraction Tested the following different momentfeatures:– Geometric, Legendre, and Zernike Recall we do not want moment invariants, since size ofoccupant depends on spatial locationTested moments basedon separation offeatures using MannWhitney distributiontest13

Feature Selection TaxonomyFeature SelectionMethodsCompleteRandomType IExhaustiveAllHeuristicType II(SA, tanceBased(RELIEF)Will test our own Mann-Whitney aswell as Mutual Information for Best FirstMann-Whitney Feature Selection1. Mann-Whitney sorts features based on separability–First compute the mean rank sumsµ n ( N 1)Aσ–AB A, and2z (SAn n ( N 1)A B12 µσA) 0.5AB.n and nare the number of samples from classes A andABSB and A is the actual sum of the ranks2. Correlation post processing tests and removescorrelated features.Cov( A, B) )–Use Spearman-R coefficient ρ ( A, B ) σ 2 ( A) σ 2 (B )Cov(A,B) is covariance of the ranks14

Discrimination Ability of FeaturesCorrelated Features Removal15

Results for Feature SelectionClassification Accuracy# of Features g all FeaturesRandom Mutation HillClimbing ( α 0.2 )Random Mutation HillClimbing ( α 0.8 ard Sequential SearchMann-Whitney w/ Correl.(all 1081)Mann-Whitney w/ Correl.(retain top 200)Mann-Whitney w/ Correl.(retain top 120)Mutual Informationw/ Correl. (all 1081)Mutual Informationw/ Correl. (retain top 200)Mutual Informationw/ Correl. (retain top 63199.5-81-92.7-16-82.8-6-Feature Selection Method* FSS results after 20 days execution, rather than after convergenceClassifier tested on extended drive sequence –FSS- l-R selected features (41% correct)16

Classifier tested on extended drive sequence –Mann-Whitney selected features (64% correct)Results for Feature Selection- Another consideration is processing time formassive scale feature selection problems such as ourswhere we started with 1081 features.MethodMann-Whitney w/ CorrelationTime 10 minutesRandom Mutation HillClimbingForward Sequential Search3.25 hours 1 day per output feature (2class) 2 day per output feature (4class)17

Approach for Occupant ionClassifierSegmenterPrior InformationCOMPLETEINCOMPLETEBayes -NN, Tree, pproachMixtureResolvingClusterAnalysis(Hard, Fuzzy)Note: we will also attempt to exploit history in the classificationsince we know the occupant cannot change type while driving.Occupant Classifier Tested the following classifiers:––––Bayesk-NNModified k-NNSVM Performed 2 types of testing– 50/50 Cross validation– Independent test data set18

Developed modified k-NN Algorithm forClassificationAverage Distance k-NNTraditional k-NNResults for Occupant ClassifierTwo-classClassifier yes99.7 %98.2 %98.4 %88.4 %k-nearest neighbor(k 9)Modified k-nearestneighbor (k 9)Support VectorMachine99.0 %95.1 %97.7 %88.2 %99.5 %96.7 %98.5 %88.2 %98.9 %96.8 %93.6 %87.8 % Performed 2 types of testing:– 50/50 Cross validation– Independent test data set19

Contextual Processing for Classifier Errors in classification output have two sources:– 1) Random errors: Due to the inherent overlap in the class decision boundaries– 2) Deterministic errors: Due to the movements and behaviors of the occupant Contextual processing provides a historicalintegration of data to reduce both types of errorAppears asadultAppears asinfantContextual Processing Details Uses Dempster-Shafer Evidential Reasoning– Explicitly allows the use of ignorance in caseswhere the occupant type suddenly changes.– Based on set theory probability mass is assigned at a subset level:2Θ {{infant}, {child}, {adult}, {empty},{infant, child}, {infant, adult}, {infant, empty},{child, adult}, {child, empty}, {adult, empty},{infant, child, adult}, {child, adult, empty}, {infant, adult, empty},{infant, child, empty}, {child, adult, empty},{infant, child, adult, empty}}20

Typical Image Sequence for ContextualProcessingResults for Contextual ProcessingIncoming ConfidencesP(class)P(class)After Dempster-ShaferImage Frame NumberClassifications After Dempster-ShaferClassPatternClassIncoming ClassificationsImage Frame NumberImage Frame NumberImage Frame Number21

Confidences after Dempster-Shafer HistoryProcessingEffects of history on the extended videosequence – (80.0% correct)22

Dynamics of ATD During Pre-crashBrakingApproach for Tracker ead/TorsoTracker&Predictor Segmentation for image sequences has two keydifferences:– 1) object motion can be used as an additional cue forsegmentation– 2) image sequence segmentation must be accomplished at aconsiderably faster (full real-time) to support tracking. Two of the more common approaches:– 1) template tracking (solid 2-D ,line, and point-set templates)– 2) optical flow– We will define a 3rd method based on mutual information inan image stream23

Results for Tracker SegmentationTemplate MatchingOptical FlowResults for Tracker Segmentation IITemplate MatchingOptical FlowNote while the optical vectors are erroneous, we are still ableto use the output for segmentation, while the template matching fails.24

Occupant Shape PredictorOccupantModelY-axisBounding Ellipse ofHead and pant Motion PredictorOccupantModelHuman esBlobs25

Occupant Motion Modeling Model human dynamics by threedistinct types of motion Commonly use HMMs to model thetransitions between states– We use an alternative method calledInteracting Multiple Models (IMMs) whichallows mixing of the separate states.HumanStationaryCrashApproach for Occupant TorsoTracker&Predictor Recall the two representations required for tracking anobject (shape and motion) Implies there should also be two trackers running:– Motion tracker Will use Interacting Multiple Model (IMM) Kalman filter– Shape tracker We have defined a new method called Shape from Deformation26

Performance of Tracker for Pre-crash BrakingIntrusion Time Results for RoboticTesterHTT Response sponse Delay (msec)MaximumMinimumAverageStandard Deviation5.3-21-7.247.1027

Performance of Tracker for Human OccupantShape from Deformation Concept All SFM algorithms rely on point correspondencesbetween image frames.– 3-D structure is computed from the relative motions ofthese points Must define image points that are suitable for tracking. Recent research on using image deformations– Infer the direction of motion of an object Still relies on point tracking body.– Point tracking is not suitable for the human body It is not rigid and hard to define good points It is also not randomly deformable so may be able to usedeformations to infer 3-D shape or orientation28