WIXLIB

Texas Tech University, Duc Dang, May 2016AndAR uses Android’s Java library to access the camera and generate videoframes. These frames will be passed to the ARToolkit to detect markers, recognizetheir patterns, and calculate camera transformation matrices relative to the detectedmarkers position and orientation.2. AndAR operationThis section will describe the operation of the AndAR framework suggested byTobias Domhan [8]. AndAR uses Java API to access the tablet camera and get thevideo stream. To reduce usage of memory bandwidth, the camera is set to 240x160pixels as default value [8]. The acquired image will be send in 2 sub processes.Sub process 1: The image will be sent to the ARToolkit Library, which willcalculate the transformation matrix for the 3D object. In this step, the ARToolkitlibrary will do marker detection and transformation matrix calculation based on theinput image. These two steps will be explained in the next section. After thetransformation matrix is available, it will be passed back to the main process throughJava Native Interface (JNI).Sub process 2: The image will be converted to an appropriate color space. Theconversion step is necessary because OpenGL ES stores its colors in RGBA format, inwhich each color has Red, Green, Blue, and Alpha components [9], while Androiduses YCbCr format for its images, including video streams [10]. The conversion isprocessed by AndAR [8]:𝐵 1.164 (𝑌 16) 2.018 (𝐶𝑏 128)𝐺 1.164 (𝑌 16) 0.813 (𝐶𝑟 128) 0.391 (𝐶𝑏 128)𝑅 1.164 (𝑌 16) 1.596 (𝐶𝑟 128)After the image is converted to the RGB color space, the resulting image willbe returned to the main process. The converted image will be used as the OpenGLtexture (apply 2D image to 3D environment as texture of a surface). This step includesmaking a full screen preview surface on the device screen to display the video streamfrom the camera. Then, an OpenGL surface will be layered on top of the preview8

Texas Tech University, Duc Dang, May 2016surface. This OpenGL surface will function as a canvas to house a video stream as its2D texture.3. ARToolkit libraryARToolkit was created by Dr. Hirokazu Kato in 1999 [11], developed by theHuman Interface Technology Laboratory (HIT Lab) at the University of Washingtonand currently released as an open-source library by DAQRI, an augmentedreality company. Released under GNU Library General Public License, this library isone of the most popular Augmented Reality libraries, used in thousands of commercialand open source projects [12].Although the ARToolkit structure includes 3 main modules (AR module,Video module and Gsub module), in this thesis, only AR module and Gsub module’stransformation matrix function will be used directly by AndAR [8]. AR module willprovide marker tracking capabilities when the transformation matrix function is usedto calculate relative position and orientation between detected marker and camera. Thestructure of the AndAR framework is shown in Figure 8.Figure 8 - ARToolKit architecture [29]9

Texas Tech University, Duc Dang, May 2016a. Marker for ARThe marker image is essential input for every application based on AndAR andthe ARToolkit library. A marker for ARToolkit can be made by developers with thefollowing constraints [13]: Must be a square marker. Must have a continuous (full black or pure white) border with pattern imageand background with contrasting colors (white background for black pattern or whitepattern image with dark background). The pattern image inside the marker border must be rotationally asymmetric.Figure 9 - Example of custom-made marker for ARToolkit [11]In Figure 9 is an example of custom-made marker for ARToolkit library. Theapplication in this thesis uses markers with alphabet characters as patterns. Thesepatterns were designed in Microsoft Paint as seen in Figure 10.10

Texas Tech University, Duc Dang, May 2016Figure 10 Marker images with alphabet characters as patternsTo detect, recognize, identify and track the marker images from a capturedvideo stream [13], ARToolkit or an application based on ARToolkit must be trainedwith pattern recognition data files (“.patt” files). These “.patt” file are generated frommarker images using ARToolkit Marker Generator Online, an online tool based onAdobe Flash [14].11

Texas Tech University, Duc Dang, May 2016Figure 11 - Marker "A" and its pattern recognition data file12

Texas Tech University, Duc Dang, May 2016b. Marker detection in ARToolkitThe marker detection process inside the ARToolkit can be broken into thefollowing steps [15] (Figure 12): First, the captured live video frame is used to extract a binary image byapplying a threshold to its grayscale image. In the AndAR framework, the binarizationthreshold was set to the value of 100 in the gray scale range of 0-255. Next, ARToolkit will find contours in the image and extract the contours thatfit inside four segments (square). For each of the extracted squares, the pattern inside it will be compared withthe pattern file pre-registered with ARToolkit to find the AR tracking marker. The coordinates of the four vertices of the square and parameters of four linesegments will be saved for estimation of the transformation matrix.Figure 12 – Marker detection steps. [15]13

Texas Tech University, Duc Dang, May 2016c. Transformation matrix estimation in ARToolkitFigure 13 shows the marker coordinate and the camera coordinate systems inARToolkit. To overlay a 3D object on top of a marker, we need to know the relativeposition and rotation between camera and marker and the relationship between MarkerCoordinate and Camera Coordinate systems (Figure 13).Figure 13 - Marker coordinate and camera coordinate systems in ARToolkit [16]Therefore, the transformation matrices that represent the relationship amongCamera Screen Coordinates (xc,yc), Marker Coordinates (XM, YM. ZM) and the CameraCoordinates (XC, YC. ZC) should be estimated (Figure 14). ARToolkit suggestsestimating the intrinsic camera parameter matrix (P) and the image distortion functionbetween Camera Screen Coordinates and Observed Screen Coordinates by doingcamera calibration. However, because the ARToolkit version we are using in thisthesis is integrated into the AndAR framework, which is designed for use withAndroid devices, this calibration step is replaced by loading default camera propertiesstored in the camera parameter file, “camera para.dat” during applicationinitialization. The rest of this section will present the process of transformation matrixestimation in ARToolkit suggested by Kato et al. [11].14

Texas Tech University, Duc Dang, May 2016Figure 14 - Coordinate systems used in ARToolKit [17]The connection between Camera Screen Coordinates and Camera Coordinatescan be expressed by:𝑋𝑐𝑃11 𝑃12ℎ𝑥𝑐𝑌0 𝑃22[ℎ𝑦𝑐 ] 𝑃 [ 𝑐 ] , where 𝑃 [𝑍𝑐0 0ℎ10 01𝑃13 0𝑃23 0]1001The transformation matrices from these marker coordinates to the cameracoordinates (TCM) can be represented by:𝑋𝑐𝑉11𝑉21𝑌𝑐[ ] [𝑍𝑐𝑉3110𝑉12𝑉22𝑉320𝑉13 𝑊𝑥𝑋𝑚𝑋𝑚𝑋𝑚𝑉23 𝑊𝑦𝑉3𝑥3 𝑊3𝑥1 𝑌𝑚𝑌𝑚𝑌][ ] [] [ ] 𝑇𝑐𝑚 [ 𝑚 ]𝑍𝑚𝑍𝑚𝑍𝑚0001𝑉33 𝑊𝑧11101(*)In addition, we have parameters of one pair of parallel sides of a square markerfrom the previous marker detection step:15

Texas Tech University, Duc Dang, May 2016𝑎1 𝑥𝑐 𝑏1 𝑦𝑐 𝑐1 0𝑎2 𝑥𝑐 𝑏2 𝑦𝑐 𝑐2 0Thus, equations of the planes that include these two sides can be representedby:𝑎1 𝑃11 𝑋𝑐 (𝑎1 𝑃12 𝑏1 𝑃22 )𝑌𝑐 (𝑎1 𝑃13 𝑏1 𝑃23 𝑐1 )𝑍𝑐 0𝑎2 𝑃11 𝑋𝑐 (𝑎2 𝑃12 𝑏2 𝑃22 )𝑌𝑐 (𝑎2 𝑃13 𝑏2 𝑃23 𝑐2 )𝑍𝑐 0From this equation, we can compute normal vectors (n1 and n2) of theseplanes:𝑛1 (𝑎1 𝑃11 , 𝑎1 𝑃12 𝑏1 𝑃22 , 𝑎1 𝑃13 𝑏1 𝑃23 𝑐1 )𝑛2 (𝑎2 𝑃11 , 𝑎2 𝑃12 𝑏2 𝑃22 , 𝑎2 𝑃13 𝑏2 𝑃23 𝑐2 )Then, direction vector u1 of two parallel sides is calculated with the productu1 n1 x n2. The directional vector u2 of the two other parallel sides can be calculatedwith the same method.Figure 15 - u1, u2 and v1, v2 vectors pairs [11]However, image processing inaccuracies mean that the vectors won't beexactly perpendicular. Therefore, a pair of perpendicular unit direction vectors v1 andv2 in the plane that includes u1 and u2 will be used instead (Figure 15). From v1 and v2we can calculate the perpendicular direction vector to v1 and v2:v3 v1 * v2From v1, v2, and v3, we now can obtain the rotation component V3x3 in thetransformation matrix TCM:16

Texas Tech University, Duc Dang, May 2016V3x3 [v1t v2t v3t]To calculate translation component w1, w2, and w3 in TCM, we simply solve thesystem of equations (*) by substituting coordinates of four vertices of the detectedmarker in camera screen coordinate system and marker coordinate system.4. AndAR summaryAs presented in this chapter, AndAR was chosen to be the foundation of the3D Augmented Reality application. Its Java API and ARToolkit libraries provide theframework, marker detection and transformation matrices calculation ability neededfor the development process. The design of markers used in this thesis was introducedin this chapter.17

Texas Tech University, Duc Dang, May 2016CHAPTER 33D HUMAN ANATOMY MODELS ANDCOMPUTERIZED AXIAL TOMOGRAPHY SCAN3D Human Anatomy models and Computerized Axial Tomography (CAT)scans are key components in this thesis. These are two elements that will besuperimposed over the human body, displayed on the device screen in real time. Inthis chapter, a brief summary of 3D file format that is used for the application will beintroduced. The CAT scan dataset and its optimization to work with the developedapplication will also be presented.1.3D modelsa.BodyParts3DIn the field of human anatomy, many methods can be used to representanatomical knowledge, and one of the most efficient and convenient is to specifyhuman body parts by 3D models. Thus, we can obtain both computable andcomprehensive representation of a physical human body [18]. Several efforts havebeen made by research groups around the world to publish a 3D human anatomy atlas,such as National Library of Medicine’s Visible Human Project and the BodyParts3Dproject. Among these atlases, BodyParts3D, a 3D anatomy atlas funded by TheIntegrated Database Project, Ministry of Education, Culture, Sports, Science andTechnology of Japan [18], provides a suitable database of 3D human body model forthis thesis. The BodyPart3D is a high resolution “dictionary-type database for anatomyin which anatomical concepts are represented by 3D structure data that specifycorresponding segments of a 3D whole-body model for an adult human male” [18].This database is made freely available under a Creative Commons Share-Alikelicense, which means that the models can be used and altered without charge under therequirement that the alternate version is also distributed under the same license. The18

Texas Tech University, Duc Dang, May 20163D models used in this thesis can be downloaded through the BodyParts3D project ftpsite: ftp://ftp.biosciencedbc.jp/archive/bodyparts3d/ [31] in Wavefront’s .OBJ format.Figure 16 - 3D model of left humerus bone in BodyPart3DFigure 17 - 3D model of left ulna bone in BodyPart3Db.Wavefront OBJ formatThe .OBJ is a geometry definition file format first developed by WavefrontTechnologies (now Alias Systems, part of Autodesk Corporation) for its AdvancedVisualizer 3D graphics software package in the 1980s [19]. Currently, the .OBJ fileformat is widely accepted by other vendors to represent 3D graphic.The application in this thesis supports parts of the .OBJ file format and itssupport material template library (.MTL) file format, which is used in 3D models ofthe BodyPart3D database and in the limited scope of this thesis. Those parts of .OBJand .MTL file formats that are used in the thesis will be described below. The fulldescription of the .OBJ file format can be found in Wavefront’s Advanced VisualizerManual [20].19

Texas Tech University, Duc Dang, May 2016i.OBJ file formatThe .OBJ objects file is written in ASCII to define polygonal objects and theirpoints, lines and faces.The objects used in this thesis have the following file structures:GroupingRenderattributesVertex dataVertex dataElementsElementsFigure 18 - OBJ file structure ofBodyPart3D models Figure 19 - Obj file structure of objectfile used to display CAT imageVertex data will provide coordinates fo

Eclipse ADT bundle. 57 b. SketchUp 2015 . 57 . Texas Tech University, Duc Dang, May 2016 v . virtual reality (VR), where the user's view is totally replaced with a computer-generated environment, Augmented Reality focuses on enhancing a user's reality . displayed on mobile device screen in real time. Texas Tech University, Duc .