WIXLIB

Table 1. Comparison between FashionAI and the existing datasets for fashion attributes recognition#images#categories#dimensions#attribute values#key pointshierarchicalWBID [18]DDAD [4]DARND [16]DeepFashion [26]FashionGen [28]FashionAI put knowledge[11]. In terms of style analysis[27, 21, 31,2, 19], the published datasets were focusing on different style analysis, e.g. five styles including hipster, bohemian,goth, preppy, and pinup in[19]. Additionally, for apparelgeneration[12, 39], the proposed datasets were designed forgenerating new fashion items. Unlike these works, we construct FashionAI to fine-grained recognize fashion items.Fine-grained Recognition Dataset. In the context of fashion recognition, there were many useful datasets that are already published for academic use. Mainstream datasets forfashion attributes recognition are summarized in Table 1.To our knowledge, the source of building current fashiondatasets[25, 9, 24, 20, 17, 37, 3, 28, 26] was all collectedfrom the websites, and the original attributes and attributesystem were used as knowledge structure directly (except asmall-scale dataset named CCP[38] which consists of 2,098fashion images).Deepfashion obtained by 800,000 images with 1,050attributes has been one of the most popular dataset forfashion related researches[26]. However, since it was designed for fashion retrieval, the attributes defined in Deepfashion were marketing-orient which was not systematical and comprehensive enough for fashion understandingtask. Recently, Rostamzadeh et al. introduced Fashion-Genwith 293,008 fashion images parried with totally 169 fashion categories[28] for text-to-image and attributes-to-imagesynthesis task. The attributes were described in text format.In contract with these datasets, FashionAI is built from theperspective of design for fashion understanding task. Thedesign regions (defined as attribute dimensions, e.g. sleevelength, sleeve cuff, or collar design etc.) and their belonging designs (defined as attribute values, e.g. cap sleeves) aresummarized in a hieratical structure.3. FashionAI DatasetWe introduce FashionAI, a high quality fashion dataset,to the academic society. It covers 6 categories of women’sclothing, a total of 41 sub-categories on the website and hasdiverse data including different seasons (e.g. winter, summer), views (e.g. front, side), and types (e.g. products images, street images). The attribute system of FashionAI thatincludes design attributes and key points are presented inFigure 3.3.1. FashionAI StructureFrom the perspective of fashion design, we contributethe knowledge structure with a top-down mechanism. Thisstructure with logical internal connections can satisfy therequirement of fashion profession and machine learning simultaneously. As shown in Figure 3, the complex fashionsemantic is dissembled into mutually exclusive design attributes.Professional knowledge: All women’s wear items aredivided into six categories, “blouse”, “pants”, “skirt”,“dress”, “jumpsuit”, and “outwear”, all located at the topof whole framework. Design attributes of apparel are divided into three parts, namely characteristic, material andpattern, on the first level. Characteristic refers to the designfeature of apparel. Material is a kind of fabric used to makeup the apparel items, such as “cotton”. Finally, pattern isreferred to color and graphic design. Meanwhile, the keypoints of each subject are defined from the perspective ofgarment making.Hierarchical structure: As shown in Figure 3, instead of asingle-layer one which used previously[7, 18, 26], FashionAI is hierarchical. All roots and leaf nodes in the attributetree are named as attribute dimensions and attribute values,respectively. The total number of annotations are not thesum of all attribute values, but the product of the number ofattribute value in each attribute dimension. For example, asshown in Figure 4, the attribute dimension of “sleeve style”is further divided into 4 sub-dimensions, including: shape,cuff, shoulder, and design. However, even there are just 5attribute dimensions with 23 attribute values, 960 differentdesigns of sleeves could still be presented. It is obvious thatthis hierarchical structure reduces the trained attributes butcould also improve the comprehensiveness of the dataset atsame time.Mutually-exclusive attribute: From the perspective offashion design, there are many attributes would appear inthe same region of a garment but belongs to different cat-

subjectFashionAI attributespantsblouseskirtdresscharacteristictop lookbottom lookneck regionsleeve regionneckcollarlapelhigh-neck rib collar notched lengthcap waist skirt typesleeve styleshapecuffjuglet bishop shoulder colorlengthskirt stylemini shapedesignbell strap tucked soliddesignred patchwork texture jumpsuitoutwearpatternmaterialgraphicdenim irregular floral FashionAI key pointsblouseneckline left/rightcenter frontshoulder left/rightcut left out/incut right out/inarmpit left.righttop hem left/rightoutwearskirttrousersdressjumpsuitFigure 3. Part of the FashionAI attribute system for demonstration4. Data Preparationsleeve egularregularcutkimonoraglanwrapbatwingslope dropanglepuffFigure 4. Demonstration the advantage of the FashionAI structurein terms of comprehensiveAs shown in Figure 2, unlike the general method usedto build fashion dataset, FashionAI Dataset is constructedin an iterative process. All images are collected from commercial website. With the back up from fashion experts,we define each attribute clearly and professionally. The standard of artificial annotation, which includes textual description and image examples, is also developed. However,since the collected data are online product images, whichare complex and devise, we still face many problems, e.g.structure noise, attributes inconsistence etc.4.1. Image collectionegories, e.g. V-neck and PeterPan Collar. Thus, to distinguish the overlap concepts, the definition of each attributedimension and attribute value are clear and mutually exclusive. To realize it, fashion knowledge is decomposed clearly to ensure that they are machine-learnable. For example,the attribute dimension of “neck” is divided into four parts:“high neck”, “neckline”, “collar”, and “lapel”. Theoretically, such definition ensures that attribute values generated from each attribute dimension can exist simultaneously. The experiment results show the advance of hierarchicalstructure compared with the single layer one.In the end, we obtain 24 different key points as well as245 attribute values in 68 attribute dimensions (noted that201 values belongs to the dimension of characteristic thatcovers almost all general designs of daily garments).According to the designed standard, all related images,which we called image pool, are directionally collected online according to the key words of attributes. Two mainproblems have been solved in this step: scarcity collectionand structure noise.Scarcity collection: The existing common methods, likekeywords searching, can retrieve most required images.However, there are still many attributes which seldom appear in the public websites. For those kind of attributes,we use model acquisition method to search similar imagesbased on the collected small-scale image set at first. Andthen, those similar images are artificial checked until wecollect enough images of the attributes.Structure noise: To ensure the objectivity of constructeddataset, except the descriptive words of attribute value, weavoid using other directional keywords. For example, “balmain sleeves” are usually used in “suits” design. However,

(a) tiled(b) model(c) “invisible”(d) “uncertain”Figure 5. Demonstrate examples in FashionAI Dataset“suits” is not used as a keyword to search data of “balmainsleeves”. Additionally, to make sure that the model recognizes “balmain sleeves” not just based on whether the apparel is “suits” or not, we train a model based on the collecteddata to examine its effectiveness. Based on the testing resulton the overall random sampling data, the standard has beenrevised accordingly.Additionally, as common in web data, the raw data fromwebsite contains certain amount of near-duplicate images.Thus, before the annotation step, the duplicate data are automatically removed.4.2. Artificial AnnotationWhen the image pool is ready, we conduct artificial annotation. Since the standard contains plenty of professionalknowledge, we assign parts of tasks to the crowdsourcingstaffs at first and revised the standard based on their feedback and the result of labeling tasks. When the accuracyrates reaches 95%, labeling tasks are fully open to the outsourcing staffs. Meanwhile, 20% data of labeling tasks ofeach attribute value are checked and the accuracy with higher than 97% is regarded as qualified labeling. However, thedata from image pool are complex and diverse since all ofthem are uploaded by different online sellers. Those imageswithout a uniform standard cause many problems at the annotation step.Attribute inconsistent problem: As shown in Figure 5(a)and 5(b), the tiled single apparel image and single modelimage are common in e-commerce fashion data. The lengthrelated attributes dimension can be easily recognized whenit has a model as a reference. However, for tiled single apparel image, no model can be used for referencing. To solvethis problem, we use key points of the armpits and the distance between two armpits as reference. According to theproportion of apparel, the length standard of the tiled singleapparel image is defined. The established standard can beverified if same results can also be obtained by comparingthe standard used in the front view of a single model image.We have tested 510 paired images, and the accuracy rate canachieve 95%.Occlusion problem: Occlusion problem is very commonin real commercial apparel images, especially the photo isuploaded by the sellers or users online. It brings troublesin attribute annotation, noticeably in the length-related at-tribute dimension. As shown in Figure 5(c), the length of apair of trousers is blocked. It is impossible to determine itslength just based on the single image. To solve this problem, a new attribute named “invisible” is added to label suchkind of situation. Thus, the recognition result of the image in Figure 5(c) can be more reasonable and enables thetrained model to have “rejection” ability instead of givingan unpredictable answer.Uncertain problem: This kind of problem usually occurs on length related attribute dimensions. As shown inFigure 5(d), if the sleeve length of an apparel is on theposition in black spot, it could be recognized as wristlength sleeves as well as three-quarter sleeves. Thus,an annotation trick named “uncertain” is created to solvethis problem instead of avoiding such uncertain images.Taking attribute dimension of sleeves length as an example, it has totally 9 attribute values, including “invisible”,“sleeveless”, “cap sleeves”, “short sleeves”, “elbow-lengthsleeves”, “three-quarter sleeves”, “wrist-length sleeves”,“long sleeves”, “extra-long sleeves”. Therefore, the attribute dimension(length of sleeves) of Figure 5(d) is annotated as “nnnnnymnn”, which means three-quarter sleeves(“y”), and can be regarded as wrist-length sleeves (“m”).Noticeably, “m” would not be punished during training.4.3. Algorithm DesignTo verify the usefulness and effectiveness of our method,we propose an AttributeNet, which simultaneously predict attributes in a hierarchical and end-to-end manner. Thenetwork structure of AttributeNet is similar to that of theresidual-50 network[13] shown in Figure 6(a).However, AttributeNet, as shown in Figure 6(b), is connected to a pooling layer (3x3 stride2) Pool5 and a convolution layer (1x1 stride1) ConvNew after the res5c layer. Then, the feature map slice of ConvNew is divided intoeight equal parts. Each part is used to represent an attributedimension. The corresponding attribute dimension layer isfollowed by a fully connected layer and a softmax loss layer for classification. Noticeably, the AttributeNet is built onour FashionAI Dataset embedded with professional fashionknowledge. In other words, without the professional definition of fine-grained apparel semantics such as length of top,sleeves length, etc., there would be no bifurcated parallelprediction structure of AttributeNet.The advantages of the AttributeNet include: (1) It canpredict all attributes of apparel in parallel with high efficiency; (2) Such multi-classification method greatly avoidsoverfitting of features, which enables the trained model toobtain good generalization ability.4.4. Model IterationAs usual, we adopt data mining strategy to collect image for annotation. However, this method will inevitably

3-way SoftMax Input ImageRes-5cMax-pooling Convolution(a) single layer attribute structure3灤3灤256224灤224灤3Input -pooling ineFC-pantsFC-skirtFC-sleeve 8-way SoftMax5-way SoftMax5-way SoftMax5-way SoftMax9-way SoftMax6-way SoftMax6-way SoftMax9-way SoftMaxslice(b) attributeNet consisting of eight SoftMaxFigure 6. Pipeline of the proposed AttributeNet and baseline netinject bias in terms of image distribution. For example, ifwe built a dataset in spring, it is unavoidable that most ofour collected data are spring products. The attributes morerelated to autumn and winter apparel like “sweater” will beinfluenced. Moreover, the collected images will naturallyrelate to the online sellers’ preference since the informationabout the products are written by themselves. As such, thetrained model based on those images cannot perform wellin real applications. Specifically, when we train a modelon the attribute of lapel, the average accuracy on testing setcan achieve up to 82% but only 36% on real applicationdataset. The classification accuracy of shawl lapel attributeis less than 10% due to its scarcity. Even though we collect enough data at multiple times, it eventually brings morebias as well. To solve the above-mentioned problems andto ensure that the trained model not only achieves good performance on testing set but also gives satisfactory results ononline products, we propose the concept of model iteration.5. Experiments and Results5.1. Dataset validationAs depicted earlier, FashionAI is created in a differentperspective from previous datasets. Thus it is hard to find afair evaluation criterion for doing comparison with the existing attribute datasets directly, we demonstrate the effectiveness of the proposed dataset in the folowing two aspects:the knowledge structure and the practical uses.Knowledge structure. To demonstrate the advance of thehierarchical structure compared with the single layer one,we firstly test the performance of those two different structures both in the FashionAI dataset. As described in Section 4.3, a 54-way residual-50 network is used to benchmark with the performance of the AttributeNet, as shownin Figure 6(b). We replace the fc level of the last 1000way output of the original res-50 net with 54-way outputs. To ensure the fairness and comparability of experiment, we use the same configuration for training. Specifically,Algorithm 1: Model IterationInput: Training set: train set Validation set: val setOutput: Testing set: app set1 while do2Training model M on train set;3Testing model M on val set, the accuracy notedas P (val set);4Testing model M on app set, the accuracy notedas P (app set);5if P (val set) P (app set) 5% then6end while;7else8Testing model M on app set, annotatesamples with low confidence;9Re-add the new annotated samples into datasetA, update train set and val set;10Continue;11end12 end1accuracy of hierarchical layeraccuracy of single layer0.90.80.70.60.50.40.30.20.10neck designcollar designlapel designnecklinesleeve lengthskirt lengthtrouser lengthtop lengthallFigure 7. Accuracy comparison between the hierarchical structureand the single layer structurethe size of each image is adjusted so that its shorter edgeis 224 pixels and then the middle 224 pixels are croppedfrom the longer edge. There are totally 78,379 images inthe training set and 1,194 images for validation. Then, wetest on 10,800 images and the accuracy results are shown inFigure 7. It can be seen that ArributeNet has consistentlybetter performance in terms of accuracy across all eight attribute dimensions. The higher accuracy indicates that theproposed AttributeNet can provide the positive sample ofeach attribute dimension with higher confidence level andgreatly avoid over fitting of features, which proves the advantage of adopting hierarchical structure.Further, we do the recognition task which using the data from DeepFashion[26] and FashionAI. The attributes including “sleeveless”, “V-neck”, and “shirt-collar” which belong to both two datasets are randomly picked. Noted thatthose attributes in DeepFashion all have the same definitions as FashionAI and their accuracy also has the comparability with ours. The source of data are collected fromboth two datasets, namely DF-sub and FAI-sub respective-

sleevelessV-neckshirt ure 8. Accuracy of attributes recognition between DF-sub andFAI-sub0.930.80.86Accuracykey points indifferent 710.60.50.510.40.30.280.2Accuracy of the real applicationsAccuracy of the iterative validation set0.10images indifferent scalesFigure 10. FashionAI for the outfit combinations generation10.9112# Iterations34Figure 9. Accuracy of “PeterPan Collar” in test set and the onlineapplicationsly. The number of images for sleeveless, v-neck, and shirtcollar in FAI-sub are 3,576, 4,207, and 1,455 respectively,which is as same as the DF-sub for the sake of fairness. Particularly, the source of the test set with total 2,522 imagesis half from DF-sub and half from FAI-sub. We adopt theDenseNet161[15] for training and the recognition accuracyof those three attributes is shown in Figure 8. The modeltrained on FAI-sub has consistently better performance inthose three attributes than the one using DF-sub for training. As discussed before, the main reason is that DF-subexists many mixed attribute with confusion concepts whilethe attributes designed in FashionAI are all mutually exclusive with clear definition.Practical uses. As described before, the structure noise andbias are unavoidable when building a dataset. Thus, an iteration process is proposed to weaken their influence andfurther improve the performance of the data-trained modelin the most real-world applications. Considering the cost ofartificial annotations, here we just taking “PeterPan Collar”as an example to verify the effectiveness of the presentedprocess. We collect 1,000 images from real applicationsand 15% images from the validation set to conduct the iterative experiments. Figure 9 shows that the gap between theaccuracy of the real applications and validation set becomesnarrower with increase number of iterations. In other words,with the increasing number of iterations, the model obtainsbetter performance in the online products. The number ofiteration is decided by the required accuracy in practice.In addition, we introduce a new application, which isoutfit composition generation, depends on the FashionAIdataset. Outfit recommendation is a trendy tool for retail-ers to cater the consumer’s pursuit of beauty and the dothe cross-selling. However, currently, the outfit composition is mostly manually generated. Based on the FashionAI dataset, the key points and position of a fashion item ineach subject could be recognized. As shown in Figure 10,the armpit key points of apparel in different scales are usedat first. The distance of the two armpits is taken as the reference for the unity of scale. Then, the useful key points forattaching apparel are adopted. Note that the used key pointsin each subject are different, e.g. we use waist key pointsfor skirt. Finally, the key points of the top and the bottomare merged to generate the outfit composition. We presentsome samples in Figure 10.5.2. Data statisticsWe released FashionAI dataset with 54 labels of designattributes in 8 dimensions and 24 key points in 324k imagesin 2018. Figure 11 presents the statistical results of the published 8 dimensions. The annotated samples are shown inFigure 12.6. DiscussionIn this paper, we present FashionAI Dataset as the basis of understanding tasks in whic

ion semantic of those items are consist of design attribute. Fashion compatibility learning is to learn the matching re-lation of a series of design attribute in fact. In view of this, systematical and comprehensive design attribute recogni-tion is the foundation of fashion understanding task