WIXLIB

DeepFashion: Powering Robust Clothes Recognition and Retrievalwith Rich AnnotationsZiwei Liu11Ping Luo3,1The Chinese University of Hong Kong2Shi Qiu2Xiaogang Wang1,3SenseTime Group Limited3Xiaoou Tang1,3Shenzhen Institutes of Advanced Technology, CAS{lz013,pluo,xtang}@ie.cuhk.edu.hk, sqiu@sensetime.com, xgwang@ee.cuhk.edu.hkAbstractRecent advances in clothes recognition have been drivenby the construction of clothes datasets. Existing datasetsare limited in the amount of annotations and are difficult to cope with the various challenges in real-worldapplications. In this work, we introduce DeepFashion1 ,a large-scale clothes dataset with comprehensive annotations. It contains over 800,000 images, which are richlyannotated with massive attributes, clothing landmarks, andcorrespondence of images taken under different scenariosincluding store, street snapshot, and consumer. Such richannotations enable the development of powerful algorithmsin clothes recognition and facilitating future researches. Todemonstrate the advantages of DeepFashion, we propose anew deep model, namely FashionNet, which learns clothingfeatures by jointly predicting clothing attributes and landmarks. The estimated landmarks are then employed to poolor gate the learned features. It is optimized in an iterativemanner. Extensive experiments demonstrate the effectiveness of FashionNet and the usefulness of DeepFashion.1. IntroductionRecently, extensive research efforts have been devotedto clothes classification [11, 1, 29], attribute prediction[3, 13, 4, 24], and clothing item retrieval [17, 6, 10, 27, 15],because of their potential values to the industry. However, clothes recognition algorithms are often confrontedwith three fundamental challenges when adopted in realworld applications [12]. First, clothes often have largevariations in style, texture, and cutting, which confuseexisting systems. Second, clothing items are frequentlysubject to deformation and occlusion. Third, clothingimages often exhibit serious variations when they are takenunder different scenarios, such as selfies vs. online shoppingphotos.Previous studies tried to handle the above challenges byannotating clothes datasets either with semantic attributes1 The dataset is available at: KimonoChinosBlazerFigure 1. (a) Additional landmark locations improve clothes recognition.(b) Massive attributes lead to better partition of the clothing feature space.(e.g. color, category, texture) [1, 3, 6], clothing locations(e.g. masks of clothes) [20, 12], or cross-domain imagecorrespondences [10, 12]. However, different datasets areannotated with different information. A unified datasetwith all the above annotations is desired. This work fillsin this gap. As illustrated in Fig.1, we show that clothesrecognition can benefit from learning these annotationsjointly. In Fig.1 (a), given the additional landmark locationsmay improve recognition. As shown in Fig.1 (b), massiveattributes lead to better partition of the clothing featurespace, facilitating the recognition and retrieval of crossdomain clothes images.To facilitate future researches, we introduce DeepFashion, a comprehensively annotated clothes dataset that contains massive attributes, clothing landmarks, as well ascross-pose/cross-domain correspondences of clothing pairs.This dataset enjoys several distinct advantages over itsprecedents. (1) Comprehensiveness - images of DeepFashion are richly annotated with categories, attributes, land-1096

DCSA [3]ACWS [1]WTBI [12]DDAN [4]DARN [10]DeepFashion# images1856145,71878,958341,021182,780 800,000# categories attributes261511671791,050# exact pairsN/AN/A39,479N/A91,390 300,000localizationN/AN/AbboxN/AN/A4 8 landmarkspublic availabilityyesyesnononoyesTable 1. Comparing DeepFashion with other existing datasets. DeepFashion offers the largest number of images and annotations.marks, and cross-pose/cross-domain pair correspondences.It has 50 fine-grained categories and 1, 000 attributes, whichare one order of magnitude larger than previous works[3, 4, 10]. Our landmark annotation is at a finer levelthan existing bounding-box label [12]. Such comprehensiveand rich information are not available in existing datasets.(2) Scale - DeepFashion contains over 800K annotatedclothing images, doubling the size of the largest one inthe literature. 3) Availability - DeepFashion will be madepublic to the research community. We believe this datasetwill greatly benefits the researches in clothes recognitionand retrieval.Meanwhile, DeepFashion also enables us to rigorouslybenchmark the performance of existing and future algorithms for clothes recognition. We create three benchmarks,namely clothing attribute prediction, in-shop clothes retrieval, and cross-domain clothes retrieval, a.k.a. street-toshop. With such benchmarks, we are able to make directcomparisons between different algorithms and gain insightsinto their pros and cons, which we hope will eventuallyfoster more powerful and robust clothes recognition andretrieval systems.To demonstrate the usefulness of DeepFashion, we design a novel deep learning structure, FashionNet, whichhandles clothing deformation/occlusion by pooling/gatingfeature maps upon estimated landmark locations. Whensupervised by massive attribute labels, FashionNet learnsmore discriminative representations for clothes recognition.We conduct extensive experiments on the above benchmarks. From the experimental results with the proposeddeep model and the state-of-the-arts, we show that theDeepFashion dataset promises more accurate and reliablealgorithms in clothes recognition and retrieval.This work has three main contributions. (1) We builda large-scale clothes dataset of over 800K images, namelyDeepFashion, which is comprehensively annotated withcategories, attributes, landmarks, and cross-pose/crossdomain pair correspondences. To our knowledge, it isthe largest clothes dataset of its kind. (2) We developFashionNet to jointly predict attributes and landmarks. Theestimated landmarks are then employed to pool/gate thelearned features. It is trained in an iterative manner.(3) We carefully define benchmark datasets and evaluation protocols for three widely accepted tasks in clothesrecognition and retrieval. Through extensive experimentswith our proposed model as well as other state-of-thearts, we demonstrate the effectiveness of DeepFashion andFashionNet.1.1. Related WorkClothing Datasets As summarized in Table 1, existingclothes recognition datasets vary in size as well as theamount of annotations. The previous datasets were labeledwith limited number of attributes, bounding boxes [12], orconsumer-to-shop pair correspondences [10]. DeepFashioncontains 800K images, which are annotated with 50 categories, 1, 000 attributes, clothing landmarks (each imagehas 4 8 landmarks), and over 300K image pairs. Itis the largest and most comprehensive clothes dataset todate. Some other datasets in the vision community werededicated to the tasks of clothes segmentation, parsing[32, 31, 23, 16, 33] and fashion modeling [24, 30], whileDeepFashion focuses on clothes recognition and retrieval.Clothing Recognition and Retrieval Earlier works [28,3, 7, 1, 6] on clothing recognition mostly relied on handcrafted features, such as SIFT [19], HOG [5] and colorhistogram etc. The performance of these methods werelimited by the expressive power of these features. In recentyears, a number of deep models have been introduced tolearn more discriminative representation in order to handlecross-scenario variations [10, 12]. Although these methodsachieved good performance, they ignored the deformationsand occlusions in the clothing images, which hinder furtherimprovement of the recognition accuracy. FashionNet handles such difficulties by explicitly predicting clothing landmarks and pooling features over the estimated landmarks,resulting in more discriminative clothes representation.2. The DeepFashion DatasetWe contribute DeepFashion, a large-scale clothesdataset, to the community. DeepFashion has several appealing properties. First, it is the largest clothing datasetto date, with over 800, 000 diverse fashion images rangingfrom well-posed shop images to unconstrained consumerphotos, making it twice the size of the previous largestclothing dataset. Second, DeepFashion is annotated withrich information of clothing items. Each image in thisdataset is labeled with 50 categories, 1, 000 descriptiveattributes, and clothing landmarks. Third, it also containsover 300, 000 cross-pose/cross-domain image pairs. Some1097

lockTweedShapeCropMidiPartBow-F Fringed-HStyleMickeyBaseballFigure 2. Example images of different categories and attributes in DeepFashion. The attributes form five groups: texture, fabric, shape, part, and style.(a) 160000(b)# Images# igure 3. (a) Image number of the top-20 categories. (b) Image number of the top-10 attributes in each group.example images along with the annotations are shown inFig.2. From the comparison items summarized in Table1, we see that DeepFashion surpasses the existing datasetsin terms of scale, richness of annotations, as well asavailability.2.1. Image CollectionShopping websites are a common source for constructingclothing datasets [10, 12]. In addition to this source, we alsocollect clothing images from image search engines, wherethe resulting images come from blogs, forums, and the otheruser-generated contents, which supplement and extend theimage set collected from the shopping websites.Collecting Images from Shopping Websites Wecrawled two representative online shopping websites, Forever212 and Mogujie3 . The former one contains imagestaken by the online store. Each clothing item has 4 5images of varied poses and viewpoints. The latter onecontains images taken by both the stores and consumers.Each clothing image in shop is accompanied by several user-taken photos of exactly the same clothing item.Therefore, these data not only cover the image distributionof professional online retailer stores, but also the otherdifferent domains such as street snapshots and selfies. Wecollected 1, 320, 078 images of 391, 482 clothing itemsfrom these two websites.Collecting Images from Google Images4 To obtainmeaningful query keywords for clothing images, we traversed the catalogue of several online retailer stores andcollected names of clothing items, such as “animal printdress”. This process results in a list of 12, 654 uniquequeries. We then feed this query set to Google Images, anddownload the returned images along with their associatedmeta-data. A total of 1, 273, 150 images are collected fromGoogle Images.Data Cleaning We identified near- and exact-duplicateimages by comparing fc7-responses after feeding them intoAlexNet [14]. After the removal of the duplicates, we askhuman annotators to remove unusable images that are oflow resolution, image quality, or whose dominant objectsare irrelevant to clothes. In total, 800, 000 clothing imagesare kept to construct DeepFashion.2.2. Image AnnotationWe advocate the following labeled information in orderto aid the tasks for clothing recognition and retrieval. Theyare: (1) Massive attributes - this type of information isessential to recognize and represent the enormous clothing items; (2) Landmarks - the landmark locations caneffectively deal with deformation and pose variation; (3)Consumer-to-shop pairs - these data is of great help in2 www.forever21.com3 www.mogujie.com4 https://www.google.com/imghp1098