WIXLIB

tion from the normalization activations. Attention normalization layer have a more compact structure and occupiesless computation resource.composed mask can be expressed as:3. Fashion Editingwhere M ′ , M and Mforeground are the composed mask, original mask and foreground mask, respectively. denoteselement-wise multiply. Besides the partial convolution encoder, we introduce a standard convolution encoder to extract semantics feature from the synthesized parsing map.The human parsing map has semantics and location information that will guide the inpainting, since the content ina region with the same semantics should be similar. Giventhe semantic features, the network can render textures onthe particular region more precisely. Two encoded featuremaps are concatenated together in a channel-wise manner.Then the concatenated feature map undergoes several dilated residual blocks. During the upsampling process, welldesigned multi-scale attention normalization layers are introduced to obtain attention maps, which are conditionedon sketch and color strokes. The learned attention maps arehelpful to select more effective feature in the forward activations. We explain the details in the next section.We propose a novel method for editing fashion image, allowing users to edit images with a few sketches and sparsecolor strokes on an interested region. The overview of ourFE-GAN is shown in Figure 2. The main components ofour FE-GAN include a free-form parsing network and aparsing-aware inpainting network with the multi-scale attention normalization layers. We first discuss the free-formparsing network. It can manipulate human parsing guidedby free-form sketch and color, and is crucial to help theparsing-aware inpainting network produce convincing interactive results. Then, we describe the attention normalization layers inserted at multiple scales in the inpaintingdecoder that can selectively extract effective features andenhance visual quality. Finally, we give a detailed description of the learning objective function used in our FE-GAN.3.1. Free-form Parsing NetworkCompared to directly restoring an incomplete image,predicting a parsing map from an incomplete parsing mapis more feasible since there are fewer details in the parsingmap. Meanwhile, the semantic information in the parsingmap can be a guidance for rendering detail textures in eachpart of an image precisely. To this end, we propose a freeform parsing network to synthesize a complete parsing mapwhen giving an incomplete parsing map and arbitrary sketchand color strokes.The architecture of the free-form parsing network is illustrated in the upper left part of Figure 2. It is based on theencoder-decoder architecture like U-net [21]. The encoderreceives five inputs: an incomplete parsing map, a binarysketch that describes the structure of the removed region, anoise sampled from the Gaussian distribution, sparse colorstrokes and a mask. It is worth noting that given the same incomplete parsing map and various sketch and color strokes,the free-form parsing network can synthesize different parsing map, which indicates that our parsing generation modelis controllable. It is significant for our fashion editing system since different parsing maps guide to render differentcontents in the edited image.M ′ (1 M ) Mforeground ,3.3. Attention Normalization LayersInspired by SPADE [23], we design a variant of conditional normalization, named Attention Normalization Layers (ANLs). However, instead of inferring an affine transformation from external data directly, ANLs learn an attention map which is used to extract the significant informationin the earlier normalized activation. The upper right partof Figure 2 illustrates the design of ANLs. The details ofANLs are shown below.Let xi denotes the activations of the layer i in the deepneural network. Let N denotes the number of samples inone batch. Let C i denotes the number of channels of xi .Let H i and W i represent the height and width of activationmap in layer i respectively. When the activations xi passingthrough ANLs, they are first normalized in a channel-wisemanner. Then the normalized activations are modulated bythe learned attention map and bias. Finally, the modulatedactivations pass through a rectified linear unit (RELU) and aconvolution layer and concatenate with the original normalized activations. The activations value before the final concatenation at position (n N, c C i , h H i , w W i ) issigned as:3.2. Parsing-aware Inpainting NetworkThe architecture of parsing-aware inpainting network isillustrated on the bottom of Figure 2. Inspired by [16],we introduce a partial convolution encoder to extract feature from the valid region in incomplete images. Instead ofusing the mask directly, we utilize the composed mask tomake the network focus only on the foreground region. The(1)if (αc,h,w(d)xin,c,h,w µici βc,h,w(d)),σci(2)where f (x) denotes RELU and convolution operations,xin,c,h,w is the activation value at particular position beforenormalization, µic and σci are the mean and standard deviation of activation in channel c. As the same of BN [9], we8123

Figure 3. Qualitative comparisons with Deepfill v1 [32], Partial Conv [16], and Edge-connect [19].formulate them as:µic X1xin,c,h,wN H iW i(3)n,h,wσci siαc,h,w(d)X1(xin,c,h,w )2 (µic )2iiNH W(4)Lfeat [27], and total variation loss LTV [14] to regularize thetraining. We define a face TV loss to remove the artifacts ofthe face by using LTV on face region. We define a mask lossby using the L1 norm on the mask area, let Igen be generated image, let Ireal be ground truth, and let M be the mask,which is computed as:Lmask Igen M Ireal M 1 ,n,h,wiβc,h,w(d)Theandare learned attention map andbias for modulating the normalization layer, which are conditioned on the external data d, namely, the sketch and colorstrokes and noise in this paper. Our implementations ofiiαn,h,wand βn,h,ware straightforward. The external data isfirst projected into an embedding space through a convolution layer. Then the bias is produced by another convolutionlayer, and the attention map is generated by a convolutionlayer and a sigmoid operation, which limits the range offeature map values between zero and one, and ensures theoutput to be an attention map. The effectiveness of ANLsis due to their inherent characteristics. Similar to SPADE[23], ANLs also can avoid washing away semantic information in activations, since the attention map and bias arespatially-varying. Moreover, the multi-scale ANLs can notonly adapt the various scales of activations during upsampling but also extract coarse-to-fine semantic informationfrom external data, which guide the fashion editing moreprecisely.3.4. Learning Objective FunctionDue to the complex textures of the incomplete imageand the variety of sketch and color strokes, the training ofthe free-form parsing network and parsing-aware inpaintingnetwork is a challenging task. To address these problems,we apply several losses to make the training easier and morestable in different aspects. Specifically, we apply adversarial loss Ladv [6], perceptual loss Lperceptual [14], style lossLstyle [14], parsing loss Lparsing [5], multi-scale feature loss(5)we also define a foreground loss to enhance the foregroundquality. Let Mforeground be the mask of foreground part, thenLforeground can be formally computed asLforeground Igen Mforeground Ireal Mforeground 1 , (6)similar to Lforeground , we formulate a face loss Lface to improve the quality of face region.The overall objective function Lfree-form-parser for freeform parsing network is formulated as:Lfree-form-parser γ1 Lparsing γ2 Lfeat γ3 Ladv ,(7)where hyper-parameters γ1 , γ2 and γ3 are weights of eachloss.The overall objective function Linpainter for parsing-awareinpainting network written as:Linpainter λ1 Lmask λ2 Lforeground λ3 Lface λ4 LfaceTV λ5 Lperceptual λ6 Lstyle λ7 Ladv ,(8)where hyper-parameters λi , (i 1, 2, 3, 4, 5, 6, 7) are theweights of each loss.4. Experiments4.1. DatasetsWe conduct our experiments on DeepFashion [35] fromFashion Image Synthesis track. It contains 38,237 imageswhich are split into a train set and a test set, 29,958 and8124

Figure 4. Example of inputs. The inputs of the free-form parsing network consist of incomplete parsing, sketch, color, mask, and noise; theinputs of parsing-aware inpainting network contain incomplete image, and composed mask. The inputs of attention normalization layersare a sketch, color, and noise. We first generate the sketches by using Canny [1] shown in the third column of the first row. Then, we use ahuman parser [5] to extract the median color of each part of the person, shown in the 5th column of the first row.8,279 images, respectively. MPV [4] contains 35,687 images which are split into a train set and a test set, 29,469and 6,218 samples. For better contributing to the fashion editing community, we collected a new fashion dataset,named FashionE. It contains 7,559 images with the size of512 320. In our experiment, we split it into a train set of6,106 images and a test set of 1,453 images. The datasetwill be released upon the publication of this work. The sizeof the image is 512 320 across all datasets.We utilize the Irregular Mask Dataset provided by [16]in our experiments. The original dataset contains 55,116masks for training and 24,866 masks for testing. We randomly select 12,000 images, splitting it into one train set of9,600 masks and one test set of 2,400 masks. To mimic thefree-form color stroke, we utilize one irregular mask datasetfrom [10] as Irregular Strokes Dataset. The mask regionstands for stroke in our experiment. In our experiment, wesplit it into a train set of 50,000 masks and a test set of10,000 masks. All the masks are resized to 512 320.4.2. Metrics. We evaluate our proposed method, as well as comparedapproaches on three metrics, PSNR (Peak Signal NoiseRatio), SSIM (Structural Similarity index) [28], and FID(Fréchet Inception Distance) [8]. We apply the AmazonMechanical Turk (AMT) for evaluating the qualitative results.4.3. Implementation DetailsTraining Procedure. The training procedure is twostage. The first stage is to train free-form parsing network.We use γ1 10, γ2 10, γ3 1 in the loss function. Thesecond stage is to train parsing-aware inpainting network.We use λ1 5.0, λ2 50, λ3 1.0, λ4 0.1, λ5 0.05,λ6 200, λ7 0.001 in the loss function. For both trainingstages, we use Adam [15] optimizer with β1 0.5 and β2 0.999 and learning rate is 0.0002. The batch sizes of stage1 is 20, and stage 2 is 8. In each training cycle, we train onestep for the generator and one step for the discriminator. Allthe experiments are conducted on 4 Nvidia 1080 Ti GPUs.Sketch & Color Domain. The way of extracting sketchand color domain from images is similar to SC-FEGAN. Instead of using HED [29], we generated sketches by CannyEdge Detector [1]. Relying on the result of human parsing,we use the median color of each segmented area to represent the color of that area. More details are presented in Figure 4. As shown in Figure 1, all sketches inputs are drawnby a human. The random mask crops the edges, colors, andnoise. Note that the inputs include a Gaussian noise, whichenhances the robustness of the model that allows to generate8125

Figure 5. Some interactive comparisons with Deepfill v1 [32], Partial Conv [16], and Edge-connect [19].image with an actual sketch drawn.Discriminators. The discriminator, used in free-formparsing network, has a similar structure as the multi-scalediscriminator in Pixel2PixelHD [27], which has two PatchGAN discriminators. The discriminator, used in parsingaware inpainting network, has a similar structure as inpainting discriminator in Edge-connect [19], with five convolutions and spectral norm blocks.Compared Approaches. To make a comprehensiveevaluation of our proposed method, we conduct three comparison experiments based on the recent state of the artapproaches at image inpainting [32, 16, 19]. The reimplementations followed the official source codes provided by authors and the same inputs to train the baselines.To make a fair comparison, all inputs consist of incompleteimages, masks, sketch, color domain, and noise across allcomparison experiments.4.4. Quantitative ResultsPSNR computes the peak signal-to-noise ratio betweenimages. SSIM measures the similarity between two images.Higher value of PSNR and SSIM mean better results. FIDis tended to replace Inception Score as one of the most significant metrics measuring the quality of generated images.It computes the Fréchet distance between two multivariateGaussians, the smaller the better. As mentioned in [28],there is no good numerical metric in image inpainting. Furthermore, our focus is even beyond the regular inpainting.We can observe from Table 1, our FE-GAN achieves thebest PSNR, SSIM, and FID scores and outperforms all othermethods among three datasets.4.5. Qualitative ResultsBeyond numerical evaluation, we present visual comparisons for image completion task among three datasets andfour methods, shown in Figure 3. Three rows, from topto bottom, are results from DeepFashion, MPV, and FashionE. The interactive results for those methods are shownin Figure 5. The last column of the Figure 5, are the results of the free-form parsing network. We can observe thatthe free-form parsing network can obtain promising parsing results by manipulating the sketch and color. Thanks tothe multi-scale attention normalization layers and the synthesized parsing result from the free-form parsing network,our FE-GAN outperforms all other baselines on visual comparisons.4.6. Human EvaluationTo further demonstrate the robustness of our proposedFE-GAN, we conduct the human evaluation deployed onthe Amazon Mechanical Turk platform on the DeepFashion [35], MPV [4], and FashionE. In each test, we provide8126

Table 1. Quantitative comparisons on DeepFashion [35], MPV [4], and FashionE datasets.DeepFashion [35]MPV eepfill v1 [32]Partial Conv [16]Edge-connect 5.182FE-GAN .246Table 2. Human evaluation results of pairwise comparison with other methods.Comparison Method PairDeepFashion [35]MPV [4]FashionEOurs vs Deepfill v1 [32]Ours vs Partial Conv [16]Ours vs Edge-connect [19]0.849 vs 0.1510.917 vs 0.0830.790 vs 0.2100.845 vs 0.1550.864 vs 0.1360.691 vs 0.3090.857 vs 0.1430.799 vs 0.2010.656 vs 0.344Table 3. Ablation studies on FashionE.MethodPSRNSSIMFIDFullw/o attention normw/o parsingw/o Lmaskw/o 00.9230.9210.9274.0925.1915.3554.7735.030achieved by the attention normalization and human parsing.We also explore the impact of our designed objective function that each of the losses can substantially improve theresults. The ablation study on FashionE dataset that suggests the proposed method has a superb performance overother methods, and confirms the quality of the results.5. Conclusiontwo images, one from compared methods, the other fromour proposed method. Workers are asked to choose themore realistic image out of two. During the evaluation, Kimages from each dataset are chosen, and n workers willonly evaluate these K images. In our case, K 100 andn 10. We can observe from Table 2, our proposed methodhas a superb performance over the other baselines. Thisconfirms the effectiveness of our FE-GAN comprised ofa free-form parsing network and a parsing-aware network,which generates more realistic fashion images.4.7. Ablation StudyTo evaluate the impact of the proposed component of ourFE-GAN, we conduct an ablation study on FashionE withusing the model of 20 epochs. As shown in Table 3 , wereport the results of the different versions of our FE-GAN.We first compare the results using attention normalization tothe results without using it. We can learn that incorporatingthe attention normalization layers into the decoder of theinpainting module significantly improves the performanceof image completion. We then verify the effectiveness ofthe proposed free-from parsing network. From Table 3 , weobserve that the performance drops dramatically without using parsing, which can depict the human layouts for guidingimage manipulation with higher-level structure constraints.Note that w/o attention norm denotes without attention normalization layers of the proposed model. As shown in Table 3, the results report that the main improved performanceIn this paper, we propose a novel Fashion Editing Generative Adversarial Network (FE-GAN), which enables usersto manipulate the fashion image with an arbitrary sketchand a few sparse color strokes. To achieve realistic interactive results, the FE-GAN incorporates a free-form parsing network to predict the complete human parsing map toguide fashion image manipulation, which is crucial to helpfor producing convincing results. Moreover, we develop aforeground-based partial convolutional encoder and designan attention normalization layer which used in the multiplescales layers of the decoder for the fashion editing network.We construct a new dataset for the fashion editing task, covering person images with more challenging styles. Extensive experiments on three fashion datasets demonstrate thatour FE-GAN outperforms the state-of-the-art methods andachieves high-quality performance with convincing detailsby controlling the sketch and color strokes.AcknowledgementsThis work is supported by the National Natural Science Foundation of China (U1711262,U1611264,U1711261,U1811261,U1811264),the National Natural Science Foundation of China (NSFC) underGrant No.U19A2073, the National Natural Science Foundation of China (NSFC) under Grant No.61976233, theNational Key R&D Program of China (2018YFB1004404),and the Key R&D Program of Guangdong Province(2018B010107005).8127

References[1] John F. Canny. A computational approach to edge detection.IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-8:679–698, 1986. 6[2] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha,Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-imagetranslation. In CVPR, 2018. 1[3] Victor Lempitsky Dmitry Ulyanov, Andrea Vedaldi. Instancenormalization: The missing ingredient for fast stylization.arXiv preprint arXiv:1607.08022, 2016. 3[4] Haoye Dong, Xiaodan Liang, Bochao Wang, Hanjiang Lai,Jia Zhu, and Jian Yin. Towards multi-pose guided virtual tryon network. arXiv preprint arXiv:1902.11026, 2019. 2, 6, 7,8[5] Ke Gong, Xiaodan Liang, Xiaohui Shen, and Liang Lin.Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing. In CVPR,pages 6757–6765, 2017. 5, 6[6] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, BingXu, David Warde-Farley, Sherjil Ozair, Aaron Courville, andYoshua Bengio. Generative adversarial nets. In NIPS, pages2672–2680, 2014. 1, 2, 5[7] Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry SDavis. Viton: An image-based virtual try-on network. arXivpreprint arXiv:1711.08447, 2017. 1[8] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner,Bernhard Nessler, and Sepp Hochreiter. Gans trained by atwo time-scale update rule converge to a local nash equilibrium. In NIPS, 2017. 6[9] Sergey Ioffe and Christian Szegedy. Batch normalization:Accelerating deep network training by reducing internal covariate shift. In ICML, pages 448–456, 2015. 3, 4[10] Kazizat T. Iskakov. Semi-parametric image inpainting. arXivpreprint arXiv:1807.02855, 2018. 6[11] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei AEfros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017. 2[12] Jimei Yang-Xiaohui Shen Xin Lu Thomas S. Huang Jiahui Yu, Zhe Lin. Free-form image inpainting with gatedconvolution. arXiv preprint arXiv:1806.03589, 2018. 1, 2[13] Geoffrey E. Hinton Jimmy Lei Ba, Jamie Ryan Kiros. Layernormalization. arXiv preprint arXiv:1607.06450, 2016. 3[14] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptuallosses for real-time style transfer and super-resolution. InECCV, pages 694–711, 2016. 1, 5[15] Diederik P Kingma

Fashion Editing with Adversarial Parsing Learning Haoye Dong1,2, Xiaodan Liang3, Yixuan Zhang5, Xujie Zhang1, Xiaohui Shen4, Zhenyu Xie1, Bowen Wu1, Jian Yin1,2, 1School of Data and Computer Science, Sun Yat-sen University 2Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou 510006, P.R.China 3School of Intelligent Systems Engineering, Sun Yat-sen University