Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over AYUSH et al (US 20210133919) in view of GOU et al (Taming the Power of Diffusion Models for High-Quality Virtual Try-On with Appearance Flow).
As per claim 1, Ayush teaches the claimed “method for generating a try-on image,” comprising: “obtaining a first image of a target model and a second image of an item of clothing to be tried on” (Ayush, [0024] - The IBVTON system 104 represents functionality of the computing device 102 to receive a reference image 106 of a person and an image of target clothing 108 and generate a synthesized image 110 that depicts the person in the reference image 106 wearing the target clothing 108 instead of clothing 112 worn by the person in the reference image 106); “performing image processing on the first image to generate a plurality of third images, each expressing different information” (Ayush, [0026] - Given the reference image 106, the representation module 114 is configured to generate a cloth-agnostic person representation of the person depicted in the reference image 106. The cloth-agnostic person representation includes information describing a body shape, a body pose, certain regions (e.g., face and hair regions) that describe an appearance of the person in the reference image 106 independent of the clothing 112); “performing clothing deformation processing on the item of clothing in the second image based on the first image to obtain a fourth image, wherein a clothing shape in the fourth image aligns with a pose of the target model” (Ayush, [0019] - The warped version of the target clothing is deformed using a learnable thin-plate spline transformation that is output by a convolutional neural network to align the target clothing with the attributes described by the clothing-agnostic person representation. Parameters for the convolutional neural network are trained from paired images of target clothing and a person wearing the target clothing, without requiring an indication of explicit corresponding interest points in the images of the target clothing and the person wearing the target clothing; [0028] - Being trained with the multi-scale patch adversarial loss, the geometric matching module 118 is representative of functionality to generate a warped version of the target clothing 108 based on the clothing-agnostic person representation of the person depicted in the reference image 106. By training the geometric matching module 118 with multi-scale patch adversarial loss, the IBVTON system 104 is able to generate the warped version of the target clothing 108 via elastic transformation to fit the body shape and pose of the person depicted in the reference image 106 while maintaining texture details of the target clothing 108, such as material properties, logos, embroidery, and so forth; [0045] - The trained geometric matching module 118 is thus configured to generate the warped target clothing 210, given the cloth-agnostic person representation 202 and the target clothing 108, and communicate the warped target clothing 210 to the try-on module 120. Although the warped target clothing 210 is roughly aligned with the body shape 206 of the person depicted in the reference image 106, the try-on module 120 represents functionality of the IBVTON system 104 to fuse the warped target clothing 210 together with the cloth-agnostic person representation 202 to output a synthesized image, such as an image of a person wearing warped target clothing 218). Ayush suggests the step of “generating a try-on image of the target model wearing the item of clothing to be tried on in the target model's pose based on the third images, the fourth image, the first image, and the second image” in the teaching of the combining the warped clothing and the cloth-agnostic person representation includes information describing a body shape, a body pose, certain regions (e.g., face and hair regions) that describe an appearance of the person in the reference image 106 (Ayush, [0029] - Using the composition mask, the try-on module 120 fuses together the warped clothing and the image of the person wearing the warped clothing to improve alignment and eliminate blurriness from the resulting image of the person wearing the warped target clothing; [0045] - The trained geometric matching module 118 is thus configured to generate the warped target clothing 210, given the cloth-agnostic person representation 202 and the target clothing 108, and communicate the warped target clothing 210 to the try-on module 120. Although the warped target clothing 210 is roughly aligned with the body shape 206 of the person depicted in the reference image 106, the try-on module 120 represents functionality of the IBVTON system 104 to fuse the warped target clothing 210 together with the cloth-agnostic person representation 202 to output a synthesized image, such as an image of a person wearing warped target clothing 218) (see also Gou, Figure 2 - The overview of our method. First, we obtain the segmentation result 𝑆𝑝, dense pose 𝑃 and clothes-agnostic 𝐼𝑎 of the target person image 𝐼𝑝 through preprocessing. The clothes image 𝐼𝑐 is roughly aligned to the person by the warping network. Then, we combine 𝐼𝑎 and 𝐼𝑐 to obtain 𝐼’0 and add noise to get 𝐼′𝑡 as input to the diffusion model, and the final output 𝐼^ produced by denoising 𝐼’𝑡). Thus, it would have been obvious, in view of Gou, to configure Ayush’s method as claimed by using the images through warped cloths according to the characteristics of the captured human image. The motivation is to improve the visual representation of try-on cloth appearance.
Claim 2 adds into claim 1 “extracting skeletal pose information from the first image to obtain a third image representing the skeletal pose of the target model in the first image; and/or performing occlusion processing on a target image region of the first image to obtain a third image with clothing expression removed; and/or performing masking processing on the first image to obtain a third image representing a redrawn region” (Ayush, [0036] - The cloth-agnostic person representation 202 is representative of information used by the IBVTON system 104 to deform target clothing in order to generate an image of the person wearing warped target clothing 218, such as synthesized image 110 illustrated in FIG. 1. Specifically, the cloth-agnostic person representation 202 includes a pose heatmap 204, a body shape 206, and one or more reserved regions 208 for a person depicted in the reference image 106, which collectively represent a set of features used as a prior to constrain the IBVTON system 104's process of generating the synthesized image 110; [0037] - In some implementations, the representation module 114 generates the pose heatmap 204 using a known pose estimator, such as the pose estimator described in Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, Realtime Multiperson 2D Pose Estimation Using Part Affinity Fields (CVPR, 2017). The pose heatmap 204 may comprise coordinates representing a number of keypoints that in turn represent the pose of the person depicted in reference image 106).
Claim 3 adds into claim 2 “wherein the target image region includes a wearing region that matches the item of clothing to be tried on” (Ayush, [0038] - The body shape 206 is representative of information that describes various regions (e.g., arms, legs, torso, etc.) of a body of the person depicted in reference image 106. In order to generate the body shape 206, the representation module 114 is configured to implement a human parser to compute a human segmentation map, where different regions represent different corresponding body parts. In some implementations, the representation module 114 implements a human parser, using known techniques. The representation module 114 then converts the segmentation map derived using the human parser to a single channel binary mask, where “ones” in the binary mask indicate human body regions (except for face and hair) and “zeroes” in the binary mask represent regions of the reference image 106 that do not depict human body parts. The representation module 114 then down-samples the binary mask to a lower resolution, which assists in avoiding artifacts that otherwise might occur when portions of a human body as indicated by the body shape 206 and the target clothing conflict. This down-sampled binary mask is then output by the representation module 114 as the body shape 206).
Claim 4 adds into claim 1 “obtaining a fifth image representing a skeletal pose of the target model in the first image, and a sixth image representing a body shape and pose of the target model in the first image” (Ayush, [0018] - In such a conditional image generation approach, given two images, one of a person and the other of target clothing, a new image is synthesized with the goal of retaining the original body shape and pose of the person while maintaining characteristics of the clothing product, such as texture, logo, text, and so forth. Some conventional approaches deform target clothing to align with the person image using conditional image generation techniques, such as image-to-image translation); “performing masking processing on the item of clothing in the second image to obtain a seventh image” (Ayush, [0020] - The convolutional neural network is described herein in the context of a geometric matching module, which takes the clothing-agnostic person representation and image of target clothing as inputs, and generates a pose-coherent warped version of the target clothing as well as a composition mask that indicates details of the target clothing to be retained in the synthesized image. The composition mask enables output of a smooth synthesized image and accommodates transformations for different body shapes and body poses, such that the synthesized image output by the IBVTON system retains a personal identity of the person depicted in the reference image); “using the fifth image and the sixth image as first branch inputs to a pre-trained appearance flow deformation network, and using the second image and the seventh image as second branch inputs to the appearance flow deformation network” (Ayush, [0019] - In order to generate a synthesized image of the person depicted in the reference image wearing the target clothing instead of the clothing originally depicted in the reference image, the IBVTON system generates a cloth-agnostic person representation of the person depicted in the reference image and uses the cloth-agnostic person representation as a model for generating a warped version of the target clothing in a manner that preserves visual characteristics of both the target clothing and the person depicted in the reference image. The warped version of the target clothing is deformed using a learnable thin-plate spline transformation that is output by a convolutional neural network to align the target clothing with the attributes described by the clothing-agnostic person representation. Parameters for the convolutional neural network are trained from paired images of target clothing and a person wearing the target clothing, without requiring an indication of explicit corresponding interest points in the images of the target clothing and the person wearing the target clothing); and “performing clothing deformation processing on the item of clothing to be tried on using the appearance flow deformation network to obtain the fourth image” (Ayush, [0036]-[0037] - The cloth-agnostic person representation 202 is representative of information used by the IBVTON system 104 to deform target clothing in order to generate an image of the person wearing warped target clothing 218, such as synthesized image 110… The pose heatmap 204 is representative of information that describes a body pose of the person depicted in the reference image 106, which consequently affects a deformation of the target clothing to be represented in the image of the person wearing warped target clothing 218).
Claim 5 adds into claim 1 “using the first image and the second image as input images for a pre-trained try-on image generation model” (Ayush, [0024] - The IBVTON system 104 represents functionality of the computing device 102 to receive a reference image 106 of a person and an image of target clothing 108 and generate a synthesized image 110 that depicts the person in the reference image 106 wearing the target clothing 108 instead of clothing 112 worn by the person in the reference image 106; [0049] - In some implementations, the convolutional neural networks 404 and 406 are each representative of a pre-trained VGG-16 convolutional neural network); “generating a denoising control signal during a denoising process of the try- on image generation model based on the second image, the third images, and the fourth image” (Gou, 3.2 Diffusion Model - Specifically, we add the warped clothes to the inpainting image 𝐼𝑎 as input for each denoising step of the diffusion model; Figure 2 - The overview of our method. First, we obtain the segmentation result 𝑆𝑝, dense pose 𝑃 and clothes-agnostic 𝐼𝑎 of the target person image 𝐼𝑝 through preprocessing. The clothes image 𝐼𝑐 is roughly aligned to the person by the warping network. Then, we combine 𝐼𝑎 and 𝐼𝑐 to obtain 𝐼’0 and add noise to get 𝐼′𝑡 as input to the diffusion model, and the final output 𝐼^ produced by denoising 𝐼’𝑡); and “generating the try-on image of the target model wearing the item of clothing in the corresponding pose through the try-on image generation model” (Ayush, [0029] - Using the composition mask, the try-on module 120 fuses together the warped clothing and the image of the person wearing the warped clothing to improve alignment and eliminate blurriness from the resulting image of the person wearing the warped target clothing; [0045] - The trained geometric matching module 118 is thus configured to generate the warped target clothing 210, given the cloth-agnostic person representation 202 and the target clothing 108, and communicate the warped target clothing 210 to the try-on module 120. Although the warped target clothing 210 is roughly aligned with the body shape 206 of the person depicted in the reference image 106, the try-on module 120 represents functionality of the IBVTON system 104 to fuse the warped target clothing 210 together with the cloth-agnostic person representation 202 to output a synthesized image, such as an image of a person wearing warped target clothing 218). Thus, it would have been obvious, in view of Gou, to configure Ayush’s method as claimed by using a defuse model to generate the try-on image produced by denoising. The motivation is to improve the visual representation of try-on cloth appearance.
Claim 6 adds into claim 5 “performing visual feature extraction on the second image to obtain a feature vector of the item of clothing to be tried on; applying noise addition to the first image using a noise addition module of the pre-trained try-on image generation model to generate a noisy image” (Gou, page 5, column 2, Reconstruction Branch - The reconstruction branch performs similarly to the vanilla diffusion model, which generates realistic images by learning the reverse diffusion process. For the target image 𝐼0, we first perform a forward diffusion process, 𝑞(·), on it, and gradually add noise to it according to the Markov chain and convert it into a Gaussian distribution); “performing multi-time-step denoising on the noisy image using a generation network of the try-on image generation model to obtain latent space vectors output at each time step” (Gou, Reconstruction Branch - To reduce computational complexity, we employ an latent diffusion model, which embeds the images from image space to latent space through a pretrained encoder E and reconstructs images by a pretrained decoder D… During denoising, an enhanced Diffusion UNet is used to predict a denoised variant of their input. The global condition 𝑐 extracted from 𝐼𝑐 is injected into diffusion UNet through cross attention mechanism), “wherein the multi-time-step denoising comprises: concatenating the third images, the fourth image, and the latent space vector output from a previous time step along an image channel dimension to obtain a concatenated vector; performing feature interaction between the concatenated vector and the feature vector based on an attention mechanism to obtain the latent space vector output for a current time step” (Ayush, [0045] - Given the concatenated input of the cloth-agnostic person representation 202 and the warped target clothing 210, the try-on module 120 simultaneously renders a person image and predicts a composition mask and fuses together the rendered person image with the warped target clothing 210 using the composition mask to output the person wearing warped target clothing 218; [0060] - The expansive path of the encoder-decoder network 502 represents an upsampling of the feature map output by the contracting path, followed by performing up-convolutions to reduce the number of feature channels and crop the feature map to account for loss. A final layer of the encoder-decoder network 502 may be used to map feature resulting feature vectors to a desired number of image class labels (e.g., classification labels assigned to each pixel of the predicted composition mask 504 and the person image 506; Gou, Reconstruction Branch - To reduce computational complexity, we employ an latent diffusion model, which embeds the images from image space to latent space through a pretrained encoder E and reconstructs images by a pretrained decoder D; 3.2 Diffusion Model - Similar to the reconstruct branch, we first employ the encoder Ɛ to extract 𝑧′0 from 𝐼′0 by 𝑧′0 = E(𝐼′0), and then perform forward process on 𝑧′0 to get 𝑧′𝑡 . Then, {𝑧′𝑡, 𝑧𝑙𝑐,𝑚} is fed into the diffusion model for denoising. When the noise 𝜖^ predicted by the model is obtained, according to the Eq.6, we can obtain the refined latent variable 𝑧^ after denoising by reverse the equation and the final image result can be recovered such that 𝐼^ = D(z^)); and generating the try-on image of the target model wearing the item of clothing in the corresponding pose based on the latent space vector output at a specified time step” (Gu, Figure 3 - The training pipeline of the diffusion model in our method. There are two branches in our training pipeline: the reconstruction branch above and the refinement branch below… For better visualization, we show the images corresponding to the variables in the latent space). Thus, it would have been obvious, in view of Gou, to configure Ayush’s method as claimed by using a defuse model to generate the try-on image produced by adding noise and denoising. The motivation is to improve the visual representation of try-on cloth appearance.
Claims 7 and 8 claim a non-transitory computer-readable storage medium and an electronic device based on the method of claim 1; therefore, they are rejected under a similar rationale.
As per claim 9, Ayush teaches the claimed “method for training a try-on image generation model,” comprising: obtaining an image pair including a sample model image and a sample clothing image, wherein the clothing in the sample model image and the sample clothing image within the same image pair is the same but differs in shape” (Ayush, [0024] - The IBVTON system 104 represents functionality of the computing device 102 to receive a reference image 106 of a person and an image of target clothing 108 and generate a synthesized image 110 that depicts the person in the reference image 106 wearing the target clothing 108 instead of clothing 112 worn by the person in the reference image 106); “using the image pair as input images for the try-on image generation model to be trained and controlling the try-on image generation model to generate a try-on image” (Ayush, [0045] - As described in further detail below with respect to FIG. 6, the IBVTON system 104 is configured to train the try-on module 120 to minimize a discrepancy between the image of the person wearing warped target clothing 218 and the ground truth warped cloth 216 corresponding to the warped target clothing 210); “iteratively optimizing the try-on image generation model to be trained with an objective of aligning the generated try-on image with the sample model image, thereby obtaining a trained try-on image generation model” (Ayush, [0075]-[0079] - FIG. 8 depicts a procedure 800 in an example implementation of training a geometric matching module to output a warped cloth for a target piece of clothing using multi-scale patch adversarial loss in accordance with the techniques described herein. A plurality of image pairs are received, with each image pair including a reference image depicting a person wearing an article of clothing and a product image depicting the article of clothing without being worn by the person (block 802)… In this manner, the geometric matching module 118 is trained to output visually realistic images of warped target clothing that roughly align with the cloth-agnostic person representation 202, for subsequent processing by the try-on module 120 to generate a synthesized image 110 depicting a person wearing warped target clothing 218); “wherein controlling the try-on image generation model to be trained to generate a try-on image comprises: performing image processing on the sample model image to obtain a plurality of model information images expressing different information” (Ayush, [0026] - Given the reference image 106, the representation module 114 is configured to generate a cloth-agnostic person representation of the person depicted in the reference image 106. The cloth-agnostic person representation includes information describing a body shape, a body pose, certain regions (e.g., face and hair regions) that describe an appearance of the person in the reference image 106 independent of the clothing 112); “performing clothing deformation processing on the clothing in the sample clothing image based on the sample model image to obtain a clothing deformation image, wherein a clothing shape in the clothing deformation image aligns with a pose of the model in the sample model image” (Ayush, [0019] - The warped version of the target clothing is deformed using a learnable thin-plate spline transformation that is output by a convolutional neural network to align the target clothing with the attributes described by the clothing-agnostic person representation. Parameters for the convolutional neural network are trained from paired images of target clothing and a person wearing the target clothing, without requiring an indication of explicit corresponding interest points in the images of the target clothing and the person wearing the target clothing; [0028] - Being trained with the multi-scale patch adversarial loss, the geometric matching module 118 is representative of functionality to generate a warped version of the target clothing 108 based on the clothing-agnostic person representation of the person depicted in the reference image 106. By training the geometric matching module 118 with multi-scale patch adversarial loss, the IBVTON system 104 is able to generate the warped version of the target clothing 108 via elastic transformation to fit the body shape and pose of the person depicted in the reference image 106 while maintaining texture details of the target clothing 108, such as material properties, logos, embroidery, and so forth; [0045] - The trained geometric matching module 118 is thus configured to generate the warped target clothing 210, given the cloth-agnostic person representation 202 and the target clothing 108, and communicate the warped target clothing 210 to the try-on module 120. Although the warped target clothing 210 is roughly aligned with the body shape 206 of the person depicted in the reference image 106, the try-on module 120 represents functionality of the IBVTON system 104 to fuse the warped target clothing 210 together with the cloth-agnostic person representation 202 to output a synthesized image, such as an image of a person wearing warped target clothing 218). Ayush suggests the step of ““controlling the try-on image generation model to be trained to generate the try-on image based on the model information images, the clothing deformation image, the sample model image, and the sample clothing image” in the teaching of fusing together the warped clothing and the image of the person wearing the warped clothing (Ayush, [0029] - Using the composition mask, the try-on module 120 fuses together the warped clothing and the image of the person wearing the warped clothing to improve alignment and eliminate blurriness from the resulting image of the person wearing the warped target clothing; [0045] - The trained geometric matching module 118 is thus configured to generate the warped target clothing 210, given the cloth-agnostic person representation 202 and the target clothing 108, and communicate the warped target clothing 210 to the try-on module 120. Although the warped target clothing 210 is roughly aligned with the body shape 206 of the person depicted in the reference image 106, the try-on module 120 represents functionality of the IBVTON system 104 to fuse the warped target clothing 210 together with the cloth-agnostic person representation 202 to output a synthesized image, such as an image of a person wearing warped target clothing 218) (see also Gou, Figure 2 - The overview of our method. First, we obtain the segmentation result 𝑆𝑝, dense pose 𝑃 and clothes-agnostic 𝐼𝑎 of the target person image 𝐼𝑝 through preprocessing. The clothes image 𝐼𝑐 is roughly aligned to the person by the warping network. Then, we combine 𝐼𝑎 and 𝐼𝑐 to obtain 𝐼’0 and add noise to get 𝐼′𝑡 as input to the diffusion model, and the final output 𝐼^ produced by denoising 𝐼’𝑡). Thus, it would have been obvious, in view of Gu, to configure Ayush’s method as claimed by fusing together the warped clothing and the image of the person wearing the warped clothing. The motivation is to improve the visual representation of try-on cloth appearance.
Claim 10 adds into claim 9 ”using the sample model image and the sample clothing image as input images for the try- on image generation model to be trained” (Ayush, [0024] - The IBVTON system 104 represents functionality of the computing device 102 to receive a reference image 106 of a person and an image of target clothing 108 and generate a synthesized image 110 that depicts the person in the reference image 106 wearing the target clothing 108 instead of clothing 112 worn by the person in the reference image 106; [0049] - In some implementations, the convolutional neural networks 404 and 406 are each representative of a pre-trained VGG-16 convolutional neural network); “generating a denoising control signal during a denoising process of the try-on image generation model based on the sample clothing image, the model information images, and the clothing deformation image” (Gou, 3.2 Diffusion Model - Specifically, we add the warped clothes to the inpainting image 𝐼𝑎 as input for each denoising step of the diffusion model; Figure 2 - The overview of our method. First, we obtain the segmentation result 𝑆𝑝, dense pose 𝑃 and clothes-agnostic 𝐼𝑎 of the target person image 𝐼𝑝 through preprocessing. The clothes image 𝐼𝑐 is roughly aligned to the person by the warping network. Then, we combine 𝐼𝑎 and 𝐼𝑐 to obtain 𝐼’0 and add noise to get 𝐼′𝑡 as input to the diffusion model, and the final output 𝐼^ produced by denoising 𝐼’𝑡); and “generating the try-on image through the try-on image generation model to be trained” (Ayush, [0029] - Using the composition mask, the try-on module 120 fuses together the warped clothing and the image of the person wearing the warped clothing to improve alignment and eliminate blurriness from the resulting image of the person wearing the warped target clothing; [0045] - The trained geometric matching module 118 is thus configured to generate the warped target clothing 210, given the cloth-agnostic person representation 202 and the target clothing 108, and communicate the warped target clothing 210 to the try-on module 120. Although the warped target clothing 210 is roughly aligned with the body shape 206 of the person depicted in the reference image 106, the try-on module 120 represents functionality of the IBVTON system 104 to fuse the warped target clothing 210 together with the cloth-agnostic person representation 202 to output a synthesized image, such as an image of a person wearing warped target clothing 218). Thus, it would have been obvious, in view of Gou, to configure Ayush’s method as claimed by using a defuse model to generate the try-on image produced by denoising. The motivation is to improve the visual representation of try-on cloth appearance.
Claim 11 adds into claim 9 “obtaining a first image pair including a sample model image and a sample clothing image, wherein the sample clothing image is a flat-laid clothing image” (Gou, Figure 2 – The clothes image 𝐼𝑐 is the images of flat-laid clothing image; [0075] - Both the reference image 106 and the target clothing 108 are representative of two-dimensional images); “performing clothing deformation processing on the sample clothing image in the same first image pair based on the sample model image to obtain a clothing deformation image corresponding to the sample model image” (Ayush, [0019] - The warped version of the target clothing is deformed using a learnable thin-plate spline transformation that is output by a convolutional neural network to align the target clothing with the attributes described by the clothing-agnostic person representation. Parameters for the convolutional neural network are trained from paired images of target clothing and a person wearing the target clothing, without requiring an indication of explicit corresponding interest points in the images of the target clothing and the person wearing the target clothing; [0028] - Being trained with the multi-scale patch adversarial loss, the geometric matching module 118 is representative of functionality to generate a warped version of the target clothing 108 based on the clothing-agnostic person representation of the person depicted in the reference image 106. By training the geometric matching module 118 with multi-scale patch adversarial loss, the IBVTON system 104 is able to generate the warped version of the target clothing 108 via elastic transformation to fit the body shape and pose of the person depicted in the reference image 106 while maintaining texture details of the target clothing 108, such as material properties, logos, embroidery, and so forth; [0045] - The trained geometric matching module 118 is thus configured to generate the warped target clothing 210, given the cloth-agnostic person representation 202 and the target clothing 108, and communicate the warped target clothing 210 to the try-on module 120. Although the warped target clothing 210 is roughly aligned with the body shape 206 of the person depicted in the reference image 106, the try-on module 120 represents functionality of the IBVTON system 104 to fuse the warped target clothing 210 together with the cloth-agnostic person representation 202 to output a synthesized image, such as an image of a person wearing warped target clothing 218; Gou, 3.2 Diffusion Model - Specifically, we add the warped clothes to the inpainting image 𝐼𝑎 as input for each denoising step of the diffusion model; Figure 2 - The overview of our method. First, we obtain the segmentation result 𝑆𝑝, dense pose 𝑃 and clothes-agnostic 𝐼𝑎 of the target person image 𝐼𝑝 through preprocessing. The clothes image 𝐼𝑐 is roughly aligned to the person by the warping network. Then, we combine 𝐼𝑎 and 𝐼𝑐 to obtain 𝐼’0 and add noise to get 𝐼′𝑡 as input to the diffusion model, and the final output 𝐼^ produced by denoising 𝐼’𝑡); “using the clothing deformation image as the sample clothing image and pairing it with the corresponding sample model image to construct a second image pair” (Ayush, [0042] - Returning to FIG. 2, the representation module 114 communicates the cloth-agnostic person representation 202 to the geometric matching module 118 together with the target clothing 108. The geometric matching module 118 is configured to generate warped target clothing 210, which a represents a result of warping the target cloth 108 to conform to the cloth-agnostic person representation 202. In order for the geometric matching module 118 to output visually realistic warped target clothing 210, the IBVTON system 104 is configured to train the geometric matching module 118 using multi-scale patch adversarial loss, which is computed using multi-scale patches 212 generated by the sampling module 116. The multi-scale patches 212 are representative of a collection of different patch pairs, where each patch pair includes one patch sampled from a region of a warped cloth 214 and another patch sampled from a corresponding same region of the ground truth warped cloth 216); and “integrating the first image pair and the second image pair to obtain an image pair including sample model image and sample clothing image” (Ayush, [0043]-[0045] - For each pair of a warped cloth 214 and a ground truth warped cloth 216, the sampling module 116 pseudo-randomly extracts patch pairs of varying sizes from corresponding same locations of the respective warped cloth 214 and ground truth warped cloth 216. Each of the multi-scale patches 212 are sampled from a region of the warped cloth 214 or ground truth warped cloth 216 that includes only cloth (e.g., does not include an edge of the depicted cloth or a portion that includes other artifacts not corresponding to cloth, such as portions of a person otherwise represented in the ground truth warped cloth, background portions of an image, and so forth)… The trained geometric matching module 118 is thus configured to generate the warped target clothing 210, given the cloth-agnostic person representation 202 and the target clothing 108, and communicate the warped target clothing 210 to the try-on module 120. Although the warped target clothing 210 is roughly aligned with the body shape 206 of the person depicted in the reference image 106, the try-on module 120 represents functionality of the IBVTON system 104 to fuse the warped target clothing 210 together with the cloth-agnostic person representation 202 to output a synthesized image, such as an image of a person wearing warped target clothing 218). Thus, it would have been obvious, in view of Gou, to configure Ayush’s method as claimed by using the sample clothing image as a flat-laid clothing image. The motivation is to improve the visual representation of try-on cloth appearance.
Claims 12 and 13 claim a non-transitory computer-readable storage medium and an electronic device based on the method of claim 9; therefore, they are rejected under a similar rationale.
As per claim 14, Ayush teaches the claimed “method for generating a try-on image, applied to a client,” comprising: “obtaining a clothing image of an item of clothing to be tried on and a user image of a current user in a target pose uploaded by the user” (Ayush, [0024] - The IBVTON system 104 represents functionality of the computing device 102 to receive a reference image 106 of a person and an image of target clothing 108 and generate a synthesized image 110 that depicts the person in the reference image 106 wearing the target clothing 108 instead of clothing 112 worn by the person in the reference image 106); “generating a try-on request based on the user image and the clothing image in response to a try-on image generation operation, and sending the try-on request to a preset server” (Ayush, [0031] - The reference image 106, the image of the target clothing 108, the synthesized image 110, and other information processed or generated by the IBVTON system 104 may be stored in storage of the computing device 102, as described in further detail below with respect to FIG. 9… For instance, the IBVTON system 104 may communicate information to remote storage 124, or directly to a different computing device, via network 126; [0081] - The computing device 902 may be, for example, a server of a service provider, a device associated with a client (e.g., a client device), an on-chip system, and/or any other suitable computing device or computing system); “obtaining a try-on image generated by the preset server in response to the try-on request, wherein the try-on image depicts the user wearing the item of clothing to be tried on in the target pose” (Ayush, [0029] - Using the composition mask, the try-on module 120 fuses together the warped clothing and the image of the person wearing the warped clothing to improve alignment and eliminate blurriness from the resulting image of the person wearing the warped target clothing); “displaying the try-on image to the current user, wherein the try-on image is generated by the preset server through the following process: performing image processing on the user image to generate a plurality of third images expressing different information” (Ayush, [0026] - Given the reference image 106, the representation module 114 is configured to generate a cloth-agnostic person representation of the person depicted in the reference image 106. The cloth-agnostic person representation includes information describing a body shape, a body pose, certain regions (e.g., face and hair regions) that describe an appearance of the person in the reference image 106 independent of the clothing 112); “performing clothing deformation processing to the item of clothing in the clothing image based on the user image to obtain a fourth image, wherein a clothing shape in the fourth image matches the target pose of the user in the user image” (Ayush, [0019] - The warped version of the target clothing is deformed using a learnable thin-plate spline transformation that is output by a convolutional neural network to align the target clothing with the attributes described by the clothing-agnostic person representation. Parameters for the convolutional neural network are trained from paired images of target clothing and a person wearing the target clothing, without requiring an indication of explicit corresponding interest points in the images of the target clothing and the person wearing the target clothing; [0028] - Being trained with the multi-scale patch adversarial loss, the geometric matching module 118 is representative of functionality to generate a warped version of the target clothing 108 based on the clothing-agnostic person representation of the person depicted in the reference image 106. By training the geometric matching module 118 with multi-scale patch adversarial loss, the IBVTON system 104 is able to generate the warped version of the target clothing 108 via elastic transformation to fit the body shape and pose of the person depicted in the reference image 106 while maintaining texture details of the target clothing 108, such as material properties, logos, embroidery, and so forth; [0045] - The trained geometric matching module 118 is thus configured to generate the warped target clothing 210, given the cloth-agnostic person representation 202 and the target clothing 108, and communicate the warped target clothing 210 to the try-on module 120. Although the warped target clothing 210 is roughly aligned with the body shape 206 of the person depicted in the reference image 106, the try-on module 120 represents functionality of the IBVTON system 104 to fuse the warped target clothing 210 together with the cloth-agnostic person representation 202 to output a synthesized image, such as an image of a person wearing warped target clothing 218). Ayush suggests the step of “generating the try-on image based on the third images, the fourth image, the user image, and the clothing image” in the teaching of the combining the warped clothing and the cloth-agnostic person representation includes information describing a body shape, a body pose, certain regions (e.g., face and hair regions) that describe an appearance of the person in the reference image 106 (Ayush, [0029] - Using the composition mask, the try-on module 120 fuses together the warped clothing and the image of the person wearing the warped clothing to improve alignment and eliminate blurriness from the resulting image of the person wearing the warped target clothing; [0045] - The trained geometric matching module 118 is thus configured to generate the warped target clothing 210, given the cloth-agnostic person representation 202 and the target clothing 108, and communicate the warped target clothing 210 to the try-on module 120. Although the warped target clothing 210 is roughly aligned with the body shape 206 of the person depicted in the reference image 106, the try-on module 120 represents functionality of the IBVTON system 104 to fuse the warped target clothing 210 together with the cloth-agnostic person representation 202 to output a synthesized image, such as an image of a person wearing warped target clothing 218) (see also Gou, Figure 2 - The overview of our method. First, we obtain the segmentation result 𝑆𝑝, dense pose 𝑃 and clothes-agnostic 𝐼𝑎 of the target person image 𝐼𝑝 through preprocessing. The clothes image 𝐼𝑐 is roughly aligned to the person by the warping network. Then, we combine 𝐼𝑎 and 𝐼𝑐 to obtain 𝐼’0 and add noise to get 𝐼′𝑡 as input to the diffusion model, and the final output 𝐼^ produced by denoising 𝐼’𝑡). Thus, it would have been obvious, in view of Gou, to configure Ayush’s method as claimed by using the images through warped cloths according to the characteristics of the captured human image. The motivation is to improve the visual representation of try-on cloth appearance.
Claim 15 adds into claim 14 “extracting skeletal pose information from the user image to obtain a third image representing the skeletal pose of the current user in the user image; and/or performing occlusion processing on a target image region of the user image to obtain a third image with clothing expression removed; and/or performing masking processing on the user image to obtain a third image representing a redrawn region” (Ayush, [0036] - The cloth-agnostic person representation 202 is representative of information used by the IBVTON system 104 to deform target clothing in order to generate an image of the person wearing warped target clothing 218, such as synthesized image 110 illustrated in FIG. 1. Specifically, the cloth-agnostic person representation 202 includes a pose heatmap 204, a body shape 206, and one or more reserved regions 208 for a person depicted in the reference image 106, which collectively represent a set of features used as a prior to constrain the IBVTON system 104's process of generating the synthesized image 110; [0037] - In some implementations, the representation module 114 generates the pose heatmap 204 using a known pose estimator, such as the pose estimator described in Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, Realtime Multiperson 2D Pose Estimation Using Part Affinity Fields (CVPR, 2017). The pose heatmap 204 may comprise coordinates representing a number of keypoints that in turn represent the pose of the person depicted in reference image 106).
Claim 16 adds into claim 15 “wherein the target image region includes a wearing region that matches the item of clothing to be tried on” (Ayush, [0038] - The body shape 206 is representative of information that describes various regions (e.g., arms, legs, torso, etc.) of a body of the person depicted in reference image 106. In order to generate the body shape 206, the representation module 114 is configured to implement a human parser to compute a human segmentation map, where different regions represent different corresponding body parts. In some implementations, the representation module 114 implements a human parser, using known techniques. The representation module 114 then converts the segmentation map derived using the human parser to a single channel binary mask, where “ones” in the binary mask indicate human body regions (except for face and hair) and “zeroes” in the binary mask represent regions of the reference image 106 that do not depict human body parts. The representation module 114 then down-samples the binary mask to a lower resolution, which assists in avoiding artifacts that otherwise might occur when portions of a human body as indicated by the body shape 206 and the target clothing conflict. This down-sampled binary mask is then output by the representation module 114 as the body shape 206).
Claim 17 adds into claim 14 “obtaining a fifth image representing a skeletal pose of the current user in the user image, and a sixth image representing a body shape and target pose of the current user in the user image” (Ayush, [0018] - In such a conditional image generation approach, given two images, one of a person and the other of target clothing, a new image is synthesized with the goal of retaining the original body shape and pose of the person while maintaining characteristics of the clothing product, such as texture, logo, text, and so forth. Some conventional approaches deform target clothing to align with the person image using conditional image generation techniques, such as image-to-image translation); “performing masking processing on the item of clothing in the clothing image to obtain a seventh image” (Ayush, [0020] - The convolutional neural network is described herein in the context of a geometric matching module, which takes the clothing-agnostic person representation and image of target clothing as inputs, and generates a pose-coherent warped version of the target clothing as well as a composition mask that indicates details of the target clothing to be retained in the synthesized image. The composition mask enables output of a smooth synthesized image and accommodates transformations for different body shapes and body poses, such that the synthesized image output by the IBVTON system retains a personal identity of the person depicted in the reference image); “using the fifth image and the sixth image as first branch inputs to a pre-trained appearance flow deformation network, and using the clothing image and the seventh image as second branch inputs to the appearance flow deformation network” (Ayush, [0019] - In order to generate a synthesized image of the person depicted in the reference image wearing the target clothing instead of the clothing originally depicted in the reference image, the IBVTON system generates a cloth-agnostic person representation of the person depicted in the reference image and uses the cloth-agnostic person representation as a model for generating a warped version of the target clothing in a manner that preserves visual characteristics of both the target clothing and the person depicted in the reference image. The warped version of the target clothing is deformed using a learnable thin-plate spline transformation that is output by a convolutional neural network to align the target clothing with the attributes described by the clothing-agnostic person representation. Parameters for the convolutional neural network are trained from paired images of target clothing and a person wearing the target clothing, without requiring an indication of explicit corresponding interest points in the images of the target clothing and the person wearing the target clothing); and “performing clothing deformation processing on the item of clothing to be tried on using the appearance flow deformation network to obtain the fourth image” (Ayush, [0036]-[0037] - The cloth-agnostic person representation 202 is representative of information used by the IBVTON system 104 to deform target clothing in order to generate an image of the person wearing warped target clothing 218, such as synthesized image 110… The pose heatmap 204 is representative of information that describes a body pose of the person depicted in the reference image 106, which consequently affects a deformation of the target clothing to be represented in the image of the person wearing warped target clothing 218).
Claim 18 adds into claim 14 “wherein generating the try-on image based on the third images, the fourth image, the user image, and the clothing image: using the clothing image and the user image as input images for a pre-trained try-on image generation model” (Ayush, [0024] - The IBVTON system 104 represents functionality of the computing device 102 to receive a reference image 106 of a person and an image of target clothing 108 and generate a synthesized image 110 that depicts the person in the reference image 106 wearing the target clothing 108 instead of clothing 112 worn by the person in the reference image 106; [0049] - In some implementations, the convolutional neural networks 404 and 406 are each representative of a pre-trained VGG-16 convolutional neural network); “generating a denoising control signal during a denoising process of the try-on image generation model based on the clothing image, the third images, and the fourth image” (Gou, 3.2 Diffusion Model - Specifically, we add the warped clothes to the inpainting image 𝐼𝑎 as input for each denoising step of the diffusion model; Figure 2 - The overview of our method. First, we obtain the segmentation result 𝑆𝑝, dense pose 𝑃 and clothes-agnostic 𝐼𝑎 of the target person image 𝐼𝑝 through preprocessing. The clothes image 𝐼𝑐 is roughly aligned to the person by the warping network. Then, we combine 𝐼𝑎 and 𝐼𝑐 to obtain 𝐼’0 and add noise to get 𝐼′𝑡 as input to the diffusion model, and the final output 𝐼^ produced by denoising 𝐼’𝑡); and “generating the try-on image of the current user wearing the item of clothing in the target pose through the try-on image generation model” (Ayush, [0029] - Using the composition mask, the try-on module 120 fuses together the warped clothing and the image of the person wearing the warped clothing to improve alignment and eliminate blurriness from the resulting image of the person wearing the warped target clothing; [0045] - The trained geometric matching module 118 is thus configured to generate the warped target clothing 210, given the cloth-agnostic person representation 202 and the target clothing 108, and communicate the warped target clothing 210 to the try-on module 120. Although the warped target clothing 210 is roughly aligned with the body shape 206 of the person depicted in the reference image 106, the try-on module 120 represents functionality of the IBVTON system 104 to fuse the warped target clothing 210 together with the cloth-agnostic person representation 202 to output a synthesized image, such as an image of a person wearing warped target clothing 218). Thus, it would have been obvious, in view of Gou, to configure Ayush’s method as claimed by using a defuse model to generate the try-on image produced by denoising. The motivation is to improve the visual representation of try-on cloth appearance.
Claim 19 adds into claim 18 “performing visual feature extraction on the clothing image to obtain a feature vector of the item of clothing to be tried on; applying noise addition to the user image using a noise addition module of the pre-trained try-on image generation model to generate a noisy image” (Gou, page 5, column 2, Reconstruction Branch - The reconstruction branch performs similarly to the vanilla diffusion model, which generates realistic images by learning the reverse diffusion process. For the target image 𝐼0, we first perform a forward diffusion process, 𝑞(·), on it, and gradually add noise to it according to the Markov chain and convert it into a Gaussian distribution); “performing multi-time-step denoising on the noisy image using a generation network of the try-on image generation model to obtain latent space vectors output at each time step” (Gou, Reconstruction Branch - To reduce computational complexity, we employ an latent diffusion model, which embeds the images from image space to latent space through a pretrained encoder E and reconstructs images by a pretrained decoder D… During denoising, an enhanced Diffusion UNet is used to predict a denoised variant of their input. The global condition 𝑐 extracted from 𝐼𝑐 is injected into diffusion UNet through cross attention mechanism), “wherein the multi-time-step denoising comprises: concatenating the third images, the fourth image, and the latent space vector output from a previous time step along an image channel dimension to obtain a concatenated vector; performing feature interaction between the concatenated vector and the feature vector based on an attention mechanism to obtain the latent space vector output for a current time step” (Ayush, [0045] - Given the concatenated input of the cloth-agnostic person representation 202 and the warped target clothing 210, the try-on module 120 simultaneously renders a person image and predicts a composition mask and fuses together the rendered person image with the warped target clothing 210 using the composition mask to output the person wearing warped target clothing 218; [0060] - The expansive path of the encoder-decoder network 502 represents an upsampling of the feature map output by the contracting path, followed by performing up-convolutions to reduce the number of feature channels and crop the feature map to account for loss. A final layer of the encoder-decoder network 502 may be used to map feature resulting feature vectors to a desired number of image class labels (e.g., classification labels assigned to each pixel of the predicted composition mask 504 and the person image 506; Gou, Reconstruction Branch - To reduce computational complexity, we employ an latent diffusion model, which embeds the images from image space to latent space through a pretrained encoder E and reconstructs images by a pretrained decoder D; 3.2 Diffusion Model - Similar to the reconstruct branch, we first employ the encoder Ɛ to extract 𝑧′0 from 𝐼′0 by 𝑧′0 = E(𝐼′0), and then perform forward process on 𝑧′0 to get 𝑧′𝑡 . Then, {𝑧′𝑡, 𝑧𝑙𝑐,𝑚} is fed into the diffusion model for denoising. When the noise 𝜖^ predicted by the model is obtained, according to the Eq.6, we can obtain the refined latent variable 𝑧^ after denoising by reverse the equation and the final image result can be recovered such that 𝐼^ = D(z^)); and “generating the try-on image of the current user wearing the item of clothing in the target pose based on the latent space vector output at a specified time step” (Gou, Figure 3 - The training pipeline of the diffusion model in our method. There are two branches in our training pipeline: the reconstruction branch above and the refinement branch below… For better visualization, we show the images corresponding to the variables in the latent space). Thus, it would have been obvious, in view of Gou, to configure Ayush’s method as claimed by using a defuse model to generate the try-on image produced by adding noise and denoising. The motivation is to improve the visual representation of try-on cloth appearance.
Claim 20 claims a non-transitory computer-readable storage medium based on the method of claim 14; therefore, it is rejected under a similar rationale.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHU K NGUYEN whose telephone number is (571)272-7645. The examiner can normally be reached M-F 8-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHU K NGUYEN/Primary Examiner, Art Unit 2616
Prosecution Insights