DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on/after Mar. 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Response to Arguments
Applicant’s arguments, see pp. 10-13, filed 30 March 2026, with respect to the rejection of claim 10 under 35 U.S.C. § 102 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Sarkar et al. (“HumanGAN: A Generative Model of Human Images”, pub. 2021). Please see the Office action below for explanation regarding the rationale(s) for the rejection(s) of the newly-amended claims.
Applicant’s arguments, see pp. 13-17, filed 30 March 2026, with respect to the rejection of claim 15 under 35 U.S.C. § 102 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Sarkar et al. (“HumanGAN: A Generative Model of Human Images”, published 2021) and Lee et al. (U.S. PG-PUB 2020/0074707). Please see the Office action below for explanation regarding the rationale(s) for the rejection(s) of the newly-amended claims.
Applicant’s arguments, see pp. 18-19, filed 30 March 2026, with respect to the rejection of claim 1 under 35 U.S.C. § 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Sarkar et al. (“HumanGAN: A Generative Model of Human Images”, published 2021). Please see the Office action below for further explanation regarding the rationale(s) for the rejection(s) of the newly-amended claims.
The Examiner notes that the GONG and the HAN references are not relied upon in this Office action.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 4, and 7-14 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Sarkar et al. (“HumanGAN: A Generative Model of Human Images”, pub. 2021, 'SARKAR').
Regarding claim 1, SARKAR discloses a method comprising:
PNG
media_image1.png
983
638
media_image1.png
Greyscale
PNG
media_image2.png
402
1533
media_image2.png
Greyscale
receiving … images comprising first/second images depicting first/second body parts (SARKAR; FIG. 6; p. 265, left col.; § 4.3 ‘Part-Based Sampling’; “We next evaluate our method for part-based sampling– the ability to produce different plausible renderings of a body part (e.g., head) while keeping the rest of the body same … We compute the aforementioned metrics for the following parts: “Head”, “Upper body” and “Lower body” for 2500 generated images … A suitable method for part sampling should generate diverse semantically meaningful parts renderings without changing the rest of the body.”);
PNG
media_image3.png
635
1334
media_image3.png
Greyscale
warping the first/second images to obtain first/second warped images, respectively (SARKAR; FIG. 2; p. 260; § 3.1 ‘Our Architecture’; “In the training stage, we take pairs of images (Is, It) of the same person (but in different poses) as input. … In the 1st step, we extract SMPL UV texture map Ts from the input image Is using the Dense-Pose correspondences. In the 2nd step, we use an encoding function E to map the human appearance Ts of the source image to the parameters of the distribution of the latent vector. In the 3rd step, we sample z from the estimated distribution of the source appearance. Given a target pose Pt, we warp the encoded latent vector z to a noise image Zt. In the 4th step, we decode the warped Zt to a realistic image I′t by a high-fidelity generator network.”);
PNG
media_image4.png
608
735
media_image4.png
Greyscale
encoding, using a texture encoder, the first/second warped images to obtain first/second texture embeddings, respectively (SARKAR; p. 261, right col.; § 3.3; “We warp z with the conditioning pose P … The appearance z can also be encoded from an input image by using the encoder on its partial texture map T, i.e., z = μ, where μ, σ = E(T).”);
PNG
media_image5.png
325
731
media_image5.png
Greyscale
PNG
media_image6.png
393
1533
media_image6.png
Greyscale
combining the first/second texture embeddings to obtain a combined texture embedding by combining the first/second texture embeddings with first/second feature selection masks, respectively, to obtain a first/second masked texture embeddings, and adding the first/second masked texture embeddings (SARKAR; FIG. 8; p. 265, left col.; § 4.3 ‘Part Sampling’; “We next evaluate our method for part-based sampling– the ability to produce different plausible renderings of a body part (e.g., head) while keeping the rest of the body same … We compute the following two metrics for a given [body] part p: 1) Variation–Part: mean pairwise L1 distance between the samples in the masked region (by Dense-Pose) of the part p normalized by the masked area [‘feature selection mask’]. 2) Variation–Rest: mean pairwise L1 distance between the samples in the masked region of all body parts [‘feature selection mask’] excluding p. We compute the aforementioned metrics for the following parts: “Head”, “Upper body” and “Lower body” for 2500 generated images and provide our result in Table 3. A suitable method for part sampling should generate diverse semantically meaningful parts renderings without changing the rest of the body.” p. 265, right col., 1st paragraph re-presented above.; The Examiner regards the ‘noise embeddings’ of ‘body parts’ and ‘garment parts’ to be analogous to first/second texture embeddings, which are depicted above in FIG. 8. The Examiner notes that in each of the three scenarios depicted in FIG. 8, a ‘Garment’ texture embedding may be transferred unto a ‘Body’ texture embedding to create the ‘Garment Transfer’ combined texture embedding) …
PNG
media_image7.png
288
731
media_image7.png
Greyscale
at a plurality of scales (SARKAR; p. 261, right col.; § 3.2 ‘Training Details’; [See screen-capture above.]); and
generating, using a generative decoder, a composite image depicting the first/second body parts based on the combined texture embedding (SARKAR; p. 261, left. Col.; “Decoding to a Photo-Realistic Image [‘composite image’]. The warped noise image in the target-pose Zt with the noise vectors correctly aligned with the body parts in the target pose, is used as an input to a generator network [‘generative decoder’] … The generator and the warping module act as the conditional decoder …”).
PNG
media_image2.png
402
1533
media_image2.png
Greyscale
Regarding claim 2, SARKAR discloses the method of claim 1, further comprising:
obtaining … body images depicting different bodies (SARKAR; FIG. 6); and
segmenting the … body images to obtain the … images (SARKAR; FIG. 6; p. 264; § 4.3 ‘Part-Based Sampling’; “When multiple elementary Dense-Pose parts (e.g., left head, right head) correspond to one logical body part for sampling (e.g., head), we sample noise in all the elementary part vectors.”).
Regarding claim 4, SARKAR discloses the method of claim 1, further comprising:
obtaining a target pose, wherein the warping is based on the target pose (SARKAR; p. 260, caption of FIG. 2; “The target pose Pt is used to warp and broadcast the latent vectors to the corresponding parts in the target image to create a noise image Zt. Finally, the generator converts Zt to a realistic image” p. 261, left col.; “Decoding to a Photo-Realistic Image. The warped noise image in the target-pose Zt with the noise vectors correctly aligned with the body parts in the target pose, is used as an input to a generator network … The generator and the warping module act as the conditional decoder …”).
PNG
media_image8.png
719
1533
media_image8.png
Greyscale
Regarding claim 7, SARKAR discloses the method of claim 1, further comprising:
obtaining … input poses (SARKAR; FIG. 2, ‘Extracted Pose (Ps)’) corresponding to the … images (SARKAR; FIG. 2, ‘Source Image (Is)’), respectively, wherein the composite image (SARKAR; FIG. 2, ‘Generated Image (I’t)’) is generated based on the … input poses (SARKAR; FIG. 2, ‘Target Pose (Pt)’).
Regarding claim 8, SARKAR discloses the method of claim 7, further comprising:
encoding the … input poses to obtain … pose embeddings, wherein the composite image is generated based on the … pose embeddings (SARKAR; FIG. 2; p. 261, left col.; “Warping Latent Space. … we sample a latent code from the predicted distribution of the encoded appearance [‘pose embedding’], zs ∼ E(Ts) ≡ N (μs, σs). Given the noise vector zs and a target pose Pt, we … reconstruct a realistic-image I′t [‘composite image’] with the appearance encoded in zs ∈ RM×N and pose from Pt.” [The Examiner notes that FIG. 2 depicts an ‘Encoder’ which indirectly takes as input the ‘Extracted Pose’ which is derived from the ‘Source Image’. The Examiner asserts that this ‘Extracted Pose’ is analogous to an input pose.]).
Regarding claim 9, SARKAR discloses the method of claim 8, further comprising:
PNG
media_image9.png
695
733
media_image9.png
Greyscale
combining each of the … pose embeddings (SARKAR; p. 265, left col.; § 4.3 ‘Part-Based Sampling’; “We next evaluate our method for part-based sampling– the ability to produce different plausible renderings of a body part (e.g., head) while keeping the rest of the body same. To this end, we vary the vector z[k] … corresponding to the part k, and keep the rest of the noise vector z[j] … fixed, and perform the decoding on a given pose.”) with a corresponding feature selection mask to obtain … masked pose embeddings (SARKAR; Fig. 8; p. 265, left col.; § 4.3 ‘Part-Based Sampling’; “We compute the following two metrics for a given part p: 1) Variation–Part: mean pairwise L1 distance between the samples in the masked region (by Dense-Pose) of the part p normalized by the masked area. 2) Variation–Rest: mean pairwise L1 distance between the samples in the masked region of all body parts excluding p. We compute the aforementioned metrics for the following parts: “Head”, “Upper body” and “Lower body” for 2500 generated images …”), wherein the composite image is generated based on the … masked pose embeddings (SARKAR; p. 265, right col.; “Using part-specific latent vectors allows us to naturally perform ‘garment transfer’ between two images representing the body and garments. … we first encode the appearance of both the body image Ib and garment image Ig, in their part-based noise vectors zb and zg … We then construct a new noise embedding that comprises of the body parts zb[p] | p ∈ Body, and garment parts zg[p] | p ∈ Garments of the two noise embeddings, and use it in the generator for the final output …”).
Regarding claim 10, SARKAR discloses a method comprising:
obtaining training data including first/second images depicting first/second body parts, and a ground truth composite image (SARKAR; FIG. 2; p. 260; § 3.1; “In the training stage, we take … images (Is, It) of the same person (but in different poses) as input.” [The Examiner asserts that the ‘pair of images’ of FIG. 2 constitute first/second images depicting first/second body parts, as the images depict a same subject in different poses. In the example of FIG. 2, the subject is captured in two poses, wherein the ‘Source Image’ {Is} is cropped above the subject’s knees, and the ‘Target Image’ {It} shows the subject with her entire body visible {uncropped}, meaning that at least her knees/lower legs/feet are different body parts from those parts in the ‘Source Image’. The Examiner further asserts that, at least in the example of FIG. 2, the ‘Target Image’ also represents a ground truth composite image, as it exhibits the subject as a captured image {ground truth} in her entirety {composite}.] § 3.2 ‘Training Details’; p. 261); and
training, using the training data, an image generation network to generate a composite image depicting … body parts based on … first/second input images (SARKAR; p. 261, left col.; § 3.1; “Decoding to a Photo-Realistic Image. The warped noise image in the target pose Zt with the noise vectors correctly aligned with the body parts in the target pose, is used as an input [‘training’] to a generator network G(·). The generator and the warping module act as the conditional decoder of our pipeline. We use the high-fidelity generator from Pix2PixHD …”), wherein the generating of the composite image comprises: ([The remaining limitations are repeated nearly verbatim from those recited in claim 1.]).
Regarding claim 11, SARKAR discloses the method of claim 10, further comprising:
obtaining … posed images corresponding to the ground truth image (SARKAR; FIG. 2; p. 260; § 3.1; “In the training stage, we take … images (Is, It) of the same person (but in different poses) as input.” [The Examiner asserts that, at least in the example of FIG. 2, the ‘Target Image’ represents a ground truth composite image, as it exhibits the subject as a captured image {ground truth} in her entirety {composite}. The ‘Source Image’ represents a differently posed image, which corresponds to the ground truth image as it depicts the same subject in the same garments in a slightly different pose and with different cropping applied to the overall image.]); and
segmenting the … posed images to obtain the first/second images (SARKAR; p. 260, right Col.; “Extracting Appearance. We use a UV texture map of the SMPL surface model to represent the subject’s appearance in the input image [‘posed image’]. The pixels of the input image Is are transformed into the UV space through a mapping predicted by Dense-Pose RCNN. The pretrained network trained on COCO-Dense-Pose dataset predicts 24 body segments and their part-specific UV coordinates of SMPL model. For easier mapping, the 24 part-specific UV maps are combined to form a single normalized UV texture map Ts in the format provided in SURREAL dataset. This normalized (partial) texture map provides us with a pose-independent appearance encoding of the subject that is located spatially according to the body parts. The 24-part segments in the texture map also provide us the placeholder for part-based noise sampling, i.e., in our case, the number of body parts M=24.”).
Regarding claim 12, SARKAR discloses the method of claim 10, further comprising:
generating the composite image based on the first/second images (SARKAR; p. 260; FIG. 2; [The Examiner asserts that the composite image is analogous to the ‘Generated Image’, which is based on both the ‘Source Image’, which is subdivided into constituent body parts in the ‘Partial Texture-Map’, as well as the ‘Target Image’. The Examiner notes that the ‘Source Image’ captures less than the subject’s entire body, whereas the ‘Target Image’ captures her entire body. Therefore, the images comprise differing sets of body parts.]); and
PNG
media_image10.png
253
731
media_image10.png
Greyscale
PNG
media_image11.png
104
730
media_image11.png
Greyscale
comparing the composite image to the ground truth image, wherein the training is based on the comparison (SARKAR; p. 260; FIG. 2; [The Examiner asserts that the ‘Target Image’ is analogous to a ground truth image, as it is a captured image of the subject’s entire body. The bi-directional arrows linking the ‘Target Image’/ground truth image to the ‘Generated Image’/composite image indicate a training … based on the comparison between the two images.]; p. 261; [See screen-captures above.]).
Regarding claim 13, SARKAR discloses the method of claim 12, further comprising:
PNG
media_image12.png
606
1532
media_image12.png
Greyscale
identifying a target pose for the ground truth image, wherein the composite image is generated based on the target pose (SARKAR; p. 260; FIG. 2; [The Examiner notes that the ‘Target Pose’/target pose is generated from the ‘Target Image’/ground truth image, which in turn is bi-directionally linked to the ‘Generated Image’/composite image.]).
Regarding claim 14, SARKAR discloses the method of claim 10, wherein:
the image generation network is pretrained using pretraining data prior to training using the training data, wherein the pretraining data includes non-segmented posed images (SARKAR; p. 260, right col.; “The pretrained network trained on COCO-Dense-Pose dataset predicts 24 body segments and their part-specific UV coordinates of SMPL model. For easier mapping, the 24 part-specific UV maps are combined to form a single normalized UV texture map Ts in the format provided in SURREAL dataset. This normalized (partial) texture map provides us with a pose-independent appearance encoding [‘non-segmented posed images’] of the subject that is located spatially according to the body parts.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 USC 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over SARKAR as applied to claim 1 above, respectively, and further in view of Grigorev et al. ("Coordinate-based Texture Inpainting for Pose-Guided Human Image Generation", published 2019, 'GRIGOREV').
Regarding claim 5, SARKAR discloses the method of claim 1; however, SARKAR does not disclose that the method of claim 1 further comprises:
generating a first visibility map and a second visibility map indicating portions of the first warped image and the second warped image based on visible portions of the first image and the second image, respectively, wherein the first texture embedding and the second texture embedding are based on the first visibility map and the second visibility map, which GRIGOREV discloses (GRIGOREV; p. 12138; “The result of this warping step is the source coordinate map C, which for each … (texel) [u, v] defines a corresponding location [x, y] = [C1[u, v], C2[u, v]] in the source image. Since only a part of a human body can be visible in the source photograph, for [most] texels, the source image location is undefined. When passing C into the network, we set the unknown values to a negative constant (-10) [‘first visibility map’], and also provide the network with the mask C′ [u, v] of known texels [‘second visibility map’].”), respectively.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of SARKAR to include the generating first/second visibility maps indicating portions of the first/second warped images based on visible portions of the first/second images, respectively, wherein the first/second texture embeddings are based on the first/second visibility maps of GRIGOREV. The motivation for this modification is to implement a deep learning approach to pose-guided resynthesis of human photographs. At the heart of this approach is the estimation of the complete body surface texture based on a single photograph. Since the input photograph always observes only a part of the surface, we suggest an inpainting method that completes the texture of the human body. Rather than working directly with colors of texture elements, the inpainting network estimates an appropriate source location in the input image for each element of the body surface (GRIGOREV; Abstract).
Regarding claim 6, SARKAR discloses the method of claim 1; however, SARKAR does not disclose that the method of claim 1 further comprises the following limitations, which GRIGOREV discloses:
generating a first/second feature selection masks corresponding to the first/second images (GRIGOREV; p. 12138; “Since only a part of a human body can be visible in the source photograph [‘first/second images’], for a big part of texels, the source image location is undefined. When passing C into the network, we set the unknown values to a negative constant (-10), and also provide the network with the mask C′ [u, v] of known texels.” [The Examiner asserts that a second feature selection mask is analogous to the leftover texels, or unknown texels.]), respectively; and
combining the first/second texture embeddings with the first/second feature selection masks, respectively, to obtain first/second masked texture embeddings (GRIGOREV; p. 12138; “The first learnable module of our pipeline is the inpainting network f(C, C′ ; φ) with learnable parameters φ that takes an incomplete coordinate map C in the texture space [‘first/second texture embeddings’] along with the mask of known texels, and outputs a completed and corrected source correspondence map D, where for each [u, v] the corresponding location in the source image is define”), wherein the composite image is generated based on the first/second masked texture embeddings (GRIGOREV; p. 12139; “Garment transfer. A slight modification of our architecture allows it to perform garment transfer [12, 15, 32, 23]. Here, given two views A and B, we want to synthesize a new view, where the pose and the person identity is taken from the view B, while the clothing is taken from view A. We achieve this by taking the architecture outlined above, and additionally conditioning the network g on the masked image N′ of the target view, where we mask out all areas except head (including face, hair, hats, and glasses) and hands (including gloves). The network g is trained on the pairs of views of the same person, and effectively learns to copy heads and hands from N′ to N. At test time, we provide the network the identity-specific image N′ and the body texture mapping MN that are both obtained from the image of a different person from the one depicted in the input view. We show that our architecture successfully generalizes to this setting and thus accomplishes the virtual re-dress task.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of SARKAR to include the generating a first/second feature selection masks corresponding to the first/second images and the combining the first/second texture embeddings with the first/second feature selection masks, respectively, to obtain first/second masked texture embeddings, wherein the composite image is generated based on the first/second masked texture embeddings of GRIGOREV. The motivation for this modification is to implement a deep learning approach to pose-guided resynthesis of human photographs. At the heart of this approach is the estimation of the complete body surface texture based on a single photograph. Since the input photograph always observes only a part of the surface, we suggest an inpainting method that completes the texture of the human body. Rather than working directly with colors of texture elements, the inpainting network estimates an appropriate source location in the input image for each element of the body surface (GRIGOREV; Abstract).
Claims 15-16, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over SARKAR in view of Lee et al. (U.S. PG-PUB 2020/0074707, 'LEE').
Regarding claim 15, SARKAR discloses a system comprising:
PNG
media_image13.png
605
520
media_image13.png
Greyscale
[LEE discloses this limitation.]);
[LEE discloses this limitation.]); and
PNG
media_image14.png
288
732
media_image14.png
Greyscale
an image generation network including parameters stored in the … memory component(s) (SARKAR; p. 261, left col.; § 3.1; [See screen-capture above.]; p. 261, right col.; § 3.2; “With reparameterization trick on sampling zs, we train the system end-to-end and optimize the parameters of the networks E, G and D.”), wherein the image generation network is trained to generate a composite image depicting … different segmented body parts based on … body part images respectively depicting the … different segmented body parts (SARKAR; p. 261, left col.; § 3.1; “Decoding [‘generate’] to a Photo-Realistic Image [‘composite image’]. The warped noise image in the target-pose Zt with the noise vectors correctly aligned with the body parts in the target pose [‘different segmented body parts’], is used as an input [‘training’] to a generator network G [‘image generation network’]. The generator and the warping module act as the conditional decoder of our pipeline. We use the high-fidelity generator from Pix2PixHD …”), wherein the generating of the composite image comprises:
using a warping prediction module (SARKAR; p. 260; FIG. 2, ‘Warp’ module) to generate … warped images based on the … body part images, respectively (SARKAR; FIG. 2; p. 261, left col. ‘Warping Latent Space’; “We also want the latent code for a specific body part to have direct influence on the same body part in the generated image. We enforce this by warping and broadcasting the part-based latent code to the corresponding part location in the target image and create a noise image Zt …”);
generating … texture-encoding vectors based on the … warped images (SARKAR; FIG. 2; p. 260, right col.; “In the first step, we extract SMPL UV texture map Ts from the input image Is using the Dense-Pose correspondences. In the second step, we use an encoding function E to map the human appearance Ts of the source image to the parameters of the distribution of the latent vector.”);
PNG
media_image15.png
362
731
media_image15.png
Greyscale
generating … pose encoding vector(s) based on … corresponding pair(s) of a source pose and a target pose (SARKAR; p. 260, left col.; § 3. ‘Method’; “Our goal is to learn a generative model of human images, which is conditioned on body pose and a low dimensional latent vector encapsulating the appearance of different body parts.” § 3.1. ‘Our Architecture’; “In the training stage, we take pairs [‘corresponding pair(s)’] of images (Is [‘source pose’], It [‘target pose’]) of the same person (but in different poses) as input.”);
generating … feature selection masks corresponding to the … body part images (SARKAR; p. 265, left col.; § 4.3. ‘Part-Based Sampling’; “We compute the following two metrics for a given part p: 1) Variation–Part: mean pairwise L1 distance between the samples in the masked region (by Dense-Pose) of the part p normalized by the masked area [‘feature selection masks’]. 2) Variation–Rest: mean pairwise L1 distance between the samples in the masked region of all body parts [‘feature selection masks’] excluding p. We compute the aforementioned metrics for the following parts: “Head”, “Upper body” and “Lower body” for 2500 generated images … A suitable method for part sampling should generate diverse semantically meaningful parts renderings without changing the rest of the body.”);
([LEE discloses this limitation.]);
([LEE discloses this limitation.]); and
PNG
media_image16.png
325
731
media_image16.png
Greyscale
generating a human image based on the masked texture combination and the masked pose combination (SARKAR; p. 265, right col.; § 4.3. ‘Part-Based Sampling’; [See the screen-capture above.]).
SARKAR does not explicitly disclose the following limitations, which LEE discloses:
… memory component(s) (LEE; FIG. 1, ‘storage 114’, ‘memory 116’; ¶ 0012-13, 0016-17);
… processing device(s) coupled to the … memory component(s) (LEE; FIG. 1, ‘storage 114’, ‘processor 102’; ¶ 0013), wherein the processing device is configured to execute instructions stored in the … memory component(s) (LEE; FIG. 1; ¶ 0013; “… processing unit(s) 102 may be any technically feasible hardware unit capable of processing data and/or executing software applications.”);
generating a masked texture combination by performing an affine combination of the … texture-encoding vectors and the feature selection masks (LEE; FIG. 2; ¶ 0022, 0029; “… random input 216 may include a random vector with a standard normal distribution that is combined (e.g., concatenated) with a semantic representation of an image to generate input to VAE 226. The semantic representation may be updated to include regions of pixels 218 and corresponding labels 220 for bounding boxes [‘masked texture/pose’] represented by affine transformations 230. Training engine 122 may apply the encoder portion of VAE 226 to the input to produce a vector in a latent space, which also has a standard normal distribution. Training engine 122 may then apply the decoder portion of VAE 226 to the vector to generate binary masks containing shapes of objects within bounding boxes represented by affine transformations 230.”); and
generating a masked pose combination by performing an affine combination of the … pose encoding vector(s) and the feature selection masks (LEE; FIG. 2; ¶ 0043; “A is an affine transformation that produces a realistic bounding box given a ground truth, Ã is a predicted affine transformation generated via supervised path 250. zA represents a vector that is encoded from parameters of a ground truth bounding box for an object [‘masked texture/pose’], EA represents an encoder that encodes parameters [‘texture-encoding vectors’] of an input affine transform A, KL denotes the Kullback-Leibler divergence, and Lsup,adv represents an adversarial loss that focuses on predicting a realistic Ã. In turn, the equation may be used to update generator model 202 so that generator model 202 maps zA to A for each ground truth.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the system of SARKAR to include the generating a masked texture combination by performing an affine combination of the … texture-encoding vectors and the feature selection masks and the generating a masked pose combination by performing an affine combination of the … pose encoding vector(s) and the feature selection masks of LEE. The motivation for this modification is to enable insertion of objects into scenes in real-world applications that include image synthesis, augmented reality, virtual reality, and/or domain randomization in machine learning. Machine learning models may insert pedestrians and/or cars into images containing roads for subsequent use in training an autonomous driving system and/or generating a video game or virtual reality environment. Inserting objects into a scene in a realistic and/or contextually meaningful way can enhance the scene for a human user for training or entertainment purposes (LEE; ¶ [0002]).
Regarding claim 16, SARKAR-LEE disclose the system of claim 15, wherein the image generation network comprises:
a feature selector component configured to generate the … feature selection masks (SARKAR; FIG. 6; p. 265, left col.; § 4.3. ‘Part-Based Sampling’; “We next evaluate our method for part-based sampling [‘feature selection masks’]– the ability to produce different plausible renderings of a body part (e.g., head) while keeping the rest of the body same … We compute the following two metrics for a given part p: 1) Variation–Part: mean pairwise L1 distance between the samples in the masked region (by Dense-Pose) of the part p normalized by the masked area [‘feature selection masks’]. 2) Variation–Rest: mean pairwise L1 distance between the samples in the masked region of all body parts [‘feature selection masks’] excluding p. We compute the aforementioned metrics for the following parts: “Head”, “Upper body” and “Lower body” for 2500 generated images … A suitable method for part sampling should generate diverse semantically meaningful parts renderings without changing the rest of the body.”).
Regarding claim 18, SARKAR-LEE disclose the system of claim 15, wherein the image generation network comprises: a texture encoder configured to encode the … body part images (SARKAR; p. 260, left col.; § 3.1; “In the training stage, we take pairs of images (Is, It) of the same person (but in different poses) as input.”) to obtain the … texture-encoding vectors (SARKAR; p. 260, right col.; § 3.1; “In the first step, we extract SMPL UV texture map Ts from the input image Is using the Dense-Pose correspondences. In the second step, we use an encoding function E to map the human appearance Ts of the source image to the parameters of the distribution of the latent vector.”).
Regarding claim 20, SARKAR-LEE disclose the system of claim 15, wherein the image generation network comprises:
PNG
media_image14.png
288
732
media_image14.png
Greyscale
a generative decoder configured to generate the composite image (SARKAR; FIG. 2; p. 260, right col.; “… we decode the warped Zt to a realistic-image I′t [‘composite image’] by a high-fidelity generator network.” p. 261, left col.; [See screen-capture above.]).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over SARKAR in view of LEE as applied to claim 15 above, and further in view of Li et al. ("Dense Intrinsic Appearance Flow for Human Pose Transfer", published June 2019, 'LI').
Regarding claim 17, SARKAR-LEE disclose the system of claim 15, wherein the image generation network comprises:
PNG
media_image17.png
989
640
media_image17.png
Greyscale
a visibility prediction module configured to generate … visibility maps based on the … body part images, respectively, wherein the … texture-encoding vectors [are] generated based on the … visibility maps (LI; p. 3694, left col.; “Figure 1 (left) illustrates our overall image generation framework. Given a reference image (and its pose) and the target pose, we first use a variant of U-Net [29] to encode the image and target pose respectively. Then our appearance flow module generates a 3D flow map from the pose pair, and further generates a visibility map to account for the missing pixels in the target pose due to self-occlusions. The visibility map proves necessary for our network to synthesize missing pixels at the correct locations. To render the final image in target pose, the encoded image features are first warped through the generated flow map, and then passed to a gating module guided by the visibility map. Finally, our pose decoder concatenates such processed image features to generate the image.” p. 3695, right col.; [See screen-shot above.]).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the system of claim 15 of SARKAR-LEE to include the generating … visibility maps based on the … body part images, respectively, wherein the … texture-encoding vectors [are] generated based on the … visibility maps of LI. The motivation for this modification is to model the large variations (in human image generation between poses) in 2-D appearance due to the change in 3-D pose. Human body self-occlusion induces ambiguities in inferring unobserved pixels for the target pose. Successful human pose transfer requires a good representation or disentangling of human pose and appearance, which is non-trivial to learn from data. The ability to infer invisible parts is also necessary. The image visual quality largely depends on whether the high frequency details can be preserved, e.g. in face regions. This is achieved by implementing a novel approach to human pose transfer that integrates implicit reasoning about 3D geometry from 2D representations only. This allows sharing of the benefits of using 3D geometry for accurate pose transfer but at much faster speed (LI, p. 3693).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over SARKAR in view of LEE as applied to claim 15 above, and further in view of Volkov et al. (U.S. PG-PUB 2020/0234480, 'VOLKOV').
Regarding claim 19, SARKAR-LEE disclose the system of claim 15, wherein the image generation network comprises:
a pose encoder configured to encode … input poses to obtain … pose embeddings (VOLKOV; FIGS. 8, 10-11; ¶ 0060; “Facial and head pose encoder 810 takes a photo of an actor [‘input pose’] and outputs [a] pose encoding … as a sequence of facial key points at each frame or a sequence of parameters of [a] parametric facial expression model. … a set of 78 facial key points is used …, but any set of facial key points may be used. The key points are 2D points that describe particular facial points (like a corner of a brow). A three-channel mask of facial key points … is used for each frame. … key points are extracted at each frame of the source video, then [a] bounding box (fixed width and height in pixels) of source actor's head is fixed [‘mask’] and these dimensions are used to draw key points mask inside. In addition to the 2D facial landmarks (represented as an RGB image), the pose can be encoded with facial landmarks represented … by segmentation masks representing face parts (hair, face, upper body) … A head view and facial expression can be encoded separately by using angles (yaw, pitch, raw) for the head view and action units (AU) for the facial expression.” ¶ 0064-66; “Person identity embedder 820 takes a set of photos with a head of the target actor (only one photo, or all frames from a video with talking target actor) and produces real-valued embedding to pass to the generator. A convolutional NN is used to take one three-channel photo of the head of the target actor and produce a [1-D] real-valued embedding. The embedding can be applied to each available photo of target actor. A final embedding is computed by averaging all embeddings for each frame. The animated head generator 825 (“Generator”) may receive identity embedding and pose encoding as an input.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the system of claim 15 of SARKAR-LEE to include the encoding … input poses to obtain … pose embeddings of VOLKOV. The motivation for this modification is to implement an animated head generator that may generate a frame sequence of a realistic and plausible-looking head of the target actor, which moves and express emotions that were extracted from the source actor (VOLKOV; ¶ [0066]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN M COFINO whose telephone number is (303)297-4268. The examiner can normally be reached Monday-Friday 10A-4P MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN M COFINO/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights