DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
2. The information disclosure statements (IDS) submitted on the following dates are in compliance with the provisions of 37 CFR 1.97 and are being considered by the Examiner: 06/11/2026.
Response to Amendment
3. Applicant’s amendments filed on 06/11/2026 have been entered. Claims 1 1, 3, 13-14, 16 and 19-20 have been amended. Claims 1-20 are pending in this application, with claims 1 and 19-20 being independent.
Response to Arguments
4. Applicant's arguments filed on 06/11/2026, with respect to the 103 rejection have been fully considered but are moot in view of the new grounds of rejection.
Examiner notes that independent claims 1 and 19-20 have been amended to include new limitation. Examiner finds these limitations to be unpatentable as can be found in below detail action.
In light of the current Office Action, the Examiner respectfully submits that independent claims 1 and 19-20 are rejected in view of newly discovered reference(s) to Hare et al., (US-2021/0233310-A1).
On page 8 of Applicant's Remarks, the Applicant argues that the dependent claims are not taught by the prior art, insomuch as they depend from claims that are not taught by the prior art.
Examiner respectfully disagrees with these arguments, for the reasons discussed below.
Claim Rejections - 35 USC § 103
5. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
6. Claims 1, 3-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over “KPE: Keypoint Pose Encoding for Transformer-based Image Generation” by Soon Yau Cheong (“Soon”) in view of Smith (“Smith”) [US-2024/0331247-A1], further in view of Hare et al. (“Hare”) [US-2021/0233310-A1]
Regarding claim 1, Soon discloses a system (Soon- Fig. 1 shows The Block diagram showing KPE encoding multiperson pose to keypoint tokens; page 9, section 5.1 Comparison with DALL-E+VQGAN, 3rd paragraph, at least discloses the KPE model’s inference speed is only 0.6 second on RTX5000 GPU) comprising:
at least one processor (Soon- page 9, section 5.1 Comparison with DALL-E+VQGAN, 3rd paragraph, at least discloses RTX5000 GPU); and
at least one memory component storing instructions that, when executed by the at least one processor (Soon- page 9, section 5.1 Comparison with DALL-E+VQGAN, 2nd paragraph, at least discloses The keypoint token length is smaller than the pose image token, requiring less computational memory. In our experiment, the image token length is 256 while there are only 25 keypoint tokens, making it at least 10more memory efficient), cause the at least one processor to perform operations comprising:
accessing a plurality of inputs comprising first input data and second input data (Soon- Fig. 1 and page 2, section 1 Introduction, 1st paragraph, at least disclose we propose to add human pose as an additional input to text [accessing a plurality of inputs comprising first input data and second input data], to improve the accuracy and fidelity of people being generated through. We can see this as enforcing disentanglement of content and style of image [2] where the content is the pose, and the text depicts the style [first input data and second input data]. Concurrent to our work, Text2Human [3] uses text and pose as input [first input data and second input data], but their method involves complex dedicated neural networks. At the same time, our approach is simple and generic to the transformer architecture. There are several families of text-to-image generation algorithms), the first input data comprising a text prompt describing a desired image and the second input data indicative of one or more structural features of the desired image (Soon- Figure 1 shows the text [text prompt], pose keypoints [structural features] and image are encoded into tokens and go into an transformer; page 2, section 1 Introduction, 1st paragraph, at least discloses we propose to add human pose as an additional input to text [the first input data comprising a text prompt describing a desired image and the second input data indicative of one or more structural features of the desired image], to improve the accuracy and fidelity of people being generated through. We can see this as enforcing disentanglement of content and style of image where the content is the pose, and the text depicts the style [a text prompt describing a desired image]. Concurrent to our work, Text2Human [3] uses text and pose as input [first input data and second input data], but their method involves complex dedicated neural networks);
generating one or more outputs via a first generative machine learning model that uses the plurality of inputs as first control signals (Soon- Figure 1 shows (b) The Architectural diagram of our pose-guided text-to-image generation model. The text, pose keypoints and image are encoded into tokens and go into an transformer; Figure 2 shows The Block diagram showing KPE encoding multiperson pose to keypoint tokens. The tokens are flattened and projected into keypoint embedding within the transformer. The skeleton image is for illustration purposes as we use the keypoints directly from the pose estimation model’s outputs; Figure 5 shows Text of “a man wore blue shirt” was used to generate images from pose that is more masculine (a) towards more feminine pose in (d); page 2, section 1 Introduction, 1st and 2nd paragraphs, at least disclose There are several families of text-to-image generation algorithms, including GANs and diffusion models and the focus of this paper is autoregressive transformer [generative machine learning model] such as DALL-E which we use as a baseline in our study […] we have devised Keypoint Pose Encoding (KPE) - a novel, efficient and accurate pose representation suitable for a transformer. Instead of using the high dimensional skeleton image mainly containing redundant information, we focus on only the body joint keypoints for pose representation. The low dimensional representation is invariant to changing the target image [one or more outputs] resolution or domain, e.g. from the natural landscape to synthetic objects; page 4, section 3 Method, at least discloses Figure 1(b) shows the overall architecture of our pose constrained text-to-image model. The first stage is to convert the text, pose keypoints and image into tokens with their respective encoders. Then the tokens are projected into an embedding space before adding positional encoding. We use learnable positional encoding for text tokens and axial positional encoding for image tokens due to its 2D structure; page 6, section 3.4 Training, at least discloses To train, the text tokens T, keypoint tokens K and image tokens I are concatenated to be fed into the transformer. The transformer output has the same length as the input, aiming to generate the same tokens as the input tokens; page 9, section 5.1 Comparison with DALL-E+VQGAN, 5th paragraph, at least discloses Increasing target image resolution or quality will require an increase in image token length hence more memory and slower running);
generating an output image via a generative machine learning model that uses at least a subset of the plurality of inputs as control signals (Soon- Figure 1 shows (b) The Architectural diagram of our pose-guided text-to-image generation model. The text, pose keypoints and image [a subset of the plurality of inputs] are encoded into tokens and go into an transformer; Figure 2 shows The Block diagram showing KPE encoding multiperson pose to keypoint tokens. The tokens are flattened and projected into keypoint embedding within the transformer. The skeleton image is for illustration purposes as we use the keypoints directly from the pose estimation model’s outputs; Figure 5 shows Text of “a man wore blue shirt” was used to generate images from pose that is more masculine (a) towards more feminine pose in (d); page 2, section 1 Introduction, 1st and 2nd paragraphs, at least disclose There are several families of text-to-image generation algorithms, including GANs and diffusion models and the focus of this paper is autoregressive transformer [generative machine learning model] such as DALL-E which we use as a baseline in our study […] we have devised Keypoint Pose Encoding (KPE) - a novel, efficient and accurate pose representation suitable for a transformer […] The low dimensional representation is invariant to changing the target image [an output image] resolution or domain, e.g. from the natural landscape to synthetic objects; page 4, section 3 Method, at least discloses Figure 1(b) shows the overall architecture of our pose constrained text-to-image model. The first stage is to convert the text, pose keypoints and image into tokens with their respective encoders. Then the tokens are projected into an embedding space before adding positional encoding. We use learnable positional encoding for text tokens and axial positional encoding for image tokens due to its 2D structure; page 6, section 3.4 Training, at least discloses To train, the text tokens T, keypoint tokens K and image tokens I are concatenated to be fed into the transformer. The transformer output has the same length as the input, aiming to generate the same tokens as the input tokens; page 9, section 5.1 Comparison with DALL-E+VQGAN, 5th paragraph, at least discloses Increasing target image resolution or quality will require an increase in image token length hence more memory and slower running); and
causing presentation of the output image at a user device of a user (Soon- Figure 3 shows KPE model can generate photorealistic people with an accurate pose. This figure shows the pose illustration, ground truth, and three generated samples; page 8, section 5 Results, at least discloses the highest FID and IS scores, indicating that KPE can generate realistic looking people. Figure 3 and 1(a) show examples with various genders and quantity of people, with different scales, poses and occluded poses with missing keypoints. Given the OKS score of 0.97, which indicates a highly accurate pose, the high Mask-SSIM score suggests the generated images have gender and clothing appearance matching the text description).
Soon does not explicitly disclose generating one or more intermediate outputs via a first generative machine learning model that uses the plurality of inputs as first control signals, the one or more intermediate outputs including an initial image; generating an output image having a higher resolution than the initial image via a second generative machine learning model that uses at least a subset of the plurality of inputs and at least a subset of the one or more intermediate outputs as second control signals.
However, Smith discloses
generating one or more intermediate outputs via a first generative machine learning model (Smith- Fig. 40 and ¶0533-0537, at least disclose the scene-based image editing system 106 generating an infill semantic map 4008 [one or more intermediate outputs] utilizing a generative semantic machine learning model 4006 [first generative machine learning model] […] the scene-based image editing system 106 utilizing the generative semantic machine learning model 4006 to generate the infill semantic map 4008 from the semantic map 4007 […] For example, FIG. 40 illustrates the scene-based image editing system 106 generating the modified digital image 4012 from the infill semantic map 4008 [one or more intermediate outputs] […] the scene-based image editing system 106 further utilizes the generative image machine learning model 4010 to generate a realistic digital image from the infill semantic map 4008. In particular, the generative image machine learning model 4010 receives as input the infill semantic map 4008 [one or more intermediate outputs] and the digital image 4002 [inputs]);
generating an output image via a second generative machine learning model that uses at least a subset of the plurality of inputs and at least a subset of the one or more intermediate outputs as second control signals (Smith- Fig. 40 and ¶0533-0538, at least disclose the scene-based image editing system 106 generating an infill semantic map 4008 [one or more intermediate outputs] utilizing a generative semantic machine learning model 4006 [first generative machine learning model] […] the scene-based image editing system 106 utilizing the generative semantic machine learning model 4006 to generate the infill semantic map 4008 from the semantic map 4007 […] the scene-based image editing system 106 utilizing a generative image machine learning model 4010 [a second generative machine learning model] to generate a modified digital image 4012 [generating an output image]. For example, FIG. 40 illustrates the scene-based image editing system 106 generating the modified digital image 4012 from the infill semantic map 4008 [one or more intermediate outputs] […] the scene-based image editing system 106 further utilizes the generative image machine learning model 4010 to generate a realistic digital image from the infill semantic map 4008. In particular, the generative image machine learning model 4010 receives as input the infill semantic map 4008 [one or more intermediate outputs] and the digital image 4002 [inputs]. The infill semantic map 4008 and the digital image 4002 guides the generative image machine learning model 4010 in generating the modified digital image 4012 […] As shown in the modified digital image 4012, the scene-based image editing system 106 infills the portion below the knees of the person portrayed in the digital image 4002 (which only portrays the human from the knees up). The results generated by the scene-based image editing system 106 illustrate an efficient, accurate, and high-quality completion of the digital image 4002 by completing the semantic map 4007 [at least a subset of the plurality of inputs] and utilizing the infill semantic map 4008 [a subset of the one or more intermediate outputs] to generate the modified digital image 4012 [generating an output image] (semantic map 4007 and input the infill semantic map 4008 corresponds to “second control signals”)).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon to incorporate the teachings of Smith, and apply the generative semantic machine learning model and generative image machine learning model into Soon’s teachings for generating one or more intermediate outputs via a first generative machine learning model that uses the plurality of inputs as first control signals; generating an output image via a second generative machine learning model that uses at least a subset of the plurality of inputs and at least a subset of the one or more intermediate outputs as second control signals.
Doing so would implement artificial intelligence models to facilitate flexible and efficient scene-based image editing.
Smith further discloses
causing presentation of the output image at a user device of a user (Smith- ¶0122, at least discloses the client devices 110 a-110 n include computing devices that access, view, modify, store, and/or provide, for display, digital images. For example, the client devices 110 a-110 n include smartphones, tablets, desktop computers, laptop computers, head-mounted-display devices, or other electronic devices. The client devices 110 a-110 n include one or more applications (e.g., the client application 112) that can access, view, modify, store, and/or provide, for display, digital images; Fig. 2 and ¶0127, at least disclose the scene-based image editing system 106 provides a graphical user interface 202 for display on a client device 204. As further shown, the scene-based image editing system 106 provides, for display within the graphical user interface 202, a digital image 206; ¶0526, at least discloses the scene-based image editing system 106 provides options to generate the modified digital image 4012 within a user interface. For instance, the scene-based image editing system 106 provides client devices with more flexibility and controllability during editing. The user interface has various options to indicate the type of modification (e.g., expanding the digital image or removing an object) and the scene-based image editing system 106 further provides customizable options for modifying the digital image 4002 […] the scene-based image editing system 106 can allow a client device to provide user input of strokes, style guides, or color patches at a desired region to guide the image generation. Indeed, by including a style input within a region of digital image 4002, the scene-based image editing system 106 can utilize a machine learning model to expand/apply the style to the entire semantic region in generating the modified digital image 4012).
The prior art does not explicitly disclose, but Hare discloses
the one or more intermediate outputs including an initial image (Smith- ¶0096, at least discloses Based on tracking the one or more facial tracking points across multiple image frames, the trajectories may grow with each successive image frame with respect to the initial image frame from the image data captured by the image capture device);
an output image having a higher resolution than the initial image (Smith- Fig. 17 and ¶0115, at least discloses In operation 1734, the modeling component 250 generates a second depth map having a second resolution. In some instances, the second resolution is a higher resolution than the first resolution of the first depth map. In some embodiments, the modeling component 250 may create more than two depth maps. The modeling component 250 may create depth maps with increasing higher resolution a predetermined number of times, until a predetermined resolution is reached, or any number of iterations for any suitable resolution).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon/Smith to incorporate the teachings of Hare, and apply the successive image frame with respect to the initial image frame and the second resolution is a higher resolution than the first resolution into Soon/Smith’s teachings for generating one or more intermediate outputs via a first generative machine learning model that uses the plurality of inputs as first control signals, the one or more intermediate outputs including an initial image; generating an output image having a higher resolution than the initial image via a second generative machine learning model that uses at least a subset of the plurality of inputs and at least a subset of the one or more intermediate outputs as second control signals.
Doing so would generate three dimensional images often perform computationally intensive operations to generate and animate the three dimensional models.
Regarding claim 3, Soon in view of Smith and Hare, discloses the system of claim 1, and further discloses wherein the one or more intermediate outputs (see Claim 1 rejection for detailed analysis) comprises at least one of: a depth map, a surface normal map, the initial image comprising a color image, a pose map, or an edge map (Soon- Figure 3 shows KPE model can generate photorealistic people with an accurate pose. This figure shows the pose illustration, ground truth, and three generated samples; Smith- ¶0267, at least discloses the image analysis graph 1000 includes edges 1006 a-1006 h arranged in a manner that organizes the plurality of nodes 1004 a-1004 g; ¶0361, at least discloses the scene-based image editing system 106 improves the accuracy of low-level features (e.g., colors, materials) that are extracted for an object portrayed in a digital image; Hare- ¶0086, at least discloses The review interface 1410 may include one or more review elements 1420 (e.g., first, second, and third review elements 1430, 1440, and 1450) […] As shown in FIGS. 14 and 15, the first review element 1430 enables selection of one or more color filters, color effects, color temperature filters, and other filters, effects, and color value adjustments configured to modify one or more of a color, a sharpness, a tint, a saturation, or a hue of the 3D model).
Regarding claim 4, Soon in view of Smith and Hare, discloses the system of claim 1, and further discloses wherein the first generative machine learning model (see Claim 1 rejection for detailed analysis) comprises a diffusion model (Soon- page 2, section 1 Introduction, 1st paragraph, at least discloses There are several families of text-to-image generation algorithms, including GANs and diffusion models and the focus of this paper is autoregressive transformer such as DALL-E [5] which we use as a baseline in our study), and the one or more intermediate outputs (see Claim 1 rejection for detailed analysis) comprise a plurality of intermediate outputs that are at least partially simultaneously denoised via the diffusion model (Smith- ¶0520, at least discloses the scene-based image editing system 106 can utilize a diffusion model (or other generative machine learning model) to complete a semantic map to progressively generate an image layout; ¶0544-0545, at least discloses the scene-based image editing system 106 utilizes various types of machine learning models. For example, FIG. 41A illustrates the scene-based image editing system 106 utilizing a diffusion neural network (also referred to as “diffusion probabilistic model” or “denoising diffusion probabilistic model”) to generate an infill semantic map in accordance with one or more embodiments. In particular, FIG. 41A illustrates the diffusion neural network generating an infill semantic map 4124 while the subsequent figure (FIG. 42 ) illustrates the diffusion neural network generating the modified digital image conditioned on the infill semantic map 4124. For example, in one or more embodiments, the scene-based image editing system 106 utilizes a diffusion model (or diffusion neural network) […] Subsequent to adding noise to the digital image at various steps of the diffusion neural network, the scene-based image editing system 106 utilizes a trained denoising neural network to recover the original data from the digital image. Specifically, the scene-based image editing system 106 utilizes a denoising neural network with a length T equal to the length of the fixed Markov chain to reverse the process of the fixed Markov chain).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon/Hare to incorporate the teachings of Smith, and apply the denoising neural network into Soon/Hare’s teachings in order the first generative machine learning model comprises a diffusion model, and the one or more intermediate outputs comprise a plurality of intermediate outputs that are at least partially simultaneously denoised via the diffusion model.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 5, Soon in view of Smith and Hare, discloses the system of claim 4, and further discloses wherein the generating of the plurality of intermediate outputs (see Claim 1 rejection for detailed analysis) comprises:
initializing a noised state associated with each of the plurality of intermediate outputs (Smith- ¶0207, at least discloses the scene-based image editing system 106 also applies a shifting tensor 632 and broadcast noise to the output of the spatial modulation operation 610; ¶0545, at least discloses a diffusion neural network receives as input a digital image and adds noise to the digital image through a series of steps. For instance, the scene-based image editing system 106 via the diffusion neural network maps a digital image to a latent space utilizing a fixed Markov chain that adds noise to the data of the digital image ); and
performing denoising by denoising the noised states conditioned on the first control signals (Smith- ¶0544-0545, at least disclose FIG. 41A illustrates the scene-based image editing system 106 utilizing a diffusion neural network (also referred to as “diffusion probabilistic model” or “denoising diffusion probabilistic model”) to generate an infill semantic map in accordance with one or more embodiments […] Subsequent to adding noise to the digital image at various steps of the diffusion neural network, the scene-based image editing system 106 utilizes a trained denoising neural network to recover the original data from the digital image. Specifically, the scene-based image editing system 106 utilizes a denoising neural network with a length T equal to the length of the fixed Markov chain to reverse the process of the fixed Markov chain; ¶0544-0549-0550, at least disclose As part of the diffusion neural network, the denoising neural network 4112 a denoises the final diffusion representation 4110 (e.g., reverses the process of adding noise to the diffusion representation performed by the diffusion process 4108) […] FIG. 41A illustrates the denoising neural network 4112 a generating a first denoised representation 4114 that partially denoises the final diffusion representation 4110 by generating a first denoised representation 4114. Furthermore, FIG. 41 also illustrates a denoising neural network 4112 b receiving the first denoised representation 4114 for further denoising to generate the second denoised representation 4116).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon/Hare to incorporate the teachings of Smith, and apply the denoising diffusion probabilistic model into Soon/Hare’s teachings for initializing a noised state associated with each of the plurality of intermediate outputs; and performing denoising by denoising the noised states conditioned on the first control signals.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 6, Soon in view of Smith and Hare, discloses the system of claim 4, and further discloses wherein the first generative machine learning model comprises a set of branches (Smith- Fig. 40 and ¶0533, at least disclose the scene-based image editing system 106 generating an infill semantic map 4008 utilizing a generative semantic machine learning model 4006; ¶0627, at least discloses the scene-based image editing system 106 utilizes a separate neural network branch to generate a segmentation map of a digital image for further use as the structure guidance map; Fig. 41A shows the denoising neural networks 4112 a and 4112 b), and each branch in the set of branches is configured to denoise a respective one of the plurality of intermediate outputs (Smith- Fig. 41A and ¶0550, at least disclose the denoising neural network 4112 a generating a first denoised representation 4114 that partially denoises the final diffusion representation 4110 by generating a first denoised representation 4114. Furthermore, FIG. 41 also illustrates a denoising neural network 4112 b receiving the first denoised representation 4114 for further denoising to generate the second denoised representation 4116).
Regarding claim 7, Soon in view of Smith and Hare, discloses the system of claim 6, and further discloses wherein the first generative machine learning model comprises one or more common neural network layers that are shared across the set of branches (Smith- Fig. 40 and ¶0533, at least disclose the scene-based image editing system 106 generating an infill semantic map 4008 utilizing a generative semantic machine learning model 4006; Fig. 41A and ¶0550, at least disclose the denoising neural network 4112 a generating a first denoised representation 4114 that partially denoises the final diffusion representation 4110 by generating a first denoised representation 4114. Furthermore, FIG. 41 also illustrates a denoising neural network 4112 b receiving the first denoised representation 4114 for further denoising to generate the second denoised representation 4116; ¶0560, at least discloses the scene-based image editing system 106 generates a final diffusion representation for the input digital image (e.g., the expected output during training). The scene-based image editing system 106 trains by comparing diffusion representations generated by the diffusion process with corresponding denoised representations generated by the denoising neural network layers).
Regarding claim 8, Soon in view of Smith and Hare, discloses the system of claim 1, and further discloses wherein the first generative machine learning model comprises a latent diffusion model (Smith- ¶0552, at least discloses the scene-based image editing system 106 utilizes the diffusion architecture and training approach described by R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, in High-Resolution Image Synthesis with Latent Diffusion Models; ¶0809, at least discloses FIG. 76 shows the scene-based image editing system 106 performing a diffusion process 7608 to add noise to the latent vector 7606 and generate a final diffusion representation 7610).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon/Hare to incorporate the teachings of Smith, and apply the latent diffusion model into Soon/Hare’s teachings in order the first generative machine learning model comprises a latent diffusion model.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 9, Soon in view of Smith and Hare, discloses the system of claim 1, and further discloses the operations further comprising:
generating the second input data based on a reference image that contains the one or more structural features of the desired image (Soon- Figure 1 shows the text [text prompt], pose keypoints [structural features] and image are encoded into tokens and go into an transformer; page 2, section 1 Introduction, 1st paragraph, at least discloses we propose to add human pose as an additional input to text [one or more structural features of the desired image], to improve the accuracy and fidelity of people being generated through. We can see this as enforcing disentanglement of content and style of image where the content is the pose, and the text depicts the style [the desired image]. Concurrent to our work, Text2Human [3] uses text and pose as input , but their method involves complex dedicated neural networks; Smith- ¶0107, at least discloses The scene-based image editing system references the semantic scene graph previously generated for the digital image to identify a relationship between that object and one or more other objects portrayed in the digital image; ¶0407, at least discloses the scene-based image editing system 106 references the semantic scene graph previously generated for the digital image 2206 to identify the relationship between the objects 2208 a-2208 b).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon/Hare to incorporate the teachings of Smith, and apply referencing the semantic scene graph into Soon/Hare’s teachings for generating the second input data based on a reference image that contains the one or more structural features of the desired image.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 10, Soon in view of Smith and Hare, discloses the system of claim 1, and further discloses wherein the second input data comprises pose data that indicates the one or more structural features of the desired image (Soon- Figure 1 shows the text, pose keypoints [structural features] and image are encoded into tokens and go into an transformer; page 2, section 1 Introduction, 1st paragraph, at least discloses we propose to add human pose as an additional input to text [one or more structural features of the desired image], to improve the accuracy and fidelity of people being generated through. We can see this as enforcing disentanglement of content and style of image where the content is the pose, and the text depicts the style [the desired image]. Concurrent to our work, Text2Human [3] uses text and pose as input , but their method involves complex dedicated neural networks), and the first generative machine learning model is trained to at least partially reflect the pose data in the one or more intermediate outputs (Soon- page 2, section 1 Introduction, 1st and 2nd paragraphs, at least disclose There are several families of text-to-image generation algorithms, including GANs and diffusion models and the focus of this paper is autoregressive transformer [generative machine learning model] such as DALL-E which we use as a baseline in our study […] we have devised Keypoint Pose Encoding (KPE) - a novel, efficient and accurate pose representation suitable for a transformer. Instead of using the high dimensional skeleton image mainly containing redundant information, we focus on only the body joint keypoints for pose representation. The low dimensional representation is invariant to changing the target image [one or more outputs] resolution or domain, e.g. from the natural landscape to synthetic objects; Smith- Fig. 40 and ¶0533-0537, at least disclose the scene-based image editing system 106 generating an infill semantic map 4008 [one or more intermediate outputs] utilizing a generative semantic machine learning model 4006 [first generative machine learning model] […] For example, FIG. 40 illustrates the scene-based image editing system 106 generating the modified digital image 4012 from the infill semantic map 4008 [one or more intermediate outputs] […] the scene-based image editing system 106 further utilizes the generative image machine learning model 4010 to generate a realistic digital image from the infill semantic map 4008. In particular, the generative image machine learning model 4010 receives as input the infill semantic map 4008 [one or more intermediate outputs] and the digital image 4002 [inputs])).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon/Hare to incorporate the teachings of Smith, and apply the pose data in the one or more intermediate outputs into Soon/Hare’s teachings in order the pose data that indicates the one or more structural features of the desired image, and the first generative machine learning model is trained to at least partially reflect the pose data in the one or more intermediate outputs.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 11, Soon in view of Smith and Hare, discloses the system of claim 10, and further discloses wherein the pose data comprises human pose data (Soon- Figure 1 shows pose constrained text-to-image model supports partial and full pose view, multiple people, different genders, at different scales), the one or more structural features of the desired image comprises a pose of at least one human in the desired image (Soon- Figure 1 shows pose constrained text-to-image model supports partial and full pose view, multiple people, different genders, at different scales), and the output image depicts the at least one human (Soon- Figure 1 shows pose constrained text-to-image model supports partial and full pose view, multiple people, different genders, at different scales; page 2, section 1 Introduction, 4th paragraph, at least discloses A new metric to measure the false error rate of generated humans in multiperson images in rendered images).
Regarding claim 12, Soon in view of Smith and Hare, discloses the system of claim 10, and further discloses wherein the pose data comprises a pose map that defines at least one body skeleton (Soon- Figure 1 shows (a) Our pose constrained text-to-image model supports partial and full pose view, multiple people, different genders, at different scales. (b) The Architectural diagram of our pose-guided text-to-image generation model. The text, pose keypoints and image are encoded into tokens and go into an transformer; Figure 2 shows The Block diagram showing KPE encoding multiperson pose to keypoint tokens).
Regarding claim 13, Soon in view of Smith and Hare, discloses the system of claim 10, and further discloses wherein the one or more intermediate outputs further comprise a depth map and a surface normal map, the first generative machine learning model is trained to generate the depth map and the surface normal map based at least partially on the pose data and the text prompt, and the depth map and the surface normal map are processed via the second generative machine learning model to generate the output image (Soon- Figure 3 shows KPE model can generate photorealistic people with an accurate pose. This figure shows the pose illustration, ground truth, and three generated samples; Smith- Fig. 40 and ¶0533-0538, at least disclose the scene-based image editing system 106 generating an infill semantic map 4008 utilizing a generative semantic machine learning model 4006 [first generative machine learning model] […] the scene-based image editing system 106 utilizing the generative semantic machine learning model 4006 to generate the infill semantic map 4008 from the semantic map 4007 […] the scene-based image editing system 106 utilizing a generative image machine learning model 4010 [the second generative machine learning model] to generate a modified digital image 4012 [output image]. For example, FIG. 40 illustrates the scene-based image editing system 106 generating the modified digital image 4012 from the infill semantic map 4008 […] the scene-based image editing system 106 further utilizes the generative image machine learning model 4010 to generate a realistic digital image from the infill semantic map 4008. In particular, the generative image machine learning model 4010 receives as input the infill semantic map 4008 and the digital image 4002. The infill semantic map 4008 and the digital image 4002 guides the generative image machine learning model 4010 in generating the modified digital image 4012 […] As shown in the modified digital image 4012, the scene-based image editing system 106 infills the portion below the knees of the person portrayed in the digital image 4002 (which only portrays the human from the knees up). The results generated by the scene-based image editing system 106 illustrate an efficient, accurate, and high-quality completion of the digital image 4002 by completing the semantic map 4007 and utilizing the infill semantic map 4008 to generate the modified digital image 4012 [output image]).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon/Hare to incorporate the teachings of Smith, and apply the generative semantic machine learning model and generative image machine learning model into Soon/Hare’s teachings in order the one or more intermediate outputs comprise a depth map and a surface normal map, the first generative machine learning model is trained to generate the depth map and the surface normal map based at least partially on the pose data and the text prompt, and the depth map and the surface normal map are processed via the second generative machine learning model to generate the output image.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 14, Soon in view of Smith and Hare, discloses the system of claim 1, and further discloses wherein the one or more intermediate outputs comprises the initial image that is a predicted color image and one or more additional intermediate outputs (Smith- ¶0361, at least discloses the scene-based image editing system 106 improves the accuracy of low-level features (e.g., colors, materials) that are extracted for an object portrayed in a digital image; Fig. 40 and ¶0533-0537, at least disclose the scene-based image editing system 106 generating an infill semantic map 4008 [one or more intermediate outputs] utilizing a generative semantic machine learning model 4006 […] the scene-based image editing system 106 utilizing the generative semantic machine learning model 4006 to generate the infill semantic map 4008 from the semantic map 4007 […] For example, FIG. 40 illustrates the scene-based image editing system 106 generating the modified digital image 4012 from the infill semantic map 4008 [one or more intermediate outputs] […] the scene-based image editing system 106 further utilizes the generative image machine learning model 4010 to generate a realistic digital image from the infill semantic map 4008; Hare- ¶0086, at least discloses The review interface 1410 may include one or more review elements 1420 (e.g., first, second, and third review elements 1430, 1440, and 1450) […] As shown in FIGS. 14 and 15, the first review element 1430 enables selection of one or more color filters, color effects, color temperature filters, and other filters, effects, and color value adjustments configured to modify one or more of a color, a sharpness, a tint, a saturation, or a hue of the 3D model), and the one or more additional intermediate outputs comprises at least one of: a depth map, a surface normal map, a pose map, or an edge map (Soon- Figure 3 shows KPE model can generate photorealistic people with an accurate pose. This figure shows the pose illustration, ground truth, and three generated samples; Smith- ¶0267, at least discloses the image analysis graph 1000 includes edges 1006 a-1006 h arranged in a manner that organizes the plurality of nodes 1004 a-1004 g).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon to incorporate the teachings of Smith and Hare, and apply the edge map into Soon’s teachings in order the one or more intermediate outputs comprises the initial image that is a predicted color image and one or more additional intermediate outputs, and the one or more additional intermediate outputs comprises at least one of: a depth map, a surface normal map, a pose map, or an edge map.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 15, Soon in view of Smith and Hare, discloses the system of claim 14, and further discloses wherein the subset of the one or more intermediate outputs includes the one or more additional intermediate outputs (Smith- Fig. 40 shows a semantic map 4007 as additional intermediate output), the predicted color image being excluded from the subset of the one or more intermediate outputs such that the predicted color image is not processed via the second generative machine learning model (Smith- Fig. 40 shows an act 4004 of determining an infill modification. For example, the infill modification includes adding pixel values to a region or replacing pixel values in a region portrayed within the digital image 4002 [Wingdings font/0xE0] (“a region or replacing pixel values in a region portrayed within the digital image 4002” corresponds to “predicted color image”). The predicted color image is not processed via the second generative machine learning model 4010).
Regarding claim 16, Soon in view of Smith and Hare, discloses the system of claim 1, and further discloses wherein the second generative machine learning model comprises a diffusion model (page 2, section 1 Introduction, 1st and 2nd paragraphs, at least disclose There are several families of text-to-image generation algorithms, including GANs and diffusion models and the focus of this paper is autoregressive transformer [generative machine learning model] such as DALL-E which we use as a baseline in our study; Smith- Fig. 40 and ¶0537, at least disclose the scene-based image editing system 106 utilizing a generative image machine learning model 4010 [second generative machine learning model] to generate a modified digital image 4012 […] the scene-based image editing system 106 further utilizes the generative image machine learning model 4010 to generate a realistic digital image from the infill semantic map 4008. In particular, the generative image machine learning model 4010 receives as input the infill semantic map 4008 and the digital image 4002. The infill semantic map 4008 and the digital image 4002 guides the generative image machine learning model 4010 in generating the modified digital image 4012), and training of the second generative machine learning model comprises implementing a dropout scheme with respect to at least one of the second control signals processed via the second generative machine learning model (Smith- ¶0171, at least discloses a content fill includes a set of pixels generated to replace another set of pixels of a digital image. Indeed, in some embodiments, a content fill includes a set of replacement pixels for replacing another set of pixels. For instance, in some embodiments, a content fill includes a set of pixels generated to fill a hole (e.g., a content void) that remains after (or if) a set of pixels (e.g., a set of pixels portraying an object) has been removed from or moved within a digital image […] a content fill includes an inpainting segment, such as an inpainting segment generated from other pixels (e.g., other background pixels) within the digital image. In some cases, a content fill includes other content (e.g., arbitrarily selected content or content selected by a user) to fill in a hole or replace another set of pixels).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon/Hare to incorporate the teachings of Smith, and apply the dropout scheme into Soon/Hare’s teachings in order the second generative machine learning model comprises a diffusion model, and training of the second generative machine learning model comprises implementing a dropout scheme with respect to at least one of the second control signals processed via the second generative machine learning model..
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 17, Soon in view of Smith and Hare, discloses the system of claim 1, and discloses the operations further comprising:
receiving user input comprising at least the text prompt from the user device, wherein the output image is caused to be presented at the user device of the user in response to receiving the user input (Soon- Figure 1 shows the text [text prompt], pose keypoints and image are encoded into tokens and go into an transformer; page 2, section 1 Introduction, 1st paragraph, at least discloses we propose to add human pose as an additional input to text, to improve the accuracy and fidelity of people being generated through. We can see this as enforcing disentanglement of content and style of image where the content is the pose, and the text depicts the style. Concurrent to our work, Text2Human [3] uses text and pose as input, but their method involves complex dedicated neural networks).
The method of claim 19 is similar in scope to the functions performed by the system of claims 1 and therefore claim 19 is rejected under the same rationale.
Regarding claims 20, all claim limitations are set forth as claim 1 in a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations and rejected as per discussion for claim 1.
7. Claim 2 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Soon in view of Smith, further in view of Hare, still further in view of “Appearance and Pose-Conditioned Human Image Generation using Deformable GANs” by Siarohin et al. (“Siarohin”)
Regarding claim 2, Soon in view of Smith and Hare, discloses the system of claim 1, and further discloses wherein the one or more intermediate outputs comprises a plurality of intermediate outputs (see Claim 1 rejection for detailed analysis) among which the one or more structural features (Soon- Figure 1 shows the text [text prompt], pose keypoints [structural features] and image are encoded into tokens and go into an transformer; page 2, section 1 Introduction, 1st paragraph, at least discloses we propose to add human pose as an additional input to text, to improve the accuracy and fidelity of people being generated through. We can see this as enforcing disentanglement of content and style of image where the content is the pose, and the text depicts the style; Smith- ¶0620, at least discloses the scene-based image editing system 106 generates an infill modification to fill in the expanded portion (e.g., completes a human/object to be cropped into the digital image so that the human/object aligns with the boundaries of the new digital image and then combines/composites the completed human with the new digital image); ¶0625, at least discloses the scene-based image editing system 106 further utilizes a parameter neural network to generate a local appearance feature tensor, which includes spatially varying scaling and shifting tensors. In other words, the scene-based image editing system 106 modulates a structural encoding generated from a structure guidance map at the human inpainting GAN. The two-dimensional modulation aids in generating accurate and high-quality human inpainted digital images).
The prior art does not explicitly discloses the one or more structural features are spatially aligned.
However, Siarohin discloses
one or more structural features are spatially aligned (Siarohin- page 3, section 3 DEFORMABLE GANS, at least discloses at testing time, the target pose is known, thus H (P(xb)) can be computed. Note also that the joint locations in Xa and Ha are spatially aligned (by construction), while in Hb they are different).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon/Smith/Hare to incorporate the teachings of Siarohin, and apply the joint locations are spatially aligned into Soon/Smith/Hare’s teachings in order a plurality of intermediate outputs among which the one or more structural features are spatially aligned.
Doing so would synthesize a new image of a person in pose, while preserving the visual details.
Regarding claim 18, Soon in view of Smith and Hare, discloses the system of claim 17, and further discloses wherein the user input is received via an interaction application executing at the user device (Smith- ¶0097, at least disclose the scene-based image editing system utilizes one or more machine learning models to process a digital image in anticipation of user interactions for modifying the digital image. For example, in some implementations, the scene-based image editing system performs operations that build a knowledge set for the digital image and/or automatically initiate workflows for certain modifications before receiving user input for those modifications. Based on the pre-processing, the scene-based image editing system facilitates user interactions with the digital image as if it were a real scene reflecting real-world conditions. For instance, the scene-based image editing system enables user interactions that target pre-processed semantic areas (e.g., objects that have been identified and/or masked via pre-processing) as distinct components for editing rather than target the individual underlying pixels […] the scene-based image editing system allows user interactions to directly indicate intended edits to the digital image rather than the various preparatory steps often utilized for making those edits).
The prior art does not explicitly discloses, but Siarohin discloses
the interaction application provides an augmented reality experience that utilizes the output image (Siarohin- Figure 1 shows An example of a "rigid" scene generation rnsk, where the conditioning and the output image local structures are well aligned; page 3, left column, 2nd paragraph, at least discloses a person generation model is specifically designed for boosting Re-ID accuracy using data augmentation. A sub-network is added to a standard U-Net GAN network in order to verify whether the identity of the person in the generated images can be distinguished from other identities; page 9, left column, section 5.5 Person generation for Re-ID data-augmentation, at least discloses The experiments in this section are motivated by the importance of using generative methods as a data-augmentation 1001 which provides additional labeled samples for training discriminative methods (see Sec. l). Specifically, we show here that the synthetic images generated by our Deformable GANs can be used to train different Re-ID networks; page 9, right column, 3rd paragraph, at least discloses For each of the tested person-generation approaches, we use the following data augmentation procedure).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Soon/Smith/Hare to incorporate the teachings of Siarohin, and apply the data-augmentation into Soon/Smith/Hare’s teachings in order the interaction application provides an augmented reality experience that utilizes the output image.
Doing so would synthesize a new image of a person in pose, while preserving the visual details.
Conclusion
8. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
9. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL LE whose telephone number is (571)272-5330. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL LE/Primary Examiner, Art Unit 2614
Prosecution Insights