DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Priority to provisional application 63/384,080 filed 11/16/2022 is acknowledged.
Response to Arguments
This final office action is in response to the amendment filed June 23, 2026. Claims 1-22 are pending in this application and have been considered below.
Applicant’s arguments with respect to claims 1-22 have been considered but are moot in view of new ground(s) of rejection because of the amendments.
Applicant argues that Corona’s segmentation mask represents the cup that is included in the input image, and that Corona is silent as to a mask representing the absent hand or the spatial arrangement of the hand interacting with the cup. Applicant is correct as to Corona, and the rejection above has been rewritten so that the record is explicit on the point: “the mask
represents” is lined through as to Corona and is supplied by Nichol. The argument does not reach the rejection, which is over Corona in view of Nichol. “[O]ne cannot show non-obviousness by attacking references individually where, as here, the rejections are based on combinations of references.” In re Keller, 642 F.2d 413,426 (CCPA 1981); see also In re Mouttet, 686 F.3d 1322 (Fed. Cir. 2012).
Applicant argues that Corona contains no teaching of using an intermediate mask when generating the segmentation mask, as recited in new claim 22. Corona concatenates its segmentation mask with the input image and feeds both to the grasp prediction network, which outputs the hand configuration parameter vector. That segmentation mask is the intermediate mask and the hand configuration is the parameter vector. Claim 22 is addressed on the merits in the rejection above.
Claims 3, 10, 13, and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 4, 7, 11-12, 15, 17, and 20-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Corona et al. (GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes – hereinafter “Corona”) in view of Nichol et al. (GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models – hereinafter “Nichol”).
Claim 1.
Corona discloses a computer-implemented method for generating an image, the method comprising:
performing one or more first denoising operations based on a first machine learning model and an input image that includes a first object to generate a mask, that is based on the first object, wherein a second object that is not included in the input image (Corona p. 5032 “Yet, the key difference with our approach and all methods discussed in this section is that in our case hands are not visible in the input images and all reasoning is done from an image of the objects alone”; p. 5034 “given an image I, we train a model M that provides a hand pose P and shape V, and grasp type C for every object of interest in I.” Where, the input I is an image of objects. No hands. The output includes the hand configuration – something not present in I. Corona p. 5035: “we represent the absolute hand rotation as R = Ro+
∆
R, where at training, Ro is the rotation from a ground truth grasp with added noise. We then build a Fully Connected Network fed with {HC, Tobject, Ro} that predicts {
∆
H,
∆
T,
∆
R}” (p. 5035). Corona’s network is trained to recover the hand configuration from a rotation corrupted by added noise, which is a denoising operation performed by a machine learning model on the input image I containing the first object.), and
a spatial arrangement associated with a second object interacting with the first object (Corona p. 5031 discloses “given a single RGB image of a scene with an arbitrary number of objects, we aim to predict human grasp affordances, i.e. predict multiple plausible solutions of how a human would grasp each one of the observed objects” and “In order to predict feasible human grasps, we introduce GanHand, a multi-task GAN architecture that given solely one input image: 1) estimates the 3D shape/pose of the objects; 2) predicts the best grasp type according to a taxonomy with 33 classes [18]; 3) refines the hand configuration”; pp. 5034-5: "Our goal is to predict how a human would naturally grasp one or several objects, given a single RGB image of these objects. This implies producing valid hand configurations showing several contact points with the target object"; p. 5035, Segmentation Mask Generation: "During training, one object is randomly selected at a time, its 3D shape is projected onto the image plane to obtain a segmentation mask that is then concatenated with the input image and fed to the grasp prediction network. The mask indicates which object has to be focused on" ); and
Corona discloses all of the subject matter as described above except for specifically teaching “performing one or more second denoising operations based on a second machine learning model, the input image, and the mask to generate an image of the second object interacting with the first object” and “the mask represents.” However, Nichol in the same field of endeavor teaches performing one or more second denoising operations based on a second machine learning model, the input image, and the mask to generate an image of the second object interacting with the first object (Fig. 2 demonstrates the model receiving an original image with an erased region and a mask and generating new objects absent from the original scene, corresponding to the an image of a second object (e.g. a hand) not present in the input, interacting with the first object. Sections 4.1-4.3 “model predicts p(xt-1|xt, c)” conditioning on both image context and mask channel at each diffusion step.).
Corona does not teach that the mask represents the second object and the spatial arrangement of the second object interacting with the first object. Corona's segmentation mask represents the first object. However, Nichol in the same field of endeavor teaches a mask that represents an object absent from the input image and the region that object is to occupy (NICHOL: "random regions of training examples are erased, and the remaining portions are fed into the model along with a mask channel as additional conditioning information"(§ 4.3); “The green region is erased, and the model fills it in conditioned on the given prompt” (Fig. 2).).
Therefore, it would have been obvious to a person of ordinary skill in the art (“POSITA”) to combine Corona and Nichol before the effective filing date of the claimed invention because Corona stops at a three-dimensional hand configuration and produces no image of the interaction, while Nichol states that, given an erased region and a mask channel, “Our model is able to match the style and lighting of the surrounding context to produce a realistic completion” (Fig. 2). This is combining prior art elements according to known methods to yield predictable results (MPEP 2143(A)). Each element is present in the prior art: Corona supplies the prediction of an absent hand’s configuration relative to an imaged object and supplies the step of projecting a predicted three-dimensional shape onto the image plane to obtain a mask; Nichol supplies the mask-conditioned denoising generation. The elements are combinable by known methods because Corona already produces a mask in the image plane in the form Nichol takes as its mask channel.
Claims 2 and 12.
The combination of Corona and Nichol discloses the computer-implemented method of claim 1, further comprising receiving an input position associated with the second object, wherein the one or more first denoising operations are further based on the input position (Corona Page 5: “we represent the absolute translation of the hand w.r.t the camera as the T = Tobject +
∆
T . Similarly we represent the absolute hand rotation as R =Ro +
∆
R … We then build a Fully Connected Network fed with (HC, Tobject,Ro) that predicts {
∆
H,
∆
T,
∆
R}, to compute the absolute rigid pose of the hand.” Corona teaches using object position (Tobject) as input to guide the grasp prediction process.).
Claim 4.
The combination of Corona and Nichol discloses the computer-implemented method of claim 1, wherein each of the one or more first denoising operations and the one or more second denoising operations includes one or more denoising diffusion operations (Nichol p. 6, Section 4.3 discloses “we explicitly fine-tune our model to perform inpainting, similar to Saharia et al. (2021a). During fine-tuning, random regions of training examples are erased, and the remaining portions are fed into the model along with a mask channel as additional conditioning information.” This shows the denoising diffusion operations to condition on both the input image and the mask channel.; p. 3, Section 2.1 Diffusion Models, “pθ(xt-1|xt) …
gradually reducing the noise” This is the backward denoising process iteratively removing noise using diffusion operations.).
In the combination set out for claim 1, the layout stage that produces the mask is carried out with Nichol's denoising diffusion model rather than with a single feed-forward correction, so that both the first and the second denoising operations are denoising diffusion operations. It would have been obvious to one of ordinary skill in the art to implement the layout stage, because Nichol's diffusion model is already present in the combination for the second stage, and applying it to the first stage requires no more than using the same known technique a second time on the same kind of data. This is use of a known technique to improve similar devices in the same way (MPEP 2143(C)).
Claims 6 and 15.
The combination of Corona and Nichol discloses the computer-implemented method of claim 1, wherein the second machine learning model comprises an encoder-decoder neural network (Nichol p. 6, Section 4.1 discloses “We adopt the ADM model architecture proposed by Dhariwal & Nichol (2021),” where, the ADM/U-Net is an encoder-decoder architecture.).
Claims 7 and 17.
The combination of Corona and Nichol discloses the computer-implemented method of claim 1, wherein the second object comprises a portion of a human body (Corona Page 1, Title: “GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes”).
Claim 11.
The combination of Corona and Nichol discloses the one or more non-transitory computer-readable media storing instructions that, when executed by at least one processor (Corona Page 5: “a pretrained and fine tuned ResNet-50 , followed by a classification network … We then build a Fully Connected Network fed with (HC, Tobject,Ro)”; Page 6: discloses “We perform a hyperparameter grid search to maximize [19] and finally train all models using LR=0.0001, BS=32 … using Adam optimizer … Training models for single object (ObMan) or multi-object (YCB-Affordance) scenes takes approximately 6 and 8 days respectively on a V100 GPU.” Where, neural networks must be stored as instructions/parameters in computer memory.), cause the at least one processor to perform steps for …
The combination of Corona and Nichol renders claim(s) 11 obvious for the reasons discussed above in claim 1, mutatis mutandis.
Claim 20.
The combination of Corona and Nichol discloses a system, comprising: one or more memories storing instructions; and one or more processors that are coupled to the one or more memories and, when executing the instructions (Corona Page 5: “a pretrained and fine tuned ResNet-50 , followed by a classification network … We then build a Fully Connected Network fed with (HC, Tobject,Ro)”; Page 6: discloses “We perform a hyperparameter grid search to maximize [19] and finally train all models using LR=0.0001, BS=32 … using Adam optimizer … Training models for single object (ObMan) or multi-object (YCB-Affordance) scenes takes approximately 6 and 8 days respectively on a V100 GPU.” Where, neural networks must be stored as instructions/parameters in computer memory.), are configured to …
The combination of Corona and Nichol renders claim(s) 20 obvious for the reasons discussed above in claim 1, mutatis mutandis.
Claims 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable Corona and Nichol as applied to claim 1 and 11 above, and further in view of Jaderberg et al. (Spatial Transformer Networks – hereinafter “Jaderberg”).
Claims 5 and 14.
The combination of Corona and Nichol discloses the computer-implemented method of claim 1, wherein the first machine learning model comprises and an encoder neural network (Corona: “We use a pre-trained ResNet-50 as image encoder.” (p. 5036); “we extract a representation of the input image using a pretrained and finetuned ResNet-50” (p. 5035)).
Corona and Nichol discloses all of the subject matter as described above except for specifically teaching “a spatial transformer neural network.” However, Jaderberg in the same field of endeavor teaches a spatial transformer neural network (Abstract: “the Spatial Transformer, which explicitly allows the spatial manipulation of data within the network. This differentiable module can be inserted into existing convolutional architectures, giving neural networks the ability to actively spatially transform feature maps, conditional on the feature map itself, without any extra training supervision or modification to the optimization process.”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Corona, Nichol, and Jaderberg before the effective filing date of the claimed invention. The motivation for this combination of references would have been to improve the (1) known limitations: fixed CNN encoders struggle with geometric variations in object poses (2) proven enhancement: Jaderberg demonstrates that spatial transformers improve spatial reasoning in CNN-based systems (3) natural integration: spatial transformers are designed to augment encoder networks as used in Corona (4) performance improvement: expected benefits include better handling of rotations, scaling, and perspective variations in object interactions.
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable Corona and Nichol as applied to claim 1 and 11 above, and further in view of Ye et al. (What’s in your hands? 3D Reconstruction of Generic Objects in Hands – hereinafter “Ye”).
Claims 8 and 16.
The combination of Corona and Nichol discloses the computer-implemented method of claim 1, further comprising
Corona and Nichol discloses all of the subject matter as described above except for specifically teaching “performing one or more operations to generate three-dimensional geometry corresponding to the second object as set forth in the image of the second object interacting with the first object.” However, Ye in the same field of endeavor teaches performing one or more operations to generate three-dimensional geometry corresponding to the second object as set forth in the image of the second object interacting with the first object (Abstract: “Our work aims to reconstruct hand-held objects given a single RGB image … our work reconstructs generic handheld object without knowing their 3D templates.”; Page 3897: “Given an image depicting a hand holding an object, we aim to reconstruct the 3D shape of the underlying object … This network … maps a query 3D point to a signed distance from the object surface, and the zero-level set of this function can be extracted as the object surface”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Corona, Nichol, and Ye before the effective filing date of the claimed invention. The motivation for this combination of references would have been to extend the 2D interaction image generation of Corona and Nichol with Ye’s 3D reconstruction techniques to provide complete 3D geometric information about the interacting objects.
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being Corona and Nichol as applied to claim 1 and 11 above, and further in view of Zhang et al. (Learning Object Placement by Inpainting for Compositional Data Augmentation – hereinafter “Zhang”).
Claims 9 and 18.
The combination of Corona and Nichol discloses the computer-implemented method of claim 1, further comprising:
Corona and Nichol discloses all of the subject matter as described above except for specifically teaching “detecting the second object as set forth in one or more training images of the second object interacting with the first object; determining one or more training parameter vectors based on the one or more training images and the second object detected in the one or more training images; and performing a plurality of operations to train the first machine learning model based on the one or more training images and the one or more training parameter vectors.” However, Zhang in the same field of endeavor teaches detecting the second object as set forth in one or more training images of the second object interacting with the first object (Page 570: “Our system leverages existing instance segmentation dataset and a self-supervised image inpainting network to generate the necessary training data for learning object placement. Our insight is that we can generate such training data by removing objects from the background scenes. With an instance segmentation mask, we first cut out the object regions and then fill in the holes with an image inpainting network.”); determining one or more training parameter vectors based on the one or more training images and the second object detected in the one or more training images (Page 570: “After that, we simultaneously obtain a clean background scene without objects in it and the corresponding ground truth plausible placement locations and scales for placing these objects into the scene.”; Fig. 2: “we first cut out the object region with the instance segmentation mask, and save the original bounding boxes as the ground truth plausible placement locations and scales."); and performing a plurality of operations to train the first machine learning model based on the one or more training images and the one or more training parameter vectors (Page 567: “The ‘free’ labeled object-background pairs are then fed into our proposed PlaceNet, which predicts the location and scale to insert the object into the background.”; Page 570: “Overall, our proposed data acquisition technique provides a way to generate large-scale training data for learning object placement without any human labeling.”).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Corona, Nichol, and Zhang before the effective filing date of the claimed invention. The motivation for this combination of references would have been to enhance Corona’s hand-object interactions training system with Zhang’s proven data quality pipeline, including automated object detection, segmentation, and artifact removal procedures, to improve the quality and scale of training data for more robust interactions prediction models, and then use Nichol diffusion-based inpainting for the final high-quality content generation.
Claim 21.
Corona and Nichol discloses the computer-implemented method of claim 1, wherein performing the one or more first denoising operations comprises: generating, via the first machine learning model, a first parameter vector indicating a spatial layout of the second object interacting with the first object (Corona p. 5034 “M: I => {C, V, P}” grasp type, hand shape, hand pose – representing the spatial layout of the hand); and generating the mask based on the first parameter vector (GLIDE-style inpainting requires a 2D mask input, admitted prior art in Fig. 8A, ¶68; CORONA: “The predicted shape is then projected onto the image plane to obtain a segmentation mask” (p. 5035, Fig. 4); “Once the 3D shape and pose is known, we compute its segmentation mask and proceed as before” (p. 5035).) and Nichol requires a mask channel as its conditioning input (§ 4.3).).
Claim 22.
The combination of Corona and Nichol discloses the computer-implemented method of claim 1, wherein performing the one or more first denoising operations comprises: generating, using the first machine learning model and based on the input image and an intermediate mask, a
parameter vector (CORONA: “The predicted shape is then projected onto the image plane to
obtain a segmentation mask that is concatenated with the input image and fed to the second subnetwork for grasp prediction” (p. 5035, Fig. 4); “The mask indicates which object has to be
focused on while the original RGB image gives contextual information about the entire scene for a more realistic grasp" (p. 5035). Corona's segmentation mask is the intermediate mask; it is concatenated with the input image and fed to the network, and the network outputs the 51-DoF hand configuration, which is the parameter vector.); and converting the parameter vector into the mask (CORONA: “The predicted shape is then projected onto the image plane to obtain a segmentation mask” (p. 5035, Fig. 4) ... and Nichol conditions generation on a mask channel (§ 4.3).).
It would have been obvious to one of ordinary skill in the art to convert Corona’s hand configuration parameter vector into a mask by the projection step Corona already applies to a predicted object shape, because the combination of claim 1 requires a mask in the image plane and Corona supplies the projection step that produces one. This is use of a known technique to improve similar devices in the same way (MPEP 2143(C)). The specification describes the same operation, the layout mask being “generated by splatting a parameter vector output by the layout model 150” (¶ 69).
Allowable Subject Matter
Claims 3, 10, 13, and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form, including all of the limitations of the base claim and any intervening claims, AND overcoming the 35 USC 112(b) rejection above.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674
Prosecution Insights