Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment received 12/16/25 has been entered in full.
Response to Arguments
Applicant's arguments filed 12/16/25 have been fully considered but they are not persuasive. Specifically the Applicant argues that “synthesis of image data and bounded boxes is not something that can be practically performed in the human mind”. The Examiner respectfully disagrees. A human looking at a pair of images could pick a location in one image, and place a smaller image on top of a larger image to synthesize an object into the image, and further draw a bounding box around the placed object. Thus the claim amendments do not recite significantly more than an abstract idea that could be carried out in the mind of a human with the aid of pen and paper, and the 101 rejection is maintained.
In regards to the 102 rejection the Applicant argues that Kim does not disclose synthesizing both the object and the bounding box annotation, and that Kim uses the bounding box as the input to make the synthesized image. However it is noted at col. 2 lines 15-17 discloses, it is still another object of the present invention to provide a method for generating synthesized images including bounding boxes. Thus it is clear that when the object is synthesized into the image the bounding box can also be synthesized with it. However the Examiner will provide an additional reference to show that it is further obvious to synthesize a bounding box into an image.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-40 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception in the form of an abstract idea without significantly more.
Independent claim 1 recites the following 3 abstract ideas:
PNG
media_image1.png
412
582
media_image1.png
Greyscale
These actions can be reasonably and practically performed as a mental process in the human mind such as a human taking an image of an object and overlaying on top a larger image to include the image in the object, and drawing/placing a bounding box around the object. See MPEP 2106:
PNG
media_image2.png
578
618
media_image2.png
Greyscale
The judicial exception is not integrated into a practical application because the additional elements of a processor comprising circuits to do the marked limitations is a generic computer implementation, see 2106.05(f)
PNG
media_image3.png
461
624
media_image3.png
Greyscale
The claim does not include any additional elements that are sufficient to amount to significantly more than the judicial exception because absent the elements addressed above there are no additional elements.
Regarding claim 2, a human can provide an input to specify a location of an object to add and supply the created image to a generic object detection neural network.
Regarding claim 3, the limitation of training using a combination of a background generated by a first network and a representation generated by a second network under its broadest reasonable interpretation, covers performance of the limitation in the mind and/or manually but for the recitation of generic computer components such as circuits and neural networks. That is other than reciting “neural network” nothing in the claim precludes the step from practically being performed in the mind and/or manually.
This judicial exception is not integrated into a practical application. In particular, the claim only recites one additional element – using a neural network and/or generic processor comprising a generic circuit to perform object detection. The processor in the single step is recited at a high-level of generality (i.e., as a generic processor performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. There is no recitation of how the object detection is performed or how a neural network is used to perform the single step. Therefore the judicial exception is not integrated into a practical application because there is no description of the process to make it practical or practicable. Accordingly, this additional element (circuit/neural network) does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Regarding claim 4, similar to claim 3 the limitation of generating a bounding box based on unlabeled two dimensional images can be carried out by a human. The recitation of a generic neural network to carry it out amounts no more than mere instructions to apply the exception using a generic computer component.
Regarding claim 5, similar to claim 3 the training using a loss can be carried out by a human. The recitation of a generic neural network to carry it out amounts no more than mere instructions to apply the exception using a generic computer component.
Claim 6 merely stipulates the inputs and outputs entered into the generic neural network thus recitation of a generic neural network to carry it out amounts no more than mere instructions to apply the exception using a generic computer component.
Regarding claim 7, a human can provide the pose of an object to be placed in a generated image.
Regarding claims 8-10 the training of the GAN is generic and amounts no more than mere instructions to apply the exception using a generic computer component.
Claims 11-20 are similarly analyzed to claims 1-10.
Claims 21-26, 28-30 are similarly analyzed to claims 1-10.
Regarding claim 27 a scene discriminator could be carried out in a person’s mind or with pen and paper.
Claims 31-40 are similarly analyzed to claims 1-10.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2, 12, 22 and 32 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 2 recites the limitation "”the assignment” in line 2. There is insufficient antecedent basis for this limitation in the claim (note the antecedent basis was removed from claim 1).
Claim 12 is similarly analyzed to claim 2.
Claim 22 is similarly analyzed to claim 2.
Claim 32 is similarly analyzed to claim 2.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 11-13, 21-23, and 31-33 are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. US 10,467,503 (hereinafter “Kim”, cited in the previous office action) in view of Deng et al. US 2022/0164578 (hereinafter “Deng”).
Regarding claim 1, Kim discloses a processor, comprising: one or more circuits to (see figure 1, col. 5 lines 51-55 which shows a computing device with various processes performed by a processor 120.
PNG
media_image4.png
529
466
media_image4.png
Greyscale
PNG
media_image5.png
83
360
media_image5.png
Greyscale
Obtain one or more input parameters indicating one or more location of one or more objects to be included within one or more images to be synthesized (see col. 2 lines 30-53, the first synthesized label is created by adding a specific label corresponding to the specific object to the original label at a location in the original label corresponding to a location of the bounding box, thus the bounding box indicates the location of the object to be included)
PNG
media_image6.png
356
361
media_image6.png
Greyscale
synthesize the one or more images to include the one or more objects based, at least in part, on the one or more input parameters (see col. 2 lines 30-53, generating a synthesized image by adding a specific object not included in the original image to the original image at a location in the original image corresponding to a location of the bounding box)
: and one or more bounding boxes annotations of the one or more objectsinvention has the effect of generating synthesized images including bounding boxes, note these are based on the location of the object as recited above in col. 2 lines 30-53).
PNG
media_image7.png
36
364
media_image7.png
Greyscale
Kim does not explicitly disclose the details of how the generated image includes the bounding box, thus does not explicitly disclose synthesizing the bounding box, although since a bounding box is generated for the synthesized object which is based on the input parameters it would be obvious to synthesize the bounding box as well.
Deng discloses that a synthesized bounding box can be generated around a synthesized 2D footprint in a data image and comparing the synthesized bounding box and the first bounding box.
PNG
media_image8.png
288
261
media_image8.png
Greyscale
Kim and Deng are analogous art because they are from the same field of endeavor of generating bounding boxes in images.
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Kim and Deng to generate a synthesized bounding box for the synthesized object to allow for improving the original bounding box
Regarding claim 2, Kim discloses the assignment of the one or more bounding boxes is controlled by an input that specifies a location of an object to add to the one or more image (see above col. 2 lines 30-53); and the synthesized one or more images is used to train an object detection neural network (see above col. 2 lines 30-34, discloses generating the image data set to be used for learning CNN capable of detecting an obstruction).
Regarding claim 3, the one or more circuits are further to perform an object detection neural network that is trained using a combination of a background image generated by a first network and a representation of the one or more objects generated by a second network (see above col. 2 lines 30-52, the synthesized image is created from the original image [background] and the object image [foreground]).
Claims 11-13 are similarly analyzed and rejected to claims 1-3.
Claims 21-23 are similarly analyzed and rejected to claims 1-3.
Claims 31-33 are similarly analyzed to claims 1-3.
Claims 4, 9-10, 14, 19-20, 24, 29-30, 34 and 39-40 are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Deng and further in view of Huynh et al. US 11,106,903 (hereinafter “Huynh”, cited in previous office action).
Regarding claim 4, as discussed Kim discloses the limitations of claim 1. Kim discloses using an original image but does not specify if the image is labeled or not.
Huynh discloses wherein generates the one or more bounding boxes based at least in part on using unlabeled two-dimensional images (see paragraph bridging cols. 3-4 and col. 6 lines 29-45).
PNG
media_image9.png
324
316
media_image9.png
Greyscale
PNG
media_image10.png
348
343
media_image10.png
Greyscale
Kim and Huynh are analogous art because they are from the same field of endeavor of determining bounding boxes.
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Kim, Deng and Huynh to use unlabeled feature data to generate the bounding boxes. The motivation would be that it could then reduce the amount of processing needed in order to obtain a bounding box.
Regarding claim 9, Huynh discloses wherein one or more GANs are trained using a loss from an object detection network (see column 9 lines 1-30).
PNG
media_image11.png
441
339
media_image11.png
Greyscale
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to use a GAN to train a loss. The motivation would be to make the processing more accurate as more training data becomes available.
Regarding claim 10, Huynh discloses one or more GANs are adapted to a target domain (see column 11 line 60 – col. 12 line 20).
PNG
media_image12.png
410
337
media_image12.png
Greyscale
Claim 14 is similarly analyzed to claim 4.
Claims 19-20 are similarly analyzed and rejected to claims 9-10.
Claim 24 is similarly analyzed to claim 4.
Claim 34 is similarly analyzed to claim 4.
Claims 29-30 are similarly analyzed and rejected to claims 9-10.
Claims 39-40 are similarly analyzed and rejected to claims 9-10.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Deng and further in view of Fisher et al. US 2019/0043003 (hereinafter “Fisher”).
Regarding claim 5, as discussed Kim discloses the limitations of claim 1.
Kim does not explicitly disclose wherein an object detection neural network is trained using a loss, the loss based at least in part on a difference between an output of the object detection neural network and in input that corresponds to the one or more bounding boxes.
Fisher discloses wherein an object detection neural network is trained using a loss, the loss based at least in part on a difference between an output of the object detection neural network and an input that corresponds to the one or more bounding boxes (see paragraph 0273).
PNG
media_image13.png
234
352
media_image13.png
Greyscale
Kim and Fisher are analogous art because they are from the same field of endeavor of object detection using neural networks.
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Kim, Deng and Fisher to use the conventional training of a neural network based on a loss function as shown by Fisher. The motivation would be to allow the neural network to improve with every iteration of processing.
Regarding claim 6, Fisher discloses the output is a first location of an object detected by the object detection neural network; the input is a second location identifies where to generate the object in an image; and the image is provided to the object detection neural network (see above paragraphs 460-461).
Claims 15-16 are similarly analyzed and rejected to claims 5-6.
Claims 25-26 are similarly analyzed and rejected to claims 5-6.
Claims 35-36 are similarly analyzed and rejected to claims 5-6.
Claims 7,17, and 37 are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Deng and further in view of Liu et al. US 2019/0012802 (hereinafter “Liu”, previously cited).
Regarding claim 7, as discussed above Kim discloses the limitations of claim 1. Kim does not explicitly disclose the indications include a pose of an object to be placed in a generated image where the indications are provided to one or more GANs
Liu discloses reconstruction of an image using GAN which inputs object pose information can be input into a GAN to reconstruct the full appearance of the object (see paragraph 0075).
PNG
media_image14.png
224
357
media_image14.png
Greyscale
Liu and Kim are analogous art because they are from the same field of endeavor of object recognition using image processing.
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Kim and Liu to input a pose to the GAN. The motivation would be to reconstruct the full appearance of an object.
Claim 17 is similarly analyzed to claim 7.
Claim 37 is similarly analyzed to claim 7.
Claims 8, 18, 28, and 38 are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Deng and further in view of Shu et al. US 2022/0222532 (hereinafter “Shu”, previously cited).
Regarding claim 8, as discussed above Kim discloses the limitations of claim 1. Claim 8 discloses the limitation which is not tied in any way to the claim 1 other than by claiming dependency that one or more GANs are trained using a foreground appearance loss, a background appearance loss, and a multi-scale object synthesis loss generated by a set of discriminative networks. Kim does not explicitly disclose this.
Shu discloses comparing a foreground modified image with an original image [interpreted as background image] to determine a differentiable loss from the original image to the foreground-modified image using a multi-scale loss [for instance the generative model determines various differentiable losses], see paragraph 0063.
PNG
media_image15.png
249
281
media_image15.png
Greyscale
Kim and Shu are analogous art because they are from the same field of endeavor of object recognition using image processing.
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Kim, Deng and Shu to disclose the limitations of claim 8 to determine a content aware differentiable loss.
Claim 18 is similarly analyzed to claim 8.
Claim 28 is similarly analyzed to claim 8.
Claim 38 is similarly analyzed to claim 8.
Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Deng and further in view of Zhao et al. US 2020/0401835 (hereinafter “Zhao”, previously cited).
Regarding claim 27, as discussed above Kim discloses the limitations of claim 1 and thus claim 21 also. Claim 27 discloses the limitation that one or more GANs are trained using a scene loss generated by a scene discriminator. Huynh does not explicitly disclose this.
Zhao discloses a scene graph generation system that can use an image discriminator of a GAN and then backpropagating the loss to train the system (see paragraph 0091).
PNG
media_image16.png
173
290
media_image16.png
Greyscale
Kim and Zhao are analogous art because they are from the same field of endeavor of object recognition.
Before the effective filing date of the invention it would have been obvious to one of ordinary skill in the art to combine Kim, Deng and Zhao to disclose the limitations of claim 27 to train a GAN effectively.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN B STREGE whose telephone number is (571)272-7457. The examiner can normally be reached M-F 9-5 (PST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571)272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN B STREGE/Primary Examiner, Art Unit 2669