DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 01/31/2024 and 02/21/2024 is/are being considered by the examiner.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Objections
Claim 5 is objected to because of the following informalities: the word “includes” has an extra “s”.
Claim 7 is objected to because of the following informalities: “or with latent space interpolation” is repeated twice at the end of the claim.
Claim 11 is objected to because of the following informalities: line 8 recites the word “lease” where it should be “least”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding independent claims 1, 11, and 12, each claim in effect recites the following steps:
“determining a digital image including an object of the second class in at least the part of the digital image…
“wherein the determining of the digital image includes determining the object of the second class with a generative model”. (emphasis added)
While “determining” seems to be used in a conventional way in the clause “determining the class for at least the part of the digital image with the classifier”, the two uses highlighted above do not appear to follow a standard definition of the verb “determine” (see attached definition from https://www.merriam-webster.com/dictionary/determine). When looking to the Specification of the published application, paragraph [0055] states
“Determining the digital image may comprise replacing or modifying a part of the digital image. In one example, the digital image is a selected digital image from the training data that is modified. In one example, the digital image is a newly generated digital image.”
This paragraph also appears to be using “determining” in a non-standard way as it describes altering a digital image. Throughout the Specification, such images are referred to as augmented images. Further looking to the specification, paragraph [0078] states
“Training the generative model for generating objects may comprise training the generative model with digital images of a training data set wherein a digital image represents a scene and contains multiple objects.”
While the word “determine” does not appear here, it is clear that the claimed generative model is generating the object, not determining it in the normal sense.
Dependent claim 5 refers back to claim 1 using “determining” as “augmenting” while dependent claims 6, 8, and 9 recite new uses “determining” as “generating”.
Where applicant acts as his or her own lexicographer to specifically define a term of a claim contrary to its ordinary meaning, the written description must clearly redefine the claim term and set forth the uncommon definition so as to put one reasonably skilled in the art on notice that the applicant intended to so redefine that claim term. Process Control Corp. v. HydReclaim Corp., 190 F.3d 1350, 1357, 52 USPQ2d 1029, 1033 (Fed. Cir. 1999). The term “determining” in claims 1, 5, 6, 8, 9, 11, and 12 is used by the claims to both mean “augmenting” and “generating” while the accepted meaning is “settling or deciding by choice of alternatives or possibilities.” The term is indefinite because the specification does not clearly redefine the term.
Further regarding independent claims 1, 11, and 12, each in effect recite
“determining the object of the second class with a generative model depending on a label for the first class”. (emphasis added)
Taking this “determining” to be “generating”, it is unclear how generating the object of the second class would depend on a label for the first class. The plain wording does not seem to make sense as it would be contradictory to use a label from a different class when generating an object. When looking to the Specification, both paragraphs [0040]-[0041] talk about a label for a pasted object in an image with a class of novel or unknown, and paragraph [0045] makes it clear that “known objects” are the first class and “novel or unknown objects” are the second class. Therefore, these paragraphs do not mention a label of the first class. Paragraph [0043] then gives a situation where no label is present, which does not fit the claim. Paragraph [0052] talks broadly about a label of the first class and paragraph [0059] merely repeats the claim language. The rest of the paragraphs describing labels continue to talk about labels for “novel or unknown objects” which would be a label for the second class.
Therefore, one of ordinary skill in the art would find the meets and bounds of “determining the object of the second class with a generative model depending on a label for the first class” to be unclear.
Dependent claims 2-10 do not cure independent claim 1 of these issues, and are similarly rejected.
Further regarding claim 5, the claim recites “wherein the determining of the digital image includess [sic] replacing or modifying a part of the digital image to form at least the part of the digital image”. (emphasis added) As currently worded, it appears that “part of the digital image” is created by modifying the same “part of the digital image” which does not make sense. The new recitation needs to be reworded to distinguish it from the previous “part of the digital image” in claim 1 on which claim 5 depends.
Further regarding claim 10, the claim refers to “the mask” and “the bounding box”. There have been two previous recitations of a bounding box, one simply as “a bounding box” in claim 4 and the other as “a predetermined bounding box” in claim 8, claim 10 depending on both. Normally, one of ordinary skill in the art would consider the recitation in claim 10 of “the bounding box” to be referring to the recitation in claim 4 i.e. without the modifier “predetermined”. However, the only previous recitation of mask was in claim 8 as “a predetermined mask”, meaning Applicant has dropped a modifier while referring back to a limitation. With this language inconsistency for the mask, it causes at least an antecedent basis issue with “the bounding box” in claim 10 since now it is unclear which bounding box this is supposed to refer to.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 2, 5, 11, and 12 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Sabatini et al, U.S. Publication No. 2025/0157174.
Regarding claim 11, Sabatini teaches a device configured to determine a class for at least a part of a digital image (see Sabatini Abstract), the device comprising:
at least one processor; and at least one storage; wherein the storage is adapted to store instructions for determining the class for the at least a part of a digital image, the instructions, when executed by the at lease one processor, causing the at least one processor to perform the following steps (see paragraph [0032]):
providing a classifier (see Figure 1, first neural network 132 and paragraph [0042], “The first neural network 132 may comprise a neural network operating as a classifier”) for a first class and a second class (see Figure 2, step 230, wherein the first class would be recognized objects and the second class would be initially unrecognized objects, which is in line with the classes used in present application as explained above),
determining a digital image including an object of the second class in at least the part of the digital image (see Figure 7, synthetic image with “rare object” and paragraph [0051], “In a rare object retrieval procedure 320, a rare object (for example, a rare vehicle) that cannot be recognized is automatically determined (detected) and a 3D bounding box corresponding to/containing the object is automatically extracted from the Lidar point clouds” indicating that rare and unrecognized are synonymous), and
determining the class for at least the part of the digital image with the classifier (see Figure 2, step 260), wherein the determining of the digital image includes determining the object of the second class (see paragraph [0064]) with a generative model (see Figure 2, second neural network 139 and paragraph [0044]) depending on a label for the first class and/or depending on at least one pixel representing an object of the first class (see Figure 7, wherein “common object” in the real image is replaced with “rare object” in the synthesized image and paragraph [0063]).
Independent claims 1 and 12 recite similar limitations as claim 11, and are rejected under similar rationale.
Regarding claim 2, Sabatini teaches all the limitations of claim 1, and further teaches training the classifier to determine the second class for the object of the second class in the digital image (see Sabatini Figure 2, step 260).
Regarding claim 5, Sabatini teaches all the limitations of claim 1, and further teaches wherein the determining of the digital image includess replacing or modifying a part of the digital image to form at least the part of the digital image (see Sabatini Figure 7, wherein “common object” in the real image is replaced with “rare object” in the synthesized image and paragraph [0063]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3, 4, 8-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sabatini et al, U.S. Publication No. 2025/0157174 in view of Erignac, U.S. Patent No. 9,547,805.
Regarding claim 3, Sabatini teaches all the limitations of claim 1, but does not expressively teach training the classifier to determine the second class for a pixel in the digital image forming at least a part of the object of the second class.
However, Erignac in a similar invention in the same field of endeavor teaches a method involving a classifier (see Erignac Figure 2, classifier 240) configured to determine a second class of an object in a digital image (see Figure 3, step 310 and column 6, “Classifier 240 may score each pixel with a roadness score that represents a likelihood that the pixel is a part of a road”) as taught in Sabatini comprising
training the classifier to determine the second class for a pixel in the digital image forming at least a part of the object of the second class (see Figure 3, step 350 and column 6, “After classifier 240 has been trained, classifier 240 is configured to process one or more images and identify roads in the one or more images. Classifier 240 may score each pixel with a roadness score that represents a likelihood that the pixel is a part of a road. Alternatively, or additionally, classifier 240 may label each pixel as a road or some other object”).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of training a classifier to label pixels as taught in Erignac with the method taught in Sabatini, the motivation being to utilize the granularity of individual pixel labeling for more accurate further analysis of a labeled image.
Regarding claim 4, Sabatini teaches all the limitations of claim 1, but does not expressively teach training the classifier to determine the second class for a bounding box in the digital image including the object of the second class.
However, Erignac in a similar invention in the same field of endeavor teaches a method involving a classifier (see Erignac Figure 2, classifier 240) configured to determine a second class of an object in a digital image (see Figure 3, step 310 and column 6, “Classifier 240 may score each pixel with a roadness score that represents a likelihood that the pixel is a part of a road”) as taught in Sabatini comprising
training the classifier to determine the second class for a bounding box in the digital image including the object of the second class (see column 6, “After classifier 240 has been trained, classifier 240 is configured to process one or more images and identify roads in the one or more images… Alternatively, or additionally, classifier 240 may output pixel coordinates, bounding boxes, or any other reference to the location of pixels determined to be a road”).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious to combine the teaching of training a classifier to output bounding boxes as taught in Erignac with the method taught in Sabatini, the motivation being to allow humans to easily visualize the determined object via such boxes.
Regarding claim 8¸ Sabatini in view of Erignac teaches all the limitations of claim 4, and further teaches
(i) determining at least the part of the digital image with the generative model depending on a predetermined mask, or
(ii) determining at least the part of the digital image with the generative model depending on a predetermined bounding box (see Sabatini paragraph [0059]).
Regarding claim 10¸ Sabatini in view of Erignac teaches all the limitations of claim 8, and further teaches training the generative model to synthesize at least the part of the digital image depending on the mask or the bounding box (see Sabatini paragraphs [0059]-[0060]).
Regarding claim 8¸ Sabatini in view of Erignac teaches all the limitations of claim 4, and but in the above embodiment does not expressively teach
determining at least the part of the digital image with the generative model depending on a predetermined mask.
However, as noted above, Sabatini does teach (ii) determining at least the part of the digital image with the generative model depending on a predetermined bounding box (see Sabatini paragraph [0059]).
Furthermore, Erignac goes on teach using a mask for outlining an object as an alternative to a bounding box (see Erignac column 6, “Classifier 240 may output a mask showing pixels having a roadness score above a pre-determined threshold and/or pixels labeled as a road. Alternatively, or additionally, classifier 240 may output pixel coordinates, bounding boxes, or any other reference to the location of pixels determined to be a road”).
Therefore, one of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of simple substitution to replace the predetermined bounding box of Sabatini with a mask as taught in Erignac to yield the predictable results of successfully switching a known object with a rare object in an image.
Regarding claim 9¸ Sabatini in view of Erignac teaches all the limitations of claim 8, and further teaches determining the mask identifying the pixels of at least the part of the digital image with the generative model (see Sabatini paragraphs [0059]-[0060] as combined with Erignac column 6, “Classifier 240 may output a mask showing pixels having a roadness score above a pre-determined threshold and/or pixels labeled as a road. Alternatively, or additionally, classifier 240 may output pixel coordinates, bounding boxes, or any other reference to the location of pixels determined to be a road”).
Claim(s) 6 and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sabatini et al, U.S. Publication No. 2025/0157174 in view of Sainburg et al, “Generative adversarial interpolative autoencoding: adversarial training on latent space interpolations encourages convex latent distributions” (published at https://arxiv.org/abs/1807.06650, April 2019).
Regarding claim 6¸Sabatini teaches all the limitations of claim 1, but does not expressively teach determining the part of the digital image with the generative model, from random noise or with latent space interpolation.
However, Sainburg in a similar invention in the same field of endeavor teaches a method involving a generative model (see Sainburg Figure 1, “Generator”) configured to determine part of a digital image (see Figure 1, middle image after “Generator” which has replaced the woman’s face with the man’s face) as taught in Sabatini comprising
teach determining the part of the digital image with the generative model, from random noise or with latent space interpolation (see section 1, final paragraph, “We propose a novel AE that hybridizes features of an AE and a GAN. Our network is trained explicitly to control for the structure of latent representations and promotes convexity in latent space by adversarially constraining interpolations between data samples in latent space to produce realistic samples”).
One of ordinary skill in the art before the effective filing date of the invention would have found it obvious as a matter of simple substitution to replace the method of determining the part of the digital image taught in Sabatini with that of Sainburg to yield the predictable results of successfully generating a synthesized image.
Regarding claim 7¸ Sabatini in view of Sainburg teaches all the limitations of claim 6, and further teaches training the generative model to synthesize at least the part of the digital image, from random noise or with latent space interpolation or with latent space interpolation (see Sainburg section 2, final paragraph, “We also train the network on interpolations in the generator, to explicitly train the generator to produce interpolations (Gd(zint)) which deceive the discriminator and are closer to the distribution in X than interpolations from an unconstrained AE”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CASEY L KRETZER whose telephone number is (571)272-5639. The examiner can normally be reached M-F 10:00-7:00 PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Payne can be reached at (571)272-3024. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CASEY L KRETZER/Primary Examiner, Art Unit 2635