CTNF 18/874,780 CTNF 70782 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Priority 02-26 AIA Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55. Specification 06-16 AIA Applicant is reminded of the proper language and format for an abstract of the disclosure. The abstract should be in narrative form and generally limited to a single paragraph on a separate sheet within the range of 50 to 150 words in length. The abstract should describe the disclosure sufficiently to assist readers in deciding whether there is a need for consulting the full patent text for details. The language should be clear and concise and should not repeat information given in the title. It should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc. In addition, the form and legal phraseology often used in patent claims, such as “means” and “said,” should be avoided. The abstract of the disclosure is objected to because 1) the abstract should be limited to a single paragraph on a separate sheet within the range of 50 to 150 words in length. 2) the abstract should not be same as Claims 1 . So delete line 1, “The present application relates to image processing.” And no space after “and” and delete a) or b) or c). A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b). Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 1-12, 15, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Andreini et al ("Image generation by GAN and style transfer for agar plate image segmentation" in IDS) in view of Bereciartua-Perez et al (US Pub 20230017425 A1) . Regarding Claim 1, Andreini et al teaches a computer-implemented method for generating synthetic training data that is usable for training a data-driven model for identifying individual objects in a surface image of a physical product that comprises at least one object (Figs. 1-12; Pages 4-8), the method comprising: a) providing image data that comprises: an object image dataset comprising a plurality of object images of the at least one object (See Fig. 1; Section 3.2.1 on Page 5 for "The distribution is modeled by a Wasserstein GAN with Gradient Penalty (WGAN-GP) [29], which is trained with the segmentation labels extracted from the training set of the MicrobIA Haemolysis Dataset", section 3.2.1, second paragraph), at least one object image being associated with a label usable for annotating a content of the object image (ibid., "is trained with the segmentation labels extracted from the training set of the MicroblA Haemolysis Dataset"; See section 3.2.1, second paragraph; see also Figs. 1 and 4), wherein the label comprises a property that describes the at least one object in the at least one object image ("The distribution is modeled by a Wasserstein GAN with Gradient Penalty (WGAN-GP) [29], which is trained with the segmentation labels extracted from the training set of the MicrobIA Haemolysis Dataset", section 3.2.1; "the MicroblA Haemolysis Dataset is currently the only public dataset containing agar plate images with segmentation annotations. MicroblA contains a segmentation ground-truth labeling obtained following the procedure reported in [22]", section 4.1; the "MicrobIA Haemolysis Dataset" contains images annotated with masks of bacteria colonies) and a background image representing a background of a surface image the physical product (See section 3.2.2; Figs. 1, 4 for "A suitable set of background images and colony prototypes is collected", section 3.2.2, first paragraph; see Fig. 1; see also second paragraph of section 3.2.2, "A set of 16 different background images and 30 patches have been used during the experiments, augmented using different scales, rotations and lighting, in order to increase the image variability"); b) generating a synthetic object image dataset from the object image dataset, wherein the synthetic object image dataset comprises a plurality of synthetic object images of the at least one object, at least one synthetic object image being associated with a label ("The distribution is modeled by a Wasserstein GAN with Gradient Penalty (WGAN-GP) [29], which is trained with the segmentation labels extracted from the training set of the MicrobIA Haemolysis Dataset", section 3.2.1, first paragraph;" At the end of the training phase, the generator is able to produce label maps representing realistic bacterial growth distributions", section 3.2.1, second paragraph; see also Figs. 6 and 11); and c) generating a plurality of first synthetic training data samples, wherein each first synthetic training data sample is generated by selecting one or more object images from the synthetic object image dataset and by plotting the selected one or more object images at one or more locations on the background image ("The output produced by the WGAN-GP is fed to the generator engine. In fact, to generate a realistic image, the engine needs a series of patches of bacterial colonies (See Fig. 7) together with empty plate images (that is, where there is no bacterial growth). The patches are rendered on the empty plates following the growth distribution modeled by the WGAN-GP", section 3.2.2, first paragraph; "The rendering procedure blends a set of random colony models on a background image, following the simulated growth distribution. Colony models are randomly positioned onto the image, ensuring that the nucleus (given by the background/foreground. colony mask) does not fall outside the growing mask generated by the WGAN-GP", p. 6, last paragraph; see Fig. 8 where the growing masks of the WGAN-GP are placed on a black background image and an empty plate image, both considered a "background" image). Andreini et al’s article is directed at the problem of segmenting bacteria colonies in agar plate images. By means of his common general knowledge, the skilled person is also aware that different computer vision tasks, require different kind of labelling when solved by means of a machine learning algorithm (e.g., neural networks). A segmentation task requires at least annotated segmentation masks, a semantic segmentation task requires at least annotated segmentation masks and class labels, etc. By the same token, a task of training a neural network for predicting properties corresponding to a plant disease would require annotation/ label/ground truth data containing such properties for training the neural network. Andreini et al does not teach the method providing image data that comprises: a property value indicative of a damage status of the at least one object in the at least one object image. Bereciartua-Perez et al teaches the method providing image data that comprises: a property value indicative of a damage status of the at least one object in the at least one object image. (Figs. 1, 2, 7; Paragraph 0045-0048, 0052-0056). Thus, it would have been obvious to a person of ordinary skill in the art when the invention was made to incorporate the teachings of Bereciartua-Perez et al into Andreini et al for predicting properties corresponding to a plant disease would require annotation/ label/ground truth data containing such properties for training the neural network. Regarding Claim 2, Andreini et al teaches the computer-implemented method wherein the at least one object comprises a plurality of objects, at least two objects of which are associated with labels that comprise different property values ("The distribution is modeled by a Wasserstein GAN with Gradient Penalty (WGAN-GP) [29], which is trained with the segmentation labels extracted from the training set of the MicrobIA Haemolysis Dataset", section 3.2.1, first paragraph;" At the end of the training phase, the generator is able to produce label maps representing realistic bacterial growth distributions", section 3.2.1, second paragraph; see also Figs. 6 and 11). Regarding Claim 3, Andreini et al teaches the computer-implemented method wherein in step b), the synthetic object image dataset is generated using a generative model (section 3.2.1 (“WGAN-GP”) and Fig. 4). Regarding Claim 4, Andreini et al teaches the computer-implemented method wherein the generative model comprises a conditional generative adversarial network, cGAN (section 3.2.1 and Fig. 4). Regarding Claim 5, Andreini et al teaches the computer-implemented method herein in step c) the selected one or more object images are plotted on the background image according to a rule derived from one or more surface image samples of the physical product (see section 3.2.2, p. 6, last paragraph ("The rendering procedure blends a set of random colony models on a background image, following the simulated growth distribution. Colony models are randomly positioned onto the image, ensuring that the nucleus (given by the background/foreground colony mask) does not fall outside the growing mask generated by the WGAN-GP"). Regarding Claim 6, Andreini et al teaches the computer-implemented method wherein step c) further comprises a step of generating a plurality of second synthetic training data samples from the plurality of first synthetic training data samples using an image-to-image translation model, wherein the image-to-image translation model has been trained to generate a synthetic surface image closer to a realistic surface image of the physical product (see section 3.2.3 ("The style transfer algorithm proposed in [42], which is particularly focused on Photorealistic image stylization, is employed to further improve the quality of the images generated by the automated engine"). Regarding Claim 7, Andreini et al teaches the computer-implemented method wherein the image-to-image translation model comprises an image-to-image generative adversarial network. (section 3.2.1 and Fig. 4). Regarding Claim 8, Bereciartua-Perez et al teaches the computer-implemented method wherein the property comprises one or more of: an annotation usable for classifying a plant disease in an image of a plant; an annotation usable for classifying cathode active material particles in an image of a battery material; an annotation usable for classifying cells in an image of a biological material; an annotation usable for classifying insects on a leaf; and an annotation usable for classifying defects in a coating. (Paragraph 0045-0048, 0052-0056). Regarding Claim 9, Bereciartua-Perez et al teaches the computer-implemented method wherein the property value comprises one or more of: a property value indicative of a plant damage; anda property value indicative of a deviation from a standard for an industrial product. (Figs. 1, 2, 7; Paragraph 0045-0048, 0052-0056). Regarding Claim 10, Bereciartua-Perez et al teaches the computer-implemented method wherein the property value is provided as a damage percentage, which is preferably usable to determine an amount of treatment to be applied to the physical product. (Figs. 1, 2, 7; Paragraph 0045-0048, 0052-0056). Regarding Claim 11, Andreini et al teaches the computer-implemented method the preceding claims, further comprising a step of providing a user interface allowing a user to provide the image data. (Figs. 1-2). Regarding Claim 12, Andreini et al teaches the computer-implemented wherein step c) further comprises providing the label for one or more first synthetic training data samples in the plurality of first synthetic training data samples. ("The distribution is modeled by a Wasserstein GAN with Gradient Penalty (WGAN-GP) [29], which is trained with the segmentation labels extracted from the training set of the MicrobIA Haemolysis Dataset", section 3.2.1, first paragraph;" At the end of the training phase, the generator is able to produce label maps representing realistic bacterial growth distributions", section 3.2.1, second paragraph; see also Figs. 6 and 11). Regarding Claim 15, Andreini et al teaches A synthetic training data generating apparatus for generating synthetic training data that is usable for training a data-driven model for analysing a surface image of a physical product that comprises at least one object, the synthetic training data generating apparatus comprising one or more processors configured to perform the steps of the method. ("The distribution is modeled by a Wasserstein GAN with Gradient Penalty (WGAN-GP) [29], which is trained with the segmentation labels extracted from the training set of the MicrobIA Haemolysis Dataset", section 3.2.1, first paragraph;" At the end of the training phase, the generator is able to produce label maps representing realistic bacterial growth distributions", section 3.2.1, second paragraph; see also Figs. 6 and 11). Regarding Claim 18, Andreini et al teaches A computer program product comprising instructions which, when the program is executed by a processing unit, cause the processing unit to carry out the steps of the method. (Figs. 1, 4) . Allowable Subject Matter 12-151-08 AIA 07-43 12-51-08 8. Claim s 13-14, 16-17, 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Conclusion 9. Examiner cites particular columns and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. 10. It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). Examiner’s Note 11. Examiner has cited particular paragraphs/columns and line numbers or figures in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested from the applicant, in preparing the responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. Applicant is reminded that the Examiner is entitled to give the broadest reasonable interpretation to the language of the claims. Furthermore, the Examiner is not limited to Applicant’s definition which is not specifically set forth in the claims. In the case of amending the claimed invention, Applicant is respectfully requested to indicate the portion(s) of the specification which dictate(s) the structure relied on for proper interpretation and also to verify and ascertain the metes and bounds of the claimed invention. 12. Any inquiry concerning this communication or earlier communications from the examiner should be directed to VIJAY SHANKAR whose telephone number is (571)272-7682. The examiner can normally be reached M-F 9 am- 6 pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Eason can be reached at 571-270-7230. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. VIJAY SHANKAR Primary Examiner Art Unit 2624 /VIJAY SHANKAR/Primary Examiner, Art Unit 2624 Application/Control Number: 18/874,780 Page 2 Art Unit: 2624 Application/Control Number: 18/874,780 Page 3 Art Unit: 2624 Application/Control Number: 18/874,780 Page 4 Art Unit: 2624 Application/Control Number: 18/874,780 Page 5 Art Unit: 2624 Application/Control Number: 18/874,780 Page 6 Art Unit: 2624 Application/Control Number: 18/874,780 Page 7 Art Unit: 2624 Application/Control Number: 18/874,780 Page 8 Art Unit: 2624 Application/Control Number: 18/874,780 Page 9 Art Unit: 2624 Application/Control Number: 18/874,780 Page 10 Art Unit: 2624 Application/Control Number: 18/874,780 Page 11 Art Unit: 2624 Application/Control Number: 18/874,780 Page 12 Art Unit: 2624
Prosecution Insights