CTNF 18/895,832 CTNF 99512 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 112 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 1. Claims 3-4, and 7 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. The term “and/or” in claims 3-4, and 7 renders the claims indefinite. The Examiner treats each instance of “and/or” as the disjunctive “or”, where only one of the options may be present to satisfy the feature provided in the claim. Claim Rejections - 35 USC § 102 07-06 AIA 15-10-15 2. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-12-aia AIA (a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention. 07-15-aia AIA Claims 1-7, and 10-11 are rejected under 35 U.S.C. 102 as being anticipated by US 202001601 78 A1: Amlan Kar et al., (herein after “Kar”). 3. Regarding cla im 1, A method [[(100)]] for providing synthetic data for use in machine learning (Kar, §Abstract: “a generative model is used to synthesize datasets for use in training a downstream machine learning model to perform an associated task.” ) , wherein the synthetic data comprise synthetic, photorealistic image information (Kar, §Abstract: “such as a probabilistic grammar—and applying the scene graph to the generative model to compute updated scene graphs more representative of object attribute distributions of real-world datasets.” ) , said method comprising the following method steps: - generating [[(101)]] simulation data, wherein the simulation data are in the form of annotated data and are generated by a simulation environment (Kar, P[0023]: “Corresponding synthetic images (e.g., images 114) and pixel level annotations (e.g., ground truth 116) may be rendered easily by placing objects or elements as described in the scene graph 106.” , and P[0030]: “The renderer 112 may render the synthetic dataset, including the images 114 and the ground truth 116.” ), - determining [[(102)]] annotations from the generated simulation data (Kar, P[0030]: “because the task may be object detection in this example, the ground truth 116 (e.g., the ground truth 116A) that may be determined—automatically, in embodiments—from the render information (e.g., the bounding shapes corresponding to the vehicles in the ground truth 116A).” , and P[0023]: “Corresponding synthetic images (e.g., images 114 ) and pixel level annotations (e.g., ground truth 116) may be rendered easily by placing objects or elements as described in the scene graph 106.” ), - generating [[(103)]] the synthetic data by means of semantic image synthesis (SIS) using the determined annotations as input for the semantic image synthesis (SIS) (Kar, P[0031]: “For example, since the objects include associated semantic information (e.g., the system knows that an object is a car, or a pedestrian), compositing or modifying the synthetic scenes will still render perfect ground truth 116.” ), - annotating [[(104)]] the generated synthetic data based on an output from the simulation environment (Kar, P[0030]: “The synthetic dataset may then be used to train a task network via a task network trainer 118.” , and P[0032]: “The task network trainer 118 may apply the image data representative of the images 114 to the task network, and the predictions of the task network may be compared against the ground truth 116.” ). Claims 10 and 11 recite features nearly identical to those recited in claim 1. Claims 10 and 11 are rejected for reasons analogous to those discussed above in conjunction with claim 1. Regarding claim 2, characterized in that the output from the simulation environment comprises the determined annotations, wherein the determined annotations are at least in part carried over into the generated synthetic data for the purpose of annotation [[(104)]] , is disclosed by Kar, §Abstract: “a generative model is used to synthesize datasets for use in training a downstream machine learning model to perform an associated task The synthesized datasets may be generated by sampling a scene graph from a scene grammar—such as a probabilistic grammar—and applying the scene graph to the generative model to compute updated scene graphs more representative of object attribute distributions of real-world datasets.” , P[0024]: “to generate a traffic scene, a centerline of a road may be defined, followed by parallel lines, then vehicles positioned within lanes, etc. The structure of the scene may be defined by the probabilistic grammar 102, while the attributes corresponding to each of the objects or features may be sampled from parametric distributions—which require careful tuning to be accurate. As such, using the scene graphs 106 sampled from the probabilistic grammar 102 to generate the images 114 and ground truth 116 for training downstream task networks does not generate as accurate results as using the transformed scene graphs 110” , and P[0030]: “ The renderer 112 may render the synthetic dataset, including the images 114 and the ground truth 116. The synthetic dataset may then be used to train a task network via a task network trainer 118. For example, assuming that a task of the task network is object detection, the transformed scene graphs may represent one or more objects such that the rendered images 114 (e.g., rendered image 114A) may depict the objects (e.g., the vehicles in the rendered image 114A). In addition, because the task may be object detection in this example, the ground truth 116 (e.g., the ground truth 116A) that may be determined—automatically, in embodiments—from the render information (e.g., the bounding shapes corresponding to the vehicles in the ground truth 116A).” , shows that the information/data output from the simulated environment contain the annotations which are carried through the image transformations to the generated synthetic data which is then further annotated for object detection. Regarding claim 3, characterized in that the generated synthetic data comprise at least one photorealistic image (Kar, P[0008]: “The generative model therefore serves as an aid in bridging the content gap that previously existed between synthetic data and real-world data. As a result, and as a non-limiting example, where conventional approaches may generate a synthetic scene including photo-realistic vehicles, pedestrians, buildings, and/or other objects distributed in an unrealistic manner (e.g., vehicles facing perpendicular to a direction of travel on a road, sidewalks wider than in a real-world environment, pedestrians too close together, etc.), the techniques of the present disclosure allow for these synthetic scenes to not only include photo-realistic renderings, but also to include distributions of objects in a more realistic, real-world analogous manner.” ), wherein, based on the annotation [[(104)]] of the generated synthetic data, the at least one photorealistic image is classified in an image- element manner, in particular pixel-wise, wherein the classification preferably comprises object detection and/or semantic segmentation (Kar, P[0024]: “when sampled by the scene graph sampler 104. For a non-limiting example, to generate a traffic scene, a centerline of a road may be defined, followed by parallel lines, then vehicles positioned within lanes, etc. The structure of the scene may be defined by the probabilistic grammar 102, while the attributes corresponding to each of the objects or features may be sampled from parametric distributions—which require careful tuning to be accurate. As such, using the scene graphs 106 sampled from the probabilistic grammar 102 to generate the images 114 and ground truth 116 for training downstream task networks does not generate as accurate results as using the transformed scene graphs 110” , and P[0031]: “For example, since the objects include associated semantic information (e.g., the system knows that an object is a car, or a pedestrian), compositing or modifying the synthetic scenes will still render perfect ground truth 116.” , and P[0023]: “Corresponding synthetic images (e.g., images 114) and pixel level annotations (e.g., ground truth 116) may be rendered easily by placing objects or elements as described in the scene graph 106.” ) Regarding claim 4, characterized in that the determined annotations comprise at least one semantic label map and/or at least one bounding box (Kar, P[0037]: “(e.g., the transformed scene graph may include semantic and/or other information indicating attributes of objects in the scenes 210). In other examples, a combination of a feature extractor, the transformed scene graphs 110, and/or another attribute determination method may be used to determine the distributions 214 corresponding to the generated synthetic scenes 210 and/or the real scenes 212.” , and P[0047]: “The distribution transformer 108 may compute or predict the transformed scene graphs 110 which may be used by the renderer 112 to generate the synthetic scenes and labels 220 (e.g., the images 114 and the ground truth 116 of FIG. 1). A task network 224A (representing an untrained task network 224) may be trained using the synthetic dataset 220 (e.g., using the task network trainer 118 of FIG. 1). Once trained, or at least partially trained, the task network 224B (representing a partially trained or fully trained task network 224) may be tested on a real-world labeled validation set 222.” ). Regarding claim 5, characterized in that the determination [[(102)]] of the annotations based on the generated simulation data is performed as a function of at least one situation condition, wherein the situation condition is specific to a type of situation represented in the generated simulation data, preferably a traffic situation of a represented traffic scene (Kar, P[0024]: “The probabilistic grammar 102, P, may define a generative process for generating the scene graphs 106, S, when sampled by the scene graph sampler 104. For a non-limiting example, to generate a traffic scene, a centerline of a road may be defined, followed by parallel lines, then vehicles positioned within lanes, etc. The structure of the scene may be defined by the probabilistic grammar 102, while the attributes corresponding to each of the objects or features may be sampled from parametric distributions” , and P[0052]: “the validation set, V, includes one-hundred random images taken from the KITTI training set which includes corresponding ground truth labels. The remaining training data (images only) forms X.sub.R. The KITTI dataset includes challenging traffic scenarios and scenes ranging from highways to urban to more rural neighborhoods.” ). Regarding claim 6, characterized in that the method steps are repeatedly performed in order to obtain a data quantity of the annotated generated synthetic data, which data quantity represents a variety of different situations and objects for use in machine learning (Kar, §Abstract: “a generative model is used to synthesize datasets for use in training a downstream machine learning model to perform an associated task. The synthesized datasets may be generated by sampling a scene graph from a scene grammar—such as a probabilistic grammar—and applying the scene graph to the generative model to compute updated scene graphs more representative of object attribute distributions of real-world datasets.” , and Fig. 4A-4B showing a variety of situations.) . Regarding claim 7, characterized in that the annotated generated synthetic data are used for machine learning by being used to train and/or validate a machine learning model (Kar, §Abstract: “a generative model is used to synthesize datasets for use in training a downstream machine learning model to perform an associated task. The synthesized datasets may be generated by sampling a scene graph from a scene grammar—such as a probabilistic grammar—and applying the scene graph to the generative model to compute updated scene graphs more representative of object attribute distributions of real-world datasets. The downstream machine learning model may be validated against a real-world validation dataset, and the performance of the model on the real-world validation dataset may be used as an additional factor in further training or fine-tuning the generative model for generating the synthesized datasets specific to the task of the downstream machine learning model.” ) , wherein the machine learning model is preferably trained for image classification, in particular object detection and/or semantic segmentation based on the annotations, preferably for application in at least partially automated driving (Kar, P[0021]: “For example, given a dataset of real imagery, X.sub.R, and a task, T (e.g., object detection, lane classification, etc.), the process 100 may be used to synthesize a training dataset, D.sub.T=(X.sub.T, Y.sub.T), with synthesized imagery, X.sub.T, and corresponding ground truth, Y.sub.T. In addition, where a validation set-” ) . Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA 4. Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Kar in view of DE 102020210711 A1: Edgar Schoenfeld et al., (herein after “Schoenfeld”) . Regarding claim 8, characterized in that the generated simulation data comprise simulated image information (Kar, P[0023]: “Corresponding synthetic images (e.g., images 114) and pixel level annotations (e.g., ground truth 116) may be rendered easily by placing objects or elements as described in the scene graph 106.” ) ,which preferably features no or less photorealism than the photorealistic image information of the synthetic data (Kar, P[0008]: “the techniques of the present disclosure allow for these synthetic scenes to not only include photo-realistic renderings, but also to include distributions of objects in a more realistic, real-world analogous manner.” ) , wherein the annotations of the generated simulation data annotate the simulated image information (Kar, P[0023]: “Corresponding synthetic images (e.g., images 114) and pixel level annotations (e.g., ground truth 116) may be rendered easily by placing objects or elements as described in the scene graph 106.” , and the annotations are kept through image transformations which annotate the simulated image data/information.) , Kar does not explicitly disclose “ wherein, out of the annotations and the simulated image information, only the annotations are used as input for the semantic image synthesis (SIS). ” However, Schoenfeld discloses wherein, out of the annotations and the simulated image information, only the annotations are used as input for the semantic image synthesis (SIS) in the §Abstract: “Method (100) for training a generator (1) for images (3) from a semantic map (2, 5a) which assigns to each pixel of the image (3) a semantic meaning (4) of an object to which this pixel belongs” , which is the pixel-wise annotated information for the image, and P[15]: “The generator creates the realistic images from a semantic map. This semantic map assigns a semantic meaning of an object to which this pixel belongs to each pixel of the realistic image to be generated.” , where only the annotated information is used to generate a semantic image. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Kar to use the annotations only as input to synthesize a semantic image, as taught by Schoenfeld, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. It is predictable that the proposed modification would have provided the benefit of reducing computational load for image generation by reducing amount of data. Conclusion 5. Any inquiry concerning this communication or earlier communications from the examiner should be directed to TY M BEATTY whose telephone number is (703)756-5370. The examiner can normally be reached Mon-Fri: 8AM-4PM EST.. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571) 272 - 3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /TY MITCHELL BEATTY/Examiner, Art Unit 2663 /GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698 Application/Control Number: 18/895,832 Page 2 Art Unit: 2663 Application/Control Number: 18/895,832 Page 3 Art Unit: 2663 Application/Control Number: 18/895,832 Page 4 Art Unit: 2663 Application/Control Number: 18/895,832 Page 5 Art Unit: 2663 Application/Control Number: 18/895,832 Page 6 Art Unit: 2663 Application/Control Number: 18/895,832 Page 7 Art Unit: 2663 Application/Control Number: 18/895,832 Page 8 Art Unit: 2663
Prosecution Insights