Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 22 and 41 are objected to because of the following informalities:
Claims 22 and 41 recite generating a refined image; it should be amended to generating a refined synthetic image.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 22-24, 26 and 41 are rejected under 35 U.S.C. 103 as being unpatentable over Miron et al. (S-Flow GAN, Computer Vision and Pattern Recognition, 2019) in view of Vinker et al. (Deep Single Image Manipulation, Computer Vision and Pattern Recognition, 2020) in view of Huang et al. (US 2021/0012486).
Regarding claim 22, Miron et al. (hereinafter Miron) discloses generating synthetic data (Miron, 4 Results, [0001], “in (fig 4) present consecutive images depicting the video to video synthesis”), comprising:
adapting at least part of a generative machine learning model for receiving primitive feature as input (Miron, 3.2 Embedding edge maps, [0001], “we provide a learnt representation of an edge map to the generator”. The edge map reads on primitive feature), feature of one or more digital images (Miron, Fig. 2 illustrates edge map including feature of a digital image), and where a primitive feature is an independent variable in a machine learning model and is derived from one or more digital images (Miron, 2.2.2 Learning edges by a neural network, [0002], “Providing the generator with deterministic edge map will produce the same scene, so we train the DNED to take as input that deterministic edge map, learn its representation and produce a variant of that edge map, as a superposition of edges seen in real datasets”. The edge map is derived from the digital image and an independent variable in a machine learning model); and
generating one or more refined synthetic images (Miron, 4 Results, [0001], “in (fig 4) present consecutive images depicting the video to video synthesis”), where the generative machine learning model uses the primitive feature as constraints when generating a refined image (Miron, 3.2 Embedding edge maps, [0002], “To allow a stable training we begin training our GAN with the edge maps from the Laplacian operator. After stabilization of the generator and discriminator, we provide our generator with edge maps from the DNED”. Fig. 3 right picture illustrates the model can generate lower-level details in the image, thus improving its photo-realism);
Miron teaches primitive feature; Miron does not expressly disclose “a plurality of primitive features”;
Vinker et al. (hereinafter Vinker) discloses a plurality of primitive features (Vinker, I Introduction, [0002], “The training image is first represented with a primitive representation, which can be unsupervised (an edge map, unsupervised segmentation), supervised (segmentation map, landmarks) or a combination of both”);
where the plurality of primitive features are a subset of a set of features of an image (Vinker, 3.3 Primitive images, [0001], “Two standard image primitives used by previous conditional generators are the edge representation of the image and the semantic instance/segmentation map of the image”);
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the plurality of primitive features of Vinker into the image generation process of Miron to generate the refined image. The motivation for doing so would have been improving image quality.
Miron teaches generating one or more refined synthetic images by providing the generative machine learning model (Miron, Fig. 3 illustrates using GAN to generate refined synthetic image); Miron as modified by Vinker does not expressly disclose “providing the generative machine learning model with one or more input images”;
Huang et al. (hereinafter Huang) discloses providing a generative machine learning model with one or more input images (Huang, [0008], “training a bidirectional generative adversarial network including a generator G that generates a target image from a source image using domain matching, texture propagation, and shape prior constraints”. In addition, in paragraph [0042], “synthesize a target domain image from a source domain image”);
a system for generating synthetic data (Huang, [0015], “a Generative Adversarial Network (GAN) to perform image synthesis”. In addition, in paragraph [0100], “Turning briefly to FIG. 8, there is shown architecture detail of one example embodiment of computer 250 that has software instructions for storage of data and programs in computer-readable media”);
at least one hardware processor (Huang, [0100], “One or more CPUs such as processor(s) 830”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to apply the concept of an input image is provided to an generative machine learning model of Huang to modify the machine learning model of Miron to generate the refined image. The motivation for doing so would have been improving image quality.
generate the refined image of Miron by providing input image to the generative machine learning model, as taught by Huang. The motivation for doing so would have been .
Regarding claim 23, Miron as modified by Vinker and Huang with the same motivation from claim 22 discloses a texture value (Huang, [0028], “Texture propagation propagates textures that are found in the source image to a target image”).
Regarding claim 24, Miron discloses an edge (Miron, 1 Introduction, [0005], “we incorporate a neural network to embed edge maps”).
Regarding claim 26, Miron as modified by Vinker with the same motivation from claim 22 discloses the plurality of primitive features is extracted from a plurality of digital images (Vinker, 3.1 Model, [0001], “extracts features from the predicted and actual images and computes the difference between them”).
Regarding claim 41, Miron discloses a method for generating synthetic data (Miron, 4 Results, [0001], “in (fig 4) present consecutive images depicting the video to video synthesis”).
The limitations recite in claim 41 are similar in scope to the functions recited in claim 22 and therefore are rejected under the same rationale.
Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Miron et al. in view of Vinker et al. in view of Huang et al. (US 2021/0012486), as applied to claim 22, in further view of Wang et al. (Deep learning-based vehicle detection with synthetic image data, IET Intelligent Transport Systems, 2019).
Regarding claim 27, Miron as modified by Vinker and Huang does not expressly disclose “providing the one or more refined synthetic images to a perception model for the purpose of training the perception model”;
Wang et al. (hereinafter Wang) discloses providing synthetic images to a perception model for the purpose of training the perception model (Wang, 3.6 Summary, [0001], “we present our method to train vehicle detectors with synthetic images”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the synthetic image generation of Miron to provide synthetic images for training a perception model, as taught by Wang. The motivation for doing so would have been improving model accuracy.
Claims 28-29 are rejected under 35 U.S.C. 103 as being unpatentable over Miron et al. in view of Vinker et al. in view of Huang et al. (US 2021/0012486) in view of Wang et al., as applied to claim 27, in further view of Zhang et al. (US 2019/0302768).
Regarding claim 28, Miron as modified by Vinker, Huang and Wang does not expressly disclose “an autonomous computerized system”;
Zhang et al. (hereinafter Zhang) discloses an autonomous computerized system (Zhang, [0020], “autonomous vehicle 101 includes, but is not limited to, perception and planning system 110”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the concept of the autonomous computerized system into the image generation system, as taught by Miron as modified by Vinker, Huang and Wang. The motivation for doing so would have been improving efficiency of the synthetic image generation and training process.
Regarding claim 29, Miron as modified by Vinker, Huang, Wang and Zhang with the same motivation from claim 28 discloses an autonomous driving system (ADS) (Zhang, [0020], “autonomous vehicle 101 includes, but is not limited to, perception and planning system 110, vehicle control system 111, wireless communication system 112, user interface system 113, infotainment system 114, and sensor system 115”).
Claim 30 is rejected under 35 U.S.C. 103 as being unpatentable over Miron et al. in view of Vinker et al. in view of Huang et al. (US 2021/0012486), as applied to claim 22, in further view of Kowalski et al. (US 2021/0390761).
Regarding claim 30, Miron as modified by Vinker and Huang does not expressly disclose “a video game”;
Kowalski et al. (hereinafter Kowalski) discloses a video game (Kowalski, [0023], “The dynamic scene image generator 100 receives a query and in response generates a synthetic output image which it sends as output image 116 to the client device. The client device uses the output image 116 for one of a variety of useful purposes including but not limited to: generating a virtual webcam stream, generating video of a computer video game”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate synthetic output images of a video game of Kowalski into the image generation system, as taught by Miron as modified by Vinker and Huang. The motivation for doing so would have been allowing generate refined synthetic images corresponding to content of a video game.
Claim 40 is rejected under 35 U.S.C. 103 as being unpatentable over Miron et al. in view of Vinker et al. in view of Huang et al. (US 2021/0012486), as applied to claim 22, in further view of Huang et al. (US 2023/0153949) hereinafter Huang 949.
Regarding claim 40, Miron as modified by Vinker and Huang does not expressly disclose “a stable diffusion model”;
Huang 949 discloses a stable diffusion model (Huang 949, [0045], “a generative model can generate an output image 104, or output image data. In at least one embodiment, this generative model can include a diffusion probabilistic model”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the image generation process of Miron using the diffusion model of Huang 949. The motivation for doing so would have been improving image quality.
Allowable Subject Matter
Claims 25 and 31-39 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYLE ZHAI whose telephone number is (571)270-3740. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ke Xiao can be reached at (571) 272 - 7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYLE ZHAI/Primary Examiner, Art Unit 2611
Prosecution Insights