DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. KR10-2023-0115734, filed on 08/31/2023.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-9 is/are rejected under 35 U.S.C. 102a(1) as being anticipated by Hassan (Deep Learning).
For claim 1, Hasan teaches a method, of training an image generation model, performed by at least one processor,
the method comprising: receiving a training image in a first domain style; “Several works on conditional image generation [27, 80, 148, 206] have exploited paired datasets of real images with semantic information, including semantic segmentation [27, 148] and body part labels [106] for training realistic image synthesis models” and “training a conditional image generation model from richer control inputs would require a large dataset of paired real images with pixel aligned intrinsic properties such as 3D structure, textures, materials and reflections” (Section 5.1, page 76).
extracting, by the at least one processor, first content information for the training image; “The goal of the data augmentation step is not to increase the dataset size but rather improve the quality of the information extracted from it by the model” (Section 2.1, page 14).
generating, by the at least one processor, a plurality of pieces of augmented content information perturbed from the first content information as part of image processing by augmenting the first content information; “Generating gray scale or low resolution images from high resolution-colored ones are an easy and exact process which can be used to generate a theoretically unlimited amount of training data for the tasks of image colorization and super-resolution” (Section 2.2, page 15).
and training an image generation model to generate a synthetic image in the first domain style from an input image in a second domain style different from the first domain style, wherein the training of the image generation model is based on: the training image, the first content information, and the plurality of pieces of augmented content information. “Learn the unknown parameters from the training data (interpreted as first content information) directly and apply them to generate images (interpreted as training image) from new input geometry (interpreted as augmented content information)” and “exploit synthetically rendered data using a state-of-the-art physically based renderer as supervised target while using an adversarial loss with real images to acquire realistic looking results. Using realistically rendered images also gives us fine-grained control over the data, which we exploit to conduct various experiments for analyzing our model” (Section 4.3, page 61).
For claim 2, wherein the first content information represents structural information on objects in the training image. “Model benefits from a rich set of input modalities, while learning realistic mappings (interpreted as structural information) which generalize to novel geometries and segmentations, and integrate the objects seamlessly into provided image content (interpreted as first content information)” (Figure 4.2).
For claim 3, wherein the generating the plurality of pieces of augmented content information comprises perturbing the first content information as part of image processing, and wherein the image processing comprises at least one of translation processing, rotation processing, flipping processing, enlargement processing, reduction processing, crop processing, brightness adjustment processing, saturation adjustment processing, or noise injection processing. “Operations include (i) independent color shifts (interpreted as saturation adjustment processing) on the ROB channels to simulate chromatic aberrations in the real lens, (ii) depth-blur operation to match the depth-of-field of the camera, (iii) radial motion-blur that matches the blur caused by the moving camera, (iv) color noise (interpreted as noise injection processing) and (v) glow effects (interpreted as brightness adjustment processing) to imitate sensor overexposure” (section 4.2) and “the translation process preserves the image content” (section 2.4.3).
For claim 4, wherein a pair of the training image and the first content information and pairs of the training image and each of the plurality of pieces of augmented content information are used as training data for training the image generation model. “conditional image-to-image generation are mostly supervised with paired data (interpreted as pairs of the training image) from both domains” (Section 5.3).
For claim 5, wherein the trained image generation model is trained to generate a synthetic image in the first domain style with second content information as a soft constraint. “The nature of the translation can vary greatly and is usually defined by the structure and constraints (interpreted as soft constraint) of the translation model” (Section 2.4). “The input information is represented in the output image, we decompose it back into its intrinsics using a second Intrinsic Image Decomposition (IID) network” and “introduce a second Decomposition cycle in which we train the IID network to decompose real images” (Section 5.1). “The second cycle (red) auto-encodes real images using image intrinsics as representation” (Section 5.2).
For claim 6, wherein the trained image generation model is trained to generate, based on second content information, a synthetic image in the first domain style. “Training a defen-ed rendering network for generating realistic images from synthetic image intrinsics and an intrinsic image decomposition network for decomposing real images of an object into its intrinsic properties” (Section 5.5).
For claim 7, wherein the second content information is extracted from an input image in a second domain style, and the first domain style and the second domain style are different from each other. “CycleGAN[228] is a generic method for translating between two domains without available paired data” (Section 5.4.1). “Learn a mapping from an intermediate representation to a natural RGB image (interpreted as first domain style) using a deep neural network” (Section 4.3.2). “Inputs to the network are intrinsic maps (interpreted as second domain style) consisting of albedo, normals and reflections (interpreted as second content information)” (Figure 5.3).
For claim 8, wherein the second domain style is a virtual domain style, and the first domain style is a real domain style. “Obtain the real images (interpreted as first domain style) from a fine grained car classification dataset” and “render 20,000 synthetic samples (interpreted as second domain style) of normals, albedo and reflections” (Section 5.4).
For claim 9, A method, of generating an image, performed by at least one processor, the method comprising:
receiving a training image in a first domain style; “Several works on conditional image generation [27, 80, 148, 206] have exploited paired datasets of real images with semantic information, including semantic segmentation [27, 148] and body part labels [106] for training realistic image synthesis models” and “training a conditional image generation model from richer control inputs would require a large dataset of paired real images with pixel aligned intrinsic properties such as 3D structure, textures, materials and reflections” (Section 5.1, page 76).
extracting second content information for the input image; “Learning image synthesis from real images and vice-versa” (Section 5.1). “Render 20,000 synthetic samples of normals, albedo and reflections” (Section 5.4). “As input layers to our network, we use the normal map, the depth map, object mask and material segmentation of the object” (Section 4.3.2).
and outputting a synthetic image in the first domain style associated with the second content information by using an image generation model, “train the generator R in our GIS framework to produce synthesized images” (Section 4.3.3) and “learn a mapping from an intermediate representation to a natural RGB image (interpreted as first domain style)” (Section 4.3.2).
wherein the image generation model:
receives a training image in the first domain style, “several works on conditional image generation [27, 80, 148, 206] have exploited paired datasets of real images with semantic information, including semantic segmentation [27, 148] and body part labels [106] for training realistic image synthesis models” and “training a conditional image generation model from richer control inputs would require a large dataset of paired real images with pixel aligned intrinsic properties such as 3D structure, textures, materials and reflections” (Section 5.1, page 76).
extracts first content information for the training image, “The goal of the data augmentation step is not to increase the dataset size but rather improve the quality of the information extracted from it by the model” (Section 2.1, page 14)
generates a plurality of pieces of augmented content information perturbed from the first content information as part of image processing by augmenting the first content information. “Generating gray scale or low resolution images from high resolution-colored ones are an easy and exact process which can be used to generate a theoretically unlimited amount of training data for the tasks of image colorization and super-resolution” (Section 2.2, page 15).
and is trained to generate the synthetic image in the first domain style from the input image in the second domain style, wherein the image generation model is trained based on the training image, the first content information, and the plurality of pieces of augmented content information. “Learn the unknown parameters from the training data (interpreted as first content information) directly and apply them to generate images (interpreted as training image) from new input geometry (interpreted as augmented content information)” and “exploit synthetically rendered data using a state-of-the-art physically based renderer as supervised target while using an adversarial loss with real images to acquire realistic looking results. Using realistically rendered images also gives us fine-grained control over the data, which we exploit to conduct various experiments for analyzing our model” (Section 4.3, page 61).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 10,11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hassan (Deep Learning) in view of Metwaly (US11941081B2).
For claim 10, Hassan doesn’t explicitly teach computer-readable medium storing instructions, when executed, cause performance of the method, but Metwaly teaches “a non-transitory computer-readable medium for training a model to stylize low-light images to improve perception and including instructions that when executed by a processor cause the processor to perform one or more functions is disclosed” (Metwaly, 7).
Hassan and Metwaly are combinable because they are in the same field of endeavor of image training. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine image generation of Hassan and computer-readable medium of Metwaly in order to train a style model for altering representations of low-light images to improve clarity (Metwaly, technical field).
For claim 11, Hassan teaches an information processing system comprising:
wherein the at least one computer-readable program comprises instructions that, when executed by the at least one processor, cause the information processing system to:
receive a training image in a first domain style. “Train our model using an small set of synthetic 3D models of an object category as well as a large unpaired dataset of real images of the same category” (Section 5.1) and “CycleGAN[228] is a generic method for translating between two domains without available paired data” (Hassan, Section 5.4.1).
extract first content information for the training image. “Learning image synthesis from real images and vice-versa” (Hassan, Section 5.1). “Render 20,000 synthetic samples of normals, albedo and reflections” (Hassan, Section 5.4). “As input layers to our network, we use the normal map, the depth map, object mask and material segmentation of the object” (Hassan, Section 4.3.2).
generate a plurality of pieces of augmented content information perturbed from the first content information as part of image processing by augmenting the first content information, “generating gray scale or low resolution images from high resolution-colored ones are an easy and exact process which can be used to generate a theoretically unlimited amount of training data for the tasks of image colorization and super-resolution” (Hassan, Section 2.2, page 15).
and train an image generation model to generate a synthetic image in the first domain style from an input image in a second domain style different from the first domain style, wherein training of the image generation model is based on:
the training image, the first content information, and the plurality of pieces of augmented content information. “Learn the unknown parameters from the training data (interpreted as first content information) directly and apply them to generate images (interpreted as training image) from new input geometry (interpreted as augmented content information)” and “exploit synthetically rendered data using a state-of-the-art physically based renderer as supervised target while using an adversarial loss with real images to acquire realistic looking results. Using realistically rendered images also gives us fine-grained control over the data, which we exploit to conduct various experiments for analyzing our model” (Section 4.3, page 61).
Hassan does not explicitly teach a communication interface; a memory; and at least one processor coupled to the memory and configured to execute at least one computer-readable program included in the memory, but Metwaly teaches “The training system includes a processor and a memory storing instructions” (6). “A non-transitory computer-readable medium for training a model to stylize low-light images to improve perception and including instructions that when executed by a processor cause the processor to perform one or more functions is disclosed” (7). “The training system 170 may access the processor(s) 110 through a data bus or another communication path” (16).
Hassan and Metwaly are combinable because they are in the same field of endeavor of image training. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine image generation of Hassan and computer-readable medium of Metwaly in order to train a style model for altering representations of low-light images to improve clarity (Metwaly, technical field).
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Huang (CN108205803A) and Kim (KR102482262B1).
“Extracting the image content information of the training image, and the image content information and the texture feature information of the second composite image” (Huang, 0018).
“The second feature information extracting unit for extracting the image content information of the third composite image, texture feature information and depth information and image content information of the training image and depth information” (Huang, 0075).
“Generates an augmented image by synthesizing a background image and a rendered object image” (Kim, 0063).
“By generating and providing various augmented images” (Kim, 0086).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIANA SZE whose telephone number is (571)272-9916. The examiner can normally be reached Monday-Thursday 6am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.S./Examiner, Art Unit 2614
/TERRELL M ROBINSON/Primary Examiner, Art Unit 2614
Prosecution Insights