DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/06/2026 has been entered.
Response to Arguments
Applicant arguments regarding claim rejections under 103 are considered, but are not moot in view of new ground of rejections. However, Examiner would like to answer the following arguments:
Applicant argues:
PNG
media_image1.png
140
632
media_image1.png
Greyscale
Examiner disagrees: Skrypnyk teaches:
generating, by a diffusion model, a depth map by denoising the noise input based on the guidance embedding.([0163], “The stable diffusion model is then conditioned on both the image template 612 and the text embedding by incorporating the text embedding into the stable diffusion model architecture or by conditioning the stable diffusion model's latent space on the textual information. The stable diffusion model generates populated image templates (such as images with color and depth information) by learning to reverse the process of adding noise to the input image template 612.”)
Arguments II and III are moot in view of the new ground rejections (see below OA).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 7, 15-16, 18-19, 21, 27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Skypnyk et al. (US 2024/0355064 A1).
Regarding claim 1, Skypnyk teaches:
A method for image generation, comprising:
obtaining a prompt and a noise input;([0147], “At operation 502, the interaction system 100 (alone or in combination with one or more elements of the personal AI agent 302) identifies a prompt of a user indicating a user's intent.” [0163], “The stable diffusion model generates populated image templates (such as images with color and depth information) by learning to reverse the process of adding noise to the input image template 612.”)
encoding, the prompt to obtain a guidance embedding; ([0162], “In the stable diffusion model, the user prompt 610 is converted into an embedding using a pre-trained language model, such as a transformer-based architecture (e.g., GPT or BERT). This embedding encodes the semantic information of the user prompt 610 and is used to condition the image/populated image template 612 generation process.”) and
generating, by a diffusion model, a depth map by denoising the noise input based on the guidance embedding.([0163], “The stable diffusion model is then conditioned on both the image template 612 and the text embedding by incorporating the text embedding into the stable diffusion model architecture or by conditioning the stable diffusion model's latent space on the textual information. The stable diffusion model generates populated image templates (such as images with color and depth information) by learning to reverse the process of adding noise to the input image template 612.”)
The above rejection mappings are from different embodiment/examples of Skypnyk. However, it would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the different embodiments/examples to generate an accurate depth map based on user prompt.
Regarding claim 7, Skypnyk teaches:
The method of claim 1, wherein: the prompt comprises a text prompt.([0147], “The user prompt includes a textual description or keyword(s) provided by the user, which defines the desired characteristics or subject of the generated image.”)
Regarding claim 15, Skypnyk teaches:
A system for image generation, comprising: one or more processors; one or more memory components coupled with the one or more processors; (FIG. 19) and
an image generation a diffusion model comprising parameters …and trained to generate an image and a depth map for the image by iteratively denoising a noise input based on a text prompt.([0302], “The neural network 2226 is iteratively trained by adjusting model parameters to minimize a specific loss function or maximize a certain objective. The system can continue to train the neural network 2226 by adjusting parameters based on the output of the validation, refinement, or retraining block 2112, and rerun the prediction 2110 on new or already run training data. The system can employ optimization techniques for these adjustments such as gradient descent algorithms, momentum algorithms, Nesterov Accelerated Gradient (NAG) algorithm, and/or the like. The system can continue to iteratively train the neural network 2226 even after deployment 2114 of the neural network 2226. The neural network 2226 can be continuously trained as new data emerges, such as based on user creation or system-generated training data.” [0162]-0163], “In stable diffusion models, such as Diffusion Probabilistic Models (DPMs), user prompts and image templates can be used together to guide the generation of populated image templates according to user preferences. In the stable diffusion model, the user prompt 610 is converted into an embedding using a pre-trained language model, such as a transformer-based architecture (e.g., GPT or BERT). This embedding encodes the semantic information of the user prompt 610 and is used to condition the image/populated image template 612 generation process. The stable diffusion model is then conditioned on both the image template 612 and the text embedding by incorporating the text embedding into the stable diffusion model architecture or by conditioning the stable diffusion model's latent space on the textual information. The stable diffusion model generates populated image templates (such as images with color and depth information) by learning to reverse the process of adding noise to the input image template 612. During the generation process, the model is guided by both the image template 612 (providing the initial structure) and the user prompt 610 (providing the desired characteristics).”[0168], “The interaction system 100 uses the denoising score matching objective to train the model to reconstruct the original image by reversing the noise injection process.”)
However, Skypnyk does not explicitly teach:
parameters stored in the one or more memory components
On the other hand, Skypnyk teaches, as shown in FIG. 19, the system comprises storage and processor to implement the image generation model. It would have been obvious for ordinary skills in the art to have stored the parameters in the storage component to make the implementation of the image generation model possible and faster.
Regarding claim 16, Skypnyk teaches:
The system of claim 15, the system further comprising: an occlusion component (FIG. 19, processor) configured to generate an occlusion area for a modified view of the image based on an image and the depth map. ([0185], “the interaction system 100 uses the color populated image template and depth populated image template generated by a stable diffusion model to modify a 3D mesh and apply it to a camera feed.” [0189], “The interaction system 100 composites the rendered 3D mesh onto the live camera feed to create the final augmented output. The interaction system 100 blends the rendered mesh with the camera image or uses depth information to handle occlusions between the mesh and the real-world objects in the scene.”)
Regarding claim 18, Skypnyk teaches:
The system of claim 15, the system further comprising: an encoder configured to generate a guidance embedding based on the text prompt. ([0162], “In the stable diffusion model, the user prompt 610 is converted into an embedding using a pre-trained language model, such as a transformer-based architecture (e.g., GPT or BERT). This embedding encodes the semantic information of the user prompt 610 and is used to condition the image/populated image template 612 generation process.”)
Regarding claim 19, Skypnyk teaches:
The system of claim 15, the system further comprising: a training component configured to train the diffusion model. ([0161], “the interaction system 100 trains a stable diffusion model to receive image templates and prompts and output populated image template that form the characteristics of the template.”)
Regarding claim 21, Skypnyk teaches:
A non-transitory computer readable medium storing code for image generation, the code comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations (FIG. 19) The rest of claims 21 recites similar limitations of claim 1, thus are rejected accordingly.
Claims 27 recites similar limitations of claim 7, thus are rejected accordingly.
Claim(s) 2, 4, 6, 17, 22, 24, 26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Skypnyk in view of Couleaud et al. (US 2025/0078347 A1).
Regarding claim 2, Skypnyk teaches:
The method of claim 1, further comprising:
generating, by the diffusion model, an image based on the guidance embedding; ([0163], “The stable diffusion model is then conditioned on both the image template 612 and the text embedding by incorporating the text embedding into the stable diffusion model architecture or by conditioning the stable diffusion model's latent space on the textual information. The stable diffusion model generates populated image templates (such as images with color and depth information) by learning to reverse the process of adding noise to the input image template 612.”)
identifying an occlusion area for a modified view of the image based on the image and the depth map; ([0185], “the interaction system 100 uses the color populated image template and depth populated image template generated by a stable diffusion model to modify a 3D mesh and apply it to a camera feed.” [0189], “The interaction system 100 composites the rendered 3D mesh onto the live camera feed to create the final augmented output. The interaction system 100 blends the rendered mesh with the camera image or uses depth information to handle occlusions between the mesh and the real-world objects in the scene.”) and
However, Skypnyk does not, but Couleaud teaches:
generating, by the diffusion model, a modified image corresponding to the modified view by inpainting the occlusion area. ([0066], “n some embodiments, as an alternative to the regeneration of such images (e.g., 216, 218, and 220 of FIG. 2) as images (e.g., 234, 236, and 238 of FIG. 2) based on a text prompt (e.g., one or more of text prompts 227-232 of FIG. 2), the image processing system may be configured to fill such holes or empty regions (e.g., 219, 221 in FIG. 2). As an example, the image processing system may perform completion (e.g., interpolation or extrapolation of image content) or inpainting 422 of such holes or empty regions in images 416, 418, and 420 to obtain updated images 434, 436, and 438. In some embodiments, such inpainting may be performed using one or more of the techniques described in Zheng et al., “Image Inpainting with Cascaded Modulation GAN and Object-Aware Training,” Computer Vision—ECCV 2022: 17th European Conference, Tel Aviv, Israel, Oct. 23-27, 2022, Proceedings, Part XVI, the contents of which is hereby incorporated by reference herein in its entirety.”)
Skypnyk teaches handling occlusion, Couleaud teaches a specific method of handling occlusion.
It would have been obvious for ordinary skills in the art to have combined the teachings of Skypnyk with the specific teachings of Couleaud to produce high quality output images.
Regarding claim 4, Skypnyk in view of Couleaud teaches:
The method of claim 2, further comprising: generating, by the diffusion model, an additional modified image based on the modified image. (Skypnyk [0191], “As such, the interaction system 100 applies the modified 3D mesh generated from a stable diffusion model to a live camera feed and create an augmented reality experience. This allows users to view and interact with the mesh in real-time, creating immersive and engaging experiences that combine the virtual and real worlds.” [0178]-[0191] teaches the details of generating modified images using a diffusion model.)
Regarding claim 6, Skypnyk in view of Couleaud teaches:
The method of claim 2, further comprising: generating a video file based on the image and the modified image.( Skypnyk [0207], “FIG. 14 illustrates the content augmentation 1402 applied to the user when the user's head is centered, according to some examples. The generated augmentation content 1402 is applied to the user's face in the live camera feed when their head is centered and aligned. The camera feed shows the live video stream with the user's face centered, similar to FIG. 12, but with the applied augmentation. The user's face remains aligned and framed within the camera feed.” [0243], “The image display driver 1820 commands and controls the image display of optical assembly 1818. The image display driver 1820 may deliver image data directly to the image display of optical assembly 1818 for presentation or may convert the image data into a signal or data format suitable for delivery to the image display device. For example, the image data may be video data formatted according to compression formats, such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, or the like, and still image data may be formatted according to compression formats such as Portable Network Group (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF) or exchangeable image file format (EXIF) or the like.”)
Regarding claim 17, Couleaud teaches:
The system of claim 16, wherein: the image generation model is further trained to
However, Skypnyk does not, but Couleaud teaches:
generate a modified image by inpainting the occlusion area. ([0066], “n some embodiments, as an alternative to the regeneration of such images (e.g., 216, 218, and 220 of FIG. 2) as images (e.g., 234, 236, and 238 of FIG. 2) based on a text prompt (e.g., one or more of text prompts 227-232 of FIG. 2), the image processing system may be configured to fill such holes or empty regions (e.g., 219, 221 in FIG. 2). As an example, the image processing system may perform completion (e.g., interpolation or extrapolation of image content) or inpainting 422 of such holes or empty regions in images 416, 418, and 420 to obtain updated images 434, 436, and 438. In some embodiments, such inpainting may be performed using one or more of the techniques described in Zheng et al., “Image Inpainting with Cascaded Modulation GAN and Object-Aware Training,” Computer Vision—ECCV 2022: 17th European Conference, Tel Aviv, Israel, Oct. 23-27, 2022, Proceedings, Part XVI, the contents of which is hereby incorporated by reference herein in its entirety.”)
Skypnyk teaches handling occlusion, Couleaud teaches a specific method of handling occlusion.
It would have been obvious for ordinary skills in the art to have combined the teachings of Skypnyk with the specific teachings of Couleaud to produce high quality output images.
claims 22, 24, 26 recites similar limitations of claim 2 4, 6, thus are rejected accordingly.
Claim(s) 3, 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Skrypnyk in view of Couleaud and further in view of Kasahara et al. (US 2024/0312166 A1).
Regarding claim 3, Skrypnyk in view of Couleaud teaches:
The method of claim 2, wherein identifying the occlusion area comprises:
However, Skrypnyk in view of Couleaud does not, but Kasahara teaches:
computing a camera view of the image; and shifting the camera view to obtain the modified view. (Abstract: “Methods and devices for processing image data for scene completion, including receiving an original image from an original viewpoint corresponding to a first direction, wherein the original image includes an object; obtaining a first image from a new viewpoint corresponding to a second direction different from the first direction by rotating the original image based on 3-dimensional (3D) information generated from 2-dimensional (2D) information which is obtained from the original image; determining an area within the first image for generating a second surface of the object based on depth information about a depth between the object and the background of the original image, wherein the determined area is expected to include an object area; and obtaining a second image by inputting the first image and the determined area to an artificial intelligence (AI) inpainting model, wherein the AI inpainting model generates the second surface of the object which occupies a portion of the determined area in the second image.” para [0090]-0102] discuss the details.)
Skrypnyk in view of Couleaud teaches an occlusion inpainting method. Kasahara provides a more comprehensive method of occlusion inpainting method that considers the different viewpoint images of a scene. This method provides a more accurate occlusion inpainting results.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have applied the specific teachings of Kasahara to the teachings of Skrypnyk in view of Couleaud to provide a more accurate occlusion inpainting results.
Claim 23 recites similar limitations of claim 3, thus is rejected accordingly.
Claim(s) 5, 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Skrypnyk in view of Couleaud and in view of Liao et al. (US 2021/0374904 A1).
Regarding claim 5, Skrypnyk in view of Couleaud teaches:
The method of claim 4, wherein generating the additional modified image comprises:
However, Skrypnyk in view of Couleaud does not teach, but Liao teaches:
averaging pixel information of the image and the modified image to obtain average pixel information, wherein the additional modified image is based on the average pixel information. ([0036], “In one embodiment, for each target pixel within the target inpainting region of the first image frame, based on the corresponding depth map, the method may further map the target pixel within the target inpainting region of the first image frame to a candidate pixel in a second image frame included in the image frames. The method may further determine a candidate color to fill the target pixel. The method may further perform Poisson image editing on the first image frame to achieve color consistency between inside and outside of the target inpainting region of the first image frame. The method may further use video fusion inpainting to inpaint occluded areas within the target inpainting region. For each pixel in the target inpainting region of the first image frame, the method may trace the pixel into neighboring frames and replacing an original color of the pixel with an average of colors sampled from the neighboring frames.”)
Skrypnyk in view of Couleaud teaches inpainting occluded regions. Liao teaches a specific method of inpainting them.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Skrypnyk in view of Couleaud with the specific teachings of Liao to effectively and accurately inpainting occlusion areas.
Claim 25 recites similar limitations of claim 5, thus is rejected accordingly.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YANNA WU whose telephone number is (571)270-0725. The examiner can normally be reached Monday-Thursday 8:00-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 5712722330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YANNA WU/ Primary Examiner, Art Unit 2615
Prosecution Insights