DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (“IDS”) filed on 01/10/2024, 01/16/2025 and 01/27/2027 has been reviewed and the listed references have been considered.
Drawings
The 21-page drawings have been considered and placed on record in the file.
Election/Restrictions
Applicant's election with traverse of Species 1 - Claims 1-14 in the reply filed on 02/03/2026 is acknowledged. The traversal is on the ground(s) that the restriction requirement does not establish that the species cannot be used together or the species have different modes of operation, function, or effect as required by MPEP 806.05(j).
Applicant argues in fourth paragraph in page 2 of the reply, that the inventions of species 1 and 2 are capable of use together and therefore, the restriction in improper. Examiner respectfully disagrees. Species 1- claims 1-15 are directed to a method and device for image outpainting and the Species 2 – claims 15-20 are directed to a method for selecting neural network structure using a proxy prediction model. As a result, each of these species contain different technical features that would require their own separate searches.
Additionally, the inventions as claimed do not encompass overlapping subject matter and there is nothing of record to show them to be obvious variants. Restriction for examination purposes as indicated is proper because all the inventions listed in this action are independent or distinct for the reasons given above and there would be a serious search and/or examination burden if restriction were not required.
Therefore, the requirement is still deemed proper and is therefore made FINAL.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “a multilabel classifier” in claims 2 and 9; and “at least one processing device” in claims 8-14.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, these are being interpreted to cover the corresponding structures described in the applicant’s drawings: schematics depicted in Fig. 1, and applicant’s specification: ¶0004: “at least one processor of an electronic device to perform the method”; and ¶0124: “applications or other software instructions that are executed by the processor 120 of the electronic device” as performing the claimed functions, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 4, 8, 9 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Fan et al. (Hierarchical Masked 3D Diffusion Model for Video Outpainting) in view of Elyahu et al. (US 2024/0355017 A1).
Regarding claim 1, Fan teaches, A method comprising: adding an outpaint (Fan, page 7896, col. 2, ¶02: “methods for horizontal video outpainting”) mask to an image to generate a masked image; (Fan, page 7894, col. 1, ¶03: “generate masked guide frames”) processing the image using an encoder neural network to generate an image representation of the image in a latent space; (Fan, page 7891, col. 1, ¶03: “encode the video frames in the latent space instead of the pixel space”) processing the masked image using a convolution neural network and adding the image representation to generate an image embedding; (Fan, page 7899, col. 2, ¶03: “2D conv layer is followed by a temporal 1D conv layer. We not only add the timestep embeddings of the noise to each layer but also add the fps rate embeddings”) processing the image representation and the image embedding using at least one of a diffusion model and an interpolation process to generate a noisy latent image representation; (Fan, page 7899, col. 2, ¶03: “utilizing Pseudo-3D convolutional and attention layers to lever age pre-trained text-to-image models within the latent diffusion models”) (Fan, page 7893, col. 2, ¶02: “the decoder D is used to map 𝑧0 back to the pixel space”) to generate an outpainted image. (Fan, page 7893, col. 2, ¶03: “the generated clips are stitched together to form the final outpainting result”). However, Fan does not explicitly teach, using a large language model to contextualize an outpainting prompt; denoising the noisy latent image representation based on the contextualized outpainting prompt to generate a denoised latent image representation.
In an analogous field of endeavor, Elyahu teaches, using a large language model to contextualize an outpainting prompt; (Elyahu, ¶0026: “incorporating knowledge from large language models or hybrid vision-language models, these text-to-image diffusion models can generate realistic high-resolution images using only a text prompt describing the scene”) denoising the noisy latent image representation based on the contextualized outpainting prompt (Elyahu, ¶0029: “Parameters of the pre-trained diffusion model 105 are then frozen and the target text embedding 120 is optimized using a denoising diffusion objective”) to generate a denoised latent image representation. (Elyahu, ¶0051: “latent representation of the input image, which can then be manipulated to obtain an output image with the desired edit specified by the given text”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fan using the teachings of Elyahu to introduce a large language model. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of contextualizing a text prompt. Therefore, it would have been obvious to combine the analogous arts Fan and Elyahu to obtain the invention in claim 1.
Regarding claim 2, Fan in view of Elyahu teaches, The method of Claim 1, wherein a multilabel classifier is employed to contextualize the outpainting prompt. (Elyahu, ¶0069: “process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fan in view of Elyahu using the additional teachings of Elyahu to introduce context classification. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of contextualizing a text prompt using natural language processing. Therefore, it would have been obvious to combine the analogous arts Fan and Elyahu to obtain the invention in claim 2.
Regarding claim 4, Fan in view of Elyahu teaches, The method of Claim 1, wherein: the masked image is processed multiple times (Fan, page 7895, col. 1, ¶01: “perform video outpainting by combining infilling and interpolation”) using an outpainting model; (Fan, page 7895, col. 2, ¶02: “masked 3D diffusion model for video outpainting”) and the outpainting model comprises the encoder neural network, (Fan, page 7893, col. 2, ¶02: “LDMs[28] additionally trained an encoder”) the convolution neural network, (Fan, page 7899, col. 2, ¶03: “Pseudo-3D convolutional and attention layers to lever age pre-trained text-to-image models”) the diffusion model, (Fan, page 7899, col. 1, ¶02: “utilizing the diffusion model”) the large language model, (Elyahu, ¶0026: “incorporating knowledge from large language models”) and the decoder neural network. (Fan, page 7893, col. 2, ¶02: “the decoder D is used to map 𝑧0 back to the pixel space”). The proposed combination as well as the motivation for combining Fan and Elyahu references presented in the rejection of claim 1, apply to claim 4 and are incorporated herein by reference. Thus, the method recited in claim 4 is met by Fan and Elyahu.
Regarding claim 8, it recites an electronic device with elements corresponding to the steps of the method recited in claim 1. Therefore, the recited elements of electronic device claim 8 are mapped to the proposed combination in the same manner as the corresponding steps in method claim 1. Additionally, the rationale and motivation to combine Fan and Elyahu presented in rejection of claim 1, apply to this claim.
Regarding claim 9, it recites an electronic device with elements corresponding to the steps of the method recited in claim 2. Therefore, the recited elements of electronic device claim 9 are mapped to the proposed combination in the same manner as the corresponding steps in method claim 2. Additionally, the rationale and motivation to combine Fan and Elyahu presented in rejection of claim 2, apply to this claim.
Regarding claim 11, it recites an electronic device with elements corresponding to the steps of the method recited in claim 4. Therefore, the recited elements of electronic device claim 11 are mapped to the proposed combination in the same manner as the corresponding steps in method claim 4. Additionally, the rationale and motivation to combine Fan and Elyahu presented in rejection of claim 1, apply to this claim.
Claims 3 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Fan et al. (Hierarchical Masked 3D Diffusion Model for Video Outpainting) in view of Elyahu et al. (US 2024/0355017 A1) and in further view of Cragg et al. (US 2024/0273670 A1).
Regarding claim 3, Fan in view of Elyahu teaches, The method of Claim 1. However, the combination of Fan and Elyahu does not explicitly teach, wherein denoising the noisy latent image representation is based on one or more personalization features.
In an analogous field of endeavor, Cragg teaches, wherein denoising the noisy latent image representation (Cragg, ¶0066: “a reverse diffusion process 325 (e.g., a U-Net ANN) gradually removes the noise from the noisy images”) is based on one or more personalization features. (Cragg, ¶0067: “guidance features 345 can be combined with the noisy images 320 at one or more layers of the reverse diffusion process”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fan in view of Elyahu using the teachings of Cragg to introduce denoising based on personalized features. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of optimizing the denoising process for a personalized denoised output. Therefore, it would have been obvious to combine the analogous arts Fan, Elyahu and Cragg to obtain the invention in claim 3.
Regarding claim 10, it recites an electronic device with elements corresponding to the steps of the method recited in claim 3. Therefore, the recited elements of electronic device claim 10 are mapped to the proposed combination in the same manner as the corresponding steps in method claim 3. Additionally, the rationale and motivation to combine Fan, Elyahu and Cragg presented in rejection of claim 3, apply to this claim.
Claims 5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Fan et al. (Hierarchical Masked 3D Diffusion Model for Video Outpainting) in view of Elyahu et al. (US 2024/0355017 A1) and in further view of Lin et al. (US 2025/0104195 A1).
Regarding claim 5, Fan in view of Elyahu teaches, The method of Claim 1, further comprising: detecting an image quality of the outpainted image; and (Fan, page 7899, table 3: “Evaluate the performance of video outpainting”). However, the combination of Fan and Elyahu does not explicitly teach, reprocessing the outpainted image based on the detected image quality.
In an analogous field of endeavor, Lin teaches, reprocessing the outpainted image based on the detected image quality. (Lin, ¶0105: “If the quality of the image is insufficient, based on specific enhancement requirements such as line continuity, contrast, and smoothness, multiple second processed image files (Y2) will be reprocessed”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fan in view of Elyahu using the teachings of Lin to introduce reprocessing low-quality images. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of ensure high quality image output. Therefore, it would have been obvious to combine the analogous arts Fan, Elyahu and Lin to obtain the invention in claim 5.
Regarding claim 12, it recites an electronic device with elements corresponding to the steps of the method recited in claim 5. Therefore, the recited elements of electronic device claim 12 are mapped to the proposed combination in the same manner as the corresponding steps in method claim 5. Additionally, the rationale and motivation to combine Fan, Elyahu and Lin presented in rejection of claim 5, apply to this claim.
Claims 6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Fan et al. (Hierarchical Masked 3D Diffusion Model for Video Outpainting), in view of Elyahu et al. (US 2024/0355017 A1), in further view of Lin et al. (US 2025/0104195 A1) and still in further view of Zhang et al. (US 12,475,565 B2).
Regarding claim 6, Fan in view of Elyahu and in further view of Lin teaches, The method of Claim 5 wherein detecting the image quality of the outpainted image comprises. However, the combination of Fan, Elyahu and Lin does not explicitly teach, processing a linear projection of image patches of the outpainted image using a transformer encoder; processing an output of the transformer encoder using a multi-layer perceptron (MLP); and applying a sigmoid function to an output of the MLP.
In an analogous field of endeavor, Zhang teaches, processing a linear projection of image patches of the outpainted image (Zhang, col. 11, lines 15-18: “A linear embedding layer can be applied to this raw-valued feature. In various embodiments, each window can contain M×M image patches”) using a transformer encoder; (Zhang, col.11, lines 4-6: “the image encoder 720 can have an architecture, including, but not limited to, ResNET, ResNeXT, Mask2Former, Swin Transformers”) processing an output of the transformer encoder using a multi-layer perceptron (MLP); (Zhang, col. 11, lines 47-50: “output of the transformer decoder 730 can be fed to output heads including one or more neural network layers 750. The neural network layers 750 can include one or more multilayer perceptrons”) and applying a sigmoid function to an output of the MLP. (Zhang, col. 13, lines 40-41: “a sigmoid activation function can be used to map the output”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fan in view of Elyahu and in further view of Lin using the teachings of Zhang to introduce applying a sigmoid function to a processed output of an MLP. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of determining the quality of the outpainting task using a probability score. Therefore, it would have been obvious to combine the analogous arts Fan, Elyahu, Lin and Zhang to obtain the invention in claim 6.
Regarding claim 13, it recites an electronic device with elements corresponding to the steps of the method recited in claim 6. Therefore, the recited elements of electronic device claim 13 are mapped to the proposed combination in the same manner as the corresponding steps in method claim 6. Additionally, the rationale and motivation to combine Fan, Elyahu, Lin and Zhang presented in rejection of claim 6, apply to this claim.
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Fan et al. (Hierarchical Masked 3D Diffusion Model for Video Outpainting), in view of Elyahu et al. (US 2024/0355017 A1), in further view of Lin et al. (US 2025/0104195 A1) and still in further view of Wang et al. (US 2022/0148293 A1).
Regarding claim 7, Fan in view of Elyahu and in further view of Lin teaches, The method of Claim 5, wherein detecting the image quality of the outpainted image comprises: using. However, the combination of Fan, Elyahu and Lin does not explicitly teach, a generative adversarial network (GAN) that includes (i) a generator configured to generate negative training examples and (ii) a discriminator trained on the negative training examples.
In an analogous field of endeavor, Wang teaches, a generative adversarial network (GAN) that includes (Wang, ¶0035: “first aspect involves training of a GAN (Generative Adversarial Network”) (i) a generator configured to generate negative training examples (Wang, ¶0020: “obtain fictitious training data with negative samples output by the mapping generator”) and (ii) a discriminator trained on the negative training examples. (Wang, ¶0020: “input training data with negative samples into the discriminator that is to be trained to obtain the discrimination result output by the discriminator”).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Fan in view of Elyahu and in further view of Lin using the teachings of Wang to introduce training a generative adversarial network discriminator. A person skilled in the art would be motivated to combine the known elements as described above and achieve the predictable result of improving the identification capabilities of the GAN discriminator. Therefore, it would have been obvious to combine the analogous arts Fan, Elyahu, Lin and Wang to obtain the invention in claim 7.
Regarding claim 14, it recites an electronic device with elements corresponding to the steps of the method recited in claim 7. Therefore, the recited elements of electronic device claim 14 are mapped to the proposed combination in the same manner as the corresponding steps in method claim 7. Additionally, the rationale and motivation to combine Fan, Elyahu, Lin and Wang presented in rejection of claim 7, apply to this claim.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MEHRAZUL ISLAM whose telephone number is (571)270-0489. The examiner can normally be reached Monday-Friday: 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Saini Amandeep can be reached on (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MEHRAZUL ISLAM/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights