DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
2. The information disclosure statement (IDS) was submitted on06/27/2025. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Allowable Subject Matter
3. Claims 10 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The claims reciting “wherein applying the third machine learning model to the forward and backward intermediate feature sets to generate the intermediate latent feature sets comprises applying a Nonlinear Activation Free Network (NAFNet) to the concatenation result to generate the intermediate latent feature sets.“ find no prior art by which NAFNet representing an AI/ML model, is applied to the concatenated results in order to generate the intermediate latent feature set..
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
4. Claims 1-9 and 11-19 are rejected under 35 U.S.C. 103 as being obvious over
Abhijeet Bhorkar at al., (hereinafter Bhorkar) US 2020/0244969, and further in view of Eddy Ilg et al., (hereinafter Ilg) “FowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks” [IDS], and Ali Razavi et al., (hereinafter Razavi) “Generating Diverse High-Fidelity Images with VQ-VAE-2” [IDS] along with Zachary Teed et al., (hereinafter Teed) “RAFT: Recurrent All-Pairs Field Transforms for Optical Flow” [IDS].
Re Claim 1. Bhorkar discloses, a method comprising:
obtaining, using at least one processing device of an electronic device, input video frames in a pixel space (obtaining at least at one processing device a space representation of video frames Par.[0046, 0068] );
compressing, (compressing at autoencoder per Fig.6 Par.[0068], Par.[0001]) using the at least one processing device, two adjacent input video frames to obtain corresponding input latent feature sets in a latent space (a pair i.e., of two adjacent video frames Par.[0048, 0052], in latent space representation, Par.[0001, 0003-0004, 0009] Fig.4);
generating, using the at least one processing device, intermediate latent feature sets in the latent space based on the input latent feature sets (sending the latest space representation/vector, z, of an intermediate frame of an encoding block, Par.[0052, 0068] based on feature space of te sequence of frames, Par.[0047]);
decompressing, using the at least one processing device, the intermediate latent feature sets to obtain interpolated video frames in the pixel space (decoding latent space buy linear interpolation Par.[0034-0035]); and
inserting, using the at least one processing device, the interpolated video frames between the two adjacent input video frames to obtain a slow motion video (generating the latent space representation associated with the latent space between the first and second frames, having fewer dimensions than the feature space of the sequence of frames of the video, Par.[0041-0043, 0047]).
Bhorkar is considered to obviate all these limitations.
Re Claim 2. Bhorkar discloses, the method of Claim 1, wherein:
the input latent feature sets comprise continuous input latent feature sets (set of features Par.[0031] Eq.2);
the latent space comprises a continuous latent space (Par/[0052]); and
compressing the two adjacent input video frames comprises (compressing the two adjacent frames, Par.[0052]):
encoding, using a machine learning model, the two adjacent input video frames to obtain corresponding continuous vector representations in the continuous latent space (at the autoencoder in Fig.6 Par.[0068]);
performing, using the machine learning model, vector quantization on the continuous vector representations to obtain corresponding discrete latent feature sets in a discrete latent space (using machine learning Par.[0046]); and
performing, using the machine learning model, vector de-quantization on the discrete latent feature sets to obtain corresponding continuous input latent feature sets in the continuous latent space (using neural networks to generate the reconstructed version of the frames, per Fig.5 Par.[0010, 0021, 0053…etc]).
Bhorkar is considered to obviate all these limitations.
Re Claim 3. Bhorkar discloses, the method of Claim 2,
Razavi teaches that, wherein performing the vector quantization comprises performing hierarchical vector quantization on the continuous vector representations to obtain, for each of the continuous vector representations, a first-level discrete latent feature set and a second-level discrete latent feature set that is conditioned on the first-level discrete latent feature set (performing multi-scale vector quantized hierarchical organization, VQ-VAE, models, Abstract, Sec.2.1).
The ordinary skilled in the art would have found obvious before the effective filing date of invention to considered the application of vector quantized variational autoencoder described in Razavi, as a method used in communications systems in order to align the vector space of the codebook to the autoencoder of Bhorkar, hence deeming such combination predictable.
Re Claim 4. Bhorkar and Razavi disclose, the method of Claim 3,
Razavi wherein performing the vector de-quantization comprises performing hierarchical vector de-quantization on each first- and second-level discrete latent feature set to obtain the corresponding continuous input latent feature set (at VQ-VAE decoder, Fig.2a(a) reconstruction Sec.3).
Re Claim 5. Bhorkar discloses, the method of Claim 2, wherein:
the intermediate latent feature sets comprise intermediate continuous latent feature sets; and
decompressing the intermediate continuous latent feature sets comprises decoding, using the machine learning model, the intermediate continuous latent feature sets to obtain corresponding interpolated video frames in the pixel space (similarly mapped at claim 2).
Re Claim 6. Bhorkar discloses, the method of Claim 1, wherein generating the intermediate latent feature sets comprises:
Ilg teaches about, estimating, using a first machine learning model, intermediate optical flows in the latent space based on the input latent feature sets (applying the Optical Flow Estimation with deep networks, FlowNet 2.0 , Title, Fig.2, Sec.1);
The ordinary skilled in the art would have found obvious before the effective filing date of invention to considered the application of vector quantized variational autoencoder described in Bhorkar, along with the well-known optical flow method of prediction identified in Ilg, hence deeming such combination predictable.
Bhorkar and Ilg, do not expressly teach the use of the occlusion mask in the latent space,
Teed teaches the occlusion mask,
estimating, using a second machine learning model, an occlusion mask in the latent space based on the intermediate optical flows (the occlusions gaps being filled by interpolation of nearest neighbor, Sec.3.3); and
performing, using a third machine learning model, frame interpolation in the latent space based on the input latent feature sets, the intermediate optical flows, and the occlusion mask to generate the intermediate latent feature sets (at a RAFT deep neural network architecture for optical flow, Abstract, Sec.1).
The ordinary skilled in the art would have found obvious before the effective filing date of invention to considered the application of vector quantized variational autoencoder described in Bhorkar, along with the well-known optical flow method of prediction identified in Ilg, along with the method of filling the image gaps by masking the occlusions taught by Teed, hence deeming such combination predictable.
Re Claim 7. Bhorkar, Ilg and Teed disclose, the method of Claim 6, wherein:
Ilg teaches about, the intermediate optical flows comprise forward and backward intermediate optical flows (estimating, using a first machine learning model, intermediate optical flows in the latent space based on the input latent feature sets by applying the Optical Flow Estimation with deep networks, FlowNet 2.0 , Title, Fig.2, Sec.1); and
estimating the intermediate optical flows comprises:
estimating, using the first machine learning model, forward and backward input optical flows for the two adjacent input video frames based on the input latent feature sets in the latent space (at Sec.6); and
performing bilinear interpolation on the estimated forward and backward input optical flows in the latent space to estimate the forward and backward intermediate optical flows (at Sec.6.1).
Re Claim 8. Bhorkar, Ilg and Teed disclose, the method of Claim 6, wherein:
Ilg teaches about, the intermediate optical flows comprise forward and backward intermediate optical flows (sec. 6.1); and
Teed teaches the occlusion mask, estimating the occlusion mask comprises estimating the occlusion mask from the estimated forward and backward intermediate optical flows in the latent space (estimating, using a second machine learning model, an occlusion mask in the latent space based on the intermediate optical flows the occlusions gaps being filled by interpolation of nearest neighbor, Sec.3.3).
Re Claim 9. Bhorkar, Ilg and Teed disclose, the method of Claim 6, wherein:
Ilg teaches about, the intermediate optical flows comprise forward and backward intermediate optical flows (at Sec.6.1); and
performing the frame interpolation comprises:
warping the input latent feature set corresponding to an earlier of the two adjacent input video frames with the estimated forward intermediate optical flow to obtain a forward intermediate feature set;
warping the input latent feature set corresponding to a later of the two adjacent input video frames with the estimated backward intermediate optical flow to obtain a backward intermediate feature set (as taught for feature warping at Abstract, Fig.2, Sec.2);
Teed teaches the occlusion mask, applying the occlusion mask to the forward and backward intermediate feature sets; and
applying the third machine learning model to the forward and backward intermediate feature sets to generate the intermediate latent feature sets (applying the same method to any subsequent machine learning model, using a second or a thirdmachine learning model, an occlusion mask in the latent space based on the intermediate optical flows the occlusions gaps being filled by interpolation of nearest neighbor, Sec.3.3).
Re Claim 10. Objected to allow
Re Claim 11. This claim represents the electronic device/apparatus, implementing each and every limitation of the method claim 1, hence it is rejected on the same mapped evidence mutatis mutandis.
Re Claim 12. This claim represents the electronic device/apparatus, implementing each and every limitation of the method claim 2, hence it is rejected on the same mapped evidence mutatis mutandis.
Re Claim 13. This claim represents the electronic device/apparatus, implementing each and every limitation of the method claim 3, hence it is rejected on the same mapped evidence mutatis mutandis.
Re Claim 14. This claim represents the electronic device/apparatus, implementing each and every limitation of the method claim 4, hence it is rejected on the same mapped evidence mutatis mutandis.
Re Claim 15. This claim represents the electronic device/apparatus, implementing each and every limitation of the method claim 5, hence it is rejected on the same mapped evidence mutatis mutandis.
Re Claim 16. This claim represents the electronic device/apparatus, implementing each and every limitation of the method claim 6, hence it is rejected on the same mapped evidence mutatis mutandis.
Re Claim 17. This claim represents the electronic device/apparatus, implementing each and every limitation of the method claim 7, hence it is rejected on the same mapped evidence mutatis mutandis.
Re Claim 18. This claim represents the electronic device/apparatus, implementing each and every limitation of the method claim 8, hence it is rejected on the same mapped evidence mutatis mutandis.
Re Claim 19. This claim represents the electronic device/apparatus, implementing each and every limitation of the method claim 9, hence it is rejected on the same mapped evidence mutatis mutandis.
Re Claim 20. Objected to allow
Conclusion
5. The prior art made of record and not relied upon, is considered pertinent to applicant's disclosure. See PTO-892 form. Applicant is required under 37 C.F.R. 1.111(c) to consider these references when responding to this action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DRAMOS KALAPODAS whose telephone number is (571)272-4622. The examiner can normally be reached on Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Czekaj can be reached on 571-272-7327. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DRAMOS KALAPODAS/Primary Examiner, Art Unit 2487
