DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 10, 12-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Pourreza et al. (US 20240022761 A1) (Hereinafter referred to as Pourreza) in view of Kong et al. (“IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation”) (Hereinafter referred to as Kong).
Regarding Claim 1, Pourreza discloses A method comprising: (See Abstract, “Techniques are described for processing video data . . .”)
performing a first encoding operation based on a first image frame at a first time point and a second image frame at a second time point to generate a first encoding feature; (See [0182], “At block 1502, the process 1500 includes obtaining a first reference frame and a second reference frame.”
See [0069], “For example, the video encoder (e.g., the encoder neural network of the motion prediction system) can use machine learning to generate latent data (e.g., a bitstream) representing predicted motion between an input frame and a reference frame . . .” In this case, a first encoding operation is can be performed on an input frame, which could be a first reference frame (first image frame) and a reference frame, which could be a second reference frame (second image frame) to generate latent data (a first encoding feature).)
performing a first decoding operation based on the first encoding feature to generate a motion vector; (See [0069], “The video decoder (e.g., the decoder neural network of the motion prediction system) can use machine learning to process the latent data and predict or reconstruct the motion between the input frame and the reference frame. . . . For instance, the predicted motion can include motion vectors (e.g., an optical flow map including a motion vector for each pixel or block of pixels of the input frame).”)
generate a first optical flow feature between the first time point and a third time point and a second optical flow feature between the second time point and the third time point; and (See [0144], “The interpolation engine 722 can perform an interpolation operation on the two reference frames {circumflex over (X)}.sub.ref.sub.0 and {circumflex over (X)}.sub.ref.sub.1 to generate an interpolated reference frame {circumflex over (X)}.sub.ref.sub.t. . . The frame interpolation engine 722 can interpolate the motion estimation information (e.g., the flow maps) to determine a motion estimation (e.g., an optical flow) for a time t associated with the interpolated reference frame {circumflex over (X)}.sub.ref.sub.t.” Here, Pourreza teaches a third time point, that being time t.
See [0183], “At block 1504, the process 1500 includes generating a third reference frame at least in part by performing interpolation between the first reference frame and the second reference frame. For instance, the process 1500 can include determining a first set of motion information representing pixel motion from the first reference frame to the third reference frame. The process 1500 can further include determining a second set of motion information representing pixel motion from the second reference frame to the third reference frame. In some aspects, the first set of motion information and the second set of motion information are determined based on pixel motion between the first reference frame and the second reference frame. In some cases, the first set of motion information includes a first optical flow map and the second set of motion information includes a second optical flow map.”
In this case, Pourreza teaches a first set of motion information representing pixel motion from the first reference frame to the third reference frame (a first optical flow feature between the first time point and a third time point) and a second set of motion information representing pixel motion from the second reference frame to the third reference frame (a second optical flow feature between the second time point and the third time point).)
generating a third image frame at the third time point based on the first optical flow feature, the second optical flow feature, and a motion vector corresponding to motion between the first image frame and the second image frame. (See [0183], “At block 1504, the process 1500 includes generating a third reference frame at least in part by performing interpolation between the first reference frame and the second reference frame. . . The process 1500 can include generating first warping information at least in part by performing a warping function on the first reference frame using the first set of motion information. In some aspects, the warping function includes a bilinear interpolation function, such as that as described above with respect to FIG. 7C. The process 1500 can further include generating second warping information at least in part by performing the warping function on the second reference frame using the second set of motion information. The process 1500 can include generating the third reference frame based on the first warping information and the second warping information.”
See [0015], “In some aspects, the first set of motion information includes a first optical flow map and the second set of motion information includes a second optical flow map.” Also see [0069], “For instance, the predicted motion can include motion vectors (e.g., an optical flow map including a motion vector for each pixel or block of pixels of the input frame).”
In summary, Pourreza teaches to generate the third reference frame (third image frame) using first and second warping information derived from the first set of motion information (first optical flow feature) and the second set of motion information (second optical flow feature), and these sets of motion information can include optical flow maps which can include a motion vector (a motion vector corresponding to motion between the first image frame and the second image frame).)
However, Pourreza fails to explicitly disclose performing a first decoding operation based on the first encoding feature to generate a first optical flow feature between the first time point and a third time point and a second optical flow feature between the second time point and the third time point;
Kong teaches performing a first decoding operation based on the first encoding feature to generate a first optical flow feature between the first time point and a third time point and a second optical flow feature between the second time point and the third time point; (See Page 1 Right Column Paragraph 2, “. . . Denoting input frames and target frame to be I0, I1 and It (0 < t < 1), existing methods either first estimate optical flow F0→1, F1→0.” Also see Page 3 Section 3.1. IFRNet, “Given two input frames I0 and I1 at adjacent time in stances, video frame interpolation aims to synthesize an intermediate frame It, where 0<t<1. To achieve this goal, proposed model performs a first extraction phase so as to retrieve a pyramid of features from each frame. . . Pyramid Encoder. To obtain contextual representation from each input frame, we design a compact encoder E to extract a pyramid of features. . . Coarse-to-Fine Decoders. After extracting meaning hierarchical representations, we then gradually refine intermediate flow fields through multiple decoders by backward warping pyramid features φk0, φk1 to generate ˜φk0, ˜φk1 according to Fkt→0 and Fkt→1,respectively.”
Lastly see Page 3 Fig. 3, “Our model is an efficient encoder-decoder based network, which first extracts pyramid context features from input frames with a shared encoder, and then gradually refines bilateral intermediate flow fields Ft→0, Ft→1 together with reconstructed intermediate feature ˆφt through coarse-to-fine decoders, until yielding the final output.”
Here, Kong also teaches an encoder-decoder based architecture for frame interpolation with optical flow estimation. Kong more directly teaches using an decoder to perform decoding operations to generate optical flows Ft→0, Ft→1 (a first optical flow feature between the first time point and a third time point and a second optical flow feature between the second time point and the third time point).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Pourreza with Kong to include using the decoder to generate a first and second optical flow feature.
The motivation to combine Pourreza with Kong would have been obvious as both arts are related to using an encoder-decoder structure to estimate optical flow, which can then be used for frame interpolation (See Kong Page 3 Fig. 3). Pourreza already teaches an encoder and decoder, and specifically finding motion information from reference frames to an interpolated image frame. Kong simply teaches that it would be a common idea to specifically use a decoder to generate the first and second optical flow feature. Note that Kong applies a pyramid structure for the encoder decoder, with the benefit of this process being that is easier to capture large motions, even when an image is scaled down.
Regarding Claim 2, Pourreza in view of Kong discloses The method of claim 1, further comprising: performing a second encoding operation based on the motion vector to generate a second encoding feature; and (See Kong Page 3 Fig. 3 showing the Pyramid Encoder, which extracts 4 levels of pyramid features. Each level would represent a different encoding operation, and thus there exists a second encoding operation to generate a second encoding feature.)
performing a second decoding operation based on the second encoding feature to generate a first motion feature between the first time point and the third time point and a second motion feature between the second time point and the third time point. (See Kong Page 3 Fig. 3 showing specifically the Coarse-to-Fine Decoders and teaching the generation of intermediate flow fields Ft→0, Ft→1 (first and second motion features).
Also see Kong Page 3 Section 3.1. IFRNet, “Coarse-to-Fine Decoders. After extracting meaning hierarchical representations, we then gradually refine intermediate flow fields through multiple decoders by backward warping pyramid features φk0, φk1 to generate ˜φk0, ˜φk1 according to Fkt→0 and Fkt→1,respectively.” The multiple decoders means that there is a “second decoder operation” and would be based on the second encoding feature, as seen by Fig. 3. The motivation to combine would have been similar that of Claim 1 rejection motivation.)
Regarding Claim 3, Pourreza in view of Kong discloses The method of claim 2, wherein the generating of the second encoding feature comprises: scaling the motion vector based on the third time point to generate a first approximated motion vector corresponding to motion between the first time point and the third time point and a second approximated motion vector corresponding to motion between the second time point and the third time point; and (See Pourreza [0152], “The optical flow prediction layer 732 can use the bidirectional motion information 740 and the time information 742 to predict 2D optical flow maps 744. In some examples, the optical flow estimation layer 730 can interpolate the bidirectional motion information 740 (e.g., f.sub.0.fwdarw.1 and f.sub.1.fwdarw.0) for reference frame {circumflex over (X)}.sub.ref.sub.0 and reference frame {circumflex over (X)}.sub.ref.sub.1, to determine the optical flow at time t.” Also see [0154] showing the equations 1 and 2 for the 2D optical flow maps which takes into account time t. In this case, the equation shows that the motion vectors are scaled based on the time. Thus the time scaled motion vectors would correspond to a first and second approximated motion vector.)
performing the second encoding operation using the first image frame, the second image frame, the first approximated motion vector, and the second approximated motion vector. (See Pourreza [0154] showing equations 1 and 2 that scales with time t resulting in the first approximated motion vector, and the second approximated motion vectors. In combination Kong Page 3 Fig. 3 showing the Pyramid Encoder, which implies a second encoding operation, the above limitations are taught, as the encoders would use the first and second image frame. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 4, Pourreza in view of Kong discloses The method of claim 2, wherein the generating of the first optical flow feature and the second optical flow feature comprises performing the first decoding operation based on the first encoding feature, the first motion feature, and the second motion feature. (See Pourreza [0069] teaching a decoder that takes into account the latent data generated by the encoder (first encoding feature).
See Kong Page 3 Section 3.1. IFRNet, “Coarse-to-Fine Decoders. After extracting meaning hierarchical representations, we then gradually refine intermediate flow fields through multiple decoders by backward warping pyramid features φk0, φk1 to generate ˜φk0, ˜φk1 according to Fkt→0 and Fkt→1, respectively.”
In summary, the pyramid architectures allows for refinement of the flow fields Fkt→0, Fkt→1 (the first motion feature and the second motion feature). This would mean that generating the first and second optical flow features can comprise performing the first decoding operation based on the first encoding feature, first motion feature, and second motion feature. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 5, Pourreza in view of Kong discloses The method of claim 2, wherein the third image frame is generated based on the first optical flow feature, the second optical flow feature, the first motion feature, and the second motion feature. (See Pourreza [0159], “In some examples, the refinement operations can also include merging the 2D optical flow maps 744 into a 3D optical flow map 748 for the interpolated frame {circumflex over (X)}.sub.ref.sub.t”. See Pourreza [0162], “In some cases, the 3D warping layer 738 can implement one or more warping functions. In one illustrative example, the following can be used to generate the interpolated frame . . .” Further see [0162] equation 3 that warps the forward and backward optical flow maps (first and second optical flow features) to generate the interpolated frame (the third image frame).
See Kong Page 3 Fig. 3 showing specifically the Coarse-to-Fine Decoders and teaching the generation of intermediate flow fields Ft→0, Ft→1 (first and second motion features) and Kong Page 3 Section 3.1. IFRNet and Fig. 3 teaching multiple encoding and decoding operations based on the pyramid architecture, resulting in the first and second motion features. Thus the combination of the Pourreza and Kong would teach the above limitations. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 6, Pourreza in view of Kong discloses The method of claim 2, wherein the second encoding operation comprises a plurality of second encoding levels including a k-th second encoding level, and the second decoding operation comprises a plurality of second decoding levels including a (k+1)-th second decoding level and a k-th second decoding level, and (See Kong Page 3 Fig. 3 showing a plurality of second encoding levels including a k-th second encoding level and a plurality of second decoding levels including a (k+1)-th second decoding level and a k-th second decoding level.)
wherein the generating of the first motion feature and the second motion feature comprises generating, at the k-th second decoding level, a (k-1)-th second decoding feature, the first motion feature at the k-th second decoding level, and the second motion feature at the k-th second decoding level, based on a k-th second encoding feature generated at the k-th second encoding level and a k-th second decoding feature generated at the (k+1)-th decoding level. (See Kong Page 3 Fig. 3 showing a plurality of second decoding levels. See Kong Page 4 Left Column Equations (1), (2), and (3) which shows that the decoders takes into account the (k-1)-th feature at the k-th second decoding level. This results in the first and second motion features at the k-th second decoding levels. Note that these would be based on the k-th second encoding feature, as shown in Fig. 3 with the decoding level having a corresponding encoding level. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 7, Pourreza in view of Kong discloses The method of claim 6, wherein the first encoding operation comprises a plurality of first encoding levels including a k-th first encoding level, and the first decoding operation comprises a plurality of first decoding levels including a (k+1)-th first decoding level and a k-th first decoding level, and (See Kong Page 3 Fig. 3 showing a plurality of second encoding levels including a k-th second encoding level and a plurality of second decoding levels including a (k+1)-th second decoding level and a k-th second decoding level.)
wherein the generating of the first optical flow feature and the second optical flow feature comprises generating, at the k-th first decoding level, a (k-1)-th first decoding feature, the first optical flow feature at the k-th first decoding level, and the second optical flow feature at the k-th first decoding level, based on a k-th first encoding feature generated at the k-th first encoding level, (See Kong Page 3 Fig. 3 showing a plurality of second decoding levels. See Kong Page 4 Left Column Equations (1), (2), and (3). Specifically, see Equation (2) showing generating at the k-th first decoding level, denoted by Dk, ~φk0 (a (k-1)th first decoding feature) and the first and second optical flow features Fk-1t->0 and Fk-1t->1. Note, as shown by Fig. 3, this would be based on the k-th first encoding feature generated at the k-th first encoding level.)
a k-th first decoding feature generated at the (k+1)-th first decoding level, the first motion feature at the (k+1)-th first decoding level, and the second motion feature at the (k+1)-th first decoding level. (See Kong 4 Left Column Equation (2). Although not shown, it should be noted that there can exist a (k+1)-th first decoding level, and subsequently, a k-th first decoding feature and first and second motion feature at the (k+1)-th first decoding level. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 10, Pourreza in view of Kong discloses The method of claim 1, wherein the generating of the third image frame comprises: warping the first image frame and the second image frame based on the first optical flow feature, the second optical flow feature, and the motion vector; and (See Pourreza [0183], “At block 1504, the process 1500 includes generating a third reference frame at least in part by performing interpolation between the first reference frame and the second reference frame. . . The process 1500 can include generating first warping information at least in part by performing a warping function on the first reference frame using the first set of motion information. In some aspects, the warping function includes a bilinear interpolation function, such as that as described above with respect to FIG. 7C. The process 1500 can further include generating second warping information at least in part by performing the warping function on the second reference frame using the second set of motion information. The process 1500 can include generating the third reference frame based on the first warping information and the second warping information.”)
merging a result of the warping with residual information based on a merging mask to generate the third image frame. (See Pourreza [0009], “In some aspects, the method, apparatuses, and non-transitory computer-readable medium and described above can include determining a residual at least in part by determining a difference between the input frame and the warped frame; and generating a predicted residual using the residual.”
See [0186], “In some examples, the process 1500 can include generating, based on the warped frame and the predicted residual, a reconstructed frame representing the input frame. The reconstructed frame includes a bidirectionally-predicted frame.” See [0162] equation 3. Further see [0163], “. . . the parameter ∝.sub.0 controls the contribution of the two input images I.sub.0 and I.sub.1 and depends on temporal consistency and occlusion reasoning” In this case, one can consider ∝.sub.0 to correspond to the merging mask as it controls the contribution of the two input,
See Kong Page 4 Right Column Paragraph 1, “M is a one channel merge mask exported by a sigmoid layer whose elements range from 0 to 1, and R is a three-channel image residual that can compensate for details.” Lastly, see Kong Page 4 Right Column Equations (4) and (5) which also includes backward warping w and using M and R to generate ^It the third image frame. The motivation to combine would have been similar to those of Claim 1 rejection motivation.)
Regarding Claim 12, Pourreza in view of Kong discloses A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1. (See Pourreza [0006], “The non-transitory computer-readable medium can include instructions stored thereon which, when executed by one or more processors, cause the one or more processors to: . . .”
The above limitations are similar to those of Claim 1 and are therefore rejected under a similar rationale as those of Claim 1.)
Regarding Claim 13, Pourreza in view of Kong discloses An electronic device comprising: one or more processors; and a memory configured to store instructions, wherein the instructions, when executed by the one or more processors, cause the electronic device to: (See Pourreza [0007], “According to another example, an apparatus for processing video data is provided.” See Pourreza [0005], “In some cases, the system can include at least one memory and one or more processors (e.g., implemented in circuitry) coupled to the memory.”)
perform a first encoding operation based on a first image frame at a first time point and a second image frame at a second time point to generate a first encoding feature; perform a first decoding operation based on the first encoding feature to generate a first optical flow feature between the first time point and a third time point and a second optical flow feature between the second time point and the third time point; and generate a third image frame at the third time point based on the first optical flow feature, the second optical flow feature, and a motion vector corresponding to motion between the first image frame and the second image frame. (The above limitations are similar to those of Claim 1 and are therefore rejected under a similar rationale as those of Claim 1.)
Regarding Claim 14, Claim 14 contains similar limitations as to Claim 2 and is therefore rejected under a similar rationale as that of Claim 2.
Regarding Claim 15, Claim 15 contains similar limitations as to Claim 3 and is therefore rejected under a similar rationale as that of Claim 3.
Regarding Claim 16, Claim 16 contains similar limitations as to Claim 4 and is therefore rejected under a similar rationale as that of Claim 4.
Regarding Claim 17, Claim 17 contains similar limitations as to Claim 5 and is therefore rejected under a similar rationale as that of Claim 5.
Regarding Claim 18, Claim 18 contains similar limitations as to Claim 10 and is therefore rejected under a similar rationale as that of Claim 10.
Regarding Claim 20, Pourreza in view of Kong discloses An electronic device comprising: a memory configured to store instructions, and one or more processors configured to execute the instructions, the instructions when executed by the one or more processors, cause the electronic device to: (See Pourreza [0007], “According to another example, an apparatus for processing video data is provided.” See Pourreza [0005], “In some cases, the system can include at least one memory and one or more processors (e.g., implemented in circuitry) coupled to the memory.”)
perform a first encoding operation based on a first image frame at a first timepoint and a second image frame at a second time point to generate a first encoding feature; perform a first decoding operation based on the first encoding feature to generate a first optical flow feature between the first time point and a third time point and a second optical flow feature between the second time point and the third time point; perform a second encoding operation based on a motion vector based on the first image and the second image to generate a second encoding feature; perform a second decoding operation based on the second encoding feature to generate a first motion feature between the first time point and the third time point and a second motion feature between the second time point and the third time point; and generate a third image frame at the third time point based on the first optical flow feature, the second optical flow feature, the first motion feature and the second motion feature. (The above limitations are similar to those of Claim 5 and are therefore rejected under a similar rationale as those of Claim 5)
Claims 11 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Pourreza in view of Kong and in further view of Kopietz (US 20180310020 A1).
Regarding Claim 11, Pourreza in view of Kong disclose The method of claim 1, wherein the first image frame and the second image frame are a result of a rendering by a rendering engine, (See Pourreza [0195], “In some examples, the processes described herein (e.g., process 1500, process 1600, and/or other process described herein) may be performed by a computing device or apparatus . . . a video game device, an extended reality device. . .” See Pourreza [0183], “At block 1504, the process 1500 includes generating a third reference frame at least in part by performing interpolation between the first reference frame and the second reference frame.”
In this case, frame interpolation for a video game or an extended reality experience would commonly imply a rendering engine, and thus the first and second image frames being a result of a rendering by a rendering engine.)
However, Pourreza in view of Kong fails to explicitly disclose the motion vector is generated in advance during the rendering of the first image frame and the second image frame by the rendering engine.
Kopietz teaches the motion vector is generated in advance during the rendering of the first image frame and the second image frame by the rendering engine. (See [0051], “In this system, a server 100 hosts video game software 102 and a graphics engine 104 which renders video output.” See [0069], “Anticipated motion vectors may be generated ahead of time or they may be generated as needed during runtime.”
Lastly, see [0078], “FIG. 10 shows an example method for generating the motion vector library and an example repetitive motion vector library for caching purposes. Since the motions selected are highly repetitive in nature, they will play in the same manner every time they are triggered. This makes it possible to generate motion vectors ahead of time and organize them into libraries.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Pourreza in view of Kong with Kopietz to have the motion vector generated in advance.
The motivation to combine would have been Pourreza in view of Kong with Kopietz would have been obvious as both Pourreza and Kopietz have similar use cases such as video games (See Kopietz [0051]). The benefit of generating the motion vector in advance is that it can save on processing power. This can be especially effective when there are repetitive motions, in which case a motion vector library can be created in advance (See Kopietz [0078]).
Regarding Claim 19, Claim 19 contains similar limitations as to Claim 11 and is therefore rejected under a similar rationale as that of Claim 11.
Allowable Subject Matter
Claim 8-9 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 8 the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: wherein a (k+1)-th weight mask is generated at the (k+1)-th second decoding level of the second decoding operation, wherein, based on the (k+1)-th weight mask, the first motion feature generated at the (k+1)-th second decoding level of the second decoding and the first optical flow feature generated at the (k+1)-th first decoding level of the first decoding operation are merged to generate a (k+1)-th merged flow, and wherein the (k+1)-th merged flow is used at the k-th first decoding level of the first decoding operation. Thus Claim 8 contains allowable subject matter.
Regarding Claim 9, Claim 9 is dependent upon Claim 8 and thus also contains allowable subject matter.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANG G HUYNH whose telephone number is (571)272-5432. The examiner can normally be reached Mon-Thu 7:30am-4:30pm EST | Fri 7:30am-11:30am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.G.H./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights