Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/20/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 10 and 18 are objected to because of the following informalities:
Claim 10 recites “applying a pyramidal block matching” in line 8. The examiner believes it should recite apply instead of “applying” to be grammatically correct. Appropriate correction is required.
Claim 18 starts by reciting “[t]he computing device of claim 10, further comprising generating an optical flow mask ... and using the optical flow mask…” Claim 18, however, is an apparatus claim that cannot further comprise additional steps as recited. The examiner suggests amending the claim to convert the recited additional steps into additional operations of the one or more processors.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 4, 6-11, 13, and 15-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over us patent application publication no. 2022/0101539 to Lin et al. (hereinafter Lin) in view of us patent application publication no. 2017/0161565 to Garud et al. (hereinafter Garud).
For claim 1, Lin as applied teaches a method, comprising:
generating an optical flow mask based, at least in part, on a calculated degree of confidence of one or more motion vectors relating a prior frame and a current frame in a rendered image sequence (see, e.g., pars. 34, 36, 44, 46, 47, 57-58, 60, and 71-74 and FIGS, 1, 3, 4A-B, and 5 which teach generating a mask identifying subsets of pixels in first and second frames, wherein the subsets of pixels are associated with feature vectors indicating pixels with high spatial confidences, e.g., a high amount of movement); and
applying a pyramidal block matching to calculate an optical flow relating the prior image frame and the current image frame (see, e.g., pars. 34, 43, 48, 49, 52-54, 61, 66-67, 69, and 78-79 and FIGS. 1, 2A-B and 5, which teach determining optical flows for the subsets of pixels within the frames identified by the mask and the associated feature vectors, wherein the optical flow determination includes pixel-wise matching between the frames and utilizes multiple scales/resolution versions of the frames and a feature pyramid),
wherein applying the pyramidal block matching comprises applying the pyramidal block matching differently for portions of an image sequence masked by the generated optical flow mask (see, e.g., pars. 34, 35, 43, 44, 52-54, 66, and 70-73 and FIGS. 1, 2A-B, 4A-B, and 5, which teach that the optical flow estimation are performed only for the subsets of pixels within the frames identified by the mask).
The examiner believes that the cited portions of Lin at least implicitly teaches, if not suggest, applying pyramidal block matching for optical flow estimation (see, e.g., pars. 34, 43, 48, 49, 52-54, 61, 66-67, 69, and 78-79 and FIGS. 1, 2A-B and 5). However, for the compact prosecution, the examiner relies, in addition and/ alternative to Lin, Garud in the analogous art that teaches applying pyramidal block matching for optical flow estimation (see, e.g., pars. 34-47 and FIGS. 2-3 of Garud).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lin to perform a pyramidal block matching as taught by Garud because doing so yield predictable results of being “faster and more tolerant to local minima as compared to an exhaustive search at high resolution” (see par. 6 of Gaud and MPEP 2143(I)(D)).
For claim 10, Lin as applied teaches a computing device (see, e.g., pars. 37 and 110-111 and FIGS. 1 and 8), comprising:
a memory comprising one more storage devices (see, e.g., pars. 37 and 112-113 and FIGS. 1 and 8); and
one or more processors coupled to the memory (see, e.g., pars. 37 and 112-113 and FIGS. 1 and 8), the one or more processors operable to execute instructions stored in the memory to, for a rendered image sequence:
generate an optical flow mask based, at least in part, on a calculated degree of confidence of one or more motion vectors relating a prior frame and a current frame in a rendered image sequence (see, e.g., pars. 34, 36, 44, 46, 47, 57-58, 60, and 71-74 and FIGS, 1, 3, 4A-B, and 5 which teach generating a mask identifying subsets of pixels in first and second frames, wherein the subsets of pixels are associated with feature vectors indicating pixels with high spatial confidences, e.g., a high amount of movement);
applying a pyramidal block matching to calculate an optical flow relating the prior frame and the current frame (see, e.g., pars. 34, 43, 48, 49, 52-54, 61, 66-67, 69, and 78-79 and FIGS. 1, 2A-B and 5, which teach determining optical flows for the subsets of pixels within the frames identified by the mask and the associated feature vectors, wherein the optical flow determination includes pixel-wise matching between the frames and utilizes multiple scales/resolution versions of the frames and a feature pyramid),
wherein applying the pyramidal block matching comprises applying the pyramidal block matching differently for portions of an image sequence masked by the generated optical flow mask (see, e.g., pars. 34, 35, 43, 44, 52-54, 66, and 70-73 and FIGS. 1, 2A-B, 4A-B, and 5, which teach that the optical flow estimation are performed only for the subsets of pixels within the frames identified by the mask).
The examiner believes that the cited portions of Lin at least implicitly teaches, if not suggest, applying pyramidal block matching for optical flow estimation (see, e.g., pars. 34, 43, 48, 49, 52-54, 61, 66-67, 69, and 78-79 and FIGS. 1, 2A-B and 5). However, for the compact prosecution, the examiner relies, in addition and/ alternative to Lin, Garud in the analogous art that teaches applying pyramidal block matching for optical flow estimation (see, e.g., pars. 34-47 and FIGS. 2-3 of Garud).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lin to perform a pyramidal block matching as taught by Garud because doing so yield predictable results of being “faster and more tolerant to local minima as compared to an exhaustive search at high resolution” (see par. 6 of Gaud and MPEP 2143(I)(D)).
For claims 2 and 11, Lin in view of Garud teaches that applying the pyramidal block matching differently for portions of an image sequence masked by the generated optical flow mask comprises excluding the portions of the image sequence masked by the generated optical flow mask from applying pyramidal block matching (see, e.g., pars. 34, 35, 43, 44, 52-54, 66, and 70-73 and FIGS. 1, 2A-B, 4A-B, and 5 of Lin, which teach that the optical flow estimation are performed only for the subsets of pixels within the frames identified by the mask).
For claims 4 and 13, Lin in view of Garud teaches that generating the optical flow mask further comprises masking one or more areas of the rendered image sequence comprising user interface graphics (see, e.g., pars. 34, 35, 43, 44, 52-54, 66, 70-73, and 114 and FIGS. 1, 2A-B, 4A-B, 5, and 8 of Lin, which teach that the mask identifies the subsets of pixels and hence masks other pixels, and that the computing system includes the input device, such as a touch-sensitive screen for graphical input).
For claims 6 and 15, while Lin as applied does not explicitly teach, Garud in the analogous art teaches that the pyramidal block matching comprises:
creating a pyramid of consecutively downsampled prior image frames (see, e.g. pars. 41-43 and FIG. 3 of Garud, which teach generating image pyramid of N levels for query and reference images, wherein a level below is half the resolution in both horizontal and vertical dimensions of a level above);
creating a pyramid of consecutively downsampled current image frames corresponding to the pyramid of consecutively downsampled prior image frames (see, e.g. pars. 41-43 and FIG. 3 of Garud, which teach generating image pyramid of N levels for query and reference images, wherein a level below is half the resolution in both horizontal and vertical dimensions of a level above);
block matching a lowest resolution downsampled prior image frame and a lowest resolution downsampled current image frame (see, e.g. pars. 35 and 45 and FIGS. 2 and 3 of Garud, which teach starting the block matching from the top of the pyramid); and
iteratively block matching consecutively higher resolution downsampled prior image frames and current image frames using block matching over corresponding smaller block matching search spaces not covered by prior lower resolution block matching iterations (see, e.g. pars. 35-39 and 45-47 and FIGS. 2 and 3 of Garud, which teach iteratively block matching at the next level, wherein at the base level, the determined optical flow is refined at fractional pixel resolution).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lin to perform a pyramidal block matching as taught by Garud because doing so yield predictable results of being “faster and more tolerant to local minima as compared to an exhaustive search at high resolution” (see par. 6 of Gaud and MPEP 2143(I)(D)).
For claims 7 and 16, while Lin as applied does not explicitly, Garud in the analogous art teaches that a number of pyramid layers, a number of tiles to search or a size of a search tile, or a combination thereof, are configurable parameters that may be selected to achieve a desired compute time, a maximum detectable optical flow vector length, or a confidence level in optical flow matches, or a combination thereof (see, e.g., pars. 6 and 37 of Garud, which teach selecting a number of layers and a size of search tile with respect to the timing and tolerance of the search and the computation and data bandwidth).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lin to parameterized as taught by Garud because doing so yield predictable results of being able to overcome the given timing/resource constraints (see MPEP 2143(I)(D)).
For claims 8 and 17, Lin in view of Garud teaches that the number of pyramid layers, the number of tiles to search or the size of a search tile, or a combination thereof, may be varied across different areas of the current frame or across different levels of the pyramid of image frames used in pyramidal block matching, or a combination thereof (see, e.g. pars. 35-39 and 45-47 and FIGS. 2 and 3 of Garud, which teach that the search is done at fractional level in the base level while the searches are done at integer level in all other levels).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Lin to parameterized as taught by Garud because doing so yield predictable results of being able to overcome the given timing/resource constraints (see MPEP 2143(I)(D)).
For claims 9 and 18, Lin in view of Garud teaches generating an optical flow mask for each level of a rendered image pyramid in pyramidal block matching, and using the optical flow mask corresponding to each level of the rendered image pyramid to applying the pyramidal block matching differently for portions of an image sequence masked by the generated optical flow mask (see, e.g., pars. 61 and 69 of Lin, which teach, for the frames and the multi-scale feature pyramid, selecting a mask based on the respective scale of the feature pyramid and the frame).
Allowable Subject Matter
Claims 19 and 20 are allowed.
Claims 3, 5, 12, and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regard to claims 3 and 12, when considered as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“wherein the calculated degree of confidence expresses a lower degree of confidence for lighting effects, particle effects, animated textures, or a combination thereof.”
In regard to claims 5 and 14, when considered as a whole, prior art of record fails to disclose or render obvious, alone or in combination:
“warping color parameters and depth parameters from the prior frame into the current frame;
creating a color difference mask based, at least in part, on differences between warped color parameters from the prior frame and color parameters from the current frame;
creating a depth difference mask based, at least in part, on differences between the warped depth parameters from the prior frame and depth parameters from the current frame;
converting the depth difference mask and the color difference mask to binary masks; and
calculating a difference between the depth difference mask and the color difference mask to generate the optical flow mask.”
Additional Citations
The following table lists several references that are relevant to the subject matter claimed and disclosed in this Application. The references are not relied on by the Examiner, but are provided to assist the Applicant in responding to this Office action.
Citation
Relevance
Garud et al. (us pat. pub. 2015/0365696)
Describes a novel process for optical flow determination using pyramidal block matching. Embodiments disclosed herein enable estimation of motion as instantaneous image velocities (pixel motion) based on a pair of images (e.g., temporally ordered images), and optionally on historic evidence of the image velocities and parametric models of the image velocities. Embodiments apply hierarchical motion estimation with a coarse-to-fine search methodology that minimizes a cost function over the images. Suitable cost functions may be sum of absolute differences (SAD) over pixel values or hamming distance over binary feature descriptors. Embodiments anticipate that over a small spatial and temporal neighborhood image velocities are constant/slowly varying and use spatial and temporal predictors to provide accurate motion estimation in small or large motion conditions. Embodiments may apply relative simple computational operations that allow for implementation using hardware of reasonable complexity while outperforming conventional optical flow estimation algorithms that are significantly more computationally intensive.
Tsoupko-Sitnikov et al. (us pat. pub. 2008/0278633)
Describes an image processing method and apparatus which utilizes corresponding point information indicating correspondence between image frames. In one embodiment, a corresponding point information generator computes matching between a source image frame and a destination image frame in image data comprising consecutive image frames so as to determine corresponding point information indicating pixel-by-pixel matching. A motion vector detector determines a motion vector for each pixel in the source image frame according to a result of matching. A reliability area isolating unit segments an image frame in which a motion vector is determined into blocks, so as to isolate, in each block, a reliable area characterized by relatively high precision of the motion vector as calculated and a non-reliable area characterized by relatively low precision of the motion vector. A motion vector improving unit calculates, when a motion vector of a reliable area is applied to a pixel in a non-reliable area adjacent to the reliable area, an error between a pixel value occurring at the destination as a result of application and a pixel value of a corresponding pixel in the destination image frame, and, when the error is equal to or smaller than a threshold, incorporates the pixel in the non-reliable area into the reliable area, and replaces the motion vector of that pixel by the motion vector of the reliable area.
Ershadi et al. (us pat. pub. 2026/0075160)
Describes systems and techniques for interpolating image data. For instance, a method for interpolating image data is provided. The method may include processing a first image frame and a second image frame using a motion estimator to generate first motion vectors, wherein the motion estimator comprises a machine-learning model trained to generate motion vectors based on image frames; projecting the first motion vectors to generate second motion vectors; and generating a third image frame based on the first image frame, the second image frame, and the second motion vectors.
Table 1
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See Table 1 and form 892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WOO RHIM whose telephone number is (571)272-6560. The examiner can normally be reached Mon - Fri 9:30 am - 6:00 pm et.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Henok Shiferaw can be reached at 571-272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WOO C RHIM/Examiner, Art Unit 2676
Prosecution Insights