DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Oath/Declaration
2. The receipt of Oath/Declaration is acknowledged.
Drawings
3. The drawing(s) filed on 12/08/2023 are accepted by the Examiner.
Status of Claims
4. Claims 1-14 are pending in this application.
Claim Rejections - 35 USC § 103
5. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
6. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
7. The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
8. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
9. Claims 1-3, 5-6, 8-10, 12-13, and 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (US 2021/0158539) in view of Anilkumar et al. (US 2019/0325561), and further in view of Cabral et al. (US 2017/0295359), hereinafter ‘Zhang’, ‘Anilkumar’ and ‘Cabral’.
Regarding Claim 1:
Zhang discloses a method for generating an optical flow for a plurality of successive image frames (Zhang: ‘Systems and methods are disclosed that can improve Optical Flow Estimate (OFE) quality when employing a hint-based algorithm.’ [0003]), comprising, by a computing device (Zhang: Fig. 6 ‘computing device 600’ [0060]):
accessing image data corresponding to a plurality of successive image frames to be displayed on a display associated with a computing device (Zhang discloses determining OFEs for “first version of an image” and a “second version of the image” across multiple hierarchical resolution levels, wherein the successive image versions correspond to image frames used in display associated applications including ADAS, virtual reality, and gaming. (Abstract; ¶¶[0026;0028)); and
generating an optical flow to represent pixel displacements from a first image frame of the plurality of successive image frames to a second image frame of the plurality of successive image frames (Zhang discloses “determining, using a first order, first optical flow estimates (OFEs) for a first version of an image” and “determining, using a second order…second OFEs for a second version of the image,” wherein the OFEs represent pixel-level displacement vectors between successive image frame versions, constituting the generated optical flow. (Abstract; ¶¶[0053-0054]; claim 1)), wherein generating the optical flow for the plurality of successive image frames comprises:.
executing an initialization process by performing a plurality of raster scans of (Zhang discloses performing a plurality of raster scans to generate an initial set of OFEs constituting the initialization process. Specifically, Zhang discloses “the first order begins at an upper left-hand corner of the image and proceeds from left to right and top to bottom” (first raster scan producing first OFEs) and the “second order begins at a lower right-hand corner…proceeds right to left and bottom to top” (second raster scan producing second OFEs). (¶¶[0021; 0043-0044]; claims 4 and 7), wherein the plurality of raster scans of the Zhang further discloses “each of the first OFEs corresponds to a respective set of one or more pixels” constituting a patch of pixels. (Zhang, ¶¶[0001; 0037]; claim 9);
Zhang does not expressly disclose performing the plurality of raster scans of a patch of pixels in parallel.
Anilkumar discloses performing the plurality of raster scans of a patch of pixels in parallel (Anilkumar discloses “initiating transfer of image data…to a graphic processing unit (GPU)” while “calculating, by the central processing unit (CPU)…optical flow of at least one predetermined coarse level of the image pyramid” The CPU and GPU operating simultaneously over the image data constituting parallel raster scan execution. (Anilkumar: ¶¶[0021; 0043-0044]). Anilkumar further discloses “dividing the image into a grid of patches” and performing the optical flow computation “for every patch of the level” explicitly constituting a plurality of raster scans of a patch of pixels. (Anilkumar: ¶¶[0029; 0033]).
Zhang in view of Anilkumar are combinable because they are from the same field of endeavor; both references are directed to accelerating optical flow computation in hierarchical motion estimation pipelines (G06T/207). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement Zhang’s varied scan order approach using Anilkumar parallel CPU/GPU execution architecture.
The suggestion/motivation for doing so is to reduce computation time. Therefore, it would have been obvious to combine Zhang with Anilkumar to obtain the invention as specified.
Zhang in view of Anilkumar further disclose executing a propagation process based on the plurality of optical flow estimates between the plurality of successive image frames (Zhang discloses wherein “a scan of an image performed with a scan order may initially leverage OFEs from a previous scan of the image…using OFEs of neighboring locations as spatial hints” and “the fourth OFE is based, at least in part, on the third OFE and at least one OFE from the first OFEs” wherein OFEs from neighboring pixel locations are propagated to compute the current pixel’s OFE (Zhang: ¶¶[0002; 0032; 0035; 0054-0056]) and,
Zhang in view of Anilkumar do not expressly disclose wherein executing the propagation process comprises propagating the plurality of optical flow estimates for one or more neighboring pixels associated with the patch of pixels; and
Cabral discloses wherein executing the propagation process comprises propagating the plurality of optical flow estimates for one or more neighboring pixels associated with the patch of pixels (Cabral discloses that “an optical flow can be generated using an iterative method which individually optimizes the optical flow vector for each pixel of a camera view and propagates chages in the optical flow to neighboring optical flow vectors” ; Abstract; ¶¶[0005; 0078]);
Zhang, Anilkumar & Cabral are combinable because they are from the same field of endeavor. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement Zhang’s spatial hint based neighbor leveraging using Cabral’s explicit neighbor propagation mechanism. The suggestion/motivation for doing so is to improve OFE quality through spatial pixel relationships. Therefore, it would have been obvious to combine Zhang, Anilkumar & Cabral to obtain the invention as specified.
Zhang, Anilkumar & Cabral further discloses executing a search process (Zhang discloses that “a scan of an image performed with a scan order may initially leverage OFEs from a previous scan of the image…the OFEs leveraged from the previous scan are more likely to be of high accuracy” and “the fourth OFE is based, at least in part, on the third OFE and at least one OFE from the first OFEs” wherein the current scan uses the neighbor derived OFEs to identify the best displacement offset, constituting the search process; (Zhang: Abstract; claim 1; ¶¶[0034; 0054-0056]) by identifying one or more offsets based at least in part on the plurality of optical flow estimates for the one or more neighboring pixels associated with the patch of pixels (Anilkumar further discloses “performing dense optical flow comprises a process of evaluating a pixel difference cost using a neighbor optical flow output” directly teaching the identification of offsets based on neighbor pixel OFEs in the search process; ¶¶[0050; 0064-0065]; claims 8 and 11).
Zhang, Anilkumar & Cabral are combinable because they are from the same field of endeavor. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement Zhang’s search process by using Anilkumar’s identification of offsets based on neighboring pixels. The suggestion/motivation for doing so is to improve the identification of offsets by using a well-known ‘neighboring pixel’ method. Therefore, it would have been obvious to combine Zhang, Anilkumar & Cabral to obtain the invention as specified in claim 1.
Regarding Claim 2:
The proposed combination of Zhang, Anilkumar & Cabral further discloses the method of Claim 1, wherein accessing the image data corresponding to the plurality of successive image frames comprises accessing one or more two-dimensional (2D) arrays of pixels corresponding to the plurality of successive image frames (Zhang discloses that “each of the first OFEs corresponds to a respective set of one or more pixels” of the image wherein pixel-level optical flow processing over successive image frames necessarily and inherently operates on two dimensional arrays of pixel data, as a digital image frame is by definition organized as a 2D spatial array of pixels; Claim 9. It is also noted that Anilkumar expressly discloses “dividing the image into a grid of patches” wherein the grid of patches is an explicit 2D spatial organization of pixel data constituting a 2D array of pixels; ¶¶[0029; 0033]).
Regarding Claim 3:
The proposed combination of Zhang, Anilkumar & Cabral further discloses the method of Claim 1, wherein executing the propagation process comprises executing the propagation process based on the plurality of optical flow estimates and in accordance with one or more predetermined metrics.
Zhang discloses that the propagation process is executed using a “hint-based algorithm” in which OFEs from neighboring locations are evaluated as spatial hints, wherein the quality and accuracy of those spatial hints constitute the predetermined metric governing the propagation process; ([0002; 0032; 0035; 0054-0056]).
Cabral further discloses “analyzing each optical flow proposal and updating the optical flow for each pixel based on an optical flow proposal” wherein the analysis of proposals against a predetermined evaluation criterion constitutes execution of the propagation process in accordance with one or more predetermined metrics; (Abstract; [0015; 0018; 0060]).
Zhang, Anilkumar & Cabral are combinable because they are from the same field of endeavor. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to implement Zhang’s ‘int based algorithm’ by using Cabrals’s analysis of proposals against a predetermined evaluation criterion. The suggestion/motivation for doing so is to improve the optimization when generating a synthetic view across multiple camera views as disclosed by Cabral at ¶[0005]. Therefore, it would have been obvious to combine Zhang, Anilkumar & Cabral to obtain the invention as specified in claim 3.
Regarding Claim 5:
The proposed combination of Zhang, Anilkumar & Cabral further discloses the method of Claim 1, wherein performing the plurality of raster scans of the patch of pixels further comprises performing a plurality of raster scans in a same vertical raster scan direction or in different horizontal raster scan directions.
Zhang discloses that “the first order comprises a first scan directioin in a first axis and a second scan direction in a second axis, and the second order comprises a third scan direction in the first axis…different from the first scan direction” Specificically, Zhang discloses the first order “begins at an upper left hand corner of the image and proceeds from left to right and top to bottom to top” constituting a plurality of raster scans in different horizontal raster scan directions (left to right, and right to left). (Zhang: ¶¶[0021; 0043-0045]; Abstract; Claims 1, 4 and 7).
Regarding Claim 6:
The proposed combination of Zhang, Anilkumar & Cabral further discloses the method of Claim 1, wherein generating the optical flow for the plurality of successive image frames further comprises performing a filtering and a scaling of the optical flow.
Zhang discloses multi-level hierarchical motion estimation across a resolution pyramid, wherein pyramid construction inherently involves scaling (downsampling to coarser resolution levels and upsampling to finer levels) and filtering (smoothing applied between pyramid levels) of the optical flow. (Zhang: ¶¶[0022; 0038-0040; 0045-0047]).
It is noted that Anilkumar also teaches “calculating, by the CPU, optical flow of at least one predetermined coarse level of the image pyramid” wherein the image pyramid structure requires filtering and scaling operations applied to the optical flow across pyramid levels; (Anilkumar:¶¶[0030; 0032; 0055]; claim 1).
Regarding Claim 8:
The proposed rejection of claim 1, over Zhang, Anilkumar & Cabral is similarly cited to reject claim 8 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 1 are equally applicable to claim 8.
It is noted that Zhang discloses one or more non-transitory computer-readable storage media including instructions (Zhang: Fig. 6 ‘memory 604’; [0064]); and one or more processors coupled to the storage media, the one or more processors configured to execute the instructions (Zhang: Fig. 6 ‘computing device 600 with one or more CPUs 606’; [0060]).
Regarding Claim 9:
The proposed rejection of claim 2, over Zhang, Anilkumar & Cabral is similarly cited to reject claim 9 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 2 are equally applicable to claim 9.
Regarding Claim 10:
The proposed rejection of claim 3, over Zhang, Anilkumar & Cabral is similarly cited to reject claim 10 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 3 are equally applicable to claim 10.
Regarding Claim 12:
The proposed rejection of claim 5, over Zhang, Anilkumar & Cabral is similarly cited to reject claim 12 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 5 are equally applicable to claim 12.
Regarding Claim 13:
The proposed rejection of claim 6, over Zhang, Anilkumar & Cabral is similarly cited to reject claim 13 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 6 are equally applicable to claim 13.
Regarding Claim 15:
The proposed rejection of claims 1 and 8, over Zhang, Anilkumar & Cabral is similarly cited to reject claim 15 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claims 1 and 8 are equally applicable to claim 15.
It is noted that Zhang discloses one or more non-transitory computer-readable storage media including instructions (Zhang: Fig. 6 ‘memory 604’; [0064]); and one or more processors coupled to the storage media, the one or more processors configured to execute the instructions (Zhang: Fig. 6 ‘computing device 600 with one or more CPUs 606’; [0060]).
Regarding Claim 16:
The proposed rejection of claims 2 and 9, over Zhang, Anilkumar & Cabral is similarly cited to reject claim 16 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claims 2 and 9 are equally applicable to claim 16.
Regarding Claim 17:
The proposed rejection of claims 3 and 10, over Zhang, Anilkumar & Cabral is similarly cited to reject claim 17 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claims 3 and 10 are equally applicable to claim 17.
Regarding Claim 18:
The proposed rejection of claims 5 and 12, over Zhang, Anilkumar & Cabral is similarly cited to reject claim 18 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claims 5 and 12 are equally applicable to claim 18.
Regarding Claim 19:
The proposed rejection of claims 6 and 13, over Zhang, Anilkumar & Cabral is similarly cited to reject claim 19 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claims 6 and 13 are equally applicable to claim 19.
10. Claims 4 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang, Anilkumar & Cabral as applied to claims 3 and 10 above, and further in view of Garud et al. (US 2019/0244036).
Regarding Claim 4.
The proposed combination of Zhang, Anilkumar & Cabral further discloses the method of Claim 3, but do not expressly disclose wherein the one or more predetermined metrics comprises one or more of a data metric, a rigidity metric, or a constraint metric.
Garud discloses wherein the one or more predetermined metrics comprises one or more of a data metric, a rigidity metric, or a constraint metric.
Garud discloses using a Hamming distance as the Cost function which is strictly considered a data metric (or distance metric); (¶0067; 0035]; claim 2).
Garud discloses a ‘motion smoothness factor’ penalizing non-rigid deviations between neighboring OFEs , constituting a rigidity metric at ¶[0035] and claim 2, which frame it as constraining the search to smooth/consistent motion, which is definitionally rigidity.
Garud discloses a Binary Census Transform (BCT) applied as a robust constraint over the patch of pixels, constituting a constraint metric; (¶¶[0035; 0066-0067] claim 2).
Zhang, Anilkumar & Cabral are combinable because they are from the same field of endeavor. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Garud’s three component metric framework to the propagation process of the Zhang, Anilkumar & Cabral combination. The suggestion/motivation for doing so is Zhang’s hint-based propagation algorithm requires a cost or quality evaluation function to select among competing OFE candidates from neighboring pixels. Garud’s explicit data, rigidity, and constraint metric framework is a well-known, finite set of predetermined metrics for OFE evaluation routinely applied in the field of hierarchical motion estimation, and improves OFE accuracy through structured cost evaluation. Therefore, it would have been obvious to combine Zhang, Anilkumar, Cabral & Garud to obtain the invention as specified in claim 4.
Regarding Claim 11:
The proposed rejection of claim 4, over Zhang, Anilkumar, Cabral & Garud is similarly cited to reject claim 11 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 4 are equally applicable to claim 11.
11. Claims 7, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang, Anilkumar & Cabral as applied to claims 1, 8 and 15 above, and further in view of Hunt (US 2011/0116584 A1).
Regarding Claim 7:
The proposed combination of Zhang, Anilkumar & Cabral further discloses the method of Claim 1, but does expressly disclose wherein generating the optical flow for the plurality of successive image frames further comprises: comparing the generated optical flow to a reference optical flow; and generating one or more confidence metrics based on the comparison of the generated optical flow and the reference optical flow, wherein the one or more confidence metrics comprises a measure of a consistency between the generated optical flow and the reference optical flow.
Hunt discloses wherein generating the optical flow for the plurality of successive image frames further comprises: comparing the generated optical flow to a reference optical flow; and generating one or more confidence metrics based on the comparison of the generated optical flow and the reference optical flow, wherein the one or more confidence metrics comprises a measure of a consistency between the generated optical flow and the reference optical flow.
Hunt discloses an optical flow engine that “derives, from a plurality of images, both optical flow output and a measure of confidence in the optical flow output”. (Hunt: ¶¶[0015; 0062]; claim 1).
Hunt further discloses “applying a first filter [generated optical flow] and a second filter [reference optical flow]…comparing the output of the first filter to the output of the second filter” and “deriving confidence estimate output representing a degree of confidence in the correctness of the first filter output” constituting comparing the generated optical flow to a reference optical flow and generating confidence metrics based on that comparison. (Hunt: ¶¶[0015; 0018; 0060]; claim 1 and Abstract).
Hunt further discloses that the confidence estimate constitutes a measure of consistency, as “the first filter has at least one failure mode not shared by the second filter” (Hunt: ¶¶[0015; 0039; 0087 (PK1/PK2 ratio); claims 8-9).
The degree to which both filters agree constitutes the consistency measure between the generated and reference optical flows. (Hunt: ¶¶[0020; 0104-0105]; claim 43 last clause).
Zhang, Anilkumar, Cabral & Hunt are combinable because they are from the same field of endeavor. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Hunt’s confidence metric framework to the Zhang/Anilkumar/Cabral optical flow pipeline, as Zhang’s mulit-scan approach generates multiple independent OFE outputs (first order and second order scan OFEs) that are natural candidates for a generated vs. reference comparison. The suggestion/motivation for doing so is improve OFE reliability through consistency verification. Therefore, it would have been obvious to combine Zhang, Anilkumar, Cabral & Hunt to obtain the invention as specified in claim 7.
Regarding Claim 14:
The proposed rejection of claim 7, over Zhang, Anilkumar, Cabral & Hunt is similarly cited to reject claim 14 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claim 7 are equally applicable to claim 14.
Regarding Claim 20:
The proposed rejection of claims 7 and 14, over Zhang, Anilkumar, Cabral & Hunt is similarly cited to reject claim 20 because these steps occur in the operation of the method as discussed above. Thus, the arguments similar to that presented above for claims 7 and 14 are equally applicable to claim 20.
Conclusion
12. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lin et al. (US 12,100,169) discloses systems and techniques for performing optical flow estimation between one or more frames. For example, a process can include determining a subset of pixels of at least one of a first frame and a second frame, and generating a mask indicating the subset of pixels. The process can include determining, based on the mask, one or more features associated with the subset of pixels of at least the first frame and the second frame. The process can include determining optical flow vectors between the subset of pixels of the first frame and corresponding pixels of a second frame. The process can include generating an optical flow map for the second frame using the optical flow vectors.
13. Any inquiry concerning this communication or earlier communications from the examiner should be directed to NEIL R MCLEAN whose telephone number is (571)270-1679. The examiner can normally be reached Monday-Thursday, 6AM - 4PM, PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Akwasi M Sarpong can be reached at 571.270.3438. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NEIL R MCLEAN/Primary Examiner, Art Unit 2681