DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed on 12/17/2025 has been entered. Claims 7 has been canceled, claims 1-6 and 8-20 remain pending in the application.
Claim Objections
Claim 1 and 18 are objected to because of the following informalities:
Claim 1 and 18 recite “perform… a single flow query for each respective patch of the plurality of patches within the first feature map, wherein a number of flow queries performed… is not a function of a number of patches included in the plurality of neighboring patches”, it sounds contradictory for the number of flow queries (“a single flow query for each respective patch” means the number of flow queries = number of patches), please amend or explain more to clarify.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 8-12 and 14-18 are rejected under 35 U.S.C. 103 as being unpatentable over Lin U.S. Patent Application 20220398747 in view of Wei U.S. Patent Application 20180268208.
Regarding claim 1, Lin discloses an apparatus to perform feature calculation, the apparatus comprising:
one or more memories (memory 1115) configured to store a plurality of feature maps (paragraph [0092]: the feature extraction engine 502 can concatenate, group, and/or otherwise store the contextual features of the neighbor pixels in connection with the contextual features of the center pixel within a data structure associated with the center pixel); and
one or more processors (processor 1110) coupled to the one or more memories, the one or more processors being configured to:
obtain flow information corresponding to a plurality of flow vectors between a first feature map and a second feature map of the plurality of feature maps (paragraph [0076]: The use of initial flow estimates in a flow target map 316 and flow search areas can be applied more generally to a source feature map F0 and at target feature map F1; paragraph [0097]: the optical flow computation engine 510 can perform optical flow estimation to determine optical flow vectors for pixels or regions of pixels);
for each respective patch of a plurality of patches within the first feature map, perform a flow query to determine a corresponding target patch within the second feature map, wherein the flow query is based on the flow information (paragraph [0082]: each feature of the source feature map, such as feature map F0, can have a corresponding flow search area. In such implementations, if the source feature map has N features, then correlation volume 432 can include N flow search (query) areas; paragraph [0089]: the feature extraction engine 502 can determine contextual features associated with the pixels of the source frame IS and/or the target frame IT; paragraph [0102]: the optical flow computation engine 510 can take the most recent pixel-level or patch-level (e.g., for a pixel and associated neighbor pixels) flow (or displacement) estimates to look up for the corresponding correlation metrics along with a suitable neighborhood context (e.g., association with neighboring pixels) in the correlation volume);
obtain, from the second feature map stored in the one or more memories, feature information for the corresponding target patch (paragraph [0089]: the feature extraction engine 502 can determine contextual features associated with the pixels of the source frame IS and/or the target frame IT; paragraph [0092]: the feature extraction engine 502 can concatenate, group, and/or otherwise store the contextual features of the neighbor pixels in connection with the contextual features of the center pixel within a data structure associated with the center pixel); and
obtain, from the second feature map stored in the one or more memories, respective feature information for a plurality of neighboring patches included within a virtual expanded range around the corresponding target patch within the second feature map, wherein the respective feature information for the plurality of neighboring patches and the feature information for the corresponding target patch are obtained based on the flow query (paragraph [0093]: Associating contextual features of neighbor pixels with contextual features of a center pixel can improve the accuracy of optical flow estimation. For instance, determining and storing the contextual features of neighbor pixels in connection with a center pixel can help the optical flow estimation system 500 accurately identify a pixel that corresponds to the center pixel within a subsequent frame; paragraph [0089]: the feature extraction engine 502 can determine contextual features associated with the pixels of the source frame IS and/or the target frame IT; paragraph [0092]: The feature extraction engine 502 can determine contextual features of any number of neighbor pixels associated with a center pixel... the feature extraction engine 502 can concatenate, group, and/or otherwise store the contextual features of the neighbor pixels in connection with the contextual features of the center pixel within a data structure associated with the center pixel; paragraph [0082]: each feature of the source feature map, such as feature map F0, can have a corresponding flow search area. In such implementations, if the source feature map has N features, then correlation volume 432 can include N flow search (query) areas); and
perform, for an iteration of the features, a single flow query for each respective patch of the plurality of patches within the first feature map, wherein a number of flow queries performed is not a function of the virtual expanded range and is not a function of a number of patches included in the plurality of neighboring patches (paragraph [0102]: the optical flow computation engine 510 can take the most recent pixel-level (not a function of the virtual expanded range and not in the plurality of neighboring patches) or patch-level (e.g., for a pixel and associated neighbor pixels) flow (or displacement) estimates to look up for the corresponding correlation metrics along with a suitable neighborhood context (e.g., association with neighboring pixels) in the correlation volume; paragraph [0069]: the optical flow map engine 106 can use incremental optical flow maps to update a cumulative optical flow map between multiple iterations of optical flow estimation between two adjacent frames).
Lin discloses all the features with respect to claim 1 as outlined above. However, Lin fails to disclose performing feature propagation explicitly.
Wei discloses performing feature propagation (paragraph [0030]: a feature propagation function is defined as...; paragraph [0048]: The method 300 then propagates 314 each of the at least one feature maps based on the flow field to approximate current locations of features identified within each of the at least one feature maps).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin’s to perform feature propagation as taught by Wei, to improve efficiency and reduce cost for processing videos.
Regarding claim 8, Lin as modified by Wei discloses the apparatus of claim 1, wherein the respective feature information for the plurality of neighboring patches is obtained from a single memory replica of the second feature map stored by the one or more memories (Lin’s paragraph [0064]: the flow target engine 102 can adaptively change the number of frames of optical flow history stored in the circular memory... the flow target map may reduce the number of frames of optical flow history 110 stored in the circular memory; paragraph [0102]: the optical flow computation engine 510 can take the most recent pixel-level or patch-level (e.g., for a pixel and associated neighbor pixels) flow (or displacement) estimates to look up for the corresponding correlation metrics along with a suitable neighborhood context (e.g., association with neighboring pixels) in the correlation volume).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin’s to perform feature propagation as taught by Wei, to improve efficiency and reduce cost for processing videos.
Regarding claim 9, Lin as modified by Wei discloses the apparatus of claim 1, wherein the respective feature information for the plurality of neighboring patches is obtained without using shifted memory replicas corresponding to the second feature map (Lin’s paragraph [0064]: the flow target engine 102 can adaptively change the number of frames of optical flow history stored in the circular memory... the flow target map may reduce the number of frames of optical flow history 110 stored in the circular memory).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin’s to perform feature propagation as taught by Wei, to improve efficiency and reduce cost for processing videos.
Regarding claim 10, Lin as modified by Wei discloses the apparatus of claim 1, wherein the one or more processors are configured to determine optical flow information corresponding to the first feature map and the second feature map, wherein the optical flow information is based on the feature propagation (Lin’s paragraph [0076]: The use of initial flow estimates in a flow target map 316 and flow search areas can be applied more generally to a source feature map F0 and at target feature map F1; paragraph [0097]: the optical flow computation engine 510 can perform optical flow estimation to determine optical flow vectors for pixels or regions of pixels; Wei’s paragraph [0030]: a feature propagation function is defined as...; paragraph [0048]: The method 300 then propagates 314 each of the at least one feature maps based on the flow field to approximate current locations of features identified within each of the at least one feature maps).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin’s to perform feature propagation as taught by Wei, to improve efficiency and reduce cost for processing videos.
Regarding claim 11, Lin as modified by Wei discloses the apparatus of claim 1, wherein each respective flow vector of the plurality of flow vectors is indicative of a displacement from a source patch within the first feature map to a target patch within the second feature map (Lin’s paragraph [0076]: The use of initial flow estimates in a flow target map 316 and flow search areas can be applied more generally to a source feature map F0 and at target feature map F1; paragraph [0097]: the optical flow computation engine 510 can perform optical flow estimation to determine optical flow vectors for pixels or regions of pixels).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin’s to perform feature propagation as taught by Wei, to improve efficiency and reduce cost for processing videos.
Regarding claim 12, Lin as modified by Wei discloses the apparatus of claim 1, wherein the one or more processors are configured to perform diffusion based on iterative propagation between the first feature map and the second feature map using the respective feature information for the plurality of neighboring patches included within the virtual expanded range (Lin’s paragraph [0089]: the feature extraction engine 502 can determine contextual features associated with the pixels of the source frame IS and/or the target frame IT; paragraph [0092]: the feature extraction engine 502 can concatenate, group, and/or otherwise store the contextual features of the neighbor pixels in connection with the contextual features of the center pixel within a data structure associated with the center pixel; paragraph [0069]: the optical flow map engine 106 can use incremental optical flow maps to update a cumulative optical flow map between multiple iterations of optical flow estimation between two adjacent frames; Wei’s paragraph [0030]: a feature propagation function is defined as...; paragraph [0048]: The method 300 then propagates 314 each of the at least one feature maps based on the flow field to approximate current locations of features identified within each of the at least one feature maps).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin’s to perform feature propagation as taught by Wei, to improve efficiency and reduce cost for processing videos.
Regarding claim 14, Lin as modified by Wei discloses the apparatus of claim 1, further comprising one or more cameras configured to capture respective images corresponding to the first feature map and the second feature map (Lin’s paragraph [0002]: a camera or a device including a camera can capture a sequence of frames of a scene (e.g., a video of a scene)).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin’s to perform feature propagation as taught by Wei, to improve efficiency and reduce cost for processing videos.
Regarding claim 15, Lin as modified by Wei discloses the apparatus of claim 14, wherein the one or more processors are configured to:
generate one or more output images corresponding to the respective images, wherein the one or more output images are generated based on iterative feature propagation using the virtual expanded range (Lin’s paragraph [0069]: the optical flow map engine 106 can use incremental optical flow maps to update a cumulative optical flow map between multiple iterations of optical flow estimation between two adjacent frames; paragraph [0076]: The use of initial flow estimates in a flow target map 316 and flow search areas can be applied more generally to a source feature map F0 and at target feature map F1).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin’s to perform feature propagation as taught by Wei, to improve efficiency and reduce cost for processing videos.
Regarding claim 16, Lin as modified by Wei discloses the apparatus of claim 15, further comprising one or more displays configured to display the one or more output images (Lin’s paragraph [0155]: Computing system 1100 can also include output device 1135, which can be one or more of a number of output mechanisms; paragraph [0002]: the sequence of frames can be processed for performing one or more functions, can be output for display).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin’s to perform feature propagation as taught by Wei, to improve efficiency and reduce cost for processing videos.
Regarding claim 17, Lin as modified by Wei discloses the apparatus of claim 14, wherein the one or more processors are configured to perform one or more of optical flow estimation, depth estimation, or motion estimation between the first feature map and the second feature map, based on iterative feature propagation using the virtual expanded range (Lin’s paragraph [0076]: The use of initial flow estimates in a flow target map 316 and flow search areas can be applied more generally to a source feature map F0 and at target feature map F1; paragraph [0097]: the optical flow computation engine 510 can perform optical flow estimation to determine optical flow vectors for pixels or regions of pixels; paragraph [0082]: each feature of the source feature map, such as feature map F0, can have a corresponding flow search area. In such implementations, if the source feature map has N features, then correlation volume 432 can include N flow search areas; paragraph [0069]: the optical flow map engine 106 can use incremental optical flow maps to update a cumulative optical flow map between multiple iterations of optical flow estimation between two adjacent frames).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin’s to perform feature propagation as taught by Wei, to improve efficiency and reduce cost for processing videos.
Claim 18 recites the functions of the apparatus recited in claim 1 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 1 applies to the method steps of claim 18.
Claim 2 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Lin U.S. Patent Application 20220398747 in view of Wei U.S. Patent Application 20180268208, and further in view of Mao U.S. Patent Application 20210365707.
Regarding claim 2, Lin as modified by Wei discloses perform feature propagation based on a similarity evaluation between feature information of the respective patch within the first feature map and the respective feature information obtained for the plurality of neighboring patches within the second feature map (Lin's paragraph [0096]: using the output of sampled feature maps from both input frames (the source frame IS and the flow search areas within the target frame IT) as input, the correlation volume engine 508 can compute pair-wise correlation in a number of pair combinations (e.g., for all possible pair combinations). Each correlation quantity denotes the correlation or in some cases the similarity between two features, one from each frame (e.g., one feature from the source frame IS and one feature from the target frame IT)). However, Lin as modified by Wei fails to disclose a current best match for feature corresponding to the respective patch within the first feature map.
Mao discloses a current best match for feature corresponding to the respective patch within the first feature map (paragraph [0139]: Every tracker-blob pair or combination between the trackers 510A and the blobs 508 includes a cost that is included in the cost matrix. Best matches between the trackers 510A and blobs 508 can be determined by identifying the lowest cost tracker-blob pairs in the matrix).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin and Wei’s to use best match as taught by Mao, to perform video analysis and generate best output.
Claim 19 recites the functions of the apparatus recited in claim 2 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 2 applies to the method steps of claim 19.
Claim 3 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lin U.S. Patent Application 20220398747 in view of Wei U.S. Patent Application 20180268208, in view of Mao U.S. Patent Application 20210365707, and further in view of Rohani U.S. Patent Application 20220215552.
Regarding claim 3, Lin as modified by Wei and Mao discloses determining a particular neighboring patch of the plurality of neighboring patches has a similarity with the respective patch within the first feature map; and configure the particular neighboring patch as the current best match for feature propagation corresponding to the respective patch within the first feature map (Mao’s paragraph [0139]: Every tracker-blob pair or combination between the trackers 510A and the blobs 508 includes a cost that is included in the cost matrix. Best matches between the trackers 510A and blobs 508 can be determined by identifying the lowest cost tracker-blob pairs in the matrix; Lin's paragraph [0096]: using the output of sampled feature maps from both input frames (the source frame IS and the flow search areas within the target frame IT) as input, the correlation volume engine 508 can compute pair-wise correlation in a number of pair combinations (e.g., for all possible pair combinations). Each correlation quantity denotes the correlation or in some cases the similarity between two features, one from each frame (e.g., one feature from the source frame IS and one feature from the target frame IT)). However, Lin as modified by Wei and Mao fails to disclose determining greater similarity with the respective patch within the first feature map than the current best match.
Rohani discloses determining greater similarity with the respective patch within the first feature map than the current best match (paragraph [0025]: determining a best match record includes generating a similarity score of infection image with images associated with each of the records of the medical databank and determining whether the similarity score is greater than a predetermined threshold).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin, Wei and Mao’s to rank similarity score as taught by Rohani, to identify a best match based on image processing.
Claim 20 recites the functions of the apparatus recited in claim 3 as method steps. Accordingly, the mapping of the prior art to the corresponding functions of the apparatus in claim 3 applies to the method steps of claim 20.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Lin U.S. Patent Application 20220398747 in view of Wei U.S. Patent Application 20180268208, and further in view of Vianello U.S. Patent Application 20230143198.
Regarding claim 4, Lin as modified by Wei discloses all the features with respect to claim 1 as outlined above. However, Lin as modified by Wei fails to disclose a configured radius value and a location of the corresponding target patch.
Vianello discloses a configured radius value and a location of the corresponding target patch (paragraph [0067]: The region encompassing the location of interest can include: a neighborhood, a radius from the location of interest (e.g., a predetermined radius, a radius determined based on the density and/or topography of the region)).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin and Wei’s to determine radius as taught by Vianello, to accurately assess various aspects of image area.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Lin U.S. Patent Application 20220398747 in view of Wei U.S. Patent Application 20180268208, and further in view of Ren U.S. Patent Application 20190147613.
Regarding claim 5, Lin as modified by Wei discloses determining the virtual expanded range based on number of patches included in the plurality of neighboring patches (Lin’s paragraph [0082]: each feature of the source feature map, such as feature map F0, can have a corresponding flow search area. In such implementations, if the source feature map has N features, then correlation volume 432 can include N flow search areas). However, Lin as modified by Wei fails to disclose a configured neighborhood size indicative of a number of patches.
Ren discloses a configured neighborhood size indicative of a number of patches (paragraph [0029]: a limitation may be placed on the number of patches by restricting the patch size to a certain range and/or requiring that the average intensity in a patch be within a specified range).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin and Wei’s to determine patch size as taught by Ren, to generate high quality images.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Lin U.S. Patent Application 20220398747 in view of Wei U.S. Patent Application 20180268208, and further in view of Schmit U.S. Patent Application 20190037240.
Regarding claim 6, Lin as modified by Wei discloses all the features with respect to claim 1 as outlined above. However, Lin as modified by Wei fails to disclose a memory requirement for warping associated with the flow query is not a function of the virtual expanded range and is not a function of a number of patches.
Schmit discloses a memory requirement for warping associated with the flow query is not a function of the virtual expanded range and is not a function of a number of patches (paragraph [0017]: in the memory, the plurality of warped images and perform a motion search around each co-located pixel block of a reference panoramic frame by limiting the motion searches in a vertical direction around the co-located pixel blocks).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin and Wei’s to determine memory requirement as taught by Schmit, to perform operation efficiently and reduce cost.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Lin U.S. Patent Application 20220398747 in view of Wei U.S. Patent Application 20180268208, and further in view of FARÅS U.S. Patent Application 20210133995.
Regarding claim 13, Lin as modified by Wei discloses all the features with respect to claim 1 as outlined above. However, Lin as modified by Wei fails to disclose a dense map of coordinate disparity information indicative of correspondence between maps.
FARÅS discloses a dense map of coordinate disparity information indicative of correspondence between maps (paragraph [0031]: Respective depths of portions of the subject 150 in the image can be computed based on the disparities, creating a dense map of the subject 150 from multiple depth maps).
Therefore, it would have been obvious before the effective filing date of the claimed invention to combine Lin and Wei’s to determine dense map as taught by FARÅS, to generate accurate and realistic images.
Response to Arguments
Applicant's arguments filed 12/17/2025, page 9 - 10, with respect to the rejection(s) of claim(s) 1 and 18 under 103, have been fully considered and are moot upon a new ground(s) of rejection made under 35 U.S.C. 103 as being unpatentable over Lin U.S. Patent Application 20220398747 in view of Wei U.S. Patent Application 20180268208, as outlined above.
Applicant argues on page 9 that "flow search area" of Lin does not combine with Wang (or any other cited reference) to disclose or make obvious "a flow query" or "a single flow query" as recited by amended claim 1.
In reply, the rejection is based on Lin and Wei combined. Lin discloses performing, for an iteration of the features, a single flow query for each respective patch of the plurality of patches within the first feature map, wherein a number of flow queries performed is not a function of the virtual expanded range and is not a function of a number of patches included in the plurality of neighboring patches (paragraph [0082]: each feature of the source feature map, such as feature map F0, can have a corresponding flow search area. In such implementations, if the source feature map has N features, then correlation volume 432 can include N flow search areas; paragraph [0102]: the optical flow computation engine 510 can take the most recent pixel-level (not a function of the virtual expanded range and not in the plurality of neighboring patches) or patch-level (e.g., for a pixel and associated neighbor pixels) flow (or displacement) estimates to look up for the corresponding correlation metrics along with a suitable neighborhood context (e.g., association with neighboring pixels) in the correlation volume; paragraph [0069]: the optical flow map engine 106 can use incremental optical flow maps to update a cumulative optical flow map between multiple iterations of optical flow estimation between two adjacent frames).
Also, “a single flow query for each respective patch”, “a number of flow queries performed is not a function of a number of patches” sounds contradictory for “the number of flow queries”, “a single flow query for each respective patch” means “the number of flow queries” = “number of patches”, please amend or explain more to clarify.
Wei discloses performing feature propagation (paragraph [0030]: a feature propagation function is defined as...; paragraph [0048]: The method 300 then propagates 314 each of the at least one feature maps based on the flow field to approximate current locations of features (flow search area) identified within each of the at least one feature maps).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Yi Yang whose telephone number is (571)272-9589. The examiner can normally be reached on Monday-Friday 9:00 AM-6:00 PM EST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/YI YANG/
Primary Examiner, Art Unit 2616