CTNF 18/771,669 CTNF 99370 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 102 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-15 AIA Claim (s) 1-4, 10, 13-14, and 16-20 are rejected under 35 U.S.C. 102( a)(1 ) as being anticipated by Ilg et al. (NPL, “Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow”, published 2018) . Regarding claim 1 , Ilg teaches an apparatus for processing image data, the apparatus comprising: one or more memories; and one or more processors coupled to one or more memories and configured to: predict, using a machine-learning model, an optical flow between a first image and a second image to generate a predicted optical flow, wherein the first image is captured at a first time and wherein the second image is captured at a second time; determine a confidence map based on the predicted optical flow; determine an error based on the confidence map and a comparison of the predicted optical flow and a ground-truth optical flow (Fig. 1, reprinted below, shows optical flow being predicted and an uncertainty map being determined, PNG media_image1.png 250 697 media_image1.png Greyscale ); and adjust at least one parameter of the machine-learning model based on the error (Pg. 6, Sec. 3.2, “We obtain a probabilistic version of FlowNet with outputs au, av, bu, bv by minimizing the negative log-likelihood of Eq. 7”, the network is trained to minimize the loss function using predicted optical flow uncertainty estimates, necessarily adjusting parameters to optimize the model ). Regarding claim 2 , Ilg teaches all of the elements of claim 1, as stated above, as well as wherein the confidence map is a learning-difficulty-balance confidence map (Fig. 6, reprinted below, shows the prediction being compared to the ground-truth to determine an error, PNG media_image2.png 401 701 media_image2.png Greyscale ). Regarding claim 3 , Ilg teaches all of the elements of claim 1, as stated above, as well as wherein the one or more processors are configured to determine the confidence map based on a comparison of the predicted optical flow with the ground-truth optical flow ( See analysis of claim 2 above ). Regarding claim 4 , Ilg teaches all of the elements of claim 3, as stated above, as well as wherein: a large difference between a first pixel of the predicted optical flow and a corresponding pixel of the ground-truth optical flow relates to a low value in the confidence map; and a small difference between a second pixel of the predicted optical flow and a corresponding pixel of the ground-truth optical flow relates to a high value in the confidence map (Fig. 1, reprinted above, Pg. 19, Fig. 1(a), reprinted below, shows that large pixel differences are given a high uncertainty (low confidence) and small pixel differences are given a low uncertainty, PNG media_image3.png 439 700 media_image3.png Greyscale ). Regarding claim 10 , Ilg teaches all of the elements of claim 1, as stated above, as well as wherein the error is determined based on the confidence map to increase error values for high-confidence pixels and to decrease error values for low-confidence pixels (Pg. 6, Sec. 3.2, Eq. 8, “As an uncertainty estimate we use the variance of the predictive distribution, which is σ2 = 2b2 in this case.”; Pg. 12, Sec. 5.3, “This is because when training against a predictive loss function, the network has the possibility to explain outliers with the uncertainty. This is known as loss attenuation. While the EPE loss tries to enforce correct solutions also for outliers, the log-likelihood loss attenuates them. The experiments confirm this effect and show that it is advantageous to let a network estimate its own uncertainty”). Regarding claim 13 , Ilg teaches all of the elements of claim 1, as stated above, as well as wherein the first image and the second image are captured by a camera (Fig. 1, reprinted above, shows the inputted image ). Regarding claim 14 , Ilg teaches all of the elements of claim 13, as stated above, as well as further comprising the camera ( A camera is necessarily used to capture the image data ). Regarding claim 16 , Ilg teaches all of the elements of claim 1, as stated above, as well as wherein the one or more processors are configured to provide at least one image based on an optical-flow prediction from the machine-learning model to a display to be displayed (Pg. 25, Fig. 7, multiple output images are showcased, as well as a video demonstrating the output. These are necessarily capable of being displayed ). Claim 17 corresponds to claim 1 and is rejected under the same analysis. Claim 18 corresponds to claim 2 and is rejected under the same analysis. Claim 19 corresponds to claim 3 and is rejected under the same analysis. Claim 20 corresponds to claim 4 and is rejected under the same analysis . Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Ilg . Regarding claim 15 , Ilg teaches all of the elements of claim 1, as stated above, as well as wherein the one or more processors are configured to adjust the at least one parameter on the apparatus ( See analysis of claim 1 ). They do not explicitly disclose to adjust the parameter in an online training process. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ilg to include adjusting the parameter in an online training process. Ilg discloses a method for performing optical flow uncertainty estimation as well as adjusting parameters to optimize the model. One of ordinary skill in the art would have understood that performing this parameter adjustment in an online training process was a predictable variation of known techniques. Performing training online was well-known in the art and would have been obvious to incorporate . 07-21-aia AIA Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Ilg in view of Lin et al. (NPL, “Focal Loss for Dense Object Detection”, published 2017) . Regarding claim 5 , Ilg teaches all of the elements of claim 1, as stated above, as well as determining a confidence map. They do not explicitly disclose wherein the error is determined based on an inverse of the confidence map to increase error values for low-confidence pixels and to decrease error values for high-confidence pixels. Lin teaches wherein the error is determined based on an inverse of the confidence map to increase error values for low-confidence pixels and to decrease error values for high-confidence pixels (Pg. 3, Col. 2, “Easily classified negatives comprise the majority of the loss and dominate the gradient. While α balances the importance of positive/negative examples, it does not differentiate between easy/hard examples. Instead, we propose to reshape the loss function to down-weight easy examples and thus focus training on hard negatives. More formally, we propose to add a modulating factor (1 − pt)γ to the cross entropy loss… Intuitively, the modulating factor reduces the loss contribution from easy examples and extends the range in which an example receives low loss… This in turn increases the importance of correcting misclassified examples”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ilg to incorporate the teachings of Lin to include wherein the error is determined based on an inverse of the confidence map to increase error values for low-confidence pixels and to decrease error values for high-confidence pixels. Ilg teaches using a confidence map to determine a training error for adjusting the model, however they do not teach to use the inverse of the confidence map. Lin teaches that scaling the training loss inversely with confidence causes the model to focus its learning on difficult predictions, improving performance by increasing the importance of correcting misclassified examples. One of ordinary skill in the art would have understood that modifying Ilg’s confidence weighted loss to incorporate Lin’s inverse confidence weighing to focus on training hard, low-confidence pixels, predictably improves accuracy and speed of detection ( Lin ; Pg. 8, Col. 2, Sec. 6). As Ilg already provides a confidence map, it would have been within ordinary skill to inverse the choice of weighing direction . 07-21-aia AIA Claim s 6-9 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Ilg in view of Meister et al. (NPL, “UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss”, published 2017) . Regarding claim 6 , Ilg teaches all of the elements of claim 1, as stated above, as well as determining an uncertainty map. They do not explicitly disclose that the confidence map is an occlusion-based confidence map. Meister teaches wherein the map is an occlusion-based map (Fig. 2, “We use forward-backward consistency based on warping the flow fields for estimating occlusion maps, which mask the differences in the data loss.”, Fig. 3). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ilg to incorporate the teachings of Meister to include wherein the confidence map is an occlusion-based confidence map. Ilg teaches a method for optical flow uncertainty prediction using a confidence map, however they do not specify any occlusion handling. Meister teaches to identify an occlusion-based map for optical flow processing. One of ordinary skill in the art would have recognized that supplementing the method of Ilg with the occlusion processing of Meister would predictably improve the identification of unreliable optical flow regions. Regarding claim 7 , Ilg teaches all of the elements of claim 1, as stated above, and when modified in view of Meister teaches wherein the one or more processors are configured to determine the confidence map based on a forward optical flow between the first image and the second image and backward optical flow between the second image and the first image ( Meister ; Fig. 2, reprinted below, shows the usage of forward and backward flow to determine a confidence map, PNG media_image4.png 329 860 media_image4.png Greyscale ). Regarding claim 8 , Ilg as modified in view of Meister teaches all of the elements of claim 7, as stated above, as well as wherein the one or more processors are configured to: predict, using the machine-learning model, the forward optical flow between the first image and the second image; and predict, using the machine-learning model, the backward optical flow between the second image and the first image ( Meister ; Fig. 2, reprinted above). Regarding claim 9 , Ilg as modified in view of Meister teaches all of the elements of claim 7, as stated above, as well as wherein: a small difference between a first forward vector between the first image and the second image and an inverse of a first backward vector between the second image and the first image relates to a high value in the confidence map; and a large difference between a second forward vector between the first image and the second image and the inverse of a second backward vector between the second image and the first image relates to a low value in the confidence map ( Meister ; Pg. 3, Col. 1, “That is, for non-occluded pixels, the forward flow should be the inverse of the backward flow at the corresponding pixel in the second frame. We mark pixels as becoming occluded whenever the mismatch between these two flows is too large.”, Fig. 3, when the mismatch (difference) between the two flows (vectors) is large, pixels are marked as occluded (low confidence value) ). Regarding claim 11 , Ilg teaches all of the elements of claim 1, as stated above, and when modified in view of Meister teaches wherein, to determine the confidence map, the one or more processors are configured to: determine a learning-difficulty-balance confidence map based on a comparison of the predicted optical flow with the ground-truth optical flow ( See analysis of claim 2 above ); and determine an occlusion-based confidence map based on the first image and the second image ( See analysis of claim 6 above ); wherein the error is determined based on the learning-difficulty-balance confidence map and the occlusion-based confidence map ( Meister ; Pg. 3, Col. 1, “We mark pixels as becoming occluded whenever the mismatch between these two flows is too large. Thus, for occlusion in the forward direction, we define the occlusion flag of ofx to be 1 whenever the constraint (Eq 1) is violated, and 0 otherwise. For the backward direction, we define obx in the same way with wf and wb exchanged.”; Pg. 2, Col. 2, “Optionally, we can also use a supervised loss for fine-tuning our networks on sparse ground truth data after unsupervised training”; Ilg ; Pg. 6, Sec. 3.2, “We obtain a probabilistic version of FlowNet with outputs au, av, bu, bv by minimizing the negative log-likelihood of Eq. 7: (Eq. 8)… As an uncertainty estimate we use the variance of the predictive distribution, which is σ2 = 2b2 in this case.”; Pg. 12, “This is because when training against a predictive loss function, the network has the possibility to explain outliers with the uncertainty. This is known as loss attenuation”, Meister teaches to perform occlusion processing using an occlusion map within the loss function (Eq. 2, PNG media_image5.png 138 384 media_image5.png Greyscale ), while Ilg teaches to dynamically attenuate the uncertainty map based on comparison with the ground truth. One of ordinary skill in the art would have understood that supplementing the confidence map of Ilg with the occlusion map processing of Meister would predictably improve the accuracy of predicted optical flow in challenging environments. Training efficiency would also improve as computational resources would not be wasted on pixel regions that are impossible to track (occluded) ) . 07-21-aia AIA Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Ilg in view of Lin further in view of Meister . Regarding claim 12 , Ilg as modified in view of Meister teaches all of the elements of claim 11, as stated above, and when further modified in view of Lin teaches wherein: the error is determined based on an inverse of the learning-difficulty-balance confidence map to increase error values for low-confidence pixels of the learning-difficulty-balance confidence map and to decrease error values for high-confidence pixels of the learning-difficulty-balance confidence map ( See analysis of claim 5 above ); and the error is determined based on the occlusion-based confidence map to increase error values for high-confidence pixels of the occlusion-based confidence map and to decrease error values for low-confidence pixels of the occlusion-based confidence map ( See analysis of claim 9, Meister ; Fig. 3, shows the error maps, where low-confidence occluded pixels have a low error and high-confidence non-occluded pixels have a high error ). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ilg to incorporate the teachings of Meister and Lin to include an error determination based on both an inverse of the confidence map and an occlusion-based map. Ilg teaches a confidence map based on predicted optical flow. Meister teaches an occlusion-based map to improve the training loss in instances where occlusion occur. Neither Ilg nor Meister teach using the inverse of the confidence map to increase error for low-confidence pixels. Lin teaches that scaling a training loss inversely with confidence causes the model to focus training on difficult predictions. One of ordinary skill in the art would have understood that applying Lin’s inverse weighing to the confidence map in the modified system of Ilg and Meister would improve the speed and accuracy of detection, as disclosed by Lin. Conclusion Pertinent Prior Art: Jeong, J., Cai, H., Garrepalli, R., & Porikli, F. (2023). DistractFlow: Improving Optical Flow Estimation via Realistic Distractions and Pseudo-Labeling. ArXiv.org. https://arxiv.org/abs/2303.14078 Common inventors and assignee. Similar architecture before filing date. Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID A WAMBST whose telephone number is (703)756-1750. The examiner can normally be reached M-F 9-6:30 EST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Gregory Morse can be reached at (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /DAVID ALEXANDER WAMBST/Examiner, Art Unit 2663 /GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698 Application/Control Number: 18/771,669 Page 2 Art Unit: 2663 Application/Control Number: 18/771,669 Page 3 Art Unit: 2663 Application/Control Number: 18/771,669 Page 4 Art Unit: 2663 Application/Control Number: 18/771,669 Page 5 Art Unit: 2663 Application/Control Number: 18/771,669 Page 6 Art Unit: 2663 Application/Control Number: 18/771,669 Page 7 Art Unit: 2663 Application/Control Number: 18/771,669 Page 8 Art Unit: 2663 Application/Control Number: 18/771,669 Page 9 Art Unit: 2663 Application/Control Number: 18/771,669 Page 10 Art Unit: 2663 Application/Control Number: 18/771,669 Page 11 Art Unit: 2663
Prosecution Insights