DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013 is being examined under the first inventor to file provisions of the AIA .
Status of the Application
This action is a first action on the merits in response to the application filed on 02/20/2024.
Status of Claims
Claims 1-20 filed on 02/20/2024 are currently pending and have been examined in this application.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/20/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections – 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-6, 9-13, and 16-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. Specifically, claims 1-6, 9-13, and 16-20 are directed to an abstract idea without additional elements to integrate the claims into a practical application or to amount to significantly more than the abstract idea.
Claims 1-6, 9-13, and 16-20 are directed to a process, machine, or manufacture (Step 1), however the claims are directed to the abstract idea of Mental Process or Mathematical Concepts.
With respect to Step 2A Prong One of the frameworks, claim 1 recites an abstract idea. Claim 1 includes limitations for “process at least one input with an efficient neural network; determine at least one performance criteria for the efficient neural network; and activate online learning for the efficient neural network based, at least partially, on the at least one performance criteria”
The limitations above recite an abstract idea under Step 2A Prong One. More particularly, the limitations above recite Mental Process or Mathematical Concepts because processing an input with a neural network, determining criteria, and activating learning are characterized as manipulating data and performing mathematical operations. As a result, claim 1 recites an abstract idea under Step 2A Prong One.
Claims 10 and 17 recite substantially similar limitations to those presented with respect to claim 1. As a result, claims 10 and 17 recite an abstract idea under Step 2A Prong One for the same reasons as stated above with respect to claim 1. Similarly, claims 2-6, 9, 11-13, 16, and 18-20 recite a Mental Process or Mathematical Concepts because the claimed elements describe a process for analyzing data and performing math. As a result, claims 2-6, 9, 11-13, 16, and 18-20 recite an abstract idea under Step 2A Prong One.
With respect to Step 2A Prong Two of the framework, claim 1 does not include additional elements that integrate the abstract idea into a practical application. Claim 1 includes additional elements that do not recite an abstract idea. The additional elements of claim 1 include “An apparatus comprising: at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to”. When considered in view of the claim as a whole, the recited computer elements do not integrate the abstract idea into a practical application because the computer elements are generic computer elements that are merely used as a tool to perform the recited abstract idea. As set forth in the 2019 Eligibility Guidance, 84 Fed. Reg. at 55 “merely include[ing] instructions to implement an abstract idea on a computer” is an example of when an abstract idea has not been integrated into a practical application. Therefore, the claim is directed to an abstract idea.
As a result, claim 1 does not include additional elements that integrate the abstract idea into a practical application under Step 2A Prong Two.
As noted above, claims 10 and 17 recite substantially similar limitations to those recited with respect to claim 1. Although claim 10 further recites “A method” and claim 17 further recites “non-transitory memory”, when considered in view of the claim as a whole, the recited computer elements do not integrate the abstract idea into a practical application because the computer elements are generic computer elements that are merely used as a tool to perform the recited abstract idea. As a result, claims 10 and 17 do not include additional elements that integrate the abstract idea into a practical application under Step 2A Prong Two.
Claims 2-6, 9, 11-13, 16, and 18-20 do not include any additional elements beyond those recited by independent claims 1, 10, and 17. As a result, claims 2-6, 9, 11-13, 16, and 18-20do not include additional elements that integrate the abstract idea into a practical application under Step 2A Prong Two.
With respect to Step 2B of the framework, claim 1 does not include additional elements amounting to significantly more than the abstract idea. As noted above, claim 1 includes additional elements that do not recite an abstract idea. The additional elements of claim 1 include “An apparatus comprising: at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to”. The recited computer elements do not amount to significantly more than the abstract idea because the computer elements are generic computer elements that are merely used as a tool to perform the recited abstract idea. As a result, claim 1 does not include additional elements that amount to significantly more than the abstract idea under Step 2B.
As noted above, claims 10 and 17 recite substantially similar limitations to those recited with respect to claim 1. Although claim 10 further recites “A method” and claim 17 further recites “non-transitory memory”, the recited computer elements do not amount to significantly more than the abstract idea because the computer elements are generic computer elements that are merely used as a tool to perform the recited abstract idea. Further, looking at the additional elements as an ordered combination adds nothing that is not already present when considering the additional elements individually. As a result, claims 10 and 17 do not include additional elements that amount to significantly more than the abstract idea under Step 2B.
Claims 2-6, 9, 11-13, 16, and 18-20do not include any additional elements beyond those recited by independent claims 1, 10, and 17. As a result, claims 2-6, 9, 11-13, 16, and 18-20do not include additional elements that amount to significantly more than the abstract idea under Step 2B.
Therefore, the claims are directed to an abstract idea without additional elements amounting to significantly more than the abstract idea. Accordingly, claims 1-6, 9-13, and 16-20 are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless – (a) (1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-7, 9-14, and 16-20 are rejected under 35 U.S.C. 102 (a) (1) as being anticipated by Zhang, Yizhe et al., "AuxAdapt: Stable and Efficient Test-Time Adaptation for Temporally Consistent Video-Semantic Segmentation", In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, January 2022
Regarding claim 1. Zhang teaches An apparatus comprising: at least one processor; and at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: [Zhang, introduction, Zhang teaches “Recent years have witnessed remarkable progress in image-based semantic segmentation. With the rising popularity and pervasiveness of videos, there is now an increasing focus on video segmentation as a necessary functionality for higher-level computer vision tasks” wherein computer implemented image semantic segmentation] process at least one input with an efficient neural network; determine at least one performance criteria for the efficient neural network; [Zhang, para. 4.1 and Table 2, Zhang teaches “We run extensive performance evaluations of our method on Cityscapes (CS)”. Further, para. 4.3 teaches “However, stopping too early leads to a large performance gap between the original and new models” wherein model performance criteria] and activate online learning for the efficient neural network based, at least partially, on the at least one performance criteria [Zhang, para. 5. conclusion, Zhang teaches “In this paper, we proposed a novel, unsupervised online adaptation method, AuxAdapt, for improving temporal consistency of video semantic segmentation in test time” wherein computer implemented image semantic segmentation].
Regarding claim 2. wherein the at least one performance criteria comprise a temporal consistency criterion [Zhang, Abstract, Zhang teaches “This paper presents an efficient, intuitive, and unsupervised online adaptation method, AuxAdapt, for improving the temporal consistency of most neural network models”].
Regarding claim 3. wherein the online learning is activated in response to the temporal consistency criteria decreasing by a threshold amount during a time period in comparison to an average observed temporal consistency criteria [Zhang, page 2340 second column, first para., Zhang teaches “Moreover, DVP relies on early stopping to a trade-off between temporal consistency and accuracy on the test video, but does not provide a clear criterion for early stopping in test time…We leverage on a simple change-detection-based adaptive momentum when performing the online adaptation, which adjusts the momentum coefficient based on the difference of two consecutive frames. We show that this provides a good balance between temporal consistency and segmentation accuracy” wherein the difference of two consecutive frames is equivalent to a threshold amount during a time period in comparison to an average observed temporal consistency criteria].
Regarding claim 4. wherein processing the at least one input with the efficient neural network comprises the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus to: perform semantic segmentation of the at least one input, wherein the at least one input comprises at least one video frame [Zhang, introduction, Zhang teaches “In other words, image pixels across consecutive video frames that belong to the same semantic class and share similar visual appearances can be labeled differently, resulting in artifacts such as flickering of segmentation. See examples in the 1st and 3rd columns of Fig. 1”].
Regarding claim 5. wherein the online learning is continually activated while at least one of the at least one performance criteria for the efficient neural network is below a threshold value [Zhang, para. 3.3, Zhang teaches “When the network is already highly confident in its prediction on a pixel, the loss (i.e., LCE for pixel (i, j)) for this spatial location will be very small. Such loss terms will not provide meaningful contributions to updating the model and will instead incur unnecessary computation. As such, we can remove such redundant spatial locations from the overall loss computation by setting a confidence threshold.” Wherein confidence threshold and contentious test time adaptation loop that runs whenever error or consistency metrics cross acceptable operation limits].
Regarding claim 6. wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus to: deactivate the online learning after a predefined period [Zhang, page 2340 column 1 second para., Zhang teaches “DVP trains a network from scratch to generate these processed frames and employs early stopping to prevent the network from overfitting to temporally-inconsistent patterns” Wherein adaptation time limits].
Regarding claim 7. wherein activating the online learning for the efficient neural network comprises the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus to: provide, to a server, one of: the at least one input, or at least one feature of the at least one input; save locally a copy of the one of the at least one input or the at least one feature; receive, from the server, at least one inference result with respect to the one of the at least one input or the at least one feature; and retrain the efficient neural network based on the at least one inference result [Zhang, page 2341 first para. Figure 2, Zhang teaches “Figure 2: Left: Deep Video Prior (DVP) [25] first applies the original network to all video frames. The outputs, together with the corresponding inputs, are collected to form a training set. A new network is then trained based on this set using at least 25 epochs. Finally, the retrained network is applied to the same video to obtain the final outputs. This means DVP applies twice inference over the entire set of frames and a full-scale, computationally expensive retraining” Wherein The outputs, together with the corresponding inputs, are collected to form a training set is equivalent to save locally a copy of the one of the at least one input or the at least one feature and “This means DVP applies twice inference over the entire set of frames and a full-scale, computationally expensive retraining” is equivalent retrain the efficient neural network based on the at least one inference result].
Regarding claim 9. wherein processing the at least one input with the efficient neural network comprises the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus to: process the at least one input with a frozen main neural network; and process the at least one input with an auxiliary neural network [Zhang, page 2340 column 1 last para., Zhang teaches “MainNet is frozen, and only AuxNet is updated as the integrated model streams through the video”].
Regarding claim 10, the claim recites analogous limitations to claim 1 above, and is therefore rejected on the same premise. Claim 1 is an Apparatus claim while claim 10 is directed to a method which is anticipated by Zhang introduction.
Regarding claims 11-14, and 16, claims 11-14, and 16` recite substantially similar limitations as claim 2-4, 7, and 9, respectively; therefore, claims 11-14, and 16 are rejected with the same rationale, reasoning, and motivation provided above for claims 2-4, 7, and 9, respectively. Claims 2-4, 7, and 9 are Apparatus claims while claims 11-14, and 16 are directed to a method which is anticipated by Zhang introduction.
Regarding claim 17, the claim recites analogous limitations to claim 1 above, and is therefore rejected on the same premise.
Regarding claim 18. wherein the at least one inference result is determined with a generic neural network [Zhang, Abstract, Zhang teaches “This paper presents an efficient, intuitive, and unsupervised online adaptation method, AuxAdapt, for improving the temporal consistency of most neural network models” wherein “most neural network models” is an indication of generic NN].
Regarding claim 19. wherein the apparatus comprises a server [Zhang, Introduction, Zhang teaches “With the rising popularity and pervasiveness of videos, there is now an increasing focus on video segmentation as a necessary functionality for higher-level computer vision tasks” wherein video segmentation requires a server].
Regarding claim 20. wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus to: train the efficient neural network [Zhang, Introduction, Zhang teaches “This paper presents an efficient, intuitive, and unsupervised online adaptation method, AuxAdapt, for improving the temporal consistency of most neural network models” wherein efficient NN].
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claims 8 and 15 are rejected under 35 U.S.C. 103 as being un-patentable over Zhang in view Park, Hyojin et al., "Real-Time, Accurate, and Consistent Video Semantic Segmentation via Unsupervised Adaptation and Cross-Unit Deployment on Mobile Device", IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), October 2021
Regarding claim 8. Zhang teaches all of the limitations of claim 1 (as above). Zhang does not specifically teach, however, Park teaches wherein activating the online learning for the efficient neural network comprises the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus to: provide, to a server, one of: the at least one input, or at least one feature of the at least one input; and receive, from the server, a weight update for the efficient neural network [Park, Abstract, Park teaches “We employ our test-time unsupervised scheme, AuxAdapt, to enable the segmentation model to adapt to a given video in an online manner. More specifically, we leverage a small auxiliary network to perform weight updates and keep the large, main segmentation network frozen” Wherein weight update]
Zhang teaches AuxAdapt: Stable and Efficient Test-Time Adaptation for Temporally Consistent Video Semantic Segmentation and Park teaches Real-Time, Accurate, and Consistent Video Semantic Segmentation via Unsupervised Adaptation and Cross-Unit Deployment on Mobile Device. The two references are in the same field of endeavor as the claimed invention of managing model adaptation for video frames. It would have been obvious for one having ordinary skill in the art before the effective filing date of the claimed invention to modify/combine utilizing the adaptation for video semantic segmentation of Zhang with the weight update of Park since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, with the predictable results of optimizing AI model adaptation.
Regarding claim 15, the claim recites analogous limitations to claim 8 above, and is therefore rejected on the same premise. Claim 8 is an Apparatus claim while claim 15 is directed to a method which is anticipated by Zhang introduction.
Conclusion
The following prior arts made of record and not relied upon are considered pertinent to applicant's disclosure. Sadek Sadek et al. (US 20170345163 A1) teaches system and computer-implemented method for editing a video sequence with temporal consistency
Berlin et al. (US 11308657 B1) teaches accessing a plurality of images of a given face; determining if the plurality of images of the given face satisfy at least a first temporal consistency criterion; and at least partly in response to determining that the plurality of images of the given face satisfy at least the first temporal consistency criterion, include the given image in a training dataset of images, configured to be used to train at least an autoencoder encoder
Reda Fitsum et al. (DE 112020003165 T5) teaches systems and techniques for enhancing a video. In at least one embodiment, one or more neural networks are used to create a second video from a first video with a higher frame rate, higher resolution, or reduced number of missing or corrupted video frames.
Any inquiry concerning this communication from the examiner should be directed to Abdallah El-Hagehassan whose contact information is (571) 272-0819 and Abdallah.el-hagehassan@uspto.gov The examiner can normally be reached on Monday- Friday 8 am to 5 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rutao Wu can be reached on (571) 272-6045. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-3734.
Information regarding the status of an application may be obtained from the patent application information retrieval (PAIR) system. Status information of published applications may be obtained from either private PAIR or public PAIR. Status information of unpublished applications is available through private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have any questions on access to the private PAIR system, contact the electronic business center (EBC) at (866) 271-9197 (toll-free). If you would like assistance from a USPTO customer service representative or access to the automated information system, call (800) 786-9199 (in US or Canada) or (571) 272-1000.
/ABDALLAH A EL-HAGE HASSAN/
Primary Examiner, Art Unit 3623
Prosecution Insights