DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception without significantly more.
Regarding Claim 1:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
comparing the first data sample to a collection of models to select one of the models that is determined as a best match for the first data sample. This limitation is directed to the abstract idea of a mental process as comparing the data sample to the models, selecting the best model is(are) analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
comparing, via the computer, metrics of data summarization for the first data sample to metrics of data summarization of the models; This limitation is directed to the abstract idea of a mental process as comparing the metrics is(are) analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
comparing, via the computer, a neural network metric for the first data sample to respective neural network metrics for the models. This limitation is directed to the abstract idea of a mental process as comparing the metrics is(are) analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
receiving, via a computer, a first data sample; This limitation recites importing data, which is considered an insignificant extra solution activity, as it is merely importing data which is a conventional computer function, therefore, it does not impose meaningful limits on the claim such that it is not nominally or tangentially related to the invention per 2106.05(g).
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
receiving, via a computer, a first data sample; This limitation recites importing data amounts to storing and retrieving information in memory, further considered well-understood, routine and conventional under MPEP 2106.05(d) II (iv).
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 2:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
comparing the embedding vector to respective embedding vectors for each of the collection of models. This limitation is directed to the abstract idea of a mental process as comparing a vector to other vectors is(are) analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
inputting the first data sample into an embedding model and, in response, receiving an embedding vector from the embedding model. This limitation recites importing data, which is considered an insignificant extra solution activity, as it is merely importing data which is a conventional computer function, therefore, it does not impose meaningful limits on the claim such that it is not nominally or tangentially related to the invention per 2106.05(g).
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
inputting the first data sample into an embedding model and, in response, receiving an embedding vector from the embedding model. This limitation is directed to inputting data into a network and receiving a vector, which the courts have recognized as well-understood, routine, conventional activity when they are claimed at a high level of generality or as insignificant extra-solution activity [see MPEP 2106.05(d) II. i] and therefore fails to amount to significantly more than the judicial exception.
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 3:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
comparing the embedding vector to respective embedding vectors for each of the collection of models. This limitation is directed to the abstract idea of a mental process as comparing a vector to other vectors is(are) analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
the neural network metric for the first data sample is extracted from a first neural network in response to inputting the first data sample into the first neural network. The claim recites the metric is extracted from a neural network in response to inputting the data sample into the network. However, the limitation merely provides instructions to apply mathematical calculations or training process, and therefore does not integrate the judicial exception into a practical application. The claim does not improve the functioning of a computer or another technology (MPEP 2106.05(f)).
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
the neural network metric for the first data sample is extracted from a first neural network in response to inputting the first data sample into the first neural network. This limitation is directed to inputting data into a network and receiving a metric, which the courts have recognized as well-understood, routine, conventional activity when they are claimed at a high level of generality or as insignificant extra-solution activity [see MPEP 2106.05(d) II. i] and therefore fails to amount to significantly more than the judicial exception.
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 4:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
the neural network metric is a logit layer energy score from the first neural network. This limitation is directed to the abstract idea of a mathematical concepts, as calculating a score is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.)
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
the neural network metric is a logit layer energy score from the first neural network. The claim recites using neural network and calculating data. However, using neural network merely provides instructions to apply the mathematical concept and therefore does not integrate the judicial exception into a practical application. The claim does not improve the functioning of a computer or another technology (MPEP 2106.05(f)).
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
the neural network metric is a logit layer energy score from the first neural network. The additional elements, neural network and calculating/training data, do not amount to significantly more than the abstract idea. A neural network, merely provides instructions to apply the mathematical concept. The claim does not improve the functioning of a computer or another technology (MPEP 2016.04(d)(1)).
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 5:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Claim 5 does not recite abstract ideas other than the ones recited at claim 1.
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
at least one of the metrics of data summarization of the models and the respective neural network metrics for the models is obtained in response to inputting training data into the respective model. This limitation recites as an insignificant extra solution activity, as inputting the data into a model and obtain a metric, under BRI, is mere data gathering per MPEP 2106.05(g)(3).
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
at least one of the metrics of data summarization of the models and the respective neural network metrics for the models is obtained in response to inputting training data into the respective model. This limitation recites obtaining data, which is considered an insignificant extra solution activity, as it is merely obtaining data which is a conventional computer function, therefore, it does not impose meaningful limits on the claim such that it is not nominally or tangentially related to the invention per 2106.05(g).
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 6:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
for the comparing of the first data sample to the collection of models the comparing of the metrics of the data summarization is given a first weight and the comparing of the neural network metric is given a second weight. This limitation is directed to the abstract idea of a mathematical concepts, as giving a weight is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.)
Regarding Claim 7:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
the comparing of the first data sample to the collection of models is performed in response to a first computer analysis indicating an out-of-distribution determination for the first data sample. This limitation is directed to the abstract idea of a mental process as indicating an out-of-distribution sample determination is(are) analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Regarding Claim 8:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Claim 8 does not recite abstract ideas other than the ones recited at claim 1.
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
in response to implementing the collection of models in a new environment. This limitation recites implementing the models in a new environment. Therefore, this limitation amounts to merely indicating a field of use or technological environment [see MPEP 2106.05(h)] and fails to integrate the judicial exception into a practical application.
in response to receiving the first data sample. This limitation is directed to mere data gathering, which is an insignificant extra-solution activity [see MPEP 2106.05(g)(3)] and therefore fails to integrate the judicial exception into a practical application.
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
in response to implementing the collection of models in a new environment. This limitation recites implementing the models in a new environment. Therefore, this limitation amounts to merely indicating a field of use or technological environment [see MPEP 2106.05(h)] and therefore fails to amount to significantly more than the judicial exception.
in response to receiving the first data sample. This limitation is directed to receiving or transmitting data over a network, which the courts have recognized as well-understood, routine, conventional activity when they are claimed at a high level of generality or as insignificant extra-solution activity [see MPEP 2106.05(d) II. i] and therefore fails to amount to significantly more than the judicial exception.
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 9:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
the comparing of the first data sample to the collection of models further comprises performing, via the computer, natural language processing to compare a text description of the first data sample to text descriptions associated with the models. This limitation is directed to the abstract idea of a mental process as comparing text descriptions is(are) analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
the comparing of the first data sample to the collection of models further comprises performing, via the computer, natural language processing to compare a text description of the first data sample to text descriptions associated with the models. This limitation recites generic computer components such as processor, which invokes a system merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to integrate the exception into a practical application
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
the comparing of the first data sample to the collection of models further comprises performing, via the computer, natural language processing to compare a text description of the first data sample to text descriptions associated with the models. This limitation invokes a system merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to amount to significantly more than the judicial exception.
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 10:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
the first data sample is a first image and the text description of the first data sample is produced via inputting the first image into an image-to-text model and, in response, receiving the text description as output from the image-to-text model. This limitation is directed to the abstract idea of a mental process as giving a text description from an image is(are) analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
the first data sample is a first image and the text description of the first data sample is produced via inputting the first image into an image-to-text model and, in response, receiving the text description as output from the image-to-text model. This limitation recites generic compute, which invokes a computer merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to integrate the exception into a practical application.
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
the first data sample is a first image and the text description of the first data sample is produced via inputting the first image into an image-to-text model and, in response, receiving the text description as output from the image-to-text model. This limitation invokes a computer merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to amount to significantly more than the judicial exception.
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 11:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
the first data sample is a first image and the text description of the first data sample is produced via inputting the first image into a classification machine learning model and, in response, receiving the text description as a class that the classification machine learning model predicts for the first image. This limitation is directed to the abstract idea of a mental process as giving text description as a classification from an image is(are) analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
the first data sample is a first image and the text description of the first data sample is produced via inputting the first image into a classification machine learning model and, in response, receiving the text description as a class that the classification machine learning model predicts for the first image. This limitation recites generic computer, which invokes a computer merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to integrate the exception into a practical application.
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
the first data sample is a first image and the text description of the first data sample is produced via inputting the first image into a classification machine learning model and, in response, receiving the text description as a class that the classification machine learning model predicts for the first image. This limitation invokes a computer merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to amount to significantly more than the judicial exception.
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 12:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Claim 12 does not recite abstract ideas other than the ones recited at claim 1.
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
uploading the first model for usage. This limitation recites uploading a model, which is considered an insignificant extra solution activity, as it is merely uploading data which is a conventional computer function, therefore, it does not impose meaningful limits on the claim such that it is not nominally or tangentially related to the invention per 2106.05(g).
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
uploading the first model for usage. This limitation recites uploading data amounts to storing and retrieving information in memory, further considered well-understood, routine and conventional under MPEP 2106.05(d) II (iv).
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding claims 13 - 18:
Claims 13 - 18 recites analogous limitations to claims 1-6 (respectively) and therefore they are rejected on the same grounds as claims 1-6.
Regarding claims 19 - 20:
Claims 19 - 20 recites analogous limitations to claims 1-2 (respectively) and therefore they are rejected on the same grounds as claims 1-2.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1 – 3, 8, 12 – 15, and 19 - 20 is/are rejected under 35 U.S.C. 102(a) (1) as being anticipated by Khani (NPL, RECL: Responsive Resource-Efficient Continuous Learning for Video Analytics, dated on 04/19/2023, by Khani et al - hereinafter Khani).
Referring to Claim 1, Khani teaches:
receiving, via a computer, a first data sample; See Khani at [Page 5, top - right]:” RECL launches a model adaptation controller on a server machine (e.g., in the cloud, edge compute cluster, etc.), which manages a set of daemons running on edge devices” Also see Khani at [Page 5, mid - right]:” In each model-update window (by default, every 30 seconds),1 each edge device sends sampled frames to the controller to query if a new model should be used.” Khani discloses a model-adaptation controller operating on a server machine. Each edge device sends sampled video frames to the controller for model selection. Accordingly, the controller receives, via the server computer, a sampled video frame corresponding to the claimed first data sample.
comparing the first data sample to a collection of models to select one of the models that is determined as a best match for the first data sample. See Khani at [Page 6, mid - right]:” Figure 5 (right-hand side) describes RECL’s online procedure to select a model from the model zoo… The gating network is a lightweight DNN that given an image, assigns a score to each model in the model zoo…A higher score indicates the model likely has higher accuracy on the image.”
PNG
media_image1.png
389
808
media_image1.png
Greyscale
Khani discloses comparing a sampled frame to a collection of expert models in a model zoo. In particular, a gating network receives an image and assigns a score to each model in the model zoo, where a higher score indicates that the model is likely to achieve higher accuracy for the image. RECL subsequently selects candidate models having the highest average scores and ultimately selects the model having the highest empirical accuracy. Thus, Khani teaches comparing the first data sample to a collection of models to select one of the models that is determined as a best match for the first data sample.
wherein the comparing comprises: comparing, via the computer, metrics of data summarization for the first data sample to metrics of data summarization of the models; See Khani at [Page 10, mid -right]:” Specifically, the average of embedding vectors of the sampled frames in a window is used as the embedding vector of that window. Also, each trained expert is assigned an embedding vector the same as its training data. We use the L2 distance between the embedding vector of a window and the models in the zoo as a measure of similarity, and the model selector returns the model with the least distance from the samples in each window as in the ODIN paper” Examiner interprets the average of embedding vectors of the sampled frames in a window as equivalent as the metrics of data summarization for the first data sample, an embedding vector assigned to each trained expert as equivalent as the metrics of data summarization of the models, and using L2 distance as measure of similarity as equivalent as comparing operation. Thus, Khani teaches the limitation.
comparing, via the computer, a neural network metric for the first data sample to respective neural network metrics for the models. See Khani at [Page 6, mid - right]:” The gating network is a lightweight DNN that given an image, assigns a score to each model in the model zoo. Logically, the gating network is similar to an image classifier, except that the labels are not object classes but models in the model zoo. A higher score indicates the model likely has higher accuracy on the image.” Also, see Khani at [Page 7, top - left]:” Instead, RECL runs the gating network on the edge device’s latest sampled frames and selects the top-K models (e.g., K =10) with the highest average scores.” Under Broadest Reasonable Interpretation (BRI), the score for each model is as equivalent as the network metric as claimed. Khani discloses a gating network implemented as a lightweight DNN. Given an image corresponding to the first data sample, the gating network assigns a respective score to each model in the model zoo. A higher score indicates that the corresponding model is likely to have higher accuracy for the image. RECL compares the scores by selecting the models having the highest average scores and ultimately selecting the model having the highest empirical accuracy since Thus, Khani discloses comparing neural-network-generated metrics for the first data sample with respective metrics assigned to the models.
Referring to Claim 2, Khani teaches:
inputting the first data sample into an embedding model and, in response, receiving an embedding vector from the embedding model; See Khani at [Page 6, bottom - right]:” An alternative approach [23,34] to model selection is to map video content to an embedding space (via an autoencoder), partition the embedding space, and map each partition to a specific expert model.” Khani discloses an autoencoder-based model-selection method in which sampled video frames are mapped into an embedding space through an autoencoder. The autoencoder produces embedding vectors for the sampled frames, and the average of the embedding vectors within a window is used as the embedding vector representing that window. Accordingly, Khani discloses inputting the first data sample into an embedding model and receiving an embedding vector in response.
comparing the embedding vector to respective embedding vectors for each of the collection of models. See Khani at [Page 10, mid - right]:” Also, each trained expert is assigned an embedding vector the same as its training data. We use the L2 distance between the embedding vector of a window and the models in the zoo as a measure of similarity, and the model selector returns the model with the least distance from the samples in each window as in the ODIN paper” Khani discloses that each trained expert model in the model zoo is assigned a respective embedding vector representative of its training data. Khani then calculates the L2 distance between the embedding vector representing the sampled-frame window and the respective embedding vectors assigned to the models in the model zoo. The L2 distance is used as a similarity measure, and the model selector returns the model having the smallest distance from the sampled frames. Therefore, Khani teaches the limitation.
Referring to Claim 3, Khani teaches:
the neural network metric for the first data sample is extracted from a first neural network in response to inputting the first data sample into the first neural network. See Khani at [Page 6, mid - right]:” Figure 5 (right-hand side) describes RECL’s online procedure to select a model from the model zoo… The gating network is a lightweight DNN that given an image, assigns a score to each model in the model zoo…A higher score indicates the model likely has higher accuracy on the image.”
PNG
media_image1.png
389
808
media_image1.png
Greyscale
Khani discloses a model selector implemented as a gating network, which is a lightweight DNN. As shown in Figure 5, the sampled frames are supplied to the gating network. When the gating network is given an image corresponding to the first data sample, it produces a score for each model in the model zoo. The score is therefore a neural-network-generated metric extracted from the gating network in response to inputting the first data sample into the gating network. Thus, Khani teaches the limitation.
Referring to Claim 8, Khani teaches:
the comparing of the first data sample to the collection of models is performed in response to implementing the collection of models in a new environment and in response to receiving the first data sample. See Khani at [Page 6, bottom - left]:” An important design choice of RECL is that rather than caching the history models of different devices separately, RECL shares the model zoo and its gating network across devices, enabling model reuse across similar video sessions of different devices that might share similar temporal-spatial correlations (e.g., in the same geographical vicinity) [33]. This reduces the need for online model retraining and improves system responsiveness when an edge device experiences a sudden scene change for which a previously trained model (probably of another device) with good accuracy is available.” Also, see Khani at [Page 5, mid - right]:” In each model-update window (by default, every 30 seconds), each edge device sends sampled frames to the controller to query if a new model should be used. (Note that the RECL controller only updates models for edge devices, which then use models to run inference on video streams.) … Based on the sampled frames, the controller performs two basic functions—model selection (§3.1), which selects a suitable expert model from a collection of history expert models to quickly respond to the edge device’s query, and model retraining (§3.2), which fine-tunes the selected model based on the sampled frames and manages GPU resources to many edge devices to retrain their models.” Khani discloses implementing a shared model zoo for edge devices operating across different video environments and teaches that the shared model zoo improves system responsiveness when an edge device experiences a sudden scene change. Khani further discloses that the edge device sends sampled frames from its current scene to the controller and that, “based on the sampled frames”, the controller performs model selection from the collection of historical expert models. Accordingly, the previously discussed model-comparison procedure is performed in response to implementing the collection of models for the new environment and in response to receiving the sampled frames as claimed. Thus, Khani teaches the limitation.
Referring to Claim 12, Khani teaches:
in response to selecting a first model of the collection of models via the comparing of the first data sample to the collection of models, uploading the first model for usage. See Khani at [Page 7, mid - left]: “Finally, among these models (top-K from the model zoo, current model, and the last retrained model), RECL selects the one with the highest empirical accuracy on the labeled images and sends it to the device.” Khani discloses that selecting the model with the highest accuracy and sending it to the device, which is as equivalent as selecting a first model and uploading the model. Thus, Khnai teaches the limitation.
Referring to Claims 13 - 15, the claim is(are) rejected on the same basis as claims 1 - 3, mutatis mutandis, since they are analogous claims.
Referring to Claims 19 - 20, the claim is(are) rejected on the same basis as claims 1 - 2, mutatis mutandis, since they are analogous claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claim(s) 4 – 5, 7 and 16 - 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Khani in view of Gholami (NPL, ETran: Energy-Based Transferability Estimation, dated on 08/03/2023, by Gholami et al - hereinafter Gholami).
Referring to Claim 4, Khani teaches the method of claim 1, however, it fails to teach: the neural network metric is a logit layer energy score from the first neural network;
Gholami teaches:
the neural network metric is a logit layer energy score from the first neural network. See Gholami at [Page 3, bottom - right]:” where Φ(y)(x) is the output logits of y-th class. Due to the deep connection between EBMs and discriminative models, we can define the energy for a given data point x as E (x, y) = −Φ(y)(x) by equating the Eq. 4 and 3. We can then compute the free energy E(x) (defined as the negative log of partition function) as follows:”
PNG
media_image2.png
125
574
media_image2.png
Greyscale
Gholami discloses inputting a data point
x
into a pre-trained source neural network
Φ
. The neural network produces respective output logits
Φ
c
x
for its output classes. Gholami calculates a free-energy score for the input data point according to
E
(
x
)
=
-
l
o
g
∑
c
e
Φ
c
(
x
)
, and expressly states that the free energy is calculated using the output logits of
Φ
. Accordingly, Gholami teaches the limitation.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khani with the above teachings of Gholami by receiving the data sample and comparing the data sample to a collection of models to select one determined model as taught by Khani, and the neural network metric is a logit layer energy score from the first neural network, as taught by Gholami. The modification would have been obvious because one of ordinary skill in the art would be motivated to use energy score method better detecting the IND or OOD target dataset for a given pre-trained model. See Gholami at [Page 2, mid - left]:” In this work, we propose an energy-empowered transferability estimation method (called ETran) that exploits energy-based models (EBM) [27] to detect whether a target dataset is IND or OOD for a given pre-trained model. To this end, the higher the energy score for a target dataset, the more IND this dataset is for the pre-trained model [30, 2, 3]. As a consequence, the corresponding model has high likelihood to provide the best accuracy after fine-tuning (for the given target dataset) compared to the models with lower energy scores. In contrast to the previous transferability metrics, the energy score is label- and optimization-free, which makes it highly efficient and easy-to-use.”
Referring to Claim 5, Khani teaches the method of claim 1, however, it fails to teach: at least one of the metrics of data summarization of the models and the respective neural network metrics for the models is obtained in response to inputting training data into the respective model;
Gholami teaches:
at least one of the metrics of data summarization of the models and the respective neural network metrics for the models is obtained in response to inputting training data into the respective model. See Gholami at [Page 3, mid - left]:” The transferability scores in our work are computed over the features extracted from the target dataset by the source models.” Also see Gholami at [Page 4, mid - left]:” Given ˆh as the dimension of features ˆf, we calculate the energy values over features:
PNG
media_image3.png
124
536
media_image3.png
Greyscale
Gholami thereby discloses inputting the target training data into each respective source model and obtaining a model-specific energy score, which is a neural-network metric. Under the BRI of “at least one of A and B”, the limitation requires only at least one of the two recited categories of metrics to be so obtained. Gholami satisfies obtaining the respective neural-network metrics, and teaches the limitation.
The same motivation that was utilized for combining Khani with Gholami as set forth in claim 4 is equally applicable to claim 5.
Referring to Claim 7, Khani teaches the method of claim 1, however, it fails to teach: the comparing of the first data sample to the collection of models is performed in response to a first computer analysis indicating an out-of-distribution determination for the first data sample.
Gholami teaches:
the comparing of the first data sample to the collection of models is performed in response to a first computer analysis indicating an out-of-distribution determination for the first data sample. See Gholami at [Page 3, mid - right]: “Energy Score. Energy-based models (EBM) introduce a function E(x): RD → R that maps input data x to a single, non-probabilistic scalar called energy [27].” Also, See Gholami at [Page 2, top - left]: “Thus, determining whether the target dataset is in-distribution (IND) or out-of- distribution (OOD) is an essential assessment factor in finding the best pre-trained model.” Gholami discloses performing an energy-based computer analysis on input data
x
, wherein the resulting energy value indicates that the input data is an OOD sample for a source model. Gholami further teaches that the OOD/IND determination is an essential assessment factor in finding the best pretrained model. Therefore, the previously discussed comparison of the first data sample to the collection of models is performed based on, and thus in response to, the computer analysis indicating the OOD determination for the first data sample. Thus, Gholami teaches the limitation.
The same motivation that was utilized for combining Khani with Gholami as set forth in claim 4 is equally applicable to claim 7.
Referring to Claims 16 - 17, the claim is rejected on the same basis as claims 4 - 5, mutatis mutandis, since they are analogous claims.
Claim(s) 6 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Khani in view of Tan (NPL, OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations, dated on 03/25/2021, by Tan et al - hereinafter Tan).
Referring to Claim 6, Khani teaches the method of claim 1, however, it fails to teach: for the comparing of the first data sample to the collection of models the comparing of the metrics of the data summarization is given a first weight and the comparing of the neural network metric is given a second weight.
Tan teaches:
for the comparing of the first data sample to the collection of models the comparing of the metrics of the data summarization is given a first weight and the comparing of the neural network metric is given a second weight. See Tan at [Page 5, mid - left]:” Thus the following step is to combine domain difference and task difference to obtain our OTCE score. Step3: Compute OTCE score. Intuitively, we model the OTCE score as a linear combination of domain difference and task difference: OTCE = λ1 WD + λ1 WT + b, (12) where λ1, λ2 are weighting coefficients for standardized domain difference WD and task difference WT respectively, and b is the bias term. Choosing the optimal weights is a challenging task since the importance of WD and WT maybe different for various cross-domain configurations, as described in Section 4.7.” Tan discloses selecting a best pretrained source model for a target task from a set of source models. Tan evaluates domain difference and task difference as separate components of a transferability score and expressly assigns respective weighting coefficients to those components according to: OTCE = λ1 WD + λ1 WT + b Tan further explains that
λ
1
and
λ
2
are weighting coefficients for the standardized domain difference and standardized task difference, respectively, and that the relative importance of the two components may differ for different cross-domain configurations.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khani with the above teachings of Tan by receiving the data sample and comparing the data sample to a collection of models to select one determined model as taught by Khani, and for the comparing of the first data sample to the collection of models the comparing of the metrics of the data summarization is given a first weight and the comparing of the neural network metric is given a second weight, as taught by Tan. The modification would have been obvious because one of ordinary skill in the art would be motivated to improve the accuracy of predicting the transfer performance. See Tan at [Page 2, bottom]:” Extensive experiments on the largest cross-domain dataset DomainNet [28] and Office31 [34] demonstrate that our OTCE score shows significantly higher correlation with transfer accuracy, i.e., predicting the transfer performance more accurately with an average of 21% gain compared to state-of-the-art metrics [24, 38, 5]. In addition, we further investigate two applications of transferability in source model selection and multi-source feature fusion.”
Referring to Claim 18, the claim is rejected on the same basis as claim 6, mutatis mutandis, since they are analogous claims.
Claim(s) 9 - 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Khani in view of Shen (NPL, HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face, dated on 05/25/2023, by Shen et al - hereinafter Shen).
Referring to Claim 9, Khani teaches the method of claim 1, however, it fails to teach: the comparing of the first data sample to the collection of models further comprises performing, via the computer, natural language processing to compare a text description of the first data sample to text descriptions associated with the models.
Shen teaches:
the comparing of the first data sample to the collection of models further comprises performing, via the computer, natural language processing to compare a text description of the first data sample to text descriptions associated with the models. See Shen at [Page 1, abstract]:” Based on this philosophy, we present HuggingGPT, a framework that leverages LLMs (e.g., ChatGPT) to connect various AI models in machine learning communities (e.g., Hugging Face) to solve AI tasks.” Also, see Shen at [Page 13, mid]:” In general, the Hugging Face Hub hosts expert models that come with detailed model descriptions, typically provided by the developers. These descriptions encompass various aspects of the model, such as its function, architecture, supported languages and domains, licensing, and other relevant details. These comprehensive model descriptions play a crucial role in aiding the decision of HuggingGPT. By assessing the user’s requests and comparing them with the model descriptions, HuggingGPT can effectively determine the most suitable model for the given task.” Shen teaches using HuggingGPT, which employs a large language model (LLM) as a computer-implemented natural-language-processing controller, to assess a textual user request and compare it with detailed textual descriptions associated with expert models. Based on this textual comparison, HuggingGPT determines the most suitable model for the task. Therefore, Shen teaches performing natural language processing (NLP) to compare a text description associated with the received data or request to respective text descriptions associated with the collection of models, as claimed.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khani with the above teachings of Shen by receiving the data sample and comparing the data sample to a collection of models to select one determined model as taught by Khani, and performing, via the computer, natural language processing to compare a text description of the first data sample to text descriptions associated with the models, as taught by Shen. The modification would have been obvious because one of ordinary skill in the art would be motivated to integrate multimodal perceptual capabilities and handle multiple complex AI tasks, absorbing the powers from task-specific experts, and enabling growable and scalable AI capabilities. See Shen at [Page 3, bottom]:” Benefiting from such a design, HuggingGPT can automatically generate plans from user requests and use external models, and thus can integrate multimodal perceptual capabilities and handle multiple complex AI tasks. More noteworthy, this pipeline also allows HuggingGPT to continue absorbing the powers from task-specific experts, enabling growable and scalable AI capabilities. Furthermore, we also point out that task planning plays a very important role in HuggingGPT, which directly determines the success of the subsequent workflow. Therefore, how to conduct planning is also a good perspective to reflect the capability of LLMs, which also opens a new door for LLM evaluation.”
Referring to Claim 10, Khani teaches the method of claim 1, however, it fails to teach: the first data sample is a first image and the text description of the first data sample is produced via inputting the first image into an image-to-text model and, in response, receiving the text description as output from the image-to-text model.
Shen teaches:
the first data sample is a first image and the text description of the first data sample is produced via inputting the first image into an image-to-text model and, in response, receiving the text description as output from the image-to-text model. See Shen at [Page 21, Fig. 6]:” For image-to-text, I chose the nlpconnect/vitgpt2- image-captioning model. This model generates an image caption with the given image. I applied this model to the input image and the developed output was: a family of four dogs are playing in the grass.” Shen teaches an image-to-text task in which example.jpg is provided as the image input to the NLP (natural language processing) connect/vit-gpt2-image-captioning model. Shen expressly states that the model generates an image caption from the given image and that, after applying the model to the input image, the resulting output is the textual description “a family of four dogs are playing in the grass.” Therefore, Shen teaches inputting the first image into an image-to-text model and, in response, receiving a text description of the first image as output from the image-to-text model, as claimed.
The same motivation that was utilized for combining Khani with Shen as set forth in claim 9 is equally applicable to claim 10.
Referring to Claim 11, Khani teaches the method of claim 1, however, it fails to teach: the first data sample is a first image and the text description of the first data sample is produced via inputting the first image into a classification machine learning model and, in response, receiving the text description as a class that the classification machine learning model predicts for the first image.
Shen teaches:
the first data sample is a first image and the text description of the first data sample is produced via inputting the first image into a classification machine learning model and, in response, receiving the text description as a class that the classification machine learning model predicts for the first image. See Shen at [Page 21, Fig. 6]:” For image-classification, I selected the google/vitbase-patch16-224 model. This model is trained on natural images dataset and it can predict the label of the image output. I applied this model to the image and get the results showing the risk of each label. It shows the highest risk at "Rhodesian ridgeback" with a score of 93.8%. Shen teaches an image-classification task in which the first image, example.jpg, is input into the google/vit-base-patch16-224 classification machine learning model. Shen explains that the model predicts labels for the input image and identifies “Rhodesian ridgeback” as the class having the highest score. Accordingly, the textual description of the first image is received as a class predicted by the classification machine learning model, as claimed.
The same motivation that was utilized for combining Khani with Shen as set forth in claim 9 is equally applicable to claim 11.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIAYUE MA whose telephone number is (571)272-9658. The examiner can normally be reached between 9 am to 5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Jiayue Ma/
Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights