DETAILED ACTION
Status of Claims
This Office action is responsive to communications filed on 2026-01-23. Claim(s) 1, 3-9, 11-15, and 17-20 is/are pending and are examined herein.
Claim(s) 1, 3-9, 11-15, and 17-20 is/are rejected under 35 USC 112(b).
Claim(s) 1, 3-9, 11-15, and 17-20 is/are rejected under 35 USC 101.
Claim(s) 1, 3-4, 8-9, 11-12, 15, and 17-18 is/are rejected under 35 USC 102.
Claim(s) 5-7, 13-14, and 19-20 is/are rejected under 35 USC 103.
Notice of Pre-AIA or AIA Status
The present application, filed on or after 2013-03-16, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Regarding rejections under 35 USC 112, the applicant’s amendments resolve the issues raised in the previous Office action. However, the amendments introduce new issues and re-introduce old issues (e.g., the issue regarding the subjectivity of “sensitive” data as described in [Office action of 2025-02-14, pages 4-5]). Issues in the pending claims are described below.
Regarding the rejections under 35 USC 101, the applicant’s arguments have been fully considered but they are unpersuasive:
Regarding Step 2A Prong 1, the applicant asserts that the claims do not need the full eligibility analysis [remarks, page 10]. The examiner respectfully disagrees. The examiner maintains that the full eligibility analysis is required as the claims recite numerous abstract ideas at Step 2A Prong 1. Regarding the improvements analysis, see below.
Regarding Step 2A Prong 2, the applicant’s remarks are an attempt to argue that the claimed invention provides an improvement to machine learning [remarks, pages 12-14]. The examiner respectfully disagrees. The claims recite numerous abstract ideas capable of being performed in the human mind (e.g., “identifying a record…; determining a characteristic of a record…; comparing the characteristic…”). Moreover, the additional elements recited in the claim are generically recited steps of data transfer, of model training, and of particular types of data to be used. A person of ordinary skill in the art would not recognize an additional element recited in the pending claims as providing an improvement to the art of machine learning.
Regarding Step 2B, the applicant argues that the pending claims include “specific unconventional approaches to model retraining” [remarks, page 15]. The examiner respectfully disagrees. As noted in the previous Office action, the applicant has not identified the purported “inventive concept” and/or “specific unconventional approaches” of the invention as claimed. As noted above, the additional elements recited in the claim are broad and generically recited.
The complete 101 analysis, updated in view of the amended claims, is given below.
Regarding rejections under 35 USC 102/103, the applicant’s remarks have been fully considered. The applicant’s remarks amount to mere assertions that claimed invention is not disclosed by the prior art made of record, without an identification of specific features that they believe to not be disclosed. Such cursory and unsubstantiated assertions fail to comply with 37 CFR 1.111(b) because they amount to general allegations that the claims define a patentable invention without specifically pointing out how the language of the claims patentably distinguishes them from the references. The examiner maintains that the claimed invention is disclosed by the prior art made of record. The grounds for rejection have been updated in view of the applicant’s amendments.
Claim Rejections - 35 USC 112(b)
The following is a quotation of 35 USC 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 USC 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 1, 3-9, 11-15, and 17-20 is/are rejected under 35 USC 112(b) or 35 USC 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 USC 112, the applicant), regards as the invention.
Claims 1, 9, and 15 were amended to recite at least the following indefinite limitations:
wherein the production data comprises sensitive images [emphasis added]. Any assessment as to whether or not images are “sensitive” is subjective: images that are sensitive for one person or in one context may not be for another person or in another context. MPEP 2173.05(b)(IV) indicates that, when a claim includes subjective terminology, “[s]ome objective standard must be provided in order to allow the public to determine the scope of the claim. A claim term that requires the exercise of subjective judgment without restriction may render the claim indefinite”. In the present application, the specification provides no such objective criterion. It indicates, for example, that “[p]roduction data may be in a data sensitive realm when there is some concern over who might see and/or analyze specific data points of the production data” and that “sensitive data includes data for which access is controlled/restricted (e.g., whether as a result of regulation, protocol, respect, or the like)” [specification, 0012; emphasis added]. Some examples [specification, 0014] and non-examples [specification, 0013] are also indicated in the specification. The examiner notes, however, that these excerpts from the specification do not constitute a special definition as they are open-ended and do not clearly and unequivocally serve to limit the scope of the term.
identifying a record in the production data that the model predicts with an inaccurate prediction having an accuracy below an accuracy threshold [emphasis added]. This limitation is not clear for at least two reasons:
First, the usage of the verb “predicts” in this limitation does not align with typical usage in the art. Machine learning models do not “predict” a record in their input data; they might predict some attribute of a record in their input data, but it does not align with standard usage in the art to say that the model is predicting the record itself. The examiner suggests “identifying a record in the production data to which the model associates an inaccurate prediction…” to clarify the meaning of the limitation and bring it into line with standard use of terminology in the art. For the purpose of compact prosecution, the claim is interpreted accordingly herein.
Second, it is not clear what it means for an “inaccurate prediction” to “hav[e] an accuracy below an accuracy threshold” as recited by the claim. The examiner notes that previously, the claim recited “identifying an accuracy of the model falling below an accuracy threshold when providing one or more predictions based on a subset of the production data” [emphasis added]. In other words, the “accuracy” which previously occurred in the claims was a property of the model (or, more specifically, of the set of predictions made by the model on a subset of the production data), not a property of a single “inaccurate prediction” as presently recited in the claim. In fact, it is not clear to the examiner what it means for a single “inaccurate prediction” to “hav[e] an accuracy below an accuracy threshold” as presently recited, or if this clause even adds any meaningful limitations on the scope of the claim. The accuracy on one single prediction is necessarily either 100% (1/1, if the one prediction was accurate) or 0% (0/1, if the one prediction was inaccurate). The claim already indicates that the prediction was “inaccurate”, which means the accuracy of the model over the single prediction is necessarily 0%, which is necessarily be below any meaningful “accuracy threshold”. For the purpose of compact prosecution, the claim is interpreted broadly as not requiring the inaccurate prediction to “hav[e] an accuracy below an accuracy threshold” because it is not clear to the examiner how to interpret this clause in view of the applicant’s amendments to the claim.
determine that the characteristic is suboptimally represented in the training dataset [emphasis added]. Any assessment as to whether or not something is “suboptimally represented” is subjective: a characteristic that is “suboptimally represented” for one person or in one context may not be for another person or in another context. The specification merely repeats the same subjective language as the claim [specification, 0009, 0017, 0019-0020, 0033, 0038, 0040-0041, 0043, 0047-0049] but provides no clear objective criterion for this assessment.
Dependent claims 3-8, 11-14, and 17-20 inherit the rejection. In particular, the examiner notes that the issues regarding the indefiniteness of the “accuracy” and the “accuracy threshold” in the independent claims cause significant impact on certain dependent claims, which have been minimally amended despite the significant amendments to the independent claims. For example, claim 3 recites the retrained model has a subsequent accuracy that satisfies the accuracy threshold, which again suggests that the “subsequent accuracy” is a property of the model, not of a single inaccurate prediction as recited in the parent claim, so it is unclear how either the “subsequent accuracy” or the “accuracy threshold” of this limitation relate to the “accuracy” and the “accuracy threshold” of the amended parent claim. Similarly, claim 7 recites in response to identifying the accuracy of the model falling below the accuracy threshold but the parent claim no longer recites a step of identifying the accuracy of the model falling below the accuracy threshold. For the purpose of compact prosecution, the indefinite limitations of these dependents are interpreted in line with the interpretation used in the previously filed claim set.
Claim 7 recites identifying a record being sent to the model for prediction, wherein the record has the characteristic [emphasis added] but an entity of the name “a record” is already introduced in the parent claim. The use of repeat terminology renders unclear whether or not the “record” of this dependent claim is bound in scope by the “record” recited by the parent claim. For the purpose of compact prosecution, the claim is interpreted broadly as encompassing at least the situation where the “record” of this dependent is the same as the “record” of the parent claim. The examiner notes that the parent claim already indicates that the model makes an inaccurate prediction on the “record” (i.e., the record has already been “sent to the model for prediction”) and the parent claim also indicates that the “characteristic” is “of the record”, so this limitation of the dependent appears to be redundant in view of the amendments made to the parent claim. Correspondingly, the examiner suggests removing this limitation from this dependent claim.
Claim Rejections - 35 USC 101
35 USC 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1, 3-9, 11-15, and 17-20 is/are rejected under 35 USC 101 because the claimed invention(s) is/are directed to abstract ideas without significantly more.
Step 1. Claims 1 and 3-8 fall under the statutory category of methods. Claims 9, 11-15, and 17-20 fall under the statutory category of machines. The applicant indicates that the “computer-readable storage media” of the claims are “not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire” [specification, 0056]. The step 2 analysis follows.
Claim 1
Step 2A Prong 1. The claim recites the following abstract ideas:
analyzing a model that is using production data operating in a production environment (This recites a mental process that can be performed in the human mind or by a human using pen and paper, since a human being can manually analyze the performance of a trained model. A human mind can, for example, checking that the model’s predictions agree with one’s prior knowledge). See MPEP 2106.04(a)(2)(III).)
identifying a record in the production data that the model predicts with an inaccurate prediction having an accuracy below an accuracy threshold; (This recites a mathematical concept and a mental process that can be performed in the human mind or by a human using pen and paper. A human mind can identify an inaccurate prediction made by a model. See MPEP 2106.04(a)(2)(I, III).)
determining a characteristic of the record that is associated with the inaccurate prediction, (This recites a mental process that can be performed in the human mind or by a human using pen and paper. A human mind can determine a characteristic of a record. See MPEP 2106.04(a)(2)(III).)
in response to determining the characteristic, comparing the characteristic to characteristics within the training dataset to determine that the characteristic is suboptimally represented in the training dataset; (This recites a mental process that can be performed in the human mind or by a human using pen and paper. A human mind can make comparisons and determine whether something is “suboptimally represented” in a dataset. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
A computer-implemented method comprising: (This recites generic computing components for performing an abstract idea. See MPEP 2106.05(f)(2).)
wherein the model is a neural network (This recites a generic type of machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
[a neural network] trained with a first training dataset, wherein the production data comprises sensitive images; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
providing the characteristic and the inaccurate prediction to a location external to the production environment; (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
autonomously creating a supplemental training dataset comprising data from the training dataset with a threshold number of transformed images that each include the characteristic; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
and autonomously retraining the model with the supplemental training dataset. (This recites a step of generically training a generic machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
A computer-implemented method comprising: (This recites generic computing components for performing an abstract idea. See MPEP 2106.05(f)(2).)
wherein the model is a neural network (This recites a generic type of machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
[a neural network] trained with a first training dataset, wherein the production data comprises sensitive images; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
providing the characteristic and the inaccurate prediction to a location external to the production environment; (This insignificant extra-solution activity is well-understood, routine, conventional as it is mere data transfer. See MPEP 2106.05(d)(II), “Receiving or transmitting data over a network” and/or “Storing and retrieving information in memory”.)
autonomously creating a supplemental training dataset comprising data from the training dataset with a threshold number of transformed images that each include the characteristic; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
and autonomously retraining the model with the supplemental training dataset. (This recites a step of generically training a generic machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Claim 3
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[The computer-implemented method of claim 1, further comprising:] identifying that the retrained model has a subsequent accuracy that satisfies the accuracy threshold; (This recites a mental process that can be performed in the human mind or by a human using pen and paper, since a human being can manually assess the accuracy of a model. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
and redeploying the retrained model (This recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
into the production environment to use production data. (This recites a general link between an abstract idea and a particular field of use or technological environment. See MPEP 2106.05(h).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
and redeploying the retrained model (This recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
into the production environment to use production data. (This recites a general link between an abstract idea and a particular field of use or technological environment. See MPEP 2106.05(h).)
Claim 4
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[The computer-implemented method of claim 1, wherein determining the characteristic comprises:] determining additional characteristics of additional records. (This recites a mental process that can be performed in the human mind or by a human using pen and paper, since a human being can manually determine characteristics of records. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
Claim 5
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[The computer-implemented method of claim 4, wherein determining the characteristic or additional characteristics comprises] a technique selected from the group consisting of a clustering algorithm, an associations algorithm, a classification model, a regression model, a statistical distribution, and bivariate statistics. (This encompasses mathematical concepts such as regressions and statistical distributions. See MPEP 2106.04(a)(2)(I).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
Claim 6
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[The computer-implemented method of claim 1, further comprising] removing the model from the production environment in response to identifying the accuracy of the model falling below the accuracy threshold. (This recites a mental process that can be performed in the human mind or by a human using pen and paper, since a human being can manually remove a model from a production environment. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
Claim 7
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
[The computer-implemented method of claim 1, further comprising:] identifying a record being sent to the model for prediction, wherein the record has the characteristic; (This recites a mental process that can be performed in the human mind or by a human using pen and paper, since a human being can manually identify records to be sent to the model having certain characteristics. See MPEP 2106.04(a)(2)(III).)
and returning a disclaimer rather than a prediction from the model in response to identifying the accuracy of the model falling below the accuracy threshold. (This recites a mental process that can be performed in the human mind or by a human using pen and paper, since a human being can handwrite a disclaimer to indicate that model accuracy has fallen below a threshold. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
Claim 8
Step 2A Prong 1. The claim recites the following abstract ideas:
The abstract idea(s) in the parent claim(s).
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
The additional element(s) in the parent claim(s).
[The computer-implemented method of claim 1, wherein] the inaccurate prediction and the characteristic are provided without providing any of the production data. (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
The additional element(s) in the parent claim(s).
[The computer-implemented method of claim 1, wherein] the inaccurate prediction and the characteristic are provided without providing any of the production data. (This insignificant extra-solution activity is well-understood, routine, conventional as it is mere data transfer. See MPEP 2106.05(d)(II), “Receiving or transmitting data over a network” and/or “Storing and retrieving information in memory”.)
Claim 9
Step 2A Prong 1. The claim recites the following abstract ideas:
analyzing a model that is using production data operating in a production environment (This recites a mental process that can be performed in the human mind or by a human using pen and paper, since a human being can manually analyze the performance of a trained model. A human mind can, for example, checking that the model’s predictions agree with one’s prior knowledge). See MPEP 2106.04(a)(2)(III).)
identifying a record in the production data that the model predicts with an inaccurate prediction having an accuracy below an accuracy threshold; (This recites a mathematical concept and a mental process that can be performed in the human mind or by a human using pen and paper. A human mind can identify an inaccurate prediction made by a model. See MPEP 2106.04(a)(2)(I, III).)
determining a characteristic of the record that is associated with the inaccurate prediction, (This recites a mental process that can be performed in the human mind or by a human using pen and paper. A human mind can determine a characteristic of a record. See MPEP 2106.04(a)(2)(III).)
in response to determining the characteristic, comparing the characteristic to characteristics within the training dataset to determine that the characteristic is suboptimally represented in the training dataset; (This recites a mental process that can be performed in the human mind or by a human using pen and paper. A human mind can make comparisons and determine whether something is “suboptimally represented” in a dataset. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
A system comprising: a processor set; one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media to cause the processor set to perform operations comprising: (This recites generic computing components for performing an abstract idea. See MPEP 2106.05(f)(2).)
wherein the model is a neural network (This recites a generic type of machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
[a neural network] trained with a first training dataset, wherein the production data comprises sensitive images; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
providing the characteristic and the inaccurate prediction to a location external to the production environment; (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
autonomously creating a supplemental training dataset comprising data from the training dataset with a threshold number of transformed images that each include the characteristic; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
and autonomously retraining the model with the supplemental training dataset. (This recites a step of generically training a generic machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
A system comprising: a processor set; one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media to cause the processor set to perform operations comprising: (This recites generic computing components for performing an abstract idea. See MPEP 2106.05(f)(2).)
wherein the model is a neural network (This recites a generic type of machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
[a neural network] trained with a first training dataset, wherein the production data comprises sensitive images; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
providing the characteristic and the inaccurate prediction to a location external to the production environment; (This insignificant extra-solution activity is well-understood, routine, conventional as it is mere data transfer. See MPEP 2106.05(d)(II), “Receiving or transmitting data over a network” and/or “Storing and retrieving information in memory”.)
autonomously creating a supplemental training dataset comprising data from the training dataset with a threshold number of transformed images that each include the characteristic; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
and autonomously retraining the model with the supplemental training dataset. (This recites a step of generically training a generic machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Claims 11-14 inherit limitations from claim 9 and recite additional limitations which are substantially similar to those recited by claims 3-4, 7, and 6, respectively, so they are rejected by the same rationale.
Claim 15
Step 2A Prong 1. The claim recites the following abstract ideas:
analyzing a model that is using production data operating in a production environment (This recites a mental process that can be performed in the human mind or by a human using pen and paper, since a human being can manually analyze the performance of a trained model. A human mind can, for example, checking that the model’s predictions agree with one’s prior knowledge). See MPEP 2106.04(a)(2)(III).)
identifying a record in the production data that the model predicts with an inaccurate prediction having an accuracy below an accuracy threshold; (This recites a mathematical concept and a mental process that can be performed in the human mind or by a human using pen and paper. A human mind can identify an inaccurate prediction made by a model. See MPEP 2106.04(a)(2)(I, III).)
determining a characteristic of the record that is associated with the inaccurate prediction, (This recites a mental process that can be performed in the human mind or by a human using pen and paper. A human mind can determine a characteristic of a record. See MPEP 2106.04(a)(2)(III).)
in response to determining the characteristic, comparing the characteristic to characteristics within the training dataset to determine that the characteristic is suboptimally represented in the training dataset; (This recites a mental process that can be performed in the human mind or by a human using pen and paper. A human mind can make comparisons and determine whether something is “suboptimally represented” in a dataset. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2. The claim recites the following additional elements which, considered individually and as an ordered combination, do not integrate the abstract idea into a practical application:
A computer program product, comprising: one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media to perform operations comprising: (This recites generic computing components for performing an abstract idea. See MPEP 2106.05(f)(2).)
wherein the model is a neural network (This recites a generic type of machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
[a neural network] trained with a first training dataset, wherein the production data comprises sensitive images; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
providing the characteristic and the inaccurate prediction to a location external to the production environment; (This recites insignificant extra-solution activity. See MPEP 2106.05(g).)
autonomously creating a supplemental training dataset comprising data from the training dataset with a threshold number of transformed images that each include the characteristic; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
and autonomously retraining the model with the supplemental training dataset. (This recites a step of generically training a generic machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B. The claim recites the following additional elements which, considered individually and as an ordered combination, do not amount to significantly more than the abstract idea:
A computer program product, comprising: one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media to perform operations comprising: (This recites generic computing components for performing an abstract idea. See MPEP 2106.05(f)(2).)
wherein the model is a neural network (This recites a generic type of machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
[a neural network] trained with a first training dataset, wherein the production data comprises sensitive images; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
providing the characteristic and the inaccurate prediction to a location external to the production environment; (This insignificant extra-solution activity is well-understood, routine, conventional as it is mere data transfer. See MPEP 2106.05(d)(II), “Receiving or transmitting data over a network” and/or “Storing and retrieving information in memory”.)
autonomously creating a supplemental training dataset comprising data from the training dataset with a threshold number of transformed images that each include the characteristic; (This recites data of a particular type or source, merely linking an abstract idea to a particular field of use. See MPEP 2106.05(h).)
and autonomously retraining the model with the supplemental training dataset. (This recites a step of generically training a generic machine learning model. In other words, this recites merely applying (or equivalent) an abstract idea, or implementing an abstract idea on a computer, or using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Claims 17-20 inherit limitations from claim 15 and recite additional limitations which are substantially similar to those recited by claims 3-4 and 6-7, respectively, so they are rejected by the same rationale.
Claim Rejections - 35 USC 102
The following is a quotation of the appropriate paragraphs of 35 USC 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 USC 102(b)(2)(C) for any potential 35 USC 102(a)(2) prior art against the later invention.
Claim(s) 1, 3-4, 8-9, 11-12, 15, and 17-18 is/are rejected under 35 USC 102(a)(1) as being anticipated by Philip KOOPMAN et al. (US20230084761A1, effectively filed 2021-02-22; hereafter, “Koopman”).
Claim 1
Koopman discloses:
A computer-implemented method comprising: ([Koopman, 0011]: Koopman discloses a “computer-implemented method” which processes an input dataset through a perception system, identifies one or more perception weaknesses, relabels the perception weaknesses, and retrains the system using the relabeled data [Koopman, 0011].)
analyzing a model ([Koopman, 0002]: The perception system of Koopman is a “learning-based system such as one implemented using machine learning” [Koopman, 0002]. In other words, the perception system maps to the “model” of the claim. The process of identifying perception weaknesses, for example, falls under the broadest reasonable of “analyzing” as recited by the claim.) that is using production data operating in a production environment, ([Koopman, 0006, 0011, 0021]: The “input dataset” for the perception system [Koopman, 0011] (also called the “baseline dataset” [Koopman, 0021]) maps to the “production data” of the claim. It may take the form of a “baseline sensor data stream” [Koopman, 0006] and thus falls under the broadest reasonable interpretation of “operating in a production environment” as recited by the claim.)
wherein the model is a neural network model trained with a training dataset, ([Koopman, 0003, 0023]: The perception system is a convolutional neural network [Koopman, 0003; see also, 0023] and is trained using “training data” [Koopman, 0003].)
wherein the production data comprises sensitive images; ([Koopman, 0011]: The input dataset of Koopman is comprised of “scene[s]” [Koopman, 0011], i.e., images.)
identifying a record in the production data that the model predicts with an inaccurate prediction having an accuracy below an accuracy threshold; determining a characteristic of the record that is associated with the inaccurate prediction; ([Koopman, 0021]: Koopman describes a “defect detection engine… to identify the presence of a perception weakness (hereinafter also called a ‘defect’) within a given set of input data” [Koopman, 0021]. Examples of perception weaknesses described in Koopman include “pedestrians wearing yellow coats” [Koopman, 0005 and 0008] and “pedestrians more than 250 pixels tall” [Koopman, 0031]. Any scene having such a perception weakness maps to the “record” of the claim, and the perception weakness itself maps to the “characteristic of the record” of the claim. As noted under the 112(b) rejections, it is not clear what it means for a single inaccurate prediction to “hav[e] an accuracy below an accuracy threshold” as presently recited in the claim, but the applicant is invited to consult previous Office actions regarding how the “accuracy threshold” that previously appeared in the claims was disclosed by the prior art made of record.)
in response to determining the characteristic, comparing the characteristic to characteristics within the training dataset ([Koopman, 0006, 0036; Koopman2, 0026]: This limitation can be mapped in multiple ways. For a mapping found directly in Koopman, note that Koopman discloses “compar[ing] a newly created set of defects with the sets of defects generated from prior labeling runs against the same source dataset” [Koopman, 0036], so the perception weaknesses from prior labeling runs (which are compared to the “newly created” perception weaknesses) can be mapped to the “characteristics” of the claim. Alternatively, Koopman incorporates Philip KOOPMAN et al. (US20220004818A1, effectively filed 2019-11-04; hereafter, “Koopman2”) in its entirety by reference, relying on Koopman2 for a more complete description of the approach to identifying systemic biases in the training data [Koopman, 0006]. The approach disclosed in Kooman2 “perform[s] analysis of multiple detections produced by the perception system” and gives an example of such an analysis in which “multiple detections compared are from a single stream of temporally different detections” [Koopman2, 0026; emphasis added]. In other words, the multiple detections being analyzed/compared could be mapped to the “characteristics” of the claim.)
to determine that the characteristic is suboptimally represented in the training dataset; ([Koopman, 0005; Koopman2, 0052]: Koopman notes that the perception weakness of a pedestrian in a yellow coat, for example, might result if the perception system was “trained thoroughly on pedestrians with red coats, black coats, and blue coats” using “training data that under-represents pedestrians in yellow coats” [Koopman, 0005; emphasis added]. This means that Koopman does in fact disclose that each perception weakness of Koopman (i.e., the “characteristic” of the claim as mapped above) appears to be “suboptimally represented in the training data” as recited by the claim. See also: “weak detections often correspond to gaps in training data, and… images containing objects that are under-represented in the machine learning training data set” [Koopman2, 0052; emphasis added].)
providing the characteristic and the inaccurate prediction to a location external to the production environment; ([Koopman, 0029-0032]: Koopman discloses that, after “candidate defects are identified” [Koopman, 0029], they are “transmi[tted] to a labeling facility” [Koopman, 0030] which “labels the scenes and artifacts requested” [Koopman, 0032]. The labeling facility maps to the “location external to the production environment” of the claim.)
autonomously creating a supplemental training dataset comprising data from the training dataset with a threshold number of transformed images that each include the characteristic; ([Koopman, 0029-0032]: As noted above, candidate defects are transmitted to a labeling facility which “labels the scenes and artifacts requested” [Koopman, 0029-0032]. Koopman further describes “adding the results [of the labeling] to the available set of labeled data at the training and validation database 250” [Koopman, 0032]. The images labeled by the labeling facility map to the “threshold number of transformed images that each include the characteristic” of the claim, and the overall training dataset containing this labeled data maps to the “supplemental training dataset” of the claim. The examiner notes that other mappings of this limitation from Koopman are also possible. For example, Koopman indicates that production data can include either real-world or simulated data [Koopman, 0027] and discusses generating “simulations with one or more altered parameters” after detecting defects [Koopman, 0053; see also, 0057], so the data from these simulations could also map to the “transformed images” of the claim.)
and autonomously retraining the model with the supplemental training dataset. ([Koopman, 0032]: Koopman discloses that, after the transformed data is added to the training database, the system “recommence[s] perception system retraining 260. Once the machine learning models associated with the newly labeled data are retrained, and a newly trained perception engine 265 is produced responsive to training data identified by the preceding steps” [Koopman, 0032].)
Claim 3
Koopman discloses the elements of the parent claim(s). It also discloses:
[The computer-implemented method of claim 1, further comprising:] identifying that the retrained model has a subsequent accuracy that satisfies the accuracy threshold; ([Koopman, 0032-0033, 0036]: The examiner notes that this limitation is rendered indefinite by amendments made to the parent claim (cf. 112(b) rejections). This limitation can be mapped in at least two ways. First, Koopman discloses that the steps in the workflow are “executed on a repeated basis” [Koopman, 0033]. In other words, in a subsequent iteration of the workflow, the same process described above is applied, and when the model has been trained using sufficient labeled examples of previously identified defects, its “accuracy” as mapped in the parent claim will eventually satisfy the “accuracy threshold” as mapped in the parent claim (because “there is an expectation that training on [labeled samples from the labeling facility] will improve perception performance” since this data “provides labeled examples of data samples that the perception engine 265 was measured to perform weakly upon” [Koopman, 0032]). Alternatively, Koopman also discloses that, after the model is retrained, the system checks to see if retraining resulted in “meaningful progress” as measured by “improvement in the Precision-Recall (PR) curve for the ML perception system. As used for this curve, “precision”… is the false positive percentage and recall is the false negative percentage. In embodiments, if the curve has improved in an amount greater than 0.5% percent, it is still considered meaningful progress” [Koopman, 0036]. In other words, the precision-recall curve falls under the broadest reasonable interpretation of being “an accuracy” of the model” as recited by the claim, and the threshold 0.5% of the exemplary embodiment can map to the “accuracy threshold” of the claim.)
and redeploying the retrained model into the production environment to use production data. ([Koopman, 0032]: Koopman discloses that “the machine learning models associated with the newly labeled data are retrained, and a newly trained perception engine 265 is produced” [Koopman, 0032]. The production of the newly trained perception engine maps to a redeployment of the retrained model as recited by the claim.)
Claim 4
Koopman discloses the elements of the parent claim(s). It also discloses:
[The computer-implemented method of claim 1, wherein determining the characteristic comprises:] determining additional characteristics of additional records. ([Koopman, 0023]: Koopman discloses that “[p]erception weaknesses, or ‘defects,’ are deposited in a defect database” by the defect detection engine [Koopman, 0023]. Observe the plural. As noted under the parent claim, one of the perception weaknesses identified by the defect detection engine maps to the “characteristic” of the claim, so the others map to the “additional characteristics” of the claim.)
Claim 8
Koopman discloses the elements of the parent claim(s). It also discloses:
[The computer-implemented method of claim 1, wherein] the inaccurate prediction and the characteristic are provided without providing any of the production data. ([Koopman, 0031-0032]: Defects identified by the defect identification engine are filtered according to filtering criteria before being transmitted/provided to the labeling facility [Koopman, 0031-0032]. The input/baseline dataset (i.e., the “production data” of the claim as mapped above) is not itself sent/provided to the labeling facility.)
Claim 9
Koopman discloses:
A system comprising: a processor set; one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media to cause the processor set to perform operations comprising: ([Koopman, 0011, 0014]: Koopman discloses a “computer-implemented method” which processes an input dataset through a perception system, identifies one or more perception weaknesses, relabels the perception weaknesses, and retrains the system using the relabeled data [Koopman, 0011] and further discloses that “these computer-enabled methods can be enabled and performed via a computer system having at least a memory or other data storage facility, one or more processors, and a non-transitory computer-readable medium to store the instructions so that when enabled by the one or more processors, they perform one or more of the aforementioned aspects” [Koopman, 0014].)
analyzing a model ([Koopman, 0002]: The perception system of Koopman is a “learning-based system such as one implemented using machine learning” [Koopman, 0002]. In other words, the perception system maps to the “model” of the claim. The process of identifying perception weaknesses, for example, falls under the broadest reasonable of “analyzing” as recited by the claim.) that is using production data operating in a production environment, ([Koopman, 0006, 0011, 0021]: The “input dataset” for the perception system [Koopman, 0011] (also called the “baseline dataset” [Koopman, 0021]) maps to the “production data” of the claim. It may take the form of a “baseline sensor data stream” [Koopman, 0006] and thus falls under the broadest reasonable interpretation of “operating in a production environment” as recited by the claim.)
wherein the model is a neural network model trained with a training dataset, ([Koopman, 0003, 0023]: The perception system is a convolutional neural network [Koopman, 0003; see also, 0023] and is trained using “training data” [Koopman, 0003].)
wherein the production data comprises sensitive images; ([Koopman, 0011]: The input dataset of Koopman is comprised of “scene[s]” [Koopman, 0011], i.e., images.)
identifying a record in the production data that the model predicts with an inaccurate prediction having an accuracy below an accuracy threshold; determining a characteristic of the record that is associated with the inaccurate prediction; ([Koopman, 0021]: Koopman describes a “defect detection engine… to identify the presence of a perception weakness (hereinafter also called a ‘defect’) within a given set of input data” [Koopman, 0021]. Examples of perception weaknesses described in Koopman include “pedestrians wearing yellow coats” [Koopman, 0005 and 0008] and “pedestrians more than 250 pixels tall” [Koopman, 0031]. Any scene having such a perception weakness maps to the “record” of the claim, and the perception weakness itself maps to the “characteristic of the record” of the claim. As noted under the 112(b) rejections, it is not clear what it means for a single inaccurate prediction to “hav[e] an accuracy below an accuracy threshold” as presently recited in the claim, but the applicant is invited to consult previous Office actions regarding how the “accuracy threshold” that previously appeared in the claims was disclosed by the prior art made of record.)
in response to determining the characteristic, comparing the characteristic to characteristics within the training dataset ([Koopman, 0006, 0036; Koopman2, 0026]: This limitation can be mapped in multiple ways. For a mapping found directly in Koopman, note that Koopman discloses “compar[ing] a newly created set of defects with the sets of defects generated from prior labeling runs against the same source dataset” [Koopman, 0036], so the perception weaknesses from prior labeling runs (which are compared to the “newly created” perception weaknesses) can be mapped to the “characteristics” of the claim. Alternatively, Koopman incorporates Philip KOOPMAN et al. (US20220004818A1, effectively filed 2019-11-04; hereafter, “Koopman2”) in its entirety by reference, relying on Koopman2 for a more complete description of the approach to identifying systemic biases in the training data [Koopman, 0006]. The approach disclosed in Kooman2 “perform[s] analysis of multiple detections produced by the perception system” and gives an example of such an analysis in which “multiple detections compared are from a single stream of temporally different detections” [Koopman2, 0026; emphasis added]. In other words, the multiple detections being analyzed/compared could be mapped to the “characteristics” of the claim.)
to determine that the characteristic is suboptimally represented in the training dataset; ([Koopman, 0005; Koopman2, 0052]: Koopman notes that the perception weakness of a pedestrian in a yellow coat, for example, might result if the perception system was “trained thoroughly on pedestrians with red coats, black coats, and blue coats” using “training data that under-represents pedestrians in yellow coats” [Koopman, 0005; emphasis added]. This means that Koopman does in fact disclose that each perception weakness of Koopman (i.e., the “characteristic” of the claim as mapped above) appears to be “suboptimally represented in the training data” as recited by the claim. See also: “weak detections often correspond to gaps in training data, and… images containing objects that are under-represented in the machine learning training data set” [Koopman2, 0052; emphasis added].)
providing the characteristic and the inaccurate prediction to a location external to the production environment; ([Koopman, 0029-0032]: Koopman discloses that, after “candidate defects are identified” [Koopman, 0029], they are “transmi[tted] to a labeling facility” [Koopman, 0030] which “labels the scenes and artifacts requested” [Koopman, 0032]. The labeling facility maps to the “location external to the production environment” of the claim.)
autonomously creating a supplemental training dataset comprising data from the training dataset with a threshold number of transformed images that each include the characteristic; ([Koopman, 0029-0032]: As noted above, candidate defects are transmitted to a labeling facility which “labels the scenes and artifacts requested” [Koopman, 0029-0032]. Koopman further describes “adding the results [of the labeling] to the available set of labeled data at the training and validation database 250” [Koopman, 0032]. The images labeled by the labeling facility map to the “threshold number of transformed images that each include the characteristic” of the claim, and the overall training dataset containing this labeled data maps to the “supplemental training dataset” of the claim. The examiner notes that other mappings of this limitation from Koopman are also possible. For example, Koopman indicates that production data can include either real-world or simulated data [Koopman, 0027] and discusses generating “simulations with one or more altered parameters” after detecting defects [Koopman, 0053; see also, 0057], so the data from these simulations could also map to the “transformed images” of the claim.)
and autonomously retraining the model with the supplemental training dataset. ([Koopman, 0032]: Koopman discloses that, after the transformed data is added to the training database, the system “recommence[s] perception system retraining 260. Once the machine learning models associated with the newly labeled data are retrained, and a newly trained perception engine 265 is produced responsive to training data identified by the preceding steps” [Koopman, 0032].)
Claims 11-12 inherit limitations from claim 9 and recite additional limitations which are substantially similar to those recited by claims 3-4, respectively, so they are rejected by the same rationale.
Claim 15
Koopman discloses:
A computer program product, comprising: one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media to perform operations comprising: ([Koopman, 0011, 0014]: Koopman discloses a “computer-implemented method” which processes an input dataset through a perception system, identifies one or more perception weaknesses, relabels the perception weaknesses, and retrains the system using the relabeled data [Koopman, 0011] and further discloses that “these computer-enabled methods can be enabled and performed via a computer system having at least a memory or other data storage facility, one or more processors, and a non-transitory computer-readable medium to store the instructions so that when enabled by the one or more processors, they perform one or more of the aforementioned aspects” [Koopman, 0014].)
analyzing a model ([Koopman, 0002]: The perception system of Koopman is a “learning-based system such as one implemented using machine learning” [Koopman, 0002]. In other words, the perception system maps to the “model” of the claim. The process of identifying perception weaknesses, for example, falls under the broadest reasonable of “analyzing” as recited by the claim.) that is using production data operating in a production environment, ([Koopman, 0006, 0011, 0021]: The “input dataset” for the perception system [Koopman, 0011] (also called the “baseline dataset” [Koopman, 0021]) maps to the “production data” of the claim. It may take the form of a “baseline sensor data stream” [Koopman, 0006] and thus falls under the broadest reasonable interpretation of “operating in a production environment” as recited by the claim.)
wherein the model is a neural network model trained with a training dataset, ([Koopman, 0003, 0023]: The perception system is a convolutional neural network [Koopman, 0003; see also, 0023] and is trained using “training data” [Koopman, 0003].)
wherein the production data comprises sensitive images; ([Koopman, 0011]: The input dataset of Koopman is comprised of “scene[s]” [Koopman, 0011], i.e., images.)
identifying a record in the production data that the model predicts with an inaccurate prediction having an accuracy below an accuracy threshold; determining a characteristic of the record that is associated with the inaccurate prediction; ([Koopman, 0021]: Koopman describes a “defect detection engine… to identify the presence of a perception weakness (hereinafter also called a ‘defect’) within a given set of input data” [Koopman, 0021]. Examples of perception weaknesses described in Koopman include “pedestrians wearing yellow coats” [Koopman, 0005 and 0008] and “pedestrians more than 250 pixels tall” [Koopman, 0031]. Any scene having such a perception weakness maps to the “record” of the claim, and the perception weakness itself maps to the “characteristic of the record” of the claim. As noted under the 112(b) rejections, it is not clear what it means for a single inaccurate prediction to “hav[e] an accuracy below an accuracy threshold” as presently recited in the claim, but the applicant is invited to consult previous Office actions regarding how the “accuracy threshold” that previously appeared in the claims was disclosed by the prior art made of record.)
in response to determining the characteristic, comparing the characteristic to characteristics within the training dataset ([Koopman, 0006, 0036; Koopman2, 0026]: This limitation can be mapped in multiple ways. For a mapping found directly in Koopman, note that Koopman discloses “compar[ing] a newly created set of defects with the sets of defects generated from prior labeling runs against the same source dataset” [Koopman, 0036], so the perception weaknesses from prior labeling runs (which are compared to the “newly created” perception weaknesses) can be mapped to the “characteristics” of the claim. Alternatively, Koopman incorporates Philip KOOPMAN et al. (US20220004818A1, effectively filed 2019-11-04; hereafter, “Koopman2”) in its entirety by reference, relying on Koopman2 for a more complete description of the approach to identifying systemic biases in the training data [Koopman, 0006]. The approach disclosed in Kooman2 “perform[s] analysis of multiple detections produced by the perception system” and gives an example of such an analysis in which “multiple detections compared are from a single stream of temporally different detections” [Koopman2, 0026; emphasis added]. In other words, the multiple detections being analyzed/compared could be mapped to the “characteristics” of the claim.)
to determine that the characteristic is suboptimally represented in the training dataset; ([Koopman, 0005; Koopman2, 0052]: Koopman notes that the perception weakness of a pedestrian in a yellow coat, for example, might result if the perception system was “trained thoroughly on pedestrians with red coats, black coats, and blue coats” using “training data that under-represents pedestrians in yellow coats” [Koopman, 0005; emphasis added]. This means that Koopman does in fact disclose that each perception weakness of Koopman (i.e., the “characteristic” of the claim as mapped above) appears to be “suboptimally represented in the training data” as recited by the claim. See also: “weak detections often correspond to gaps in training data, and… images containing objects that are under-represented in the machine learning training data set” [Koopman2, 0052; emphasis added].)
providing the characteristic and the inaccurate prediction to a location external to the production environment; ([Koopman, 0029-0032]: Koopman discloses that, after “candidate defects are identified” [Koopman, 0029], they are “transmi[tted] to a labeling facility” [Koopman, 0030] which “labels the scenes and artifacts requested” [Koopman, 0032]. The labeling facility maps to the “location external to the production environment” of the claim.)
autonomously creating a supplemental training dataset comprising data from the training dataset with a threshold number of transformed images that each include the characteristic; ([Koopman, 0029-0032]: As noted above, candidate defects are transmitted to a labeling facility which “labels the scenes and artifacts requested” [Koopman, 0029-0032]. Koopman further describes “adding the results [of the labeling] to the available set of labeled data at the training and validation database 250” [Koopman, 0032]. The images labeled by the labeling facility map to the “threshold number of transformed images that each include the characteristic” of the claim, and the overall training dataset containing this labeled data maps to the “supplemental training dataset” of the claim. The examiner notes that other mappings of this limitation from Koopman are also possible. For example, Koopman indicates that production data can include either real-world or simulated data [Koopman, 0027] and discusses generating “simulations with one or more altered parameters” after detecting defects [Koopman, 0053; see also, 0057], so the data from these simulations could also map to the “transformed images” of the claim.)
and autonomously retraining the model with the supplemental training dataset. ([Koopman, 0032]: Koopman discloses that, after the transformed data is added to the training database, the system “recommence[s] perception system retraining 260. Once the machine learning models associated with the newly labeled data are retrained, and a newly trained perception engine 265 is produced responsive to training data identified by the preceding steps” [Koopman, 0032].)
Claims 17-18 inherit limitations from claim 15 and recite additional limitations which are substantially similar to those recited by claims 3-4, respectively, so they are rejected by the same rationale.
Claim Rejections - 35 USC 103
The following is a quotation of 35 USC 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 USC 102(b)(2)(C) for any potential 35 USC 102(a)(2) prior art against the later invention.
Claim(s) 5 is/are rejected under 35 USC 103 as being unpatentable over Koopman in view of Matthew DECARLO et al. (Graduate research methods in social work, published 2020-08-15; hereafter, “Decarlo”).
Claim 5
Koopman discloses the elements of the parent claim(s). It might not distinctly disclose:
[The computer-implemented method of claim 4, wherein determining the characteristic or additional characteristics comprises] a technique selected from the group consisting of a clustering algorithm, an associations algorithm, a classification model, a regression model, a statistical distribution, and bivariate statistics.
Decarlo is in the field of data analysis. Moreover, Koopman in view of Decarlo discloses:
[The computer-implemented method of claim 4, wherein determining the characteristic or additional characteristics comprises] a technique selected from the group consisting of a clustering algorithm, an associations algorithm, a classification model, a regression model, a statistical distribution, and bivariate statistics. ([Decarlo, section 24.15]: Decarlo discusses statistical techniques of bivariate analysis (i.e., the “bivariate statistics” of the claims), and in particular discusses the chi-square test for independence, which makes use of the chi-square distribution (i.e., the “statistical distribution” of the claim). In the combination with Koopman, a distribution of characteristics in the overall input/baseline dataset can be compared against an analogous distribution of characteristics for the subset identified as having a defect, and any characteristics that are overrepresented in the subset can be used for the filtering criteria.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the method of identifying defects and retraining models as disclosed by Koopman with a chi-square test as disclosed by Decarlo because it “foundational to analyzing relationships between nominal or ordinal variables” [Decarlo, section 24.15], so the combination would allow one to identify filtering criteria in more systematic way, thereby increasing the robustness and efficiency of the system.
Claim(s) 6, 14, and 19 is/are rejected under 35 USC 103 as being unpatentable over Koopman in view of Li CAO et al. (US20220342887A1, published 2021-04-26; hereafter, “Cao”).
Claim 6
Koopman discloses the elements of the parent claim(s). It might not distinctly disclose:
[The computer-implemented method of claim 1, further comprising] removing the model from the production environment in response to identifying the accuracy of the model falling below the accuracy threshold.
Cao is in the field of machine learning. Moreover, Koopman and Cao discloses:
[The computer-implemented method of claim 1, further comprising] removing the model from the production environment in response to identifying the accuracy of the model falling below the accuracy threshold. ([Cao, 0096]: Cao discloses a system which “can set a status flag for a predictive model to active when testing the predictive model indicates that the model is producing predictions with a threshold satisfying level of accuracy and can set the status flag to inactive when testing of a predictive model at block 1102 indicates that the predictive model is not producing predictions within a threshold satisfying level of accuracy” [Cao, 0096]. Setting the status flag of a model to inactive when it is not achieving a threshold level of accuracy falls under the broadest reasonable interpretation of “removing the model from the production environment” as recited by the claim.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the method of identifying defects and retraining models as disclosed by Koopman with the use of inactive flags as in Cao because this would ensure that inaccurate predictions are not being provided to the end user.
Claims 14 and 19 inherit limitations from claims 9 and 15, respectively, and recite additional limitations which are substantially similar to those recited by claim 6, so they are rejected by the same rationale.
Claim(s) 7, 13, and 20 is/are rejected under 35 USC 103 as being unpatentable over Koopman in view of SUM Prasad DHANYAMRAJU et al. (US20180005139A1, published 2018-01-04; hereafter, “Dhanyamraju”).
Claim 7
Koopman discloses the elements of the parent claim(s). It also discloses:
[The computer-implemented method of claim 1, further comprising:] identifying a record being sent to the model for prediction, wherein the record has the characteristic; ([Koopman, 0029-0032]: As noted under the parent claim, any scene having such a perception weakness maps to the “record” of the claim, and the perception weakness itself maps to the “characteristic” of the claim.)
Koopman might not distinctly disclose:
and returning a disclaimer rather than a prediction from the model in response to identifying the accuracy of the model falling below the accuracy threshold.
Dhanyamraju is in the field of machine learning. Moreover, Koopman in view Dhanyamraju discloses:
and returning a disclaimer rather than a prediction from the model in response to identifying the accuracy of the model falling below the accuracy threshold. ([Dhanyamraju, 0014]: The examiner notes that this limitation is indefinite in view of amendments made to the independent claims (cf. 112(b) rejections). Dhanyamraju discloses “monitoring the ML model and generate alerts for updating the ML model whenever the accuracy of the model is dropped below a predefined threshold level” [Dhanyamraju, 0014]. In other words, Koopman in view of Dhanyamraju discloses generating an alert (i.e., “returning a disclaimer”) whenever accuracy falls below a threshold and a defect is detected.)
Before the effective filing date of the invention, it would have been obvious to a person of ordinary skill in the art to combine the method of identifying defects and retraining models as disclosed by Koopman in view of Feinstein with the method of monitoring models as disclosed by Dhanyamraju because the latter “ensure[s] the Machine Learning (ML) model accuracy in production environments with live users and active streaming data” [Dhanyamraju, 0013], so the system would be more effective overall.
Claims 13 and 20 inherit limitations from claims 9 and 15, respectively, and recite additional limitations which are substantially similar to those recited by claim 7, so they are rejected by the same rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Shishir AGRAWAL whose telephone number is +1 703-756-1183. The examiner can normally be reached Monday through Thursday, 08:30-14:30 Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey SHMATOV can be reached on +1 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is +1 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at +1 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call +1 800-786-9199 (IN USA OR CANADA) or +1 571-272-1000.
/S.A./Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights