Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
The present application is being examined under the claims filed on May 27, 2022.
Claims 1-20 are pending.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 5-27-2022 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
FIGS. 2A, 2B, 2C, 2D, 2E, 2F, 2G, 2H, 2I, and 2J of the drawings are objected to because contain text that are oriented in different directions [see CFR 1.84(p)(1)]. Additionally, FIG. 9 is objected to for having text placed upon hatched or shaded surfaces [see CFR 1.84(p)(3)]. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The use of the term GTX® in paras. [0126] and [0203], which is a trade name or a mark used in commerce, has been noted in this application. The term should be accompanied by the generic terminology; furthermore the term should be capitalized wherever it appears or, where appropriate, include a proper symbol indicating use in commerce such as ™, SM , or ® following the term.
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
Claim Objections
Claims 1-20 are objected to because of the following informalities:
In claim 1, l. 8, “wherein the model builder engine is configured to:” should read, “wherein the model builder engine is further configured to:” to clarify the recitation of additional elements.
In claim 1, l. 14, “the model-in-training” should read, “the model-in-training
Φ
” in view of antecedent basis claim 1, l. 10, which recites “a model-in-training
Φ
”.
In claim 1, instances of “model after-convergence
Φ
” should read, “model-after-convergence
Φ
”.
Claims 2-8 are objected to for inheriting the deficiencies of claim 1.
In claims 4 and 8, the first recitation of “BALD” should read, “Bayesian Active Learning by Disagreement (BALD)” for clarity because acronyms should be spelled out at their first use. The full term is taken in view of para. [0038] of the specification filed May 27, 2022.
In claim 9, “a first
β
for a first estimated y” should read, “a first
β
for a first estimated value y”.
In claim 10, instances of “model after-convergence
Φ
” should read, “model-after-convergence
Φ
”.
Claims 11-20 are objected to for inheriting the deficiencies of claim 10.
Appropriate correction is required.
Claim Interpretation
Claims 1, 9, and 12 recite an oracle. In view of the specification, the oracle is interpreted as an algorithm or a human. In particular, para. [0047], “the selected points x* are provided by a human (the oracle of FIGS. 1A and 1B)” describes the oracle as a human, and para. [0063], “Under this formulation, the oracle (active learning algorithm)” describes the oracle as an algorithm.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding Claims 1:
Claim 1, l. 5, recites “a model after-convergence
Φ
”. Claim 1, l. 10, further recites “a model-in-training
Φ
”. The same character, “
Φ
”, is used to reference both “a model after-convergence” and “a model-in-training”, making it unclear which model “
Φ
” refers to.
The terms “low information” and “probably has some information” in claim 1 are relative terms which renders the claim indefinite. The terms “low” and “probably has some” are not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim 1, ll. 9-10, recites “a training value x*”. Claim 1, ll. 12-13, further cites “for x* according to the information measure, provide x* to the oracle”. It is unclear if the “x*” from ll. 12-13 are referring to the “training value x*” from ll. 9-10 or if “x*” is referring to something entirely different.
Claim 1, ll. 11-12, recites “a correct classification Y*”. Claim 1, ll. 13-14, further cites “to obtain Y*, and update the model-in-training with Y*”. It is unclear if the “Y*” from ll. 13-14 are referring to the “correct classification Y*” from ll. 11-12 or if “Y*” is referring to something entirely different.
Regarding Claims 2-8:
Claims 2-8 are rejected for inheriting the deficiencies of claim 1.
Regarding Claim 2:
Claim 1, l. 5, recites “a model after-convergence
Φ
”. Claim 2 further recites “a model-in-training
Φ
”. The same character, “
Φ
”, is used to reference both “a model after-convergence” and “a model-in-training”, making it unclear which model “
Φ
” refers to.
Regarding Claim 3:
Claim 3 is rejected for inheriting the deficiencies of claim 2.
Regarding Claim 5:
Claim 5, 1. 2, recites “the following equation in which i ranges over a simplex of C classes”. Claim 5, l. 3, recites “assigning a data point x to a class i”. The same character, “i” is used to reference the range of C classes. Here at l. 3, the claim does not use a definite article, such as “the” or “said”, but instead uses “a”, making it unclear if “i” is intended to refer to “i” from l. 2 or to something different altogether.
Regarding Claims 6 and 8:
Claims 6 and 8 are rejected for inheriting the deficiencies of claim 5.
Regarding Claim 7:
Claim 7, l. 2, recites “the training value x*”. Claim 7, l. 3, further recites (1) “x* has some information”, and l. 4 also recites, (2) “such data x* has some correlation”. Regarding (1), it is unclear if the “x*” is referring to the “training value x*” or if it referring to something entirely different. Regarding (2), “x*” is now being referring to as “such data x*”. In this limitation, the same character, “x*” is used to reference both “the training value” and “such data”, making it unclear what x* is referring to in claim 7.
The terms “approximately zero”, “has some information”, and “has some correlation or compatibility” in claim 7 are relative terms which renders the claim indefinite. The terms “approximately” and “has some” are not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Regarding Claim 9:
Claim 9, ll. 5-6, recites “a data set comprising
D
t
r
a
i
n
i
n
g
and
D
p
o
o
l
”. Claim 9, l. 11, recites “A) sample a first plurality of data points from
D
t
r
a
i
n
i
n
g
”. Here at l. 11, the claim does not use a definite article, such as “the” or “said”, to reference
D
t
r
a
i
n
i
n
g
. Additionally, the difference in formatting (italicized and non-italicized) makes it unclear if
D
t
r
a
i
n
i
n
g
from l. 11 is referring to
D
t
r
a
i
n
i
n
g
from ll. 5-6, or to a new data set
D
t
r
a
i
n
i
n
g
. The examiner is interpreting
D
t
r
a
i
n
i
n
g
to be referring to
D
t
r
a
i
n
i
n
g
.
Claim 9, ll. 13-14, recites “generate M dropout samples of estimated values y to form a plurality of estimated values y, wherein the estimated values y”. Claim 9, l. 16-17, recites “the plurality of estimated values y” Claim 9, l. 18, recites “a first estimated value y”. The same character “y” is used to reference three different limitations, (1) “estimated values” and (2) “a plurality of estimated values”, and (3) “a first estimated value”, making unclear what “y” refers to.
Claim 9, ll. 27-28, recites “F) provide x* to the oracle to obtain Y*, and G) update the model-in-training with Y*”. It is unclear what the character “Y*” is referring to. The examiner is interpreting “Y*” has referring to a correct classification or proper label in view of para. [0046] of the instant specification filed May 27, 2022.
Regarding Claim 10:
Claim 10, ll. 4, recites “a data set
D
t
r
a
i
n
i
n
g
and
D
p
o
o
l
”. Claim 10, ll. 11-12, recites “a plurality of samples from
D
p
o
o
l
, wherein the plurality of samples are not in
D
t
r
a
i
n
i
n
g
”. Here at ll. 11-12, the claim does not use a definite article, such as “the” or “said”, to reference
D
p
o
o
l
and
D
t
r
a
i
n
i
n
g
. Additionally, the difference in formatting (italicized and non-italicized) makes it unclear if
D
p
o
o
l
and
D
t
r
a
i
n
i
n
g
from ll. 11-12 is referring to
D
p
o
o
l
and
D
t
r
a
i
n
i
n
g
from l. 4, or to a new data set
D
p
o
o
l
and
D
t
r
a
i
n
i
n
g
. The examiner is interpreting
D
p
o
o
l
and
D
t
r
a
i
n
i
n
g
to be referring to
D
p
o
o
l
and
D
t
r
a
i
n
i
n
g
.
Regarding Claims 11-20:
Claims 11-20 are rejected for inheriting the deficiencies of claim 10.
Regarding Claim 15:
The terms “is more than about 50” and “is not more than about 1000” in claim 15 are relative terms which renders the claim indefinite. The term “about” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-20 are directed to an active learning classifier engine [machine].
Regarding Claim 1:
Step 2A, Prong 1: The following limitations are directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind or with pen and paper (including an observation, evaluation, judgement, or opinion).
(a) …operate on a data set
D
t
r
a
i
n
i
n
g
and
D
p
o
o
l
and consult an oracle
(b) …to act on the observation to produce the label
(c) …identify, based on an entropy measure and an information measure, a training value x* for which a model-in-training
Φ
provides low information according to the entropy measure and the training value x* probably has some information about a correct classification Y* for x* according to the information measure
provide x* to the oracle to obtain Y*
As drafted, under their broadest reasonable interpretation (BRI), in view of the specification, the above limitations cover concepts performed in the human mind (observation, evaluation, judgement, or opinion). Given a sufficiently small set of data, nothing in the claim prohibits this process from being performed mentally or with pen and paper.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
An active learning classifier engine for classifying an observation under a label using machine learning, the active learning classifier engine comprising one or more processors executing instructions from one or more memories to implement:
(a) a model builder engine configured to…produce a model after-convergence
Φ
(b) a classification engine configured to use the model after-convergence
Φ
…
(c) wherein the model builder engine is configured to…
update the model-in-training with Y* to obtain the model after-convergence
Φ
, thereby reducing a training time of the active learning classifier engine to obtain the label
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
An active learning classifier engine for classifying an observation under a label using machine learning, the active learning classifier engine comprising one or more processors executing instructions from one or more memories to implement:
(a) a model builder engine configured to…produce a model after-convergence
Φ
(b) a classification engine configured to use the model after-convergence
Φ
…
(c) wherein the model builder engine is configured to…
update the model-in-training with Y* to obtain the model after-convergence
Φ
, thereby reducing a training time of the active learning classifier engine to obtain the label
Regarding Claim 2:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 1. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
…identify, based on the entropy measure and the information measure, the training value x* for which first information below a first bit precision threshold according to the entropy measure and the training value x* is associated with second information above a second bit threshold about the correct classification Y* for x* according to the information measure
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
wherein the model builder engine is further configured to…
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
wherein the model builder engine is further configured to…
Regarding Claim 3:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 2.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
wherein the first bit threshold is 0.1 bits given precision of the first information per a ranked unlabeled data point and the second bit threshold is 0.1 bits per the ranked unlabeled data point
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
wherein the first bit threshold is 0.1 bits given precision of the first information per a ranked unlabeled data point and the second bit threshold is 0.1 bits per the ranked unlabeled data point
Regarding Claim 4:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 4.
The following limitations are/remain directed to the abstract idea of a mathematical concept (i.e., mathematical relationships, mathematical formulas or equations, or mathematical calculations) [see MPEP 2106.04(a)(2) I.C.].
wherein the information measure is BALD
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 5:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 1. Additionally, the following limitations are directed to the abstract idea of a mathematical concept (i.e., mathematical relationships, mathematical formulas or equations, or mathematical calculations) [see MPEP 2106.04(a)(2) I.C.].
wherein the entropy measure is expressed as the following equation in which i ranges over a simplex of C classes,
P
i
is an estimated probability of assigning a data point x to a class i,
f
P
i
is a density function and
E
P
i
is an expectation with respect to
P
i
treated as a random variable:
M
J
E
n
t
x
=
-
∑
i
E
P
i
P
i
log
P
i
f
P
i
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 6:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 5. Additionally, the following limitations are/remain directed to the abstract idea of a mathematical concept (i.e., mathematical relationships, mathematical formulas or equations, or mathematical calculations) [see MPEP 2106.04(a)(2) I.C.].
wherein
f
P
i
is a Beta distribution
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 7:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 1. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
(a) …identify the training value x* for which the model-in-training provides approximately zero information value and x* has some information about the correct classification, Y, and such data x* has some correlation or compatibility with previously-used training data.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
(a) wherein the model builder engine is further configured to…
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
(a) wherein the model builder engine is further configured to…
Regarding Claim 8:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 5. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
(a) identify, based on the entropy measure
M
J
E
n
t
[
x
]
and BALD, the training value x* for which the model-in-training provides third information below a third bit threshold
The following limitations are/remain directed to the abstract idea of a mathematical concept (i.e., mathematical relationships, mathematical formulas or equations, or mathematical calculations) [see MPEP 2106.04(a)(2) I.C.].
wherein the information measure is BALD
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
(a) the model builder engine is further configured to…
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
(a) the model builder engine is further configured to…
Regarding Claim 9:
Step 2A, Prong 1: The following limitations are directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind or with pen and paper (including an observation, evaluation, judgement, or opinion).
(a) …operate on a data set
D
t
r
a
i
n
i
n
g
and
D
p
o
o
l
and consult an oracle…
(b) …to act on the observation to produce the label
A) sample a first data point in the first plurality of data points from
D
t
r
a
i
n
i
n
g
B) for a first data point in the first plurality of data points:
a) generate M dropout samples of estimate values y to form a plurality of estimate values y, wherein the estimated values y are samples of a domain of a simplex of classes
b) calculate a plurality of model statistics of the plurality of estimated values y, wherein the plurality of model statistics includes a first
α
and a first
β
for a first estimated value y corresponding to a first class in the simplex of classesc) calculate a first acquisition function measure for the first data point
C) repeat a) through c) for each data point in the first plurality of data points, thereby forming a plurality of acquisition function measures corresponding, respectively, to each data point in the first plurality of data points
D) rank the plurality of acquisition function measures
E) identify a top K data points of the first plurality of data points as x*
F) provide x* to the oracle to obtain Y*
As drafted, under their broadest reasonable interpretation (BRI), in view of the specification, the above limitations cover concepts performed in the human mind (observation, evaluation, judgement, or opinion). Given a sufficiently small set of data, nothing in the claim prohibits this process from being performed mentally or with pen and paper.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
An active learning classifier engine for classifying an observation under a label using machine learning, the active learning classifier engine comprising: one or more memories storing instructions; and one or more processors configured to execute the instructions to implement:
(a) a model builder engine configured to…train a model-in-training…and produce a model after-convergence
Φ
(b) a classification engine configured to use the model-after-convergence
Φ
…
wherein the model builder engine is further configured to:
G) update the model-in-training with Y* to obtain the model-after-convergence
thereby reducing a training time of the active learning classifier engine to obtain the label
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
An active learning classifier engine for classifying an observation under a label using machine learning, the active learning classifier engine comprising: one or more memories storing instructions; and one or more processors configured to execute the instructions to implement:
(a) a model builder engine configured to…train a model-in-training…and produce a model after-convergence
Φ
(b) a classification engine configured to use the model-after-convergence
Φ
…
wherein the model builder engine is further configured to:
G) update the model-in-training with Y* to obtain the model-after-convergence
thereby reducing a training time of the active learning classifier engine to obtain the label
Regarding Claim 10:
Step 2A, Prong 1: The following limitations are directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind or with pen and paper (including an observation, evaluation, judgement, or opinion).
(a) …operate on a data set
D
t
r
a
i
n
i
n
g
and
D
p
o
o
l
for N epochs…
(b) …act on the observation to produce the label
(c) …identify a tentative decision boundary between at least two classes
(d) evaluate uncertainty…of a plurality of samples from
D
p
o
o
l
, wherein the plurality of samples are not in
D
t
r
a
i
n
i
n
g
…
determine, based on the uncertainty, a plurality of information values respectively corresponding to the plurality of samples
select a second plurality of samples as a first number of top-ranked samples of the plurality of samples, wherein the second plurality of samples is approximately uniformly distributed along an extent of the tentative decision boundary
As drafted, under their broadest reasonable interpretation (BRI), in view of the specification, the above limitations cover concepts performed in the human mind (observation, evaluation, judgement, or opinion). Given a sufficiently small set of data, nothing in the claim prohibits this process from being performed mentally or with pen and paper.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
An active learning classifier engine for classifying an observation under a label using machine learning, the active learning classifier engine comprising one or more processors executing instructions from one or more memories to implement:
(a) a model builder engine configured to…and produce a model-after-convergence
Φ
, wherein convergence corresponds to a classification accuracy measure exceeding an accuracy threshold
(b) a classification engine configured to use the model after-convergence
Φ
to…
(c) wherein the model builder engine is further configured to:…
(d) …in a trial classification…, wherein the trial classification is based on a model-in-training
update the model-in-training based on the second plurality of samples
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
An active learning classifier engine for classifying an observation under a label using machine learning, the active learning classifier engine comprising one or more processors executing instructions from one or more memories to implement:
(a) a model builder engine configured to…and produce a model-after-convergence
Φ
, wherein convergence corresponds to a classification accuracy measure exceeding an accuracy threshold
(b) a classification engine configured to use the model after-convergence
Φ
to…
(c) wherein the model builder engine is further configured to:…
(d) …in a trial classification…, wherein the trial classification is based on a model-in-training
update the model-in-training based on the second plurality of samples
Regarding Claim 11:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 10.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
wherein the model builder engine is further configured to obtain a plurality of labels for the second plurality of samples, respectively, before the model-in-training is updated
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
wherein the model builder engine is further configured to obtain a plurality of labels for the second plurality of samples, respectively, before the model-in-training is updated
Regarding Claim 12:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 11. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
(a) …obtain the plurality of labels from an oracle
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
(a) wherein the model builder engine is further configured to obtain…
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
(a) wherein the model builder engine is further configured to obtain…
Regarding Claim 13:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 10. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
wherein the first number of top-ranked samples is 50
As drafted, under their broadest reasonable interpretation (BRI), in view of the specification, the above limitations cover concepts performed in the human mind (observation, evaluation, judgement, or opinion). Given a sufficiently small set of data, nothing in the claim prohibits this process from being performed mentally or with pen and paper.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 14:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 10. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
wherein the first number of top-ranked samples is 250
As drafted, under their broadest reasonable interpretation (BRI), in view of the specification, the above limitations cover concepts performed in the human mind (observation, evaluation, judgement, or opinion). Given a sufficiently small set of data, nothing in the claim prohibits this process from being performed mentally or with pen and paper.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 15:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 10. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
wherein the first number of top-ranked samples is more than about 50 and the first number of top-ranked samples is not more than about 1000
As drafted, under their broadest reasonable interpretation (BRI), in view of the specification, the above limitations cover concepts performed in the human mind (observation, evaluation, judgement, or opinion). Given a sufficiently small set of data, nothing in the claim prohibits this process from being performed mentally or with pen and paper.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 16:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 10. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
wherein the N epochs correspond to 100 epochs or less
As drafted, under their broadest reasonable interpretation (BRI), in view of the specification, the above limitations cover concepts performed in the human mind (observation, evaluation, judgement, or opinion). Given a sufficiently small set of data, nothing in the claim prohibits this process from being performed mentally or with pen and paper.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 17:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 10.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
wherein the accuracy threshold corresponds to 0.9 or better precision score on a benchmark data set
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
wherein the accuracy threshold corresponds to 0.9 or better precision score on a benchmark data set
Regarding Claim 18:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 10.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
wherein the accuracy threshold corresponds to 0.9 or better recall score on a benchmark data set
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
wherein the accuracy threshold corresponds to 0.9 or better recall score on a benchmark data set
Regarding Claim 19:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 17.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
wherein the benchmark data set is CIFAR-10, CIFAR-100, or Caltech-256
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
wherein the benchmark data set is CIFAR-10, CIFAR-100, or Caltech-256
Regarding Claim 20:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 18.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
wherein the benchmark data set is CIFAR-10, CIFAR-100, or Caltech-256
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
wherein the benchmark data set is CIFAR-10, CIFAR-100, or Caltech-256
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 1 is rejected under 35 U.S.C. 103 as being unpatentable over Zhao et al. (US 20200250527), hereinafter Zhao, in view of Das et al. (US 20210192335), hereinafter Das.
Regarding Claim 1:
Zhao discloses:
An active learning classifier engine for classifying an observation under a label using machine learning
Zhao, [0008], “One example aspect of the present disclosure is directed to a computer-implemented method for performing active learning on a training dataset that comprises a plurality of unlabeled datapoints and a plurality of labeled datapoints. The method includes, for each of one or more training iterations: training, by one or more computing devices, a machine-learned classifier model…obtaining, by the one or more computing devices, a respective label for each unlabeled datapoint included in the label gathering slots to transform the unlabeled datapoints included in the label gathering slots into labeled datapoints.”
Zhao discloses a computer system for active learning using a machine-learned classifier model [active learning classifier engine] to label unlabeled datapoints [classifying an observation under a label using machine learning].
the active learning classifier engine comprising one or more processors executing instructions from one or more memories to implement:
Zhao, [0081], “The training computing system 150 includes one or more processors 152 and a memory 154…The memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations.”
a model builder engine configured to operate on a data set
D
t
r
a
i
n
i
n
g
and
D
p
o
o
l
Zhao, [0096], “At 201, a computing system trains a machine-learned classifier model using a training dataset. The training dataset can include a plurality of labeled datapoints and a plurality of unlabeled datapoints…”
Zhao discloses training a machine-learned classifier model using a training dataset that includes labeled datapoints [operate on a data set
D
t
r
a
i
n
i
n
g
] and unlabeled datapoints [
D
p
o
o
l
]
and consult an oracle to produce a model after-convergence
Φ
Zhao, [0101], “In some implementations, obtaining a label can include providing the unlabeled datapoints selected for the label gathering slots to an expert and receiving the labeled datapoints that have been assigned a label by an expert. The labels obtained for each unlabeled datapoint can be used to update the training dataset 209 by removing the unlabeled datapoints selected for the label gathering slots 207 from the unlabeled datapoints and adding the labeled datapoints assigned a label by an expert to the labeled datapoints. The process 200 can be continued in an iterative manner by retraining the machine learned classifier 201 using the updated training dataset.”
Zhao discloses using an expert [consult an oracle] in order to help label unlabeled datapoints that is used to update and train the model [produce a model after-convergence
Φ
].
wherein the model builder engine is configured to: identify, based on…an information measure,…the training value x* probably has some information about a correct classification Y* for x* according to the information measure
Zhao, [0098], “At 203, the computing system determines an exploration score for each unlabeled datapoint. As an example implementation, the exploration scores can be determined using information from the feature vectors that describe the dataset. For example, the exploration score can be determined for each unlabeled datapoint based at least in part on a respective distance between the unlabeled point and each of the labeled datapoints.”
Zhao discloses for the unlabeled datapoints, using information that describes the dataset [the training value x* probably has some information] based on a distance between the unlabeled datapoint and the labeled datapoint [about a correct classification Y* for x* according to the information measure].
provide x* to the oracle to obtain Y*
Zhao, [0101], “In some implementations, obtaining a label can include providing the unlabeled datapoints selected for the label gathering slots to an expert and receiving the labeled datapoints that have been assigned a label by an expert. The labels obtained for each unlabeled datapoint can be used to update the training dataset 209 by removing the unlabeled datapoints selected for the label gathering slots 207 from the unlabeled datapoints and adding the labeled datapoints assigned a label by an expert to the labeled datapoints. The process 200 can be continued in an iterative manner by retraining the machine learned classifier 201 using the updated training dataset.”
Zhao discloses providing unlabeled datapoints to an expert [provide x* to the oracle] to assign labels [to obtain Y*].
update the model-in-training with Y*…
As cited above in para. 101, Zhao discloses adding the labeled datapoints to the dataset to iteratively retrain the classifier.
thereby reducing a training time of the active learning classifier engine to obtain the label
Zhao, [0022], “In particular, the systems and methods described can provide active learning techniques to improve the performance of machine learning models at reduced costs. In some cases, labeling data can be expensive and/or time intensive and the examples and implementations described herein can provide a solution by intelligently selecting unlabeled datapoints to assign a label and training the machine learning model using the updated labeled datapoints that include the assigned labels.”
Zhao discloses reducing the costs of machine learning models [reducing a training time of the active learning classifier engine] that are used for labeling data [to obtain the label].
Zhao does not explicitly disclose:
a classification engine configured to use the model after-convergence
Φ
to act on the observation to produce the label
…based on an entropy measure and an information measure, a training value x* for which a model-in-training
Φ
provides low information according to the entropy measure…
…to obtain the model after-convergence
Φ
However, in the same field, analogous art Das teaches:
a classification engine configured to use the model after-convergence
Φ
to act on the observation to produce the label
Das, [0038], “After the machine-learning algorithm 110 achieves a predetermined performance level (e.g., 100% agreement with the outcomes associated with the training dataset 112), the machine-learning algorithm 110 may be executed using data that is not in the training dataset 112. The trained machine-learning algorithm 110 may be applied to new datasets to generate annotated data.”
Das discloses using the trained machine-learning algorithm [model after-convergence
Φ
] to new datasets to generate annotated data [act on the observation to produce the label].
…based on an entropy measure and an information measure, a training value x* for which a model-in-training
Φ
provides low information according to the entropy measure…
Das, [0060], “It is contemplated that the active learning sampling strategy may be implemented using a maximum entropy-based heuristic to drive the active learning process of the machine learning algorithm.”
Das discloses using a maximum entropy-based heuristic for their active learning machine learning algorithm [identifying, based on an entropy measure…a training value x* for which a model-in-training
Φ
provides low information according to the entropy measure]. Maximum entropy refers to the principle of determining the distribution that maximizes uncertainty, and the maximized uncertainty is interpreted as corresponding to low information.
…to obtain the model after-convergence
Φ
Das, [0038], “After the machine-learning algorithm 110 achieves a predetermined performance level (e.g., 100% agreement with the outcomes associated with the training dataset 112), the machine-learning algorithm 110 may be executed using data that is not in the training dataset 112.”
Das discloses a machine-learning algorithm that achieves a predetermined performance level [obtain the model after-convergence
Φ
].
Zhao, Das, and the instant application are analogous art because they are all directed to active learning (Zhao [0002]; Das, [0001]).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao with Das in order to quickly achieve desired performance levels with a small amount of labeled data. “Active learning, (i.e., maximum entropy-based heuristics and batch active learning) may be used to quickly achieve classifier performance with a small number of target samples being labeled” (Das, [0023]).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Zhao in view of Das as applied to claim 1 above, and further in view of Huszar et al. (US 20180240031), hereinafter Huszar.
Regarding Claim 4:
As discussed above Zhao in view of Das teach [the] active learning classifier engine of claim 1, and but do not explicitly disclose:
wherein the information measure is BALD
However, in the same field, analogous art Huszar teaches:
wherein the information measure is BALD
Huszar, [0010], “In some implementations the diversity metric is a Bayesian Active Learning by Disagreement (BALD) score. An unlabeled data object that satisfies the diversity metric is an informative object. The method may include identifying several informative objects.”
Huszar discloses that BALD is used to identify informative objects [information measure is BALD].
Zhao, Das, Huszar, and the instant application are analogous art because they are all directed to active learning (Huszar, [0009]).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao and Das with Huszar to implement BALD in order to identify information about the datapoint. “The BALD criterion aims at maximizing the mutual information between the newly acquired labelled example and the parameters of the neural network” and “the system may use the BALD criterion to analyze how much more is there to gain from any new example to be labeled. For example, the system may evaluate BALD on each of a universe of unlabeled objects and determine the maximum BALD. The maximal BALD score on the outstanding unlabeled objects should decrease over time. Accordingly, the system may monitor the BALD score of the items selected by active learning and terminate the iterations when this falls below a certain value.” (Huszar, [0028], [0030]).
Claims 10-12 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao in view of Das, and further in view of Nguyen et al. (US 20220083878), hereinafter Nguyen.
Regarding Claim 10:
Zhao discloses:
An active learning classifier engine for classifying an observation under a label using machine learning
Zhao, [0008], “One example aspect of the present disclosure is directed to a computer-implemented method for performing active learning on a training dataset that comprises a plurality of unlabeled datapoints and a plurality of labeled datapoints. The method includes, for each of one or more training iterations: training, by one or more computing devices, a machine-learned classifier model…obtaining, by the one or more computing devices, a respective label for each unlabeled datapoint included in the label gathering slots to transform the unlabeled datapoints included in the label gathering slots into labeled datapoints.”
Zhao discloses a computer system for active learning using a machine-learned classifier model [active learning classifier engine] to label unlabeled datapoints [classifying an observation under a label using machine learning].
the active learning classifier engine comprising one or more processors executing instructions from one or more memories to implement:
Zhao, [0081], “The training computing system 150 includes one or more processors 152 and a memory 154…The memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations.”
a model builder engine configured to operate on a data set
D
t
r
a
i
n
i
n
g
and
D
p
o
o
l
for N epochs and produce a model-after-convergence
Φ
Zhao, [0101], “In some implementations, obtaining a label can include providing the unlabeled datapoints selected for the label gathering slots to an expert and receiving the labeled datapoints that have been assigned a label by an expert. The labels obtained for each unlabeled datapoint can be used to update the training dataset 209 by removing the unlabeled datapoints selected for the label gathering slots 207 from the unlabeled datapoints and adding the labeled datapoints assigned a label by an expert to the labeled datapoints. The process 200 can be continued in an iterative manner by retraining the machine learned classifier 201 using the updated training dataset.”
Zhao discloses using an expert [consult an oracle] in order to help label unlabeled datapoints that is used to update and train the model [produce a model after-convergence
Φ
].
evaluate uncertainty in a trial classification of a plurality of samples from
D
p
o
o
l
, wherein the plurality of samples are not in
D
t
r
a
i
n
i
n
g
, wherein the trial classification is based on a model-in-training
Zhao [0038], “As described above, certain implementations of the method can include iterating the method to improve the machine learning model by selecting unlabeled datapoints that the classifier is more uncertain about labeling, based in part on the confidence values, to assign a label.”
Zhao discloses uncertainty [evaluate uncertainty] for the unlabeled datapoints [a plurality of samples from
D
p
o
o
l
, wherein the plurality of samples are not in
D
t
r
a
i
n
i
n
g
] that are selected by iteratively improving [in a trial classification…wherein the trial classification is based on a model-in-training] the machine learning model.
Zhao does not explicitly disclose:
wherein convergence corresponds to a classification accuracy measure exceeding an accuracy threshold
a classification engine configured to use the model after-convergence
Φ
to act on the observation to produce the label
wherein the model builder engine is further configured to: identify a tentative decision boundary between at least two classes
determine, based on the uncertainty, a plurality of information values respectively corresponding to the plurality of samples
select a second plurality of samples as a first number of top-ranked samples of the plurality of samples
wherein the second plurality of samples is approximately uniformly distributed along an extent of the tentative decision boundary
update the model-in-training based on the second plurality of samples
However, in the same field, analogous art Das teaches:
wherein convergence corresponds to a classification accuracy measure exceeding an accuracy threshold
Das, [0038], “After the machine-learning algorithm 110 achieves a predetermined performance level (e.g., 100% agreement with the outcomes associated with the training dataset 112), the machine-learning algorithm 110 may be executed using data that is not in the training dataset 112. The trained machine-learning algorithm 110 may be applied to new datasets to generate annotated data.”
Das discloses a trained machine-learning algorithm as achieving a predetermined performance level [convergence corresponds to a classification accuracy measure exceeding an accuracy threshold].
a classification engine configured to use the model after-convergence
Φ
to act on the observation to produce the label
As cited above in para. 38, Das discloses using the trained machine-learning algorithm [model after-convergence
Φ
] to new datasets to generate annotated data [act on the observation to produce the label].
update the model-in-training based on the second plurality of samples
As cited above in para. 38, Das discloses using the trained machine-learning algorithm [model after-convergence
Φ
] to new datasets to generate annotated data [act on the observation to produce the label].
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao with Das in order to quickly achieve desired performance levels with a small amount of labeled data. “Active learning, (i.e., maximum entropy-based heuristics and batch active learning) may be used to quickly achieve classifier performance with a small number of target samples being labeled” (Das, [0023]).
Zhao in view of Das do not explicitly disclose:
wherein the model builder engine is further configured to: identify a tentative decision boundary between at least two classes
evaluate uncertainty in a trial classification of a plurality of samples from
D
p
o
o
l
, wherein the plurality of samples are not in
D
t
r
a
i
n
i
n
g
, wherein the trial classification is based on a model-in-training
determine, based on the uncertainty, a plurality of information values respectively corresponding to the plurality of samples
select a second plurality of samples as a first number of top-ranked samples of the plurality of samples
wherein the second plurality of samples is approximately uniformly distributed along an extent of the tentative decision boundary
However, in the same field, analogous art Nguyen teaches:
wherein the model builder engine is further configured to: identify a tentative decision boundary between at least two classes
Nguyen, [0072], “FIG. 3 is a diagram illustrating a distribution of samples with respect to a decision boundary. Referring to FIG. 3, a high-confidence sample group (high-confidence discriminative instances) already contains discriminatory features and thus is non-informative, whereas features of a sample group near the decision boundary contain unknown information and thus are more informative. In conclusion, as illustrated in FIG. 3, when instances are selected near the decision boundary, best performance can be obtained, compared to when instances are randomly selected.”
In FIG. 3 and para. 72, Nguyen depicts a decision boundary between two classifications [identify a tentative decision boundary between at least two classes].
determine, based on the uncertainty, a plurality of information values respectively corresponding to the plurality of samples
Nguyen, [0068], “A general approach to active learning includes calculating an uncertainty score uncertain(x) using entropy according to Equation 3 below, assigning a label to instances with a low uncertainty score, and randomly selecting instances for further annotation.”
In para. 68, Nguyen discloses uncertainty for the data instances [based on the uncertainty]. In view of para. 72 cited above, high-confidence samples have discriminatory features [determine…a plurality of information values respectively corresponding to the plurality of samples].
select a second plurality of samples as a first number of top-ranked samples of the plurality of samples
Nguyen, [0084], “Therefore, it can be seen that the number of instances with a confidence score near a decision boundary is quite reasonable, compared to selecting 10,000 batch data for further labeling.”
Nguyen discloses selecting a batch of 10,000 data near the decision boundary [select a second plurality of samples as a first number of top-ranked samples of the plurality of samples].
wherein the second plurality of samples is approximately uniformly distributed along an extent of the tentative decision boundary
In FIG. 3, cited above, Nguyen depicts the non-discriminative instances which are uniformly distributed along the decision boundary.
Zhao, Das, Nguyen and the instant application are analogous art because they are all directed to active learning (Nguyen [0002]).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao and Das with Nguyen to use a decision boundary and uncertainty in order to improve performance. “FIG. 3 is a diagram illustrating a distribution of samples with respect to a decision boundary. Referring to FIG. 3, a high-confidence sample group (high-confidence discriminative instances) already contains discriminatory features and thus is non-informative, whereas features of a sample group near the decision boundary contain unknown information and thus are more informative. In conclusion, as illustrated in FIG. 3, when instances are selected near the decision boundary, best performance can be obtained, compared to when instances are randomly selected” (Nguyen, [0072]). Nguyen discloses that selecting a sample group near the decision boundary achieves best performance because there is more information.
Regarding Claim 11:
As discussed above Zhao in view of Das, further in view of Nguyen teach [the] active learning classifier of claim 10, and Nguyen further discloses:
wherein the model builder engine is further configured to obtain a plurality of labels for the second plurality of samples, respectively, before the model-in-training is updated
Nguyen, [0063], “Next, the data inference unit 14 may perform inference using the updated training set and test set, which are modified by relabeling, and the updated unlabeled data set. After relabeling performed by an user, the data inference unit 14 trains a new snapshot of a model
θ
t
for some iterations t using a triplet (the updated unlabeled data set, the training set, and the test set) of the modified data set.”
Nguyen discloses performing updating the training set by relabeling, and the updated unlabeled data set [obtain a plurality of labels for the second plurality of samples, respectively] before training a new snapshot of the model on the modified data set [before the model-in-training is updated].
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao, Das, and Nguyen further with Nguyen in order to minimize external intervention. “The label inference system 10 according to an embodiment may comprise two-phase data construction flow which consists of a initial phase for constructing a reasonably modest model and a iterative phase involving incremental human-annotation in which an online update operation plays an important role in minimizing necessary external intervention” (Nguyen, [0049]).
Regarding Claim 12:
As discussed above Zhao in view of Das, further in view of Nguyen teach [the] active learning classifier of claim 11, and Zhao further discloses:
wherein the model builder engine is further configured to obtain the plurality of labels from an oracle
Nguyen, [0063], “Next, the data inference unit 14 may perform inference using the updated training set and test set, which are modified by relabeling, and the updated unlabeled data set. After relabeling performed by an user, the data inference unit 14 trains a new snapshot of a model
θ
t
for some iterations t using a triplet (the updated unlabeled data set, the training set, and the test set) of the modified data set.”
Nguyen discloses relabeling is performed by the user [obtain the plurality of labels from an oracle].
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao, Das, and Nguyen further with Nguyen in order to minimize external intervention. “The label inference system 10 according to an embodiment may comprise two-phase data construction flow which consists of a initial phase for constructing a reasonably modest model and a iterative phase involving incremental human-annotation in which an online update operation plays an important role in minimizing necessary external intervention” (Nguyen, [0049]).
Regarding Claim 17:
As discussed above Zhao in view of Das, further in view of Nguyen teach [the] active learning classifier of claim 10, and Zhao further discloses:
wherein the accuracy threshold corresponds to a 0.9 or better precision score on a benchmark data set
Zhao, [0043], “Depending on the application, the performance of the ML model can be measured in terms of different metrics such as its accuracy, area under precision-recall curve and recall at a certain precision.”
[0058], “Reported in the Appendix are two performance metrics for the classifiers trained at each time step—the area under the precision-recall curve (AUC-PR) and the recall at precision of 0.9.”
[0062], “To verify the above observations on real-world data sets, the evaluations were applied to a modified MNIST data set.”
In para. 43, Zhao discloses measuring the accuracy of the model [accuracy threshold] using a precision-recall curve. Para. 58 specifies the precision-recall performance metric of 0.9 [corresponds to a 0.9 or better precision score]. Lastly, para. 62 discloses applying the evaluations to real-world data sets [on a benchmark dataset].
Regarding Claim 18:
As discussed above Zhao in view of Das, further in view of Nguyen teach [the] active learning classifier of claim 10, and Zhao further discloses:
wherein the accuracy threshold corresponds to a 0.9 or better recall score on a benchmark data set
Zhao, [0043], “Depending on the application, the performance of the ML model can be measured in terms of different metrics such as its accuracy, area under precision-recall curve and recall at a certain precision.”
[0058], “Reported in the Appendix are two performance metrics for the classifiers trained at each time step—the area under the precision-recall curve (AUC-PR) and the recall at precision of 0.9.”
[0062], “To verify the above observations on real-world data sets, the evaluations were applied to a modified MNIST data set.”
In para. 43, Zhao discloses measuring the accuracy of the model [accuracy threshold] using a precision-recall curve. Para. 58 specifies the precision-recall performance metric of 0.9 [corresponds to a 0.9 or better recall score]. Lastly, para. 62 discloses applying the evaluations to real-world data sets [on a benchmark dataset].
Claims 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao in view of Das, and further in view of Nguyen as applied to claim 10 above, and further in view of Shekar et al. (US 20220253630), hereinafter Shekar.
Regarding Claim 15:
As discussed above Zhao in view of Das, further in view of Nguyen teach [the] active learning classifier of claim 10, and Zhao further discloses:
wherein the first number of top-ranked samples is more than about 50 and the first number of top-ranked samples is not more than about 1000
Shekar, [0140], “In each of the 10 active learning cycles, 64 samples are selected in the case of detection tasks and 25 samples in the case of NER, from unlabeled dataset for labeling”
Shekar discloses selection of 64 samples [the first number of top-ranked samples is more than about 50 and the first number of top-ranked samples is not more than about 1000]
Zhao, Das, Nguyen, Shekar, and the instant application are analogous art because they are all directed to active learning (Shekar, [0140]).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao, Das, and Nguyen with Shekar in order to increase the performance of the model. “According to some embodiments, the system is configured to make an iterative selection of the samples from an unlabeled pool Xu, which would maximally increase the performance metric of the underlying model Θ (i.e., the object detection network) until the annotation budget custom-character is consumed” (Shekar, [0093]).
Regarding Claim 16:
As discussed above Zhao in view of Das, further in view of Nguyen teach [the] active learning classifier of claim 10, and Zhao further discloses:
wherein the N epochs correspond to 100 epochs or less
Shekar, [0140], “Both the Faster-RCNN model and BiLSTM-CRF models are trained for 10 epochs on the labeled set in an active learning cycle. In each of the 10 active learning cycles…The system runs 10 episodes of these active learning cycles to train the policy network.”
Shekar, discloses 10 epochs for their active learning cycle [100 epochs or less].
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao, Das, and Nguyen with Shekar in order to increase the performance of the model. “According to some embodiments, the system is configured to make an iterative selection of the samples from an unlabeled pool Xu, which would maximally increase the performance metric of the underlying model Θ (i.e., the object detection network) until the annotation budget custom-character is consumed” (Shekar, [0093]).
Claims 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhao in view of Das, and further in view of Nguyen as applied to claims 17 and 18 above, respectively, and further in view of Ghorbani et al. (US 20220156519), hereinafter Ghorbani.
Regarding Claim 19:
As discussed above Zhao in view of Das, further in view of Nguyen teach [the] active learning classifier of claim 17, and Zhao further discloses:
wherein the benchmark data set is CIFAR-10, CIFAR-100, or Caltech-256
Ghorbani, [0045], “at process 410, the ADS module may receive, via a data interface, a training dataset of unlabeled data. For example, the unlabeled data may be standard datasets of images such as but not limited to CIFAR-10”
Zhao, Das, Nuygen, Ghorbani and the instant application are analogous art because they are all directed to active learning (Ghorbani, [0003]).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao, Das, and Nguyen with Ghorbani to use CIFAR-10 in order to evaluate the performance of the model using a standardized dataset. “As noted above, in some embodiments, FIG. 5A shows results comparing the performances of the ADS-based batch active learning method, the core-set selection method, the entropy method and the random method when the baseline dataset used is the CIFAR-10 dataset” (Ghorbani, [0060]). As disclosed by Ghorbani, using a standardized baseline dataset allows the performance evaluation of multiple methods in order to determine the best performance.
Regarding Claim 20:
As discussed above Zhao in view of Das, further in view of Nguyen teach [the] active learning classifier of claim 18, and Zhao further discloses:
wherein the benchmark data set is CIFAR-10, CIFAR-100, or Caltech-256
Ghorbani, [0045], “at process 410, the ADS module may receive, via a data interface, a training dataset of unlabeled data. For example, the unlabeled data may be standard datasets of images such as but not limited to CIFAR-10”
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Zhao, Das, and Nguyen with Ghorbani to use CIFAR-10 in order to evaluate the performance of the model using a standardized dataset. “As noted above, in some embodiments, FIG. 5A shows results comparing the performances of the ADS-based batch active learning method, the core-set selection method, the entropy method and the random method when the baseline dataset used is the CIFAR-10 dataset” (Ghorbani, [0060]). As disclosed by Ghorbani, using a standardized baseline dataset allows the performance evaluation of multiple methods in order to determine the best performance.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEVEN PHUNG whose telephone number is (703) 756-1499. The examiner can normally be reached Monday-Thursday: 9:00AM-4:00PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KAMRAN AFSHAR can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/STEVEN PHUNG/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights