DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/09/2026 has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claim 15 is non-elected via election by original presentation rejection (See Final Rejection dated 12/08/2025 for rejection rationale).
Claims 1-15 are pending and have been examined, claims 1-14 are rejected with claim 15 withdrawn from consideration.
This action is in reply to the papers filed on 03/09/2026 (effective filing date 10/23/2019).
Information Disclosure Statement
The information disclosure statement(s) submitted: 03/04/2024, has/have been considered by the Examiner and made of record in the application file.
Amendment
The present Office Action is based upon the original patent application filed on 03/04/2024 as modified by the amendment filed on 03/09/2026.
Election by Original Presentation
Newly submitted claim 15 is directed to an invention that is independent or distinct from the invention originally claimed and examined for the following reasons:
Group I. Claims 1-14 drawn to: 1. (Currently Amended) A computer-implemented method for identifying a stopping point of a machine learning-assisted review process, comprising: calculating, using a machine learning model, a first uncertain rank count and a second uncertain rank count; and based on a first estimated error rate, a second estimated error rate, the first uncertain rank count, the second uncertain rank count, and a target error rate, displaying, in a display of a computing device, an indication that the stopping point has been reached. (G06N3/091)
Group II. Claim 15 drawn to: 15. (New) A computer-implemented method for identifying a stopping point of a machine learning-assisted review process, comprising: (a) receiving, by a computing device, a corpus of electronic documents for review; (b) training, by the computing device, a machine learning model to predict relevance ranks for documents in the corpus of electronic documents; (c) executing the trained machine learning model to assign a relevance rank to each document of the corpus, wherein the relevance rank is a numerical score indicating a likelihood of being relevant; (d1) tracking, by the computing device, the coding of documents by a user and periodically updating the machine learning model in response to each coding decision; (d2) deriving, based upon classification predictions of the trained machine learning model, a first estimated error rate for a first set of recently coded documents and a second estimated error rate for a second set of recently coded documents; (e) calculating, using the trained machine learning model, (i) a first uncertain rank count comprising a count of documents within a predetermined uncertain relevance score range for a first model build, and (ii) a second uncertain rank count comprising a count of documents within a predetermined uncertain relevance score range for a second model build; (f) determining, by the computing device, whether the first estimated error rate and the second estimated error rate have each remained below a configurable target error rate for at least a predetermined number of sequential model builds, and whether the uncertain rank counts are steady or decreasing across a configurable number of previous builds; (g) in response to the determination in step (f), displaying, in a display of the computing device, an indication that the stopping point for conducting an elusion test has been reached. (G06N5/00).
The inventions are each distinct from the other because of the following reasons:
SUBCOMBINATIONS USABLE TOGETHER
Inventions I and II are related as subcombinations disclosed as usable together in a single combination. The subcombinations are distinct if they do not overlap in scope and are not obvious variants, and if it is shown that at least one subcombination is separately usable.
In the instant case, subcombination I has separate utility such as Group I (Claim 1) discloses “calculating, using a machine learning model, a first uncertain rank count and a second uncertain rank count; and based on a first estimated error rate, a second estimated error rate, the first uncertain rank count, the second uncertain rank count, and a target error rate, displaying, in a display of a computing device, an indication that the stopping point has been reached.”
Subcombination II has separate utility such as Group II (Claim 15) discloses “(a) receiving, by a computing device, a corpus of electronic documents for review; (b) training, by the computing device, a machine learning model to predict relevance ranks for documents in the corpus of electronic documents; (c) executing the trained machine learning model to assign a relevance rank to each document of the corpus, wherein the relevance rank is a numerical score indicating a likelihood of being relevant; (d1) tracking, by the computing device, the coding of documents by a user and periodically updating the machine learning model in response to each coding decision; (d2) deriving, based upon classification predictions of the trained machine learning model, a first estimated error rate for a first set of recently coded documents and a second estimated error rate for a second set of recently coded documents; (e) calculating, using the trained machine learning model, (i) a first uncertain rank count comprising a count of documents within a predetermined uncertain relevance score range for a first model build, and (ii) a second uncertain rank count comprising a count of documents within a predetermined uncertain relevance score range for a second model build; (f) determining, by the computing device, whether the first estimated error rate and the second estimated error rate have each remained below a configurable target error rate for at least a predetermined number of sequential model builds, and whether the uncertain rank counts are steady or decreasing across a configurable number of previous builds; (g) in response to the determination in step (f), displaying, in a display of the computing device, an indication that the stopping point for conducting an elusion test has been reached.”
See MPEP §806.05(d).
Because these inventions are distinct for the reasons given above and have acquired a separate status in the art because of their recognized divergent subject matter, restriction for examination purposes as indicated is proper.
Since applicant has received an action on the merits for the originally presented invention, this invention has been constructively elected by original presentation for prosecution on the merits. Accordingly, claim 15 is withdrawn from consideration as being directed to a non-elected invention. See 37 CFR 1.142(b) and MPEP § 821.03.
Terminal Disclaimer
The terminal disclaimer filed on xxx disclaiming the terminal portion of any patent granted on this application which would extend beyond the expiration date of US Pat. No. xxxx has been reviewed and has been placed in the file.
Examiner acknowledges Applicant’s filed Terminal Disclaimer to prior art patent McCauley et al. US Pat. No. 5,930,775. A terminal disclaimer may be filed to overcome or obviate a nonstatutory double patenting rejection (37 CFR 1.321; MPEP 706.02; 1490).
Double Patenting - Withdrawn
The double patenting rejection is withdrawn per the filed terminal disclaimer noted above.
Reasons For Allowance
Prior-Art Rejection withdrawn
Claims xxx are allowed. The closest prior art (See PTO-892, Notice of References Cited) does not teach the claimed:
The invention teaches… and the prior-art teaches…, however, the prior-art does not teach…
The closest prior-art (xxx) teach the features as disclosed in Non-final Rejection (xxxx), however, these cited references do not teach and the prior-art does not teach at least the following:
determining, at a second time after associating the information corresponding to the first loyalty card with the logged location, that a second user computing device is located within a specified distance of the logged location using a second positioning system of the second user computing device; in response to determining that the second user computing device is located within the specified distance of the logged location of the first user computing device at the first time of detecting: retrieving information corresponding to a second loyalty card, the second loyalty card being associated with the merchant and the second user computing device; and displaying, by the second user computing device, data describing the second loyalty card.
Claim Rejections - 35 USC §101 - Withdrawn
Per Applicant’s amendments and arguments and considering new guidance in the MPEP, the rejections are withdrawn. Specifically, in Applicant’s Remarks (dated 03/14/2017, pgs. 8-11), Applicant traverses the 35 USC §101 rejections arguing that the amended claims recite new limitations that are not abstract, amount to significantly more, are directed to a practical application, etc… For example, Applicant argues….
In support of their arguments, Applicant cites to the following recent Fed. Cir. court cases (i.e., Alice Corp. v. CLS Bank Int’l, SRI Int’l, Inc. v. Cisco Systems, Inc., Ultramercial, Inc. v. Hulu, LLC, Berkheimer, Core Wireless, McRO, Enfish, Bascom, DDR, etc…).
Claim Rejections - 35 USC § 112
Per Applicants’ amendments/arguments, the rejections are withdrawn.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1-14 are rejected on the ground of anticipatory-nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 11,921,568.
18/595,261 – Claim 1. (Currently Amended) A computer-implemented method for identifying a stopping point of a machine learning-assisted review process, comprising:
US 11,921,568 – Claim 1. A computer-implemented method for identifying a stopping point of an active learning process, comprising:
18/595,261 – Claim 1. calculating, using a machine learning model, a first uncertain rank count and a second uncertain rank count; and
US 11,921,568 – Claim 1. calculating a first estimated error rate, a second estimated error rate, a first uncertain rank count and a second uncertain rank count; and
18/595,261 – Claim 1. based on a first estimated error rate, a second estimated error rate, the first uncertain rank count, the second uncertain rank count, and a target error rate, displaying, in a display of a computing device, an indication that the stopping point has been reached.
US 11,921,568 – Claim 1. based on the first estimated error rate, the second estimated error rate, the first uncertain rank count, the second uncertain rank count, and a target error rate, displaying, in a display of a computing device, an indication that the stopping point has been reached.
18/595,261 – Claim 2. (Currently Amended) The computer-implemented method of claim 1 further comprising calculating the first estimated error rate, wherein calculating the first estimated error rate includes receiving a coverage review indication from a user.
US 11,921,568 – Claim 2. The computer-implemented method of claim 1, wherein calculating the first estimated error rate includes receiving a coverage review indication from a user.
18/595,261 – Claim 3. (Currently Amended) The computer-implemented method of claim 1 further comprising calculating the first estimated error rate, wherein calculating the first estimated error rate includes determining whether a user has coded a minimum number of documents.
US 11,921,568 – Claim 3. The computer-implemented method of claim 1, wherein calculating the first estimated error rate includes determining whether a user has coded a minimum number of documents.
18/595,261 – Claim 4. (Currently Amended) The computer-implemented method of claim 1 further comprising calculating the first estimated error rate, wherein calculating the first estimated error rate includes determining whether a user has coded a minimum number of document groups.
US 11,921,568 – Claim 4. The computer-implemented method of claim 1, wherein calculating the first estimated error rate includes determining whether a user has coded a minimum number of document groups.
18/595,261 – Claim 5. (Original) The computer-implemented method of claim 1, wherein the target error rate is a configurable constant.
US 11,921,568 – Claim 5. The computer-implemented method of claim 1, wherein the target error rate is a configurable constant.
18/595,261 – Claim 6. (Original) The computer-implemented method of claim 1, wherein calculating the first uncertain rank count and the second uncertain rank count includes comparing uncertain rank counts across a configurable number of previous builds.
US 11,921,568 – Claim 6. The computer-implemented method of claim 1, wherein calculating the first uncertain rank count and the second uncertain rank count includes comparing uncertain rank counts across a configurable number of previous builds.
18/595,261 – Claim 7. (Original) The computer-implemented method of claim 1, wherein displaying, in the display of the computing device, the indication that the stopping point has been reached includes generating a message indicating that the stopping point has been reached; and transmitting the message via one or both of (i) a push message, and (ii) an email message.
US 11,921,568 – Claim 7. The computer-implemented method of claim 1, wherein displaying, in the display of the computing device, the indication that the stopping point has been reached includes generating a message indicating that the stopping point has been reached; and transmitting the message via one or both of (i) a push message, and (ii) an email message.
18/595,261 – Claim 8. (Currently Amended) A computing system for determining a stopping point of a machine learning process, comprising: one or more processors; and a memory storing instructions that, when executed, cause the computing system to: calculate, using a machine learning model, a first uncertain rank count and a second uncertain rank count; and based on a first estimated error rate, a second estimated error rate, the first uncertain rank count, the second uncertain rank count, and a target error rate, display, in a display of a computing device, an indication that the stopping point has been reached.
US 11,921,568 – Claim 8. A computing system for determining a stopping point of an active learning process, comprising: one or more processors; and a memory storing instructions that, when executed, cause the computing system to: calculate a first estimated error rate, a second estimated error rate, a first uncertain rank count and a second uncertain rank count; and based on the first estimated error rate, the second estimated error rate, the first uncertain rank count, the second uncertain rank count, and a target error rate, display, in a display of a computing device, an indication that the stopping point has been reached.
18/595,261 – Claim 9. (Original) The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to: receive a coverage review indication from a user.
US 11,921,568 – Claim 9. The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to: receive a coverage review indication from a user.
18/595,261 – Claim 10. (Original) The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to: determine whether a user has coded a minimum number of documents.
US 11,921,568 – Claim 10. The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to: determine whether a user has coded a minimum number of documents.
18/595,261 – Claim 11. (Original) The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to: determine whether a user has coded a minimum number of document groups.
US 11,921,568 – Claim 11. The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to: determine whether a user has coded a minimum number of document groups.
18/595,261 – Claim 12. (Original) The computing system of claim 8, wherein the target error rate is a configurable constant.
US 11,921,568 – Claim 12. The computing system of claim 8, wherein the target error rate is a configurable constant.
18/595,261 – Claim 13. (Original) The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to: compare uncertain rank counts across a configurable number of previous builds.
US 11,921,568 – Claim 13. The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to: compare uncertain rank counts across a configurable number of previous builds.
18/595,261 – Claim 14. (Original) The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to: generate a message indicating that the stopping point has been reached; and transmit the message via one or both of (i) a push message, and (ii) an email message.
US 11,921,568 – Claim 14. The computing system of claim 8, the memory including further instructions that when executed, cause the computing system to: generate a message indicating that the stopping point has been reached; and transmit the message via one or both of (i) a push message, and (ii) an email message.
The remaining independent claims contain feature like that of claim 1 and are rejected accordingly. The dependent claims are further rejected for their dependency upon a rejected independent base claim.
Claim Rejections - 35 USC § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-14 are rejected under 35 U.S.C. § 101 as being directed to non-statutory subject matter because the claimed invention is directed to an abstract idea without significantly more. These claims recite a method and a system for determining a stopping point.
Claim 1 is INELIGIBLE under 35 U.S.C. § 101
Claim 1 recites [a] computer-implemented method for identifying a stopping point of a machine learning-assisted review process, comprising: training, by a computing device and based on one or more coding decisions, a machine learning model to predict relevance ranks for documents in a corpus of electronic documents; calculating, using the trained machine learning model, a first uncertain rank count for a first model build and a second uncertain rank count for a second model build; deriving, based upon classification predictions of the trained machine learning model and document coding values, a first estimated error rate for a first set of recently coded documents and a second estimated error rate for a second set of recently coded documents; determining, by the computing device, whether the first estimated error rate and the second estimated error rate have each remained below a target error rate for at least a predetermined number of sequential model builds, and whether the first uncertain rank count and the second uncertain rank count are not increasing across a predetermined number of previous builds; and transmitting an indication that the stopping point has been reached.
The claim is directed to patent-ineligible subject matter under 35 U.S.C. § 101 because it is directed to an abstract idea (mathematical concepts and/or mental processes) and does not recite elements that integrate the abstract idea into a practical application (Step 2A, Prong 2) or amount to significantly more than the abstract idea (Step 2B).
Specifically:
The claim recites the abstract idea of calculating error rates and counts to determine when to stop a review process—a combination of mathematical calculations and mental processes (evaluating/determining).
The additional element (“computing device”) is a generic computer component performing generic functions that are well-understood, routine, and conventional.
The claim amounts to the application of the abstract idea using a generic computer as a tool.
Claim 8 is INELIGIBLE under 35 U.S.C. § 101
Claim 8 recites [a] computing system for determining a stopping point of a machine learning process, comprising: one or more processors; and a memory storing instructions that, when executed, cause the computing system to: train, based on one or more coding decisions, a machine learning model to predict relevance ranks for documents in a corpus of electronic documents; calculate, using the trained machine learning model, a first uncertain rank count for a first model build and a second uncertain rank count for a second model build; derive, based upon classification predictions of the trained machine learning model and document coding values, a first estimated error rate for a first set of recently coded documents and a second estimated error rate for a second set of recently coded documents; determine whether the first estimated error rate and the second estimated error rate have each remained below a target error rate for at least a predetermined number of sequential model builds, and whether the first uncertain rank count and the second uncertain rank count are not increasing across a predetermined number of previous builds; and transmit an indication that the stopping point has been reached.
The claim is directed to patent-ineligible subject matter under 35 U.S.C. § 101 because it is directed to an abstract idea (mathematical concepts and/or mental processes) and does not recite elements that integrate the abstract idea into a practical application (Step 2A, Prong 2) or amount to significantly more than the abstract idea (Step 2B).
Specifically:
The claim’s functional limitations recite the same abstract idea as Claim 1: calculating error rates/counts to determine when to stop a review process.
The additional structural elements (“processors,” “memory”) are generic computer components performing generic functions and are WURC.
The claim amounts to the application of the abstract idea using a generic computer, which is insufficient under Alice and its progeny.
DEPENDENT CLAIMS 2-7 and 9-14 are INELIGIBLE under 35 U.S.C. § 101
The dependent claims add additional limitations but do not cure the § 101 defects because they:
Add data gathering/input steps (claims 2-4, 9-11): receiving user indications, determining minimum document counts—these are insignificant data gathering steps per Flook and Bilski.
Add parameters (claims 5, 12): “configurable constant”—a parameter value does not transform abstract idea into practical application.
Add calculation details (claims 6, 13): comparing across previous builds—additional abstract calculation steps.
Add output formatting (claims 7, 14): transmitting via push/email—insignificant extra-solution activity (output formatting) per MPEP 2106.05(g).
All dependent claims are INELIGIBLE for the same reasons as their parent independent claims.
No Prior-art Rejection
Following an extensive search, Examiner has been unable to find a sufficient combination of references to anticipate every limitation of the independent claim. The following patent prior-art is deemed to be relevant to Applicant’s disclosure:
Hickey et al. 2020/0410440 [0062 - the stopping criteria data is used to determine whether the machine learning has reached an acceptable level of error rate]
Torkkola et al. 2004/0252027 [0042 – machine learning… stopping criteria… error…]
Glyman et al. 2019/0005389 [0110 - This training process can continue until a stopping point is reached. This stopping point may depend on an error rate, a number of training iterations, or an elapsed time.]
However, none of these references explicitly describe a method or system for identifying a stopping point of a machine learning-assisted review process, comprising: training, by a computing device and based on one or more coding decisions, a machine learning model to predict relevance ranks for documents in a corpus of electronic documents; calculating, using the trained machine learning model, a first uncertain rank count for a first model build and a second uncertain rank count for a second model build; deriving, based upon classification predictions of the trained machine learning model and document coding values, a first estimated error rate for a first set of recently coded documents and a second estimated error rate for a second set of recently coded documents; determining, by the computing device, whether the first estimated error rate and the second estimated error rate have each remained below a target error rate for at least a predetermined number of sequential model builds, and whether the first uncertain rank count and the second uncertain rank count are not increasing across a predetermined number of previous builds; and transmitting an indication that the stopping point has been reached.
While the above noted references use ML models, stopping criteria, and error rates, they do not disclose all the features as claimed in the order and combination claimed.
For at least this reason, a rejection under 35 USC 102 and/or 35 USC 103 is not warranted.
Examiner’s Response to Arguments
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Examiner’s Response: Claim Rejections – 35 USC §112
Per Applicants’ amendments/arguments, the rejections are withdrawn.
Examiner’s Response: Claim Rejections – 35 USC §101
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Regarding Claims 1-14, on page(s) 7-10 of Applicant’s Remarks (dated 03/09/2026), Applicants traverse the 35 USC §101 rejections generally arguing the following: the amended claims are not abstract, integrates the judicial exception into a practical application, and provides an inventive concept. Applicant’s primary arguments for patent eligibility is directed to integration of the abstract idea into a practical application because the claim reflects an improvement to technology.
Respectfully, the Office disagrees, and the claims remain rejected as subject matter ineligible under 35 USC §101 as outlined in the above rejection and further detailed herein below.
CLAIM PARSING
Independent Claim 1 (Method)
Limitation
Label
Claim 1. (Currently Amended) A computer-implemented method for identifying a stopping point of a machine learning-assisted review process, comprising:
Preamble
(L1) training, by a computing device and based on one or more coding decisions, a machine learning model to predict relevance ranks for documents in a corpus of electronic documents;
Data-gathering / ML training
(L2) calculating, using the trained machine learning model, a first uncertain rank count for a first model build and a second uncertain rank count for a second model build;
Mathematical calculation
(L3) deriving, based upon classification predictions of the trained machine learning model and document coding values, a first estimated error rate for a first set of recently coded documents and a second estimated error rate for a second set of recently coded documents;
Mathematical calculation
(L4) determining, by the computing device, whether the first estimated error rate and the second estimated error rate have each remained below a target error rate for at least a predetermined number of sequential model builds, and whether the first uncertain rank count and the second uncertain rank count are not increasing across a predetermined number of previous builds; and
Mathematical comparison / evaluation / mental process
(L5) transmitting an indication that the stopping point has been reached.
Insignificant post-solution activity (outputting result)
Independent Claim 8 (System)
Limitation
Label
Claim 8. (Currently Amended) A computing system for determining a stopping point of a machine learning process, comprising: one or more processors; and a memory storing instructions that, when executed, cause the computing system to:
Preamble + generic hardware
(L1’) train, based on one or more coding decisions, a machine learning model to predict relevance ranks for documents in a corpus of electronic documents;
Data-gathering / ML training
(L2’) calculate, using the trained machine learning model, a first uncertain rank count for a first model build and a second uncertain rank count for a second model build;
Mathematical calculation
(L3’) derive, based upon classification predictions of the trained machine learning model and document coding values, a first estimated error rate for a first set of recently coded documents and a second estimated error rate for a second set of recently coded documents;
Mathematical calculation
(L4’) determine whether the first estimated error rate and the second estimated error rate have each remained below a target error rate for at least a predetermined number of sequential model builds, and whether the first uncertain rank count and the second uncertain rank count are not increasing across a predetermined number of previous builds; and
Mathematical comparison / evaluation / mental process
(L5’) transmit an indication that the stopping point has been reached.
Insignificant post-solution activity (outputting result)
Dependent Claims Summary
Claim
Additional Limitation
Character
2, 9
Receiving a coverage review indication from a user
Data-gathering / field-of-use
3, 10
Determining whether a user has coded a minimum number of documents
Threshold check / mental process
4, 11
Determining whether a user has coded a minimum number of document groups
Threshold check / mental process
5, 12
Target error rate is a configurable constant
Descriptive characteristic of data
6, 13
Comparing uncertain rank counts across a configurable number of previous builds
Mathematical comparison
7, 14
Generating a message indicating the stopping point has been reached; transmitting via push
message and/or email
Insignificant extra-solution activity
(outputting result)
STEP 1 – Statutory Category
Claim
Category
Determination
Claims 1-7
Process (method)
✅ Falls within § 101 statutory category
Claims 8-14
Machine (system: processor + memory)
✅ Falls within § 101 statutory category
Step 1: Statutory Category: All claims satisfy Step 1. Proceed to Step 2A.
Step 2A, Prong 1: Judicial Exception: The claim recites a judicial exception—specifically, an abstract idea falling within multiple groupings under the MPEP:
Mathematical Concepts (Mathematical Relationships, Calculations, Formulas) (MPEP 2106.04(a)). The core of each independent claim is a series of mathematical calculations and threshold comparisons.
Offending Clause (L2 / L2’): “calculating, using the trained machine learning model, a first uncertain rank count for a first model build and a second uncertain rank count for a second model build”
This is a counting operation — tallying documents that fall within a numerical range (e.g., rank 40–60). Applicant’s specification confirms this at ¶ [0053]:
[0053] Generally, the method 700 depicts a metric that is easily interpreted by a project administrator, depending on the administrator's project needs, to determine when the administrator should stop review and being an elusion test. The metric may be summarized as determining whether an estimated error rate is at or below a configurable target rate, determining whether an estimated error rate has a configurable number of sequential data points at or below the target rate, and determining a number of “uncertain” documents (e.g., rank 40-60) is steady or decreasing over a configurable number of data points. When a rolling load occurs (i.e., the index size changes) then the method 700 may reset the sequential data point count to 0. …
Counting items satisfying a numerical predicate is a mathematical calculation.
Offending Clause (L3 / L3’): “deriving, based upon classification predictions of the trained machine learning model and document coding values, a first estimated error rate for a first set of recently coded documents and a second estimated error rate for a second set of recently coded documents”
The error rate is expressly defined mathematically in ¶ [0053] as:
PNG
media_image1.png
74
445
media_image1.png
Greyscale
This is a pure mathematical formula applied to data.
Offending Clause (L4 / L4’): “determining … whether the first estimated error rate and the second estimated error rate have each remained below a target error rate for at least a predetermined number of sequential model builds, and whether the first uncertain rank count and the second uncertain rank count are not increasing across a predetermined number of previous builds”
This limitation recites: (i) a threshold comparison (error rate < target rate), (ii) a sequential count check (at least N builds below threshold), and (iii) a trend comparison (uncertain count not increasing). All three are mathematical comparisons / evaluations.
Mental Processes (Observations, Evaluations, Judgments).
The determining step (L4/L4’) can also be characterized as a mental process — a human project administrator could, and historically did, perform these evaluations manually by:
Observing the error rate of recent batches of coded documents;
Comparing the error rates to a target threshold;
Noting whether the count of uncertain documents is trending downward;
Deciding that the project has stabilized.
The specification itself acknowledges this at ¶ [0049]:
[0049] As noted above, an elusion test is generally run when the project has stabilized and low-ranking documents have an acceptably low relevance rate. However, an elusion test may be run at any point during the active learning process. Thus, before an elusion test can be executed, the user must determine an appropriate stopping point. Doing so is subjective and often difficult. When an elusion test is performed too soon, then the relevance score of many documents will be uncertain, and the elusion test will lack sufficient confidence. When the elusion test is performed too late, then the reviewer may unnecessarily review too many documents. The present techniques include displaying an indication to the user when the active learning process is ready for an elusion test. In some embodiments, the indication is transmitted (e.g., by the backend server) to the user (e.g., via an email, an SMS message, a push notification, etc.).
This confirms the claimed process automates what was previously a subjective human judgment — i.e., a mental process.
Treatment of Machine Learning Training Limitation (L1/L1’)
The ML training steps (e.g., “training … a machine learning model to predict relevance ranks”) are recited at a high level of generality. No specific model architecture, training algorithm, loss function, or implementation detail is claimed. Under BRI, this is a generic invocation of ML as a tool. Critically, the focus of the claim is not on an improvement to how the model is trained, but on the post-training mathematical evaluation of error rates and uncertain rank counts to decide when to stop reviewing. The ML model is merely the source of the data being mathematically evaluated; the judicial exception resides in the calculations and comparisons applied to that data.
See Intellectual Ventures I LLC v. Erie Indemnity Co., 850 F.3d 1315 (Fed. Cir. 2017) (claims that used generic data processing / indexing as a tool to feed an abstract evaluation were directed to the abstract idea).
Prong 1 Conclusion
All independent claims (1 and 8) recite judicial exceptions under the mathematical concepts and mental processes groupings. The dependent claims (2–7, 9–14) add only: data-gathering prerequisites (claims 2–4, 9–11), data characterizations (claims 5, 12), additional mathematical comparisons (claims 6, 13), or result-outputting steps (claims 7, 14) — none of which negate the recitation of a judicial exception.
STEP 2A — PRONG 2: Integration into a Practical Application?
Under Prong 2, we evaluate whether the claim as a whole integrates the judicial exception into a practical application, looking for indicia such as:
An improvement to the functioning of a computer or other technology;
Application of the exception with a particular machine;
Transformation of an article to a different state or thing;
Other meaningful limitations beyond generally linking the exception to a technological environment.
Analysis of Additional Elements
Additional Element
Integration?
Reasoning
“by a computing device” / “one or more processors; and a memory” (Claims 1, 8)
❌ No
Generic computer components recited at the highest level of abstraction. Alice Corp.
v. CLS Bank, 573 U.S. 208 (2014).
“training … a machine learning model to predict relevance ranks” (L1/L1’)
❌ No
Recited generically without any specific architecture, algorithm, or technical improvement to the training process. Operates as mere data-gathering / pre-solution activity — the model generates the data that the abstract idea then evaluates.
See MPEP § 2106.05(g).
“transmitting an indication that the stopping point has been reached” (L5/L5’)
❌ No
Post-solution activity — merely outputting the result of the abstract analysis. MPEP § 2106.05(g). See Electric Power Group v. Alstom, 830 F.3d 1350 (Fed. Cir. 2016) (displaying results of abstract analysis is insignificant extra-solution activity).
“corpus of electronic documents” (all independent claims)
❌ No
Field-of-use limitation tying the abstract idea to eDiscovery / document review. Does not integrate. MPEP § 2106.05(h).
Claimed Technological Improvement?
Applicant’s primary arguments for patent eligibility are directed to integration of the abstract idea into a practical application because the claim reflects an improvement to technology. Remarks pgs. 8-10. Applicant argues that amended claim 1 recites specific criteria for assessing ML model readiness in monitoring error rates and uncertain rank counts across sequential model builds to determine when the model has achieved sufficient predictive accuracy thereby improving functioning of ML-assisted document review systems by ensuring validation testing occurs only when the ML has stabilized. Remarks pg. 10.
With respect, the Office disagrees. The “improvement” described is an improvement in the accuracy or timing of a human decision (when to stop reviewing), not an improvement to the functioning of a computer or to a particular technology. The claims do not recite:
Reduced computational resource consumption (no metrics claimed);
Faster processing (no benchmarks or architectural changes);
Reduced memory usage or network bandwidth;
A novel model architecture or training methodology;
A specific technological mechanism achieving any improvement.
The improvement is in the outcome of a business/legal process — deciding when a document review can cease — which is an improvement to the process of conducting litigation discovery, not to computer technology per se. See BSG Tech LLC v. BuySeasons, Inc., 899 F.3d 1281, 1290 (Fed. Cir. 2018) (“The claims do not recite any specific technology for performing those functions.”); RecogniCorp, LLC v. Nintendo Co., 855 F.3d 1322 (Fed. Cir. 2017) (encoding/decoding abstract data manipulations are not improvements to technology).
Furthermore, the specification acknowledges at ¶ [0053] that the error rate formula simply counts misclassified documents — a straightforward mathematical ratio — and at ¶ [0054] notes that “the error rate may correlate with elusion test results” — indicating a statistical correlation, not a technical mechanism.
Prong 2 Conclusion
The additional elements, individually and in ordered combination, amount to:
1. Generic computer implementation (processor, memory, computing device, display);
2. Pre-solution data-gathering activity (training ML model, receiving documents, tracking coding);
3. Post-solution output activity (transmitting/displaying indication);
4. Field-of-use limitation (eDiscovery document corpus).
None of these integrate the judicial exception into a practical application.
Step 2B: Significantly More (WURC Analysis): Because the claim is directed to an abstract idea and does not integrate it into a practical application, the analysis proceeds to Step 2B.
Analysis: The additional elements do NOT amount to significantly more than the judicial exception.
Well-Understood, Routine, Conventional (WURC): The additional claim elements beyond the abstract idea are:
“a computing device”
“a display”
“displaying… an indication”
“calculating” (computer-performed calculations)
These elements are WURC in the relevant art (computer-implemented document review systems, eDiscovery technology). Factual support:
Generic computing components: The specification describes conventional computing systems ([0080-0096], Figure 10). The client device 1002 includes generic components: processor 1050, RAM 1052, input devices 1054, display 1056. These are standard computer components performing standard functions (processing, storing, displaying data).
Conventional calculation/display functions: [0038-0043] describe using computers to display documents and collect user input—standard computing operations. [0086-0087] describe conventional machine learning and neural network technologies.
Simply claiming "machine learning" in a patent claim is not patent-eligible because it is considered an abstract idea. To be eligible, the claim must go beyond the generic application of machine learning and show a specific improvement to computer technology or a practical, non-abstract application. Simply training a machine learning model on data is often considered routine and well-understood in the field, and does not, by itself, transform the claim into a patent-eligible invention.
No atypical implementation: The specification provides no evidence that the claimed calculating and displaying are performed in an unconventional manner. The calculations follow standard formulas [0050-0053]. The display is a standard UI element ([0051], Figure 7 block 722).
Field evidence: Per Berkheimer, the record must include factual evidence that elements are not WURC to survive a § 101 rejection. Here, the specification itself describes the computer implementation as conventional. Courts have repeatedly held that performing calculations on generic computers and displaying results are routine. Elec. Power Grp. v. Alstom S.A., 830 F.3d 1350 (Fed. Cir. 2016) (collecting and displaying information on generic computer is abstract); Credit Acceptance Corp. v. Westlake Services, 859 F.3d 1044 (Fed. Cir. 2017) (using generic computer components to perform abstract idea is insufficient).
Elec. Power Grp. v. Alstom S.A., 830 F.3d 1350 (Fed. Cir. 2016) held that a patent for a method of monitoring and alarming about the health of an electric power grid was patent-ineligible. The court reasoned that merely collecting, analyzing, and displaying information—even information specific to a technical field—on a generic computer was an abstract idea. The court found no "inventive concept" in the specific implementation details that elevated it beyond routine data gathering and presentation.
Credit Acceptance Corp. v. Westlake Services, 859 F.3d 1044 (Fed. Cir. 2017) similarly found patents related to a method for determining loan application decisions using a multi-tiered financing system to be abstract ideas. The court emphasized that merely using generic computer components or common software functions (like retrieving data, running calculations, and displaying results) to perform an otherwise abstract business process does not add enough substance to make the claim patent-eligible.
Unconventional Arrangement/Application:
The claim does not recite an unconventional arrangement or application of known elements. The process is a straightforward application: calculate values, compare to threshold, display result. This is a conventional data processing workflow.
No Meaningful Limitation:
The limitations do not impose meaningful constraints on practicing the abstract idea. The abstract idea (determining stopping point via error rate calculations) could be performed manually; adding generic computer implementation does not transform it into patent-eligible subject matter. Alice, 573 U.S. at 223.
No Other Considerations:
The claim does not add: specific unconventional steps; improvements to another technology; application to a particular machine beyond generic computers; or other elements that courts have found constitute “significantly more.”
Conclusion for Step 2B: The additional elements do NOT amount to significantly more than the abstract idea. They are WURC elements that amount to generic computer implementation of the abstract idea.
CONCLUSION FOR CLAIM 1
Claim 1 is INELIGIBLE under 35 U.S.C. § 101.
Basis for Rejection: The claim is directed to patent-ineligible subject matter under 35 U.S.C. § 101 because it is directed to an abstract idea (mathematical concepts and/or mental processes) and does not recite elements that integrate the abstract idea into a practical application (Step 2A, Prong 2) or amount to significantly more than the abstract idea (Step 2B).
Specifically:
The claim recites the abstract idea of calculating error rates and counts to determine when to stop a review process—a combination of mathematical calculations and/or mental processes (evaluating/determining).
The additional elements (“computing device,” “display,” “displaying”) are generic computer components performing generic functions and are well-understood, routine, and conventional.
The claim amounts to the application of the abstract idea using a generic computer as a tool.
CONCLUSION FOR CLAIM 8
Claim 8 is INELIGIBLE under 35 U.S.C. § 101.
Basis for Rejection: The claim is directed to patent-ineligible subject matter under 35 U.S.C. § 101 because it is directed to an abstract idea (mathematical concepts and/or mental processes) and does not recite elements that integrate the abstract idea into a practical application (Step 2A, Prong 2) or amount to significantly more than the abstract idea (Step 2B).
Specifically:
The claim’s functional limitations recite the same abstract idea as Claim 1: calculating error rates/counts to determine when to stop a review process.
The additional structural elements (“processors,” “memory”) are generic computer components performing generic functions and are WURC.
The claim amounts to the application of the abstract idea using a generic computer, which is insufficient under Alice and its progeny.
DEPENDENT CLAIMS 2-7 and 9-14
The dependent claims add additional limitations but do not cure the § 101 defects because they:
Add data gathering/input steps (claims 2-4, 9-11): receiving user indications, determining minimum document counts—these are insignificant data gathering steps per Flook and Bilski.
Add parameters (claims 5, 12): “configurable constant”—a parameter value does not transform abstract idea into practical application.
Add calculation details (claims 6, 13): comparing across previous builds—additional abstract calculation steps.
Add output formatting (claims 7, 14): transmitting via push/email—insignificant extra-solution activity (output formatting) per MPEP 2106.05(g).
All dependent claims are INELIGIBLE for the same reasons as their parent independent claims.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
PERTINENT PRIOR ART – Patent Literature
The prior-art made of record and considered pertinent to applicant's disclosure.
See Information Disclosure Statement by Applicant (IDS) filed 03/04/2024.
Hickey et al. 2020/0410440 [0062 - the stopping criteria data is used to determine whether the machine learning has reached an acceptable level of error rate]
Torkkola et al. 2004/0252027 [0042 – machine learning… stopping criteria… error…]
Glyman et al. 2019/0005389 [0110 - This training process can continue until a stopping point is reached. This stopping point may depend on an error rate, a number of training iterations, or an elapsed time.]
PERTINENT PRIOR ART – Non-Patent Literature (NPL)
The NPL prior-art made of record and considered pertinent to applicant's disclosure.
See Information Disclosure Statement by Applicant (IDS) filed 03/04/2024.
A review of data mining applications in crime. By: Hassani, Hossein; Huang, Xu; Silva, Emmanuel S.; In: Statistical Analysis & Data Mining, Jun2016.
A Bayesian Failure Prediction Network Based on Text Sequence Mining and Clustering. By: Chang, Wenbing; Xu, Zhenzhong; You, Meng; In: Entropy, Dec2018.
THIS ACTION IS MADE FINAL
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW T. SITTNER whose telephone number is (571) 270-7137 and email: matthew.sittner@uspto.gov. The examiner can normally be reached on Monday-Friday, 8:00am - 5:00pm (Mountain Time Zone).
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sarah M. Monfeldt can be reached on (571) 270-1833.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW T SITTNER/
Primary Examiner, Art Unit 3629b
Prosecution Insights