Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/14/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description:
Fig. 1 and 7 element 122.
Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claim 5 and 8 is objected to because of the following informalities:
Claim 5 recites “...wherein based on a determination that all of the subgroup the subgroup performance measures” should recite “wherein based on a determination that all of the subgroup of the subgroup performance measures.
Claim 8 line 2 recites “ vali date” should recite “validate” without spacing.
Appropriate correction is required.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-4 and 6-11 of the instant application are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-3 and 7-13 of U.S. Patent No. 11,610,152 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims limitations in U.S. Patent No. 11,610,152 B2 are substantially similar as highlighted by the difference in the table below. The claims of the instant application claims 1-4 and 6-11 are anticipated by claims 1-3 and 7-13 of U.S. Patent No. 11,610,152 B2, as highlighted by the difference table as follow:
Instant Application 18/176,985
U.S. Patent No. 11,610,152 B2
Claim 1
Claim 1
A system, comprising:
A system, comprising:
a memory that stores computer executable components; and
a memory that stores computer executable components; and
a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise:
a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise:
a model training component that trains a machine learning model to perform an inferencing task on an initial medical dataset comprising medical data samples;
a grouping component that generates different subgroups of the medical data samples based on different metadata factors, wherein the inferencing task relates to evaluating a medical condition and wherein the different metadata factors comprise different clinical and non-clinical factors;
a performance evaluation component evaluates performance of a machine learning model trained to perform an inferencing task regarding an assessment of medical data samples, wherein the performance evaluation component determines subgroup performance measures for different subgroups of the medical data samples grouped based on different metadata factors comprising clinical and non-clinical metadata factors, and wherein the subgroup performance measures reflect measures of performance accuracy of the machine learning model with respect to the different subgroups; and
a performance evaluation component that determines subgroup performance measures for the different subgroups of the medical data samples, wherein the subgroup performance measures reflect measures of performance accuracy of the machine learning model with respect to the different subgroups of the medical data samples; and
an approval regulation component that determines whether the machine learning model meets an acceptable level of performance for deployment in a field environment based on whether the subgroup performance measures respectively satisfy a threshold subgroup performance measure.
an approval regulation component that determines whether the machine learning model meets an acceptable level of performance for deployment in a field environment based on whether the subgroup performance measures respectively satisfy a threshold subgroup performance measure.
Instant Application 18/176,985
U.S. Patent No. 11,610,152 B2
Claim 2
Claim 2
The system of claim 1, wherein the subgroup performance measures respectively comprise uncertainty estimate values representative of a degree of uncertainty in the performance accuracy of the machine learning model with respect to the different subgroups, and wherein the threshold subgroup performance measure comprises a maximum uncertainty value.
The system of claim 1, wherein the subgroup performance measures respectively comprise uncertainty estimate values representative of a degree of uncertainty in the performance accuracy of the machine learning model with respect to the different subgroups of the medical data samples, and wherein the threshold subgroup performance measure comprises a maximum uncertainty value.
Instant Application 18/176,985
U.S. Patent No. 11,610,152 B2
Claim 3
Claim 3
The system of claim 1, wherein the subgroup performance measures respectively comprise lower prediction bound values of the performance accuracy of the machine learning model with respect to the different subgroups, and wherein the threshold subgroup performance measure comprises a minimum lower prediction bound value.
The system of claim 1, wherein the subgroup performance measures respectively comprise lower prediction bound values of the performance accuracy of the machine learning model with respect to the different subgroups of the medical data samples, and wherein the threshold subgroup performance measure comprises a minimum lower prediction bound value.
Instant Application 18/176,985
U.S. Patent No. 11,610,152 B2
Claim 4
Claim 7
The system of claim 1, wherein based on a determination that a subgroup of the different subgroups has a subgroup performance measure that fails to satisfy the threshold subgroup performance measure, the approval regulation component disapproves the machine learning model as having the acceptable level of performance for deployment in the field environment on new data samples included in the subgroup.
The system of claim 1, wherein based on a determination that a subgroup of the different subgroups of the medical data samples has a subgroup performance measure that fails to satisfy the threshold subgroup performance measure, the approval regulation component disapproves the machine learning model as having the acceptable level of performance for deployment in the field environment on new data samples included in the subgroup.
Instant Application 18/176,985
U.S. Patent No. 11,610,152 B2
Claim 6
Claim 8
The system of claim 1, wherein the computer executable components further comprise:
The system of claim 1, wherein the computer executable components further comprise:
an active learning component that identifies underperforming subgroups of the different subgroups of the medical data samples based on respective subgroup performance measures associated with the underperforming subgroups failing to satisfy the threshold subgroup performance measure; and
an active learning component that identifies underperforming subgroups of the different subgroups of the medical data samples based on respective subgroup performance measures associated with the underperforming subgroups failing to satisfy the threshold subgroup performance measure; and
an active sampling component that retrieves additional data samples for the underperforming subgroups from a collection of population data samples and adds the additional data samples to at least one of a training dataset, a test dataset, a validation dataset or a regulatory validation dataset.
an active sampling component that retrieves additional data samples for the underperforming subgroups from a collection of population data samples and adds the additional data samples to at least one of a training dataset, test dataset, validation dataset or a regulatory validation dataset
Instant Application 18/176,985
U.S. Patent No. 11,610,152 B2
Claim 7
Claim 9
The system of claim 6, wherein the computer executable components further comprise: a training component the updates the machine learning model using at least one of the training dataset, the test dataset, the validation dataset or the regulatory validation dataset, resulting in an updated machine learning model, and wherein the performance evaluation component further updates the respective subgroup performance measures based on new measures of performance accuracy of the updated machine learning model with respect to the underperforming subgroups.
The system of claim 8, wherein the model training component further updates the machine learning model using at least one of the training dataset, the test dataset, the validation dataset or the regulatory validation dataset, resulting in an updated machine learning model, and wherein the performance evaluation component further updates the respective subgroup performance measures based on new measures of performance accuracy of the updated machine learning model with respect to the underperforming subgroups.
Instant Application 18/176,985
U.S. Patent No. 11,610,152 B2
Claim 8
Claim 10
The system of claim 7, wherein the active sampling component continues to retrieve the additional data samples and the model training component continues to train, update and validate the machine learning model using the additional data samples until all of the subgroup performance measures respectively satisfy the threshold subgroup performance measure or a maximum amount, by cost or count, of the additional data samples authorized for retrieval has been reached.
The system of claim 9, wherein the active sampling component continues to retrieve the additional data samples and the model training component continues to train, update and validate the machine learning model using the additional data samples until all of the subgroup performance measures respectively satisfy the threshold subgroup performance measure or a maximum amount, by cost or count, of the additional data samples authorized for retrieval has been reached.
Instant Application 18/176,985
U.S. Patent No. 11,610,152 B2
Claim 9
Claim 11
The system of claim 6, wherein the active sampling component further determines a difficulty score for the underperforming subgroups, and wherein the active sampling component further selects the additional data samples that maximize a change to the difficulty score.
The system of claim 8, wherein the active sampling component further determines a difficulty score for the underperforming subgroups, and wherein the active sampling component further selects the additional data samples that maximize a change to the difficulty score.
Instant Application 18/176,985
U.S. Patent No. 11,610,152 B2
Claim 10
Claim 12
The system of claim 6, wherein the active sampling component determines priority scores for potential new data samples based on the respective subgroup performance measures of the underperforming subgroups that the potential new data samples respectively belong, and wherein the active sampling component further selects the additional data samples from the potential new data samples based on the priority scores.
The system of claim 8, wherein the active sampling component determines priority scores for potential new data samples based on the respective subgroup performance measures of the underperforming subgroups that the potential new data samples respectively belong, and wherein the active sampling component further selects the additional data samples from the potential new data samples based on the priority scores.
Instant Application 18/176,985
U.S. Patent No. 11,610,152 B2
Claim 11
Claim 13
The system of claim 6, wherein the active sampling component further determines an amount of the additional data samples to retrieve using an entitlement function, including first amount of additional training data samples of the additional data samples and a second amount of additional validation data samples of the additional data samples.
The system of claim 8, wherein the active sampling component further determines an amount of the additional data samples to retrieve using an entitlement function, including first amount of additional training data samples of the additional data samples and a second amount of additional validation data samples of the additional data samples.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
STEP 1
Claims 1-11 are a system type claim. Claims 12-19 are a method type claim. Therefore, claims 1-19 are directed to either a process, machine, manufacture or composition of matter.
Claim 20 is signal per se type claim. Thus, is not directed to either a process, machine, manufacture or composition of matter.
Regarding claim 1: 2A Prong 1:
(mental process – of evaluating the performance of a machine learning model trained to perform an inferencing task regarding an assessment of medical data samples can be performed by the human mind with the help of pen and paper. For example a human can evaluate the performance of the machine learning model while is performing inference regarding medical data and can determines subgroup performance measures for different subgroups of the medical data grouped based on different metadata factors (e.g., evaluation )).
(mental process – of determines whether the machine learning model meets an acceptable level of performance for deployment in a field environment based on whether the subgroup performance measures respectively satisfy a threshold subgroup performance measure can be performed by the human mind with the help of pen and paper. For example, a human can evaluate the performance of the machine learning model and comparing it to a threshold by observing how the model performs while being deployed in production environment (e.g., evaluation and observation)).
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
A system, comprising: (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
a memory that stores computer executable components; and (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise: (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
a performance evaluation component...; and an approval regulation component that (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above.
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
A system, comprising: (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
a memory that stores computer executable components; and (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise: (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
a performance evaluation component...; and an approval regulation component that (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
The additional elements as disclosed above in combination of the abstract idea are not sufficient to amount to significantly more than the judicial exception as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above.
Regarding claim 2: Depends on claim 1, thus the rejection of claim 1 is incorporated.2A Prong 1: none.
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the subgroup performance measures respectively comprise uncertainty estimate values representative of a degree of uncertainty in the performance accuracy of the machine learning model with respect to the different subgroups, and wherein the threshold subgroup performance measure comprises a maximum uncertainty value (The specification of data to be stored is understood to be a field of use limitation. See MEPE 2106.05(h)).
Regarding claim 3: Depends on claim 1, thus the rejection of claim 1 is incorporated.2A Prong 1: none.
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the subgroup performance measures respectively comprise lower prediction bound values of the performance accuracy of the machine learning model with respect to the different subgroups, and wherein the threshold subgroup performance measure comprises a minimum lower prediction bound value (The specification of data to be stored is understood to be a field of use limitation. See MEPE 2106.05(h)).
Regarding claim 4: Depends on claim 1, thus the rejection of claim 1 is incorporated.2A Prong 1:
wherein based on a determination that a subgroup of the different subgroups has a subgroup performance measure that fails to satisfy the threshold subgroup performance measure, the approval regulation (mental process – of disapproves the machine learning model as having the acceptable level of performance for deployment in the field environment on new data samples included in the subgroup based on a determination that a subgroup of the different subgroups has a subgroup performance measure that fails to satisfy the threshold subgroup performance measure can be performed by the human mind with the help of pen and paper. For example, a human can deny the machine learning model as having the acceptable level of performance for deployment in the production environment on new data samples included in the subgroup based on determining that a subgroup of the different subgroups has a subgroup performance measure that fails to satisfy the threshold subgroup performance measure (e.g., evaluation and observation )).
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
...component... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 5: Depends on claim 1, thus the rejection of claim 1 is incorporated.2A Prong 1:
wherein based on a determination that all of the subgroup the subgroup performance measures satisfy the threshold subgroup performance measure, (mental process – of approves the machine learning model as having the acceptable level of performance for deployment in the field environment based on a determination that all of the subgroup the subgroup performance measures satisfy the threshold subgroup performance measure can be performed by the human mind with the help of pen and paper. For example, a human can determine if a machine learning model is to be deployed in production environment based on performance thresholds (e.g., evaluation and observation )).
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
...the approval regulation component approves... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 6: Depends on claim 1, thus the rejection of claim 1 is incorporated.2A Prong 1:
(mental process – of identifies underperforming subgroups of the different subgroups of the medical data samples based on respective subgroup performance measures associated with the underperforming subgroups failing to satisfy the threshold subgroup performance measure can be performed by the human mind with the help of pen and paper. For example, a human can identify underperforming datasets based on performance measures such as thresholds (e.g., evaluation)).
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
wherein the computer executable components further comprise: (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
an active learning component that... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
an active sampling component that... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
(This is understood to be insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g)).
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above.
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the computer executable components further comprise: (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
an active learning component that... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
an active sampling component that... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
( This is directed to well understood, routine of storing and retrieving information in memory. See MPEP 2106.05 (d)(II)).
The additional elements as disclosed above in combination of the abstract idea are not sufficient to amount to significantly more than the judicial exception as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above.
Regarding claim 7: Depends on claim 6, thus the rejection of claim 6 is incorporated.2A Prong 1: none.
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the computer executable components further comprise: a training component the updates the machine learning model using at least one of the training dataset, the test dataset, the validation dataset or the regulatory validation dataset, resulting in an updated machine learning model, and wherein the performance evaluation component further updates the respective subgroup performance measures based on new measures of performance accuracy of the updated machine learning model with respect to the underperforming subgroups (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 8: Depends on claim 7, thus the rejection of claim 7 is incorporated.2A Prong 1: none.
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the active sampling component continues to retrieve the additional data samples and the model training component continues to train, update and validate the machine learning model using the additional data samples until all of the subgroup performance measures respectively satisfy the threshold subgroup performance measure or a maximum amount, by cost or count, of the additional data samples authorized for retrieval has been reached (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 9: Depends on claim 6 thus the rejection of claim 6 is incorporated.2A Prong 1:
...determines a difficulty score for the underperforming subgroups, and ...further selects the additional data samples that maximize a change to the difficulty score (mental process – of determines a difficulty score for the underperforming subgroups and selects the additional data samples that maximize a change to the difficulty score can be performed by the human mind with the help of pen and paper (e.g., evaluation and judgment )).
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the active sampling component further ... wherein the active sampling component... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 10: Depends on claim 6 thus the rejection of claim 6 is incorporated.2A Prong 1:
(mental process – of determining priority scores for potential new data samples based on the respective subgroup performance measures of the underperforming subgroups that the potential new data samples respectively belong, and further selecting the additional data samples from the potential new data samples based on the priority scores can be performed by the human mind with the help of pen and paper (e.g., evaluation and judgment )).
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the active sampling component... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 11: Depends on claim 6 thus the rejection of claim 6 is incorporated.2A Prong 1:
(mathematical concept – of determines an amount of the additional data samples to retrieve using an entitlement function. Applicant specification [0046] and equation 1 states the entitlement function is as calculated as follow
PNG
media_image1.png
94
525
media_image1.png
Greyscale
(e.g., mathematical calculation)).
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the active sampling component further... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 12: 2A Prong 1:
evaluating, (mental process – of evaluating the performance of a machine learning model trained to perform an inferencing task regarding an assessment of medical data samples can be performed by the human mind with the help of pen and paper. For example a human can evaluate the performance of the machine learning model while is performing inference regarding medical data and can determines subgroup performance measures for different subgroups of the medical data grouped based on different metadata factors (e.g., evaluation )).
determining, (mental process – of determines whether the machine learning model meets an acceptable level of performance for deployment in a field environment based on whether the subgroup performance measures respectively satisfy a threshold subgroup performance measure can be performed by the human mind with the help of pen and paper. For example, a human can evaluate the performance of the machine learning model and comparing it to a threshold by observing how the model performs while being deployed in production environment (e.g., evaluation and observation)).
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
...by a system operatively coupled to a processor,... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
...by the system,... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above.
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
...by a system operatively coupled to a processor,... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
...by the system,... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
The additional elements as disclosed above in combination of the abstract idea are not sufficient to amount to significantly more than the judicial exception as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above.
Regarding claim 13: See rejection of claim 2, same rational applies.
Regarding claim 14: Depends on claim 12, thus the rejection of claim 12 is incorporated.2A Prong 1:
wherein based on a determination that a subgroup of the different subgroups has a subgroup performance measure that fails to satisfy the threshold subgroup performance measure, the method further comprises: disapproving, (mental process – of disapproving the machine learning model as having the acceptable level of performance for deployment in the field environment on new data samples included in the subgroup based on a determination that a subgroup of the different subgroups has a subgroup performance measure that fails to satisfy the threshold subgroup performance measure can be performed by the human mind with the help of pen and paper. For example, a human can deny a machine learning model to be deployed based on the level of performance (e.g., evaluation and observation )).
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
...by the system,... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 15: Depends on claim 12, thus the rejection of claim 12 is incorporated.2A Prong 1:
wherein based on a determination that all of the subgroup the subgroup performance measures satisfy the threshold subgroup performance measure, the method further comprises: approving, (mental process – of approving the machine learning model as having the acceptable level of performance for deployment in the field environment based on a determination that all of the subgroup the subgroup performance measures satisfy the threshold subgroup performance measure can be performed by the human mind with the help of pen and paper. For example, a human can approve a machine learning model to be deployed based on the level of performance (e.g., evaluation and observation ))
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
...by the system,... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 16: Depends on claim 12, thus the rejection of claim 12 is incorporated.2A Prong 1:
identifying, (mental process – of identifies underperforming subgroups of the different subgroups of the medical data samples based on respective subgroup performance measures associated with the underperforming subgroups failing to satisfy the threshold subgroup performance measure can be performed by the human mind with the help of pen and paper. For example, a human can identify underperforming datasets based on performance measures such as thresholds (e.g., evaluation)).
retrieving,
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
...by the system,... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
retrieving, (This is understood to be insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g). Further, this is directed to well understood, routine of storing and retrieving information in memory. See MPEP 2106.05 (d)(II)).
Regarding claim 17: Depends on claim 16, thus the rejection of claim 16 is incorporated.2A Prong 1: none.
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
further comprising: training, by the system, the machine learning model using at least one of the training dataset, the test dataset, the validation dataset or the regulatory validation dataset, resulting in an updated machine learning model; and (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
updating, by the system, the respective subgroup performance measures based on new measures of performance accuracy of the updated machine learning model with respect to the underperforming subgroups (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 18: Depends on claim 16, thus the rejection of claim 16 is incorporated.2A Prong 1: none.
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
further comprising: continuing, by the system, the retrieving, the training and the updating until all of the subgroup performance measures respectively satisfy the threshold subgroup performance measure or a maximum amount, by cost or count, of the additional data samples authorized for retrieval has been reached (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 19: Depends on claim 16, thus the rejection of claim 16 is incorporated.2A Prong 1:
further comprising: determining, (mental process – of determines a difficulty score for the underperforming subgroups and selects the additional data samples that maximize a change to the difficulty score can be performed by the human mind with the help of pen and paper (e.g., evaluation and judgment )).
2A Prong 2 and 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
...by the system,... (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
Regarding claim 20:
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least of the four categories of patent eligible subject matter because the broadest reasonable interpretation of the “machine-readable storage medium” of claim 20 encompasses signals per se. Paragraph [0093] of the specification states that “The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device” and can be “can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. ” which clearly includes propagating electromagnetic waves. The further recitation of “instructions” in claim 1 only serves to limit the content carried by the electromagnetic waves. As understood in light of the specification, the broadest reasonable interpretation of claim 20 encompasses signals which are not within one of the four statutory categories of invention. See MPEP 2106.03(I). It is suggested that claim 20 be amended to recite a “non-transitory” computer readable medium to overcome this rejection. 2A Prong 1:
evaluating performance of a machine learning model trained to perform an inferencing task regarding an assessment of medical data samples, wherein the evaluating comprises determining subgroup performance measures for different subgroups of the medical data samples grouped based on different metadata factors comprising clinical and non-clinical metadata factors, and wherein the subgroup performance measures reflect measures of performance accuracy of the machine learning model with respect to the different subgroups; and (mental process – of evaluating the performance of a machine learning model trained to perform an inferencing task regarding an assessment of medical data samples can be performed by the human mind with the help of pen and paper. For example a human can evaluate the performance of the machine learning model while is performing inference regarding medical data and can determines subgroup performance measures for different subgroups of the medical data grouped based on different metadata factors (e.g., evaluation )).
determining whether the machine learning model meets an acceptable level of performance for deployment in a field environment based on whether the subgroup performance measures respectively satisfy a threshold subgroup performance measure (mental process – of determines whether the machine learning model meets an acceptable level of performance for deployment in a field environment based on whether the subgroup performance measures respectively satisfy a threshold subgroup performance measure can be performed by the human mind with the help of pen and paper. For example, a human can evaluate the performance of the machine learning model and comparing it to a threshold by observing how the model performs while being deployed in production environment (e.g., evaluation and observation)).
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
A machine-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising: (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
The additional elements as disclosed above alone or in combination do not integrate the judicial exception into practical application as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above.
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
A machine-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising: (This is directed to using computers or other machinery merely as a tool to perform an existing process. See MPEP 2106.05(f)).
The additional elements as disclosed above in combination of the abstract idea are not sufficient to amount to significantly more than the judicial exception as they are mere insignificant extra solution activity in combination of generic computer functions being implemented with generic computer elements in a high level of generality to perform the disclosed abstract idea above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1,3-5, 12, 14-15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable Wubbels et al. US 10,599,984 B1 (Hereinafter Wubbels) in view of Tanaka US 2019/0164014 A1.
Regarding claim 1:
A system, comprising: (Wubbels Fig. 16 teaches a processing system element 1600).
a memory that stores computer executable components; and (Wubbels Fig. 16 teaches a memory element 1606).
a processor that executes the computer executable components stored in the memory, wherein the computer executable components comprise: (Wubbels Fig. 16 teaches a processor element 1602).
a performance evaluation component evaluates performance of a machine learning model trained to perform an inferencing task regarding an assessment of medical data samples, ( Wubbels col 4:26-28, teaches a validator module (i.e., performance evaluation component) can be used to evaluate the performance of the machine learning module and lines 29-32 teaches “model performance in pre - deployment can be evaluated based on the accuracy of inferences and / or based on the latency of inferences performed by the machine learning”. In addition, col 11:1-6, teaches the machine learning model produces an inference regarding an assessment of medical data samples such inference can be a “diagnosis of various diseases”).
wherein the performance evaluation component determines subgroup performance measures for different subgroups of the medical data samples grouped based on different metadata factors comprising clinical and non-clinical metadata factors, and wherein the subgroup performance measures reflect measures of performance accuracy of the machine learning model with respect to the different subgroups; and (Wubbels teaches the performance evaluation component can select one or more dimension (col 6:35) and evaluate the performance of the machine learning model using only using the inputs associated with the dimensions (col 6: 40-41). The dimension can include subject attributes such as race, gender, ethnicity, current health conditions, health history, age, location of residence; input attributes such as modality of the input, a field of view of the input and an eye position; and an attribute of the input device (see col 6: 9-26). Thus, this represent type of metadata related with the sample that are being grouped based on different metadata factors. Further, Wubbels col: 6: 26-31 teaches the metadata factors include clinical metadata factor such as subject suffering with HIV/AIDS and non-clinical metadata factor such as the subject age. In addition Wubbels col 5:64-66, teaches the validator module (i.e., performance evaluation component) can ensure that the dataset set used to calculate the performance of the machine learning module covers all predefined categories of patient, thus teaching the proposed system can be used to evaluate the performance of the machine leaning model with respect to the different subgroups).
an approval regulation component that determines whether the machine learning model meets an acceptable level of performance for deployment in a field environment inference by deciding to deploy the machine learning model upon validating the performance of the machine learning model”).
While Wubbels teaches the machine learning model is deployed in the field environment based on an acceptable level of performance. Wubbels does not explicitly teaches the machine learning model meets an acceptable level of performance based on whether the subgroup performance measures respectively satisfy a threshold subgroup performance measure.
Nonetheless, Tanaka teaches the following:
an approval regulation component that determines whether the machine learning model meets an acceptable level of performance based on whether the subgroup performance measures respectively satisfy a threshold subgroup performance measure (Tanaka Fig. 18 and [0088-90] teaches the evaluation data is divided into groups and subgroups; performance evaluation is calculated for each subgroup; each subgroup performance evaluation value is compared with a “threshold value set for the concerned subgroup” (i.e., subgroup performance measures); and teaches the subgroups values must satisfy thresholds in order for the model to be updated/deployed).
Tanaka is also in the same field of endeavor as Wubbels (performance in machine learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of subgroup performance metrics, as being disclosed and taught by Tanaka, in the system taught by Wubbels to yield the predictable results of generating a new model having improved performance ( see Tanaka [0079]).
Regarding claim 3:
Wubbels teaches The system of claim 1. Wubbels does not specify wherein the subgroup performance measures respectively comprise lower prediction bound values of the performance accuracy of the machine learning model with respect to the different subgroups, and wherein the threshold subgroup performance measure comprises a minimum lower prediction bound value.
Nonetheless, Tanaka teaches the following:
wherein the subgroup performance measures respectively comprise lower prediction bound values of the performance accuracy of the machine learning model with respect to the different subgroups, and wherein the threshold subgroup performance measure comprises a minimum lower prediction bound value (Tanaka teaches each subgroup has a threshold value set and further teaches “If the first performance evaluation value of each subgroup ...is equal to greater than the first threshold value ...then it is determined that the existing model should be updated with the new model”, thus under the broasted reasonable interpretation such threshold therefore can be view as a “minimum lower prediction bound value” for the subgroup (see Fig. 18 teaches & [0088]).
Regarding claim 4:
Wubbels and Tanaka teach The system of claim 1. Wubbels specifically teaches wherein based on a determination that a subgroup of the different subgroups has a subgroup performance measure that fails to satisfy the threshold subgroup performance measure, the approval regulation component disapproves the machine learning model as having the acceptable level of performance for deployment in the field environment on new data samples included in the subgroup (Wubbels col 14:39-46, col 14:47-57 , teaches the validator component (i.e., approval regulation component) disapproves the machine learning model as having the acceptable level of performance for deployment in the field environment based on exceeding a predetermined value (i.e., threshold)).
Regarding claim 5:
Wubbels and Tanaka teach The system of claim 1. Wubbels specifically teaches ...the approval regulation component approves the machine learning model as having the acceptable level of performance for deployment in the field environment (Wubbels col 4: 28-32 teaches the machine learning model performance is evaluated based on its inference prior to being deployed, therefore this suggest only if the machine learning model has an acceptable level of performance it would be deployed).
While Wubbels does not disclose wherein based on a determination that all of the subgroup the subgroup performance measures satisfy the threshold subgroup performance measure.
Tanaka overcome this deficiencies and teaches the following:
wherein based on a determination that all of the subgroup the subgroup performance measures satisfy the threshold subgroup performance measure, the approval regulation component approves the machine learning model as having the acceptable level of performance... (Tanaka Fig. 18 and [0088-90] teaches the evaluation data is divided into groups and subgroups; performance evaluation is calculated for each subgroup; each subgroup performance evaluation value is compared with a “threshold value set for the concerned subgroup” (i.e., subgroup performance measures); and teaches the subgroups values must satisfy thresholds in order for the model to be updated/deployed).
Regarding claim 12:
A method, comprising: (Wubbels Fig. 15 teaches a method).
evaluating, by a system operatively coupled to a processor, performance of a machine learning model trained to perform an inferencing task regarding an assessment of medical data samples, ( Wubbels Fig. 16 teaches a system – element 1600 operatively coupled to a processor -element 1006. In addition, Wubbels col 4:26-28, teaches a validator module can be used to evaluate the performance of the machine learning module and lines 29-32 teaches “model performance in pre - deployment can be evaluated based on the accuracy of inferences and / or based on the latency of inferences performed by the machine learning”. In addition, col 11:1-6, teaches the machine learning model produces an inference regarding an assessment of medical data samples such inference can be a “diagnosis of various diseases”).
wherein the performance evaluation component determines subgroup performance measures for different subgroups of the medical data samples grouped based on different metadata factors comprising clinical and non-clinical metadata factors, and wherein the subgroup performance measures reflect measures of performance accuracy of the machine learning model with respect to the different subgroups; and (Wubbels teaches the performance evaluation component can select one or more dimension (col 6:35) and evaluate the performance of the machine learning model using only using the inputs associated with the dimensions (col 6: 40-41). The dimension can include subject attributes such as race, gender, ethnicity, current health conditions, health history, age, location of residence; input attributes such as modality of the input, a field of view of the input and an eye position; and an attribute of the input device (see col 6: 9-26). Thus, this represent type of metadata related with the sample that are being grouped based on different metadata factors. Further, Wubbels col: 6: 26-31 teaches the metadata factors include clinical metadata factor such as subject suffering with HIV/AIDS and non-clinical metadata factor such as the subject age. In addition Wubbels col 5:64-66, teaches the validator module (i.e., performance evaluation component) can ensure that the dataset set used to calculate the performance of the machine learning module covers all predefined categories of patient, thus teaching the proposed system can be used to evaluate the performance of the machine leaning model with respect to the different subgroups).
determining, by the system, whether the machine learning model meets an acceptable level of performance for deployment in a field environment such as a gender, a race, an ethnicity, and an age, a health condition, a type of device used to generate the input, field of view of the input, etc., to identify an area in which the machine learning model is underperforming” (Col 12: 49-56) such that “the processor can increase an accuracy and can decrease a latency of generating the inference by deciding to deploy the machine learning model upon validating the performance of the machine learning model”).
While Wubbels teaches the machine learning model is deployed in the field environment based on an acceptable level of performance. Wubbels does not explicitly teaches the machine learning model meets an acceptable level of performance based on whether the subgroup performance measures respectively satisfy a threshold subgroup performance measure.
Nonetheless, Tanaka teaches the following:
determining, by the system, whether the machine learning model meets an acceptable level of performance based on whether the subgroup performance measures respectively satisfy a threshold subgroup performance measure (Tanaka Fig. 18 and [0088-90] teaches the evaluation data is divided into groups and subgroups; performance evaluation is calculated for each subgroup; each subgroup performance evaluation value is compared with a “threshold value set for the concerned subgroup” (i.e., subgroup performance measures); and teaches the subgroups values must satisfy thresholds in order for the model to be updated/deployed).
Regarding claim 14: is a method type claim comprising limitations similar to those of claim 4 , therefore is rejected under the same rational of claim 4.
Regarding claim 15: is a method type claim comprising limitations similar to those of claim 5 , therefore is rejected under the same rational of claim 5.
Regarding claim 20: is A machine-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations comprising limitations similar to those of claim 12 , therefore is rejected under the same rational of claim 12.
Claims 2 and 13 are rejected under 35 U.S.C. 103 as being unpatentable Wubbels, Tanaka in view of Erenrich et al. US 10,325,224 B1 (hereinafter Erenrich).
Regarding claim 2:
Wubbels and Tanaka teaches The system of claim 1. Wubbels and Tanaka does not specify wherein the subgroup performance measures respectively comprise uncertainty estimate values representative of a degree of uncertainty in the performance accuracy of the machine learning model with respect to the different subgroups, and wherein the threshold subgroup performance measure comprises a maximum uncertainty value.
Nonetheless, Erenrich teaches the following:
wherein the subgroup performance measures respectively comprise uncertainty estimate values representative of a degree of uncertainty in the performance accuracy of the machine learning model with respect to the different subgroups, and wherein the threshold subgroup performance measure comprises a maximum uncertainty value ( Erenrich col 18: 29-31 & 61-65 teaches “An uncertainty score of a data example may indicate a 30 level of uncertainty in a model's evaluation of the data example” , thus the uncertainty score (i.e., uncertainty estimate values) presents how uncertain the machine learning model is regarding its evaluation inference for all or some of the data examples (i.e., different subgroups). In addition, Erenrich col 18: 34-36 teaches the “uncertainty score may represent a confidence level in the determination result” and col 20: 43-55 teaches “a certainty level of the model, e.g., a factor based on the uncertainty scores of the data examples in the training data set... may be a maximum uncertainty score”).
Erenrich is also in the same field of endeavor as Wubbels and Tanaka (machine learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of uncertainty scores, as being disclosed and taught by Erenrich, in the system taught by Wubbels and Tanaka to yield the predictable results of developing and improving machine learning models (Erenrich col 3: 34-35).
Regarding claim 13: is a method type claim comprising limitations similar to those of claim 2 , therefore is rejected under the same rational of claim 2.
Claims 6, 7, 11 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable Wubbels, Tanaka in further view of Horvitz et al. US 2008/0319727 A1 (hereinafter Horvitz).
Regarding claim 6:
Wubbels and Tanaka teach The system of claim 1. Wubbels specifically teaches wherein the computer executable components further comprise: an learning component that identifies underperforming subgroups of the different subgroups of the medical data samples based on respective subgroup performance measures associated with the underperforming subgroups failing to satisfy the threshold subgroup performance measure; (Wubbels col 9:18-25, teaches identifying underperforming dimension (subgroups) based on dimension performance measure failing to satisfy a threshold and teaches retrieving additional sample for training).
Neither Wubbels or Tanaka explicitly teaches an active learning component and an active sampling component that retrieves additional data samples for the underperforming subgroups from a collection of population data samples and adds the additional data samples to at least one of a training dataset, a test dataset, a validation dataset or a regulatory validation dataset.
Nonetheless, Horvitz teaches the following:
an active learning component... (Horvitz Fig. 3 element 302 and [0038] teach an analysis component (i.e., active learning component) that facilitates analysis of how well or how poorly the model runs based on the current sets of data).
an active sampling component that retrieves additional data samples for the underperforming subgroups from a collection of population data samples and adds the additional data samples to at least one of a training dataset, a test dataset, a validation dataset or a regulatory validation dataset (Horvitz Fig. 4 element 106 and [0043] teaches a sampling component for sampling data of the data store and teaches a selection component -element 402 that “that facilitates ...the collection of additional data ...for improving on model execution over an existing set of data”. Furthermore, Horvitz [0046] teaches the selection component can selects related data from data source from underperforming area (i.e., where the model performs poorly) and [0048] teaches the new set of data (i.e., additional data samples) is added to training dataset ( i.e., the original set of data)).
Horvitz is also in the same field of endeavor as Wubbels and Tanaka (machine learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of the sampling, analysis and selection components, as being disclosed and taught by Horvitz, in the system taught by Wubbels and Tanaka to yield the predictable results of “enhance the model performance where the model testing is deemed to be poor” (Horvitz Abstract).
Regarding claim 7:
Wubbels, Tanaka and Horvitz teach The system of claim 6. Wubbels specifically teaches wherein the computer executable components further comprise: a training component the updates the machine learning model using at least one of the training dataset, the test dataset, the validation dataset or the regulatory validation dataset, resulting in an updated machine learning model, and wherein the performance evaluation component further updates the respective subgroup performance measures based on new measures of performance accuracy of the updated machine learning model with respect to the underperforming subgroups (A person skilled in the relevant art will recognize a “retraining module” is used to update the machine learning model for which Wubbels col 15: 26-30 teaches a retraining module that can train the machine learning model using multiple input, thus resulting in an updated model and Wubbels col 9:18-25, further teaches dimension (subgroup) retraining based on underperformance).
Regarding claim 11:
Wubbels, Tanaka and Horvitz teach The system of claim 6. Horvitz specifically teaches wherein the active sampling component further determines an amount of the additional data samples to retrieve using an entitlement function, including first amount of additional training data samples of the additional data samples and a second amount of additional validation data samples of the additional data samples ( Horvitz [0044] teaches a data store that can store various types and kinds of data (denoted DATA1, DATA2, DATA3, . . . , DATAN, where N is an integer) and teaches the selection component (i.e., sampling component ) can retrieve additional data for which the amount can be determined by the particular implementation, thus suggesting an evaluation function can be used to retrieve additional amount of data).
Regarding claim 16:
Wubbels and Tanaka teach The method of claim 12. Wubbels specifically teaches further comprising: identifying, by the system, underperforming subgroups of the different subgroups of the medical data samples based on respective subgroup performance measures associated with the underperforming subgroups failing to satisfy the threshold subgroup performance measure; (Wubbels col 9:18-25, teaches identifying underperforming dimension (subgroups) based on dimension performance measure failing to satisfy a threshold).
While Wubbels teaches retrieving additional sample for training. Neither Wubbels or Tanaka explicitly teach or suggest retrieving, by the system, additional data samples for the underperforming subgroups from a collection of population data samples and adds the additional data samples to at least one of a training dataset, a test dataset, a validation dataset or a regulatory validation dataset.
Nonetheless, Horvitz teaches the following:
and retrieving, by the system, additional data samples for the underperforming subgroups from a collection of population data samples and adds the additional data samples to at least one of a training dataset, a test dataset, a validation dataset or a regulatory validation dataset (Horvitz Fig. 4 element 106 and [0043] teaches a sampling component for sampling data of the data store and teaches a selection component -element 402 that “that facilitates ...the collection of additional data ...for improving on model execution over an existing set of data”. Furthermore, Horvitz [0046] teaches the selection component can selects related data from data source from underperforming area (i.e., where the model performs poorly) and [0048] teaches the new set of data (i.e., additional data samples) is added to training dataset ( i.e., the original set of data)).
Regarding claim 17:
Wubbels, Tanaka and Horvitz teach The method of claim 16. Wubbels specifically teaches further comprising: training, by the system, the machine learning model using at least one of the training dataset, the test dataset, the validation dataset or the regulatory validation dataset, resulting in an updated machine learning model; and updating, by the system, the respective subgroup performance measures based on new measures of performance accuracy of the updated machine learning model with respect to the underperforming subgroups (A person skilled in the relevant art will recognize a “retraining module” is used to update the machine learning model for which Wubbels col 15: 26-30 teaches a retraining module that can train the machine learning model using multiple input, thus resulting in an updated model and Wubbels col 9:18-25, further teaches dimension (subgroup) retraining based on underperformance).
Claims 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable Wubbels, Tanaka, Horvitz in further view of Plumbley et al. US 2021/0027864 A1 (hereinafter Plumbley).
Regarding claim 8:
Wubbels, Tanaka and Horvitz teach The system of claim 7.
While Wubbels col 9:18-25 teaches evaluate the performance measure of subgroup, identifying an underperforming subgroup; retrieving additional data samples for a subgroup; retraining the model and validating the retrained model.
Neither Wubbels, Tanaka or Horvitz specifically teach wherein the active sampling component continues to retrieve the additional data samples and the model training component continues to train, update and validate the machine learning model using the additional data samples until all of the subgroup performance measures respectively satisfy the threshold subgroup performance measure or a maximum amount, by cost or count, of the additional data samples authorized for retrieval has been reached.
Nonetheless Plumbley teaches the following:
wherein the active sampling component continues to retrieve the additional data samples and the model training component continues to train, update and validate the machine learning model using the additional data samples until all of the subgroup performance measures respectively satisfy the threshold subgroup performance measure or a maximum amount, by cost or count, of the additional data samples authorized for retrieval has been reached (Examiner will like to emphasize, the claim as presented recites “continues to train, update and validate the machine learning model using the additional data samples until all of the subgroup performance measures respectively satisfy the threshold subgroup performance measure or a maximum amount, by cost or count, of the additional data samples authorized for retrieval has been reached” (emphasis added) for which Plumbley teaches continues to train, update and validate the machine learning model using the additional data samples until a maximum amount, by count, of the additional data samples authorized for retrieval has been reached. Specifically, Plumbley [0003] teaches training, update and validating a machine learning model using selected compounds ( i.e., additional data) and teaches using an iterative procedure/feedback loop that may be performed for generating the ML model until it is considered to be validly trained. Further, Plumbey [0036] teaches “the method may include further training the property model by iterating over the steps of generating, validating and updating the property model until it is determined the property model has been validly trained or when a stopping criterion has been reached or met” such suitable stopping criterion include for example “maximum number of iterations, plateau in property model score, a peak in property model score, and the like”).
Plumbey is also in the same field of endeavor as Wubbels, Tanaka and Horvitz (machine learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of continuing to train, update and validate a machine learning model using additional data samples until a maximum amount, by count, of the additional data samples authorized for retrieval has been reached, as being disclosed and taught by Plumbey, in the system taught by Wubbels, Tanaka and Horvitz to yield the predictable results of improve the machine learning model by utilizing an iterative procedure/feedback loop (see Plumbey [0007] & [0015]).
Regarding claim 18:
Wubbels, Tanaka and Horvitz teach The method of claim 16.
While Wubbels col 9:18-25 teaches evaluate the performance measure of subgroup, identifying an underperforming subgroup; retrieving additional data samples for a subgroup; retraining the model and validating the retrained model. Neither Wubbels, Tanaka or Horvitz specifically teach further comprising: continuing, by the system, the retrieving, the training and the updating until all of the subgroup performance measures respectively satisfy the threshold subgroup performance measure or a maximum amount, by cost or count, of the additional data samples authorized for retrieval has been reached.
Nonetheless Plumbley teaches the following:
further comprising: continuing, by the system, the retrieving, the training and the updating until all of the subgroup performance measures respectively satisfy the threshold subgroup performance measure or a maximum amount, by cost or count, of the additional data samples authorized for retrieval has been reached (Examiner will like to emphasize, the claim as presented recites “continues to train, update and validate the machine learning model using the additional data samples until all of the subgroup performance measures respectively satisfy the threshold subgroup performance measure or a maximum amount, by cost or count, of the additional data samples authorized for retrieval has been reached” (emphasis added) for which Plumbley teaches continues to train, update and validate the machine learning model using the additional data samples until a maximum amount, by count, of the additional data samples authorized for retrieval has been reached. Specifically, Plumbley [0003] teaches training, update and validating a machine learning model using selected compounds ( i.e., additional data) and teaches using an iterative procedure/feedback loop that may be performed for generating the ML model until it is considered to be validly trained. Further, Plumbey [0036] teaches “the method may include further training the property model by iterating over the steps of generating, validating and updating the property model until it is determined the property model has been validly trained or when a stopping criterion has been reached or met” such suitable stopping criterion include for example “maximum number of iterations, plateau in property model score, a peak in property model score, and the like”).
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable Wubbels, Tanaka, Horvitz in further view of Ghalaty et al. US 10,867,245 B1 (hereinafter Ghalaty).
Regarding claim 9:
Wubbels, Tanaka and Horvitz teach The system of claim 6. Neither Wubbels, Tanaka or Horvitz teach wherein the active sampling component further determines a difficulty score for the underperforming subgroups, and wherein the active sampling component further selects the additional data samples that maximize a change to the difficulty score .
Nonetheless, Ghalaty analogues in the art teaches the following:
wherein the active sampling component further determines a difficulty score for the underperforming subgroups, and wherein the active sampling component further selects the additional data samples that maximize a change to the difficulty score (Ghalaty col 11: 21-24 teaches determining “relevancy scores” (i.e., a difficulty score) for training data that include plurality of features (i.e., subgroups) and col 11: 53-62 teaches selecting additional datasets that has a “threshold amount of influence on a machine learning model” (col 14: 14)).
Ghalaty is also in the same field of endeavor as Wubbels, Tanaka and Horvitz (machine learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of relevancy scores as difficulty scores as being disclosed and taught by Ghalaty, in the system taught by Wubbels, Tanaka and Horvitz to yield the predictable results for facilitating prediction model training (see Ghalaty Fig. 1).
Regarding claim 19:
Wubbels, Tanaka and Horvitz teach The method of claim 16. Neither Wubbels, Tanaka or Horvitz teach further comprising: determining, by the system, a difficulty score for the underperforming subgroups; and selecting, by the system, the additional data samples that maximize a change to the difficulty score.
Nonetheless, Ghalaty analogues in the art teaches the following:
further comprising: determining, by the system, a difficulty score for the underperforming subgroups; and selecting, by the system, the additional data samples that maximize a change to the difficulty score (Ghalaty col 11: 21-24 teaches determining “relevancy scores” (i.e., a difficulty score) for training data that include plurality of features (i.e., subgroups) and col 11: 53-62 teaches selecting additional datasets that has a “threshold amount of influence on a machine learning model” (col 14: 14)).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable Wubbels, Tanaka in view of Hughes US 2020/0202171 A1 (hereinafter Hughes) as cited in the information disclosure statement (IDS) dated 04/14/2026.
Regarding claim 10:
Wubbels, Tanaka and Horvitz teach The system of claim 6. Neither Wubbels, Tanaka or Horvitz explicitly teach wherein the active sampling component determines priority scores for potential new data samples based on the respective subgroup performance measures of the underperforming subgroups that the potential new data samples respectively belong, and wherein the active sampling component further selects the additional data samples from the potential new data samples based on the priority scores.
However, Hughes teaches the following:
wherein the active sampling component determines priority scores for potential new data samples based on the respective subgroup performance measures of the underperforming subgroups that the potential new data samples respectively belong, and wherein the active sampling component further selects the additional data samples from the potential new data samples based on the priority scores ( Hughes [0011-12] teaches implementing an active sampling algorithm. Further Hughes, [0189] teaches new data sample is retrieved from the unnoted data or the set of training candidate and its pre-processed. Further, Hughes [0190] teaches the new data sample is streamed to generate vector of scores (i.e., priority scores). To be specific, Hughes [0194] teaches calculate necessary metric (e.g., scores) such as entropy, minimum margin and so on from the classification predictions and teaches these score can be compared to the scores already stored for each type of sampler. In the event that a prediction meets certain criteria, it is kept and the results are stored in one or more of the priority queues. Further, [0197] teaches as new prediction is received, each of the priority queues evaluate the sampling score (i.e., priority score) for the new prediction and selects additional sample based on the sampling score).
Hughes is also in the same field of endeavor as Wubbels and Tanaka (machine learning). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the functionality of vector scores, as being disclosed and taught by Hughes, in the system taught by Wubbels and Tanaka to yield the predictable results of “speed predictions and reduce hardware requirements” (see Hughes [0119]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GISEL G FACCENDA whose telephone number is (703)756-1919. The examiner can normally be reached Monday - Friday 8:00 am - 4:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Al Kawsar can be reached at (571) 270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/G.G.F./Examiner, Art Unit 2127
/ABDULLAH AL KAWSAR/Supervisory Patent Examiner, Art Unit 2127
Prosecution Insights