DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is made final.
This Office Action is in response to the amendments filed on December 30th, 2025.
Claims 1, 3, 4, 5, 6, 8, 9, 10, and 12 have been amended.
Claim 2 has been cancelled.
Response to Amendment
The amendment filed December 30th, 2025 has been entered. Claims 1, and 3-12 remain pending in the application.
Response to Arguments
Response to 112(b) arguments
Applicant’s arguments, see page 7, filed December 30th, 2025, with respect to claims 6,. and 11 have been fully considered and are persuasive. The 35 U.S.C. 112(b) of claims 6, and 11 has been withdrawn.
Applicant's arguments filed December 30th, 2025 have been fully considered but they are not persuasive.
Response to 101 arguments
Applicant argues:
As amended, independent claims 1 and 12 no longer recite a mental process or an abstract idea implemented using generic computer components, but instead recite a concrete machine-learning training architecture that is rooted in computer technology and directed to improving the training of medical decision-making models.
At Step 2A, Prong 1, the Office Action asserts that the claims recite a mental process on the basis that a human could observe another human's behavior and subjectively assess reliability. However, the claims, as amended, now expressly require training a reliability level determination model that is a neural network having defined input, hidden, and output layers, where the model is trained based on collected status data and correct answer labels and outputs a reliability level of the correct answer label. The claims therefore require machine-learning model training and inference based on multi-modal operator status data, rather than a human observation or judgment. The determination of a reliability level is not performed by a human mind, but by a trained neural network that learns a relationship between operator behavior and label reliability. Accordingly, the amended claims do not recite a mental process as described in MPEP § 2106.04(a)(2)(III), and the judicial exception identified by the Examiner is no longer present.
Even if the claims were viewed as reciting an abstract idea, the claims integrate any such idea into a practical application under Step 2A, Prong 2. As amended, the claims require two distinct machine-learning models having different training objectives and different inputs and outputs, namely, a reliability level determination model trained to output a reliability level based on operator status data, and a decision making model trained based on input data, correct answer labels, and the reliability level output by the reliability level determination model. The reliability level is not merely generated or displayed, but is explicitly used as a training input to the decision making model. This interdependent training architecture constitutes a specific technical implementation that improves how medical decision-making models are trained by conditioning model learning on the reliability of human-provided labels. The claims therefore apply any alleged abstract idea in a concrete and meaningful way within a medical Al training system, rather than merely instructing that an abstract idea be performed on a generic computer.
The Office Action characterizes the collection of status data and the obtaining of input data as extra-solution activity. However, in the claims, as amended, the collected status data is not ancillary or preparatory data, but is a required training input to the reliability level determination neural network, and the resulting reliability level is a required training input to the decision making model. These steps form part of the claimed solution itself, rather than data gathering performed before or after the solution. As such, these features cannot be dismissed as extra-solution activity under MPEP §2106.05(g).
At Step 2B, the Office Action concludes that the claims lack significantly more than the alleged judicial exception because the recited machine-learning models are characterized as generic. However, as amended, the claims recite a specific arrangement in which a first neural network is trained to quantify label reliability based on operator behavior and a second machine-learning model is trained using that quantified reliability. This configuration alters the manner in which training data influences the learning behavior of the medical decision- making model and improves the robustness of model training in the presence of variable human labeling quality. The Office Action does not identify evidence that such reliability- conditioned training of medical decision-making models is well-understood, routine, or conventional. Accordingly, the amended claims recite additional elements that amount to significantly more than any alleged abstract idea.
For at least the foregoing reasons, Applicant respectfully submits that amended independent claims 1 and 12 recite patent eligible subject matter and requests that the rejection of claims 1-12 under 35 USC § 101 be withdrawn.
Examiners Response:
Applicant's arguments filed December 30th, 2025 have been fully considered but they are not persuasive. As amended, claim 1 continues to recite an abstract idea in the form of a mental process, namely evaluating the reliability of human-provided information based on observed operator behavior and using that evaluation to influence a decision-making process. The claim recites collecting operator status data reflecting a decision-making process, determining a reliability level for a correct answer label based on that data, and using the reliability level during training of a decision making model. These limitations collectively recite the abstract concept of assessing trustworthiness of information and weighting it accordingly. Merely implementing this evaluative process using a machine-learning model or neural network does not remove the claim form the mental-process category, as automating a mental process using generic computing techniques does not render it patent eligible.
The abstract idea is not integrated into a practical application. The claim does not improve the functioning a computer or machine-learning technology itself, but instead improves only the quality or weighting of training data use by a generic decision-making model. The recited machine-learning models and neural-network layers are described as a high level and perform their conventional functions of receiving data, processing data, and outputting results. No specific technical improvement to model training, computation, or system performance is claimed. Limiting the claims to a medical care field and using generic machine-learning components does not amount to significantly more than the abstract idea. Accordingly, the rejection of claim 1 under 35 U.S.C. § 101 is maintained, and the dependent claims fall therewith.
Response to 103 arguments
Applicant argues:
Applicant respectfully submits that Garnavi, Welinder and Kano, either individually or in combination, fail to disclose or suggest at least the aforementioned features recited in amended independent claims 1 and 12.
In rejecting the claims, the Office Action acknowledges that Garnavi "does not teach, train, based on the status data and the correct answer label, a reliability level determination model, which is a machine learning model which accepts status data and outputs a reliability level of the correct answer label and train, based on the input data, the correct answer label, and the reliability level, the decision making model which accepts the input data and outputs output data that is data indicating a result of the decision making." See Office Action, page 21, lines 1-6. Applicant respectfully submits that Garnavi likewise fails to disclose or suggest the features of: "add a correct answer label used for training a decision making model, which is a machine learning model used for decision making in a medical care field, in accordance with an operator's input instruction, wherein the decision making model is a multi-class classification model having an input layer for inputting input data, a hidden layer for converting the input data into output data, and an output layer for outputting the output data"; 'collect status data indicating a status of the operator while doing the work of adding, wherein the status data is data relating to an operator's operations, lines of sight, speech, and/or facial expressions that reflect a process of an operator's decision making at the time of doing the work"; and "train, based on the status data and the correct answer label, a reliability level determination model, which is a machine learning model which accepts status data and outputs a reliability level of the correct answer label, wherein the reliability level determination model is a neural network having an input layer for inputting the status data, a hidden layer for converting the status data into the reliability level, and an output layer for outputting the reliability level," as now recited in the claims.
Welinder fails to make up for the deficiencies of Garnavi. Specifically, Welinder discloses that the distributed data annotation system 110 is configured to obtain pieces of source data and store the pieces of source data using source data database 120. "Source data database 120 can obtain source data from any of a variety of sources, including content sources, customers, and any of a variety of providers of source data as appropriate to the requirements of specific applications in accordance with embodiments of the invention. In a variety of embodiments, source data database 120 includes one or more references (such as a uniform resource locator) to source data that is stored in a distributed fashion. Source data database 120 includes one or more sets of source data to be categorized using distributed data annotation server system 110. A set of source data includes one or more pieces of source data including, but not limited to, image data, audio data, signal data, and text data. In several embodiments, one or more pieces of source data in source data database 120 includes source data metadata describing attributes of the piece of source data." See Welinder, col. 8, lines 18-35. In other words, Welinder determines the reliability level based on source data. However, since source data is the subject of adding annotation, it is different from the claimed subject matter, in which the reliability level is determined from the status data, which is different from the subject of adding annotation. Accordingly, Applicant respectfully submits that Welinder fails to disclose or suggest at least the aforementioned features recited in independent claims 1 and 12.
Kano, either individually or in combination with Garnavi and Welinder, likewise fails to disclose or suggest at least the aforementioned features recited in claims 1 and 12, and as such, fails to make up for the deficiencies of Garnavi
In view of the above, Applicant respectfully submits that independent claims 1 and 12 are allowable. The dependent claims are therefore also allowable by virtue of their dependencies, as well as for the additional distinguishing features recited therein. Accordingly, reconsideration and withdrawal of the rejections are respectfully requested.
Examiners Response:
Applicant argues that the references do not teach the claims as amended. Garnavi teaches a medial decision-making model implemented as a multi-layer neural network classifier that receives input data (e.g., image data), processes the data through multiple layers (convolution, pooling, and activation), and outputs a predicted classification label. Such multi-layer neural network inherently includes an input layer, one or more hidden layers, and an output layer, as recited in amended claimm1. Explicit recitation of these layers does not patentably distinguish the claimed decision-making model form Garnavi.
Welinder teaches determining a confidence or reliability level for human-provided labels and using that reliability in model training. While Welinder does not describe a multi-layer neural network implementation, it would have been obvious to one of ordinary skill in the art to implement Welinders reliability determination using a conventional multi-layer neural network, as taught by Garnavi, doing so would have enabled the system to more effectively model the relationship between operator-related status data and a resulting reliability level. The claimed reliability level determination model therefore would have been obvious as a predictable implementation choice.
Accordingly, the combination of Garnavi and Welinder teaches or renders obvious all limitations of amended claims 1 and 12, and the rejection under 35 U.S.C. § 103 is maintained.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
To determine if a claim is directed to patent ineligible subject matter, the Court has guided the Office to apply the Alice/Mayo test, which requires:
Step 1: Determining if the claim falls within a statutory category.
Step 2A: Determining if the claim is directed to a patent ineligible judicial exception consisting of a law of nature, a natural phenomenon, or abstract idea; and Step 2A is a two prong inquiry. MPEP 2106.04(II)(A). Under the first prong, examiners evaluate whether a law of nature, natural phenomenon, or abstract idea is set forth or described in the claim. Abstract ideas include mathematical concepts, certain methods of organizing human activity, and mental processes. MPEP 2104.04(a)(2). The second prong is an inquiry into whether the claim integrates a judicial exception into a practical application. MPEP 2106.04(d).
Step 2B: If the claim is directed to a judicial exception, determining if the claim recites limitations or elements that amount to significantly more than the judicial exception. (See MPEP 2106).
Claims 1, and 3-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1, and 3-11 are directed to an information processing apparatus (a machine), and Claim 12 is directed to a computing device comprising one or more processors (a process). Therefore, Claims 1, and 3-12 are directed to a process, machine or manufacture or composition of matter.
Regarding claim 1
Step 2A Prong 1
Claim 1 recites the following mental processes, that in each case under the broadest reasonable interpretation, covers performance of the limitation in the mind (including observation, evaluation, judgement, opinion) or with the aid of pencil and paper but for recitation of generic computer components (e.g., “decision making model”, “machine learning model” and “reliability level determination model”) [see MPEP 2106.04(a)(2)(III)].
“train, based on the status data and the correct answer label, … outputs a reliability level of the correct answer label” (e.g., a human can observe facial recognitions of another human and record their expressions during a task to produce a score)
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of a “decision making model”, “machine learning model” and “reliability level determination model” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)). The Examiner notes that this is used throughout the claim limitations, and is rejected thusly for each claim which recites the same language.
Regarding the “add a correct answer label used for training a decision making model, which is a machine learning model used for decision making in a medical care field, in accordance with an operator’s input instruction” and “collect status data indicating a status of the operator while doing the work of adding, wherein the status data is data relating to an operator's operations, lines of sight, speech, and/or facial expressions that reflect a process of an operator's decision making at the time of doing the work” limitations, these additional elements are recited at a high-level of generality and amounts to extra-solution activity of inputting a user selected label for model training, i.e., pre-solution activity of data gathering (see MPEP 2106.05(g)).
Regarding the “wherein the decision making model is a multi-class classification model having an input layer for inputting input data, a hidden layer for converting the input data into output data, and an output layer for outputting the output data”, “a reliability level determination model, which is a machine learning model which accepts status data”, and “wherein the reliability level determination model is a neural network having an input layer for inputting the status data, a hidden laver for converting the status data into the reliability level, and an output layer for outputting the reliability level” which are merely defining a generic machine-learning/neural-network implementation at a high level of generality and do not specify any particular technical mechanism for achieving the claimed result. Accordingly, these amount to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)).
Regarding the “obtain input data of the decision making model” limitation, this additional element is recited at a high-level of generality and amounts to extra-solution activity of receiving the trained decision making model to use for further model training, i.e., post-solution activity of data gathering (see MPEP 2106.05(g)).
Regarding the “train, based on the input data, the correct answer label, and the reliability level, the decision making model which accepts the input data and outputs output data that is data indicating a result of the decision making” limitation, this additional element is merely describing the training step of the decision making model and is mere instructions to apply the abstract idea with generic computer components (see MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of a “decision making model”, “machine learning model” and “reliability level determination model”, which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Regarding the “add a correct answer label used for training a decision making model, which is a machine learning model used for decision making in a medical care field, in accordance with an operator’s input instruction” and “collect status data indicating a status of the operator while doing the work of adding, wherein the status data is data relating to an operator's operations, lines of sight, speech, and/or facial expressions that reflect a process of an operator's decision making at the time of doing the work” limitations, as discussed above, these additional elements are recited at a high level of generality and amounts to extra-solution activity of data gathering, for use in the claimed process. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (See MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding the “wherein the decision making model is a multi-class classification model having an input layer for inputting input data, a hidden layer for converting the input data into output data, and an output layer for outputting the output data”, “a reliability level determination model, which is a machine learning model which accepts status data”, and “wherein the reliability level determination model is a neural network having an input layer for inputting the status data, a hidden laver for converting the status data into the reliability level, and an output layer for outputting the reliability level” which are merely defining a generic machine-learning/neural-network implementation at a high level of generality and do not specify any particular technical mechanism for achieving the claimed result. Accordingly, these amount to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)).
Regarding the “obtain input data of the decision making model” limitation, this additional element is recited at a high-level of generality and amounts to extra-solution activity of post-solution activity data gathering. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (See MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding the “train, based on the input data, the correct answer label, and the reliability level, the decision making model which accepts the input data and outputs output data that is data indicating a result of the decision making” limitation, this additional element is merely describing the training step of the decision making model and is mere instructions to apply the abstract idea with generic computer components (see MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Regarding claim 2 (Cancelled)
Regarding claim 3
Step 2A Prong 1
Claim 3 does not introduce any new abstract ideas, but does inherit the same abstract ideas as claim 1.
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of a “the processing circuitry is further configured to collect, as data relating to the line of sight, reference item data which is data relating to an item on which the line of sight of the operator focuses among various items displayed on a display screen for the addition work” limitation, this additional element is recited at a high-level of generality and amounts to extra-solution activity of receiving a user’s expressions for model training, i.e., pre-solution activity of data gathering (see MPEP 2106.05(g)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional element of “the processing circuitry is further configured to collect, as data relating to the line of sight, reference item data which is data relating to an item on which the line of sight of the operator focuses among various items displayed on a display screen for the addition work”, limitation, as discussed above, this additional element is recited at a high level of generality and amounts to extra-solution activity of data gathering, for use in the claimed process. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (See MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Accordingly, at Step 2B, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Regarding claim 4
Step 2A Prong 1
Claim 4 does not introduce any new abstract ideas, but does inherit the same abstract ideas as claim 1.
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of a “the processing circuitry is further configured to collect as the reference item data, an identifier of a reference item which is an item on which the operator’s line of sight focuses” limitation, this additional element is recited at a high-level of generality and amounts to extra-solution activity of receiving a user’s expressions for model training, i.e., pre-solution activity of data gathering (see MPEP 2106.05(g)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional element of “the processing circuitry is further configured to collect as the reference item data, an identifier of a reference item which is an item on which the operator’s line of sight focuses”, limitation, as discussed above, this additional element is recited at a high level of generality and amounts to extra-solution activity of data gathering, for use in the claimed process. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (See MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Accordingly, at Step 2B, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Regarding claim 5
Step 2A Prong 1
Claim 5 does not introduce any new abstract ideas, but does inherit the same abstract ideas as claim 1.
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of “collect ability data which is data relating to an ability of the operator” limitation, this additional element is recited at a high-level of generality and amounts to extra-solution activity of observing a user’s expressions and inputting them for model training, i.e., pre-solution activity of data gathering (see MPEP 2106.05(g)).
Regarding the “train, based on the status data, the ability data and the correct answer label, the reliability level determination model which accepts the status data and the ability data, and outputs the reliability level” limitation, this additional element is merely describing the training step of the decision making model and is mere instructions to apply the abstract idea with generic computer components (see MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of a “collect ability data which is data relating to an ability of the operator”, limitation, as discussed above, this additional element is recited at a high level of generality and amounts to extra-solution activity of data gathering, for use in the claimed process. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (See MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding the “train, based on the status data, the ability data and the correct answer label, the reliability level determination model which accepts the status data and the ability data, and outputs the reliability level” limitation, this additional element is merely describing the training step of the decision making model and is mere instructions to apply the abstract idea with generic computer components (see MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Regarding claim 6
Step 2A Prong 1
Claim 6 does not introduce any new abstract ideas, but does inherit the same abstract ideas as claim 1.
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of “collect additional data which is data relating to a freshness level, a confidence level, quality, and/or a required time” limitation, this additional element is recited at a high-level of generality and amounts to extra-solution activity of observing a user’s expressions and inputting them for model training, i.e., pre-solution activity of data gathering (see MPEP 2106.05(g)).
Regarding the “train, based on the status data, the ability data, the additional data, and the correct answer label, the reliability level determination model which accepts the status data, the ability data, and the additional data, and outputs the reliability level” limitation, this additional element is merely describing the training step of the decision making model and is mere instructions to apply the abstract idea with generic computer components (see MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of a “collect additional data which is data relating to a freshness level, a confidence level, quality, and/or a required time”, limitation, as discussed above, this additional element is recited at a high level of generality and amounts to extra-solution activity of data gathering, for use in the claimed process. The courts have found limitations directed to obtaining information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (See MPEP 2106.05(d)(II), “receiving or transmitting data over a network”, "electronic record keeping," and "storing and retrieving information in memory").
Regarding the “train, based on the status data, the ability data, the additional data, and the correct answer label, the reliability level determination model which accepts the status data, the ability data, and the additional data, and outputs the reliability level” limitation, this additional element is merely describing the training step of the decision making model and is mere instructions to apply the abstract idea with generic computer components (see MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Regarding claim 7
Step 2A Prong 1
Claim 7 does not introduce any new abstract ideas, but does inherit the same abstract ideas as claim 1.
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of a “the reliability level determination model is a multi-class classification model that outputs the probability of each of multiple classes relating to a result of the decision making as the reliability level” limitation, which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)). The Examiner notes that this is merely defining the model and is just defining the kind of model to execute the type of scoring.
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of a “the reliability level determination model is a multi-class classification model that outputs the probability of each of multiple classes relating to a result of the decision making as the reliability level” limitation, which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Regarding claim 8
Step 2A Prong 1
Claim 8 inherits the same abstract idea as its parent claims, and further recites the following mathematical concepts, that in each case under the broadest reasonable interpretation, involves mathematical relationships, formulas, calculations, or algorithms implemented using generic computer components (e.g., “decision making model”, “machine learning model” and “reliability level determination model”) [see MPEP 2106.04(a)(2)(I)].
“the processing circuitry is further configured to train the decision making model by minimizing a loss function” (e.g., mathematical relationship/calculation)
“the loss function includes an error between an output of the decision making model and the correct answer label weighted by the reliability level” (e.g., mathematical relationship/calculation by comparing two outputs and deciding a probability in their difference)
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of a “processing circuitry” which is recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of a “processing circuitry”, which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Regarding claim 9
Step 2A Prong 1
Claim 9 does not introduce any new abstract ideas, but does inherit the same abstract ideas as claim 1.
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of a “the processing circuitry is further configured to display the reliability level via a display device” which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)). The Examiner notes that this is merely stating that by using a monitor or other display device the system relays output from the model.
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of a “the processing circuitry is further configured to display the reliability level via a display device”, which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Regarding claim 10
Step 2A Prong 1
Claim 10 does not introduce any new abstract ideas, but does inherit the same abstract ideas as claim 1.
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of a “the decision making is an addition of annotation of a disease candidate area to a medical image”, and “the processing circuitry is further configured to display the annotation with a color value according to the reliability level” limitations, which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)). The Examiner notes that this is merely stating that a medical image is annotated and that their reliability levels are differential by color.
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of the decision making is an addition of annotation of a disease candidate area to a medical image”, and “the processing circuitry is further configured to display the annotation with a color value according to the reliability level” limitations, which are recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Regarding claim 11
Step 2A Prong 1
Claim 11 does not introduce any new abstract ideas, but does inherit the same abstract ideas as claim 1.
Accordingly, at Step 2A, prong one, the claim recites an abstract idea.
Step 2A Prong 2
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of a “the correspondence between the reliability level and the color value is set in accordance with a difficulty level of the addition of the annotation” limitation, which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (See MPEP 2106.05(f)). The Examiner notes that this is used throughout the claim limitations, and is rejected thusly for each claim which recites the same language.
Accordingly, at Step 2A, prong two, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Step 2B
In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of a “the correspondence between the reliability level and the color value is set in accordance with a difficulty level of the addition of the annotation” limitation, which is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Accordingly, at Step 2B, the additional elements individually or in combination do not integrate the judicial exception into a practical application.
Regarding claim 12
Claim 12 recites information processing method. Which corresponds directly to the apparatus steps of claim 1, respectively, with the addition of instructions and computer-executable instructions which are insufficient to render the claims subject matter eligible for the same reasons described above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-9, and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garnavi et al. (US 10657838 B2, referred to as Garnavi) in view of Welinder et al. (US 9704106 B2, referred to as Welinder).
Regarding claim 1, Garnavi teaches, a medical information processing apparatus comprising processing circuitry configured to(Col. 3, lines 19-54: Describes that this is a system, which compromises computer hardware components. Configured to help to train individuals on identifying medical issues form pictures annotated by experts.):
add a correct answer label used for training a decision making model, which is a machine learning model used for decision making in a medical care field, in accordance with an operator’s input instruction (Col. 10, lines 40-58: “ Annotations are received via a UI receiving input signals from an input device by which the user may annotate the image on a display screen or device” a user expert can make annotations on images and that those annotations with the labels are then processed. These annotations are done by the expert to give a correct answer for learners to distinguish ; Col. 11, lines 22-26: Describes that the annotations are then processed in a CNN for model training. In response to an expert/operator; Col 8, lines 13-25: Describes that the model for training is a diabetic retinopathy(DR) vs Non-DR and disease detection model, detailing that the model to be trained, is for use in the medical care field.), wherein the decision making model is a multi-class classification model having an input layer for inputting input data, a hidden layer for converting the input data into output data, and an output layer for outputting the output data (Col. 8, lines 9-50; Describes a multi-class classification model implemented as a deep neural network that receives input image data, processes the data through multiple intermediate layers (e.g., convolution, pooling, and activation), and produces a predicted classification label, which corresponds to a model having an input layer, or more hidden layers, and an output layer.);
collect status data indicating a status of the operator while doing the work of adding (Col. 10, lines 14-39: Describes that the system is designed to record audio data in synchronization with eye tracking data. This data is used to track an expert’s annotations of images to learn from the user expert for learning and training of a model. This data reflects an indication of an operator while that operator is assigning correct labels to a input source for the model.), wherein the status data is data relating to an operator's operations, lines of sight, speech, and/or facial expressions that reflect a process of an operator's decision making at the time of doing the work (Col. 10, lines 40-59: Describes that the learning system receives input data of audio or eye tracking data, in combination with expert annotations. The examiner notes that the system does not say ‘facial expressions’ it is noted that the system could be configured to track those as it is already watching the users’ eyes.);
obtain input data of the decision making model (Col. 15, lines 28-43: Describes that once a learned model from expert annotations and training, students are then monitored to determine their ability in reference to the experts. Showing that a trained model is used and obtained to determine student decision making.)
Although Garnavi teaches how to add a correct answer label used for training a decision making model, which is a machine learning model used for decision making in a medical care field, in accordance with an operator’s input instruction, wherein the decision making model is a multi-class classification model having an input layer for inputting input data, a hidden layer for converting the input data into output data, and an output layer for outputting the output data, collect status data indicating a status of the operator while doing the work of adding and to obtain input data of the decision making model. It does not teach, train, based on the status data and the correct answer label, wherein the reliability level determination model is a neural network having an input layer for inputting the status data, a hidden laver for converting the status data into the reliability level, and an output layer for outputting the reliability level, a reliability level determination model, which is a machine learning model which accepts status data and outputs a reliability level of the correct answer label and train, based on the input data, the correct answer label, and the reliability level, the decision making model which accepts the input data and outputs output data that is data indicating a result of the decision making.
Welinder teaches, train, based on the status data and the correct answer label, a reliability level determination model, which is a machine learning model which accepts status data and outputs a reliability level of the correct answer label (Col. 5, lines 28-67 cont. Col 6, lines 1-24: Describes that the system is used to assign a confidence label based on source data, which can include audio data. Source data has associated labels from annotation devices to assist in confidence scoring.; FIG. 1, Col. 8, lines 36-65: Describes that Distributed data annotation server system 110, is configured to receive annotated data and then assign confidence labels/reliability levels to determine the source data ground truth.; Col. 16, lines 11-23: Describes that the confidence label system is configured to run probabilistic/optimization framework to determine a confidence label. The confidence labeling corresponds to the reliability level as it is using the models output for further use downstream in additional training as a weight. This is a well-known method for machine learning and corresponds to the system being a reliability level determination model. The Examiner notes that confidence scores are a well-known and used method to label input data for model learning.)
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to combine Garnavi’s facial expression and expert annotator labeling with Welinder’s confidence scoring model. Doing so would have enabled the system to rank different experts’ labels, allowing for more prominent labels to be favored for better learning and generation of the decision making model, enabling it to be an efficient labeling system.
Welinder further teaches, train, based on the input data, the correct answer label, and the reliability level, the decision making model which accepts the input data and outputs output data that is data indicating a result of the decision making (Col. 8, lines 9-50: Describes that the CNN architecture receives input data to train a deep neural network. ; Col. 10, lines 40-59: Describes that the learning system receives input data of audio or eye tracking data, in combination with expert annotations.).
Although Welinder teaches a reliability level determination model, it does not teach that the reliability level determination model is a neural network having an input layer for inputting the status data, a hidden laver for converting the status data into the reliability level, and an output layer for outputting the reliability level.
Garnavi teaches a model is a neural network having an input layer for inputting the status data, a hidden laver for converting the status data into the reliability level, and an output layer for outputting the reliability level (Col. 8, lines 9-50; Describes a layered neural network classifier used for medical decision making, that receives input image data, processes the data through multiple intermediate layers (e.g., convolution, pooling, and activation), and produces a predicted classification label, which corresponds to a model having an input layer, or more hidden layers, and an output layer.)
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to combine Garnavi’s multi layered deep learning architecture with Welinder’s confidence scoring model. Doing so would have enabled the system to more effectively model complex relationships between operator-related status data and the resulting reliability level.
Regarding claim 2. (Cancelled)
Regarding claim 3, Garnavi, in view of Welinder, teaches the medical information processing apparatus according to claim 2.
Garnavi further teaches, the processing circuitry is further configured to collect, as data relating to the line of sight, reference item data which is data relating to an item on which the line of sight of the operator focuses among various items displayed on a display screen for the addition work (Col. 8, lines 51-67 cont. Col. 10 liens 1-14: Details the Eye tracking system, specifically on the system is tracking the experts gaze while they are looking at different images indicating a pattern. Which allows for the system to analyze the eyes focus on the images to better learn what data is relevant.).
Regarding claim 4, Garnavi, in view of Welinder, teaches the medical information processing apparatus according to claim 3.
Garnavi further teaches, the processing circuitry is further configured to collect as the reference item data, an identifier of a reference item which is an item on which the operator’s line of sight focuses (Col 9, lines14-38: Describes that the system times the gaze of the expert and can then save and determine if the image of interest is ambiguous. Indicating that the reference image is of interest in relation to the expert’s gaze.).
Regarding claim 5, Garnavi, in view of Welinder, teaches the medical information processing apparatus according to claim 1.
Weidner further teaches, the processing circuitry is further configured to (Col. 3, lines 14-37: Describes a system comprising computer hardware components to execute instructions.):
collect ability data which is data relating to an ability of the operator (Welinder Col 3, lines 66-67 cont. Col 4, lines 1-3: Describes that there is an annotator accuracy, indicating that annotator data is ranked to determine the ability of an expert label.; FIG. 3, Col. 11, lines 8-31: Describes that annotators labels can be measured according to their confidence. The system can assign scores to the annotators labels and then use the annotators scored labels for further training so as to give weights to the training data.); and
train, based on the status data, the ability data and the correct answer label, the reliability level determination model which accepts the status data and the ability data, and outputs the reliability level (The Examiner notes that the training as described in claim 1, is now introducing ‘ability data’ to train the decision making model. Garnavi, teaches the status data, and correct answer label, in view of Welinder, who teaches the reliability model and ability data. Merely adding this extra step, is merely adding additional data for training a model, which is well known in the art based on experts ranking and the confidence score of labeled data.).
Regarding claim 6, Garnavi in view of Welinder teaches, the medical information processing apparatus according to claim 5.
Garnavi, in view of Welinder further teaches, the processing circuitry is further configured to:
collect additional data which is data relating to a freshness level (Garnavi Col. 7, lines 5-50: Describes assigning time codes to annotations, in response to an expert’s gaze and other expert annotations of an image. The examiner notes that although they do not discuss that these time codes are used for explicit recency of the annotations of the expert. But by keeping time stamps of the annotations indicates that the system tracks when an expert makes an annotation, and keeps that with the associated image. It would have been obvious to have the system update the images with new annotations from experts which could better assist in training and update the training data with newer information for better image processing and identification of medical data as in Welinder Col. 10, lines 35-64: which describes that the system receives annotated pieces of data with updated information from new annotations, indicating that the system can receive newly updated data for continued training.), a confidence level (Welinder’s, Col. 5, lines 28-60: Describes that confidence scoring is labeling the experts ranking for the source data, indicating that the input data is ranked to determine its level of truthfulness for training. The confidence level is an operator entered, subjective confidence about the label the expert entered. The confidence label allows for an expert to particularly choose if that assigned label should be more or less emphasized depending on the annotators accuracy.), quality (Welinder Col. 11, lines 32-64: Describes that “confidence labels are determined (314) based on features of the obtained (310) source data”, which indicates that source images are received with varying degrees of quality, and correlates to time taken by experts on determining a images potential medical issues. When an expert takes longer on an image to determine if there are any medical issues, that can correlate to several factors, such as cases that may indicate the beginning of a medical condition, wherein the start of a diagnosis is difficult as newly formed issues are just beginning and therefore difficult to find. Or, the image quality is of varying enhancement indicating that the image may be interpreted differently, depending on the expert’s own experience and needing to take longer on lower quality images to make a correct label on the image.), and/or a required time (Garnavi Col 5, lines 8-31: Describes that some data used for training can include how long an expert takes to analyze a particular region of a data image.); and
train, based on the status data, the ability data, the additional data, and the correct answer label, the reliability level determination model which accepts the status data, the ability data, and the additional data, and outputs the reliability level (The Examiner notes that the training as described in claim 1, is now introducing ‘a freshness level’, ‘confidence level’, ‘quality’ and ‘required time’ to further train the reliability level determination model. Garnavi, in view of Welinder teaches these new additional data requirements. These are merely adding extra steps, to identify and label the training data, which is then used for training a model. Merely saying that data has additional labels, to input into a model, is well known in the art).
Regarding claim 7, Garnavi, in view of Welinder teaches the medical information processing apparatus according to claim 1.
Welinder further teaches, the reliability level determination model is a multi-class classification model that outputs the probability of each of multiple classes relating to a result of the decision making as the reliability level (Col 11, lines 32-64: Describes that confidence labels can include a scale from strongly disagree to strongly agree or a numerical scale. Col. 13, lines54-67 cont. Cl. 14, lines 1-30: Describes that each confidence class t via a thresholds T (i.e. a standard multi class discretization) The classes related to the decision result attached to the operator’s label. Annotations include the label (decision result) and a confidence label describing certainty of that label. The confidence then emphasizes/de-emphasizes the annotation.).
Regarding claim 8, Garnavi, in view of Welinder teaches the medical information processing apparatus according to claim 1.
Garnavi further teaches, the processing circuitry is further configured to train the decision making model by minimizing a loss function (Col. 8, lines 36-50: Describes that through back propagation between the ground truth label and the predicted labels, the CNN is trained to predict a label form input which is known supervised training. This corresponds to a decision making model (CNN) trained by minimizing an error between its output and the label (loss).)
Although Garnavi teaches, the processing circuitry is configured to train the decision making model by minimizing a loss function it does not teach the loss function includes an error between an output of the decision making model and the correct answer label weighted by the reliability level.
Welinder teaches, the loss function includes an error between an output of the decision making model and the correct answer label weighted by the reliability level. (Col. 8, lines 36-65: Describes that using confidence/reliability to weight annotations, using per-annotation reliability (confidence) to weight labels/annotations, weights derived from reliability/confidence are applied when learning form labels.)
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to combine Garnavis loss function with Welinders per annotation reliability weighted levels. Doing so would have enabled the system to weigh expert labels and determine preferred answers over others for a more efficient and reliable training pipeline.
Regarding claim 9, Garnavi, in view of Welinder teaches the medical information processing apparatus according to claim 1.
Garnvai further teaches, the processing circuitry is further configured to display the reliability level via a display device (Col. 16, lines 40-47: Describes that the system can display the experts search patterns are then displayed on a monitor for viewing.).
Regarding claim 12, which recites substantially the same limitations as claim 1. Claim 12 further cites a medical information processing method(Garnavi, Col. 2, lines 12-49: Describes that the system is comprised of method steps to execute the computer instructions for image processing.) to perform the system steps of claim 1, and is therefore rejected on the same premise.
Claim(s) 10, and 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Garnavi et al. (US 10657838 B2, referred to as Garnavi) in view of Welinder et al. (US 9704106 B2, referred to as Welinder), in view of Kano et al. (US 20220301716 A1, referred to as Kano).
Regarding claim 10, Garnavi in view of Welinder teaches, the medical information processing apparatus according to claim 9.
Garnavi further teaches, the decision making is an addition of annotation of a disease candidate area to a medical image (Col. 8, lines 9-50: Describes that an expert makes annotations on an image, particularly the image is meant for use on the medical field by an expert to mark up to identify potential medical issues in the image of a patient for training.)
Although Garnavi in view of Welinder teaches the medical information processing apparatus according to claim 9. And Garnavi further teaches the decision making is an addition of annotation of a disease candidate area to a medical image, they do not teach that the processing circuitry is configured to display the annotation with a color value.
Kano teaches, the processing circuitry is further configured to display the annotation with a color value([0083] : Describes that a medical UI using a scalar value is mapped to color, linking a value (reliability level to color, the color is set according to quantitative features when generating the graph visualization. The UI displays a probability alongside the visualization.) according to the reliability level (As described above in claim 1, taught by Welinder.).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to combine the medical image annotation of Garnavi, with the confidence scoring of Welinder with the color coding of Kano. Doing so would have enabled the system to label different images with color coded references based on experts’ confidence labeling. Allowing for trainees or others to interpret if a labeled image is marked up by a more capable expert to better distinguish images and medical diagnosis.
Regarding claim 11. Garnavi in view of Welinder, in view of Kano teaches the medical information processing apparatus according to claim 10.
Garnavi further teaches, the correspondence between the reliability level and the color value is set in accordance with a difficulty level of the addition of the annotation (Col. 10, lines 59-67 cont. Col. 11, lines 1-13: Describes that the concept extraction component, takes in all the information captured form an expert, while they are annotating an image and assigns a difficulty label to the experts ability to identify any issues, “attaches a difficulty and/or importance label (e.g., difficult to identify pathology in region, easy to identify pathology in region, region highly important for the diagnosis of the image, region of medium importance for image diagnosis, region of low importance for image diagnosis) to the image regions. The difficulty label (e.g., difficult to identify pathology in region, easy to identify pathology in region) is derived using the time spent on a region (e.g., difficult if the time spent on the region is greater than a specific threshold) gained through eye tracking, and keywords (e.g., a threshold number of keywords matched against a list of known words to describe difficulty).”).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONALD T RODEN whose telephone number is (571)272-6441. The examiner can normally be reached Mon-Thur 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at (571) 272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.T.R./Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128