Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is in response to the application filed 20 January 2026. Claims 2, 11, and 19 were previously canceled. Claims 1, 8, 9, 10, 17, 18, and 21 are amended. Claims 1, 3-10, 12-18. 20, and 21 are pending and have been examined.
Response to Arguments
Applicant's arguments, see page 11, filed 20 January 2026, with respect to the rejections of Claims 1, 3-10, 12-18, 20, and 21 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The rejections of Claims 1, 3-10, 12-18. 20, and 21 under 35 U.S.C. 112(b) of the previous Office Action have been withdrawn. However, a new grounds of rejection of amended Claims 1, 3-10, 12-18, 20, and 21 is made below under 35 U.S.C. 112(b).
APPLICANT'S ARGUMENT: Applicant argues (page 11, paragraph 7) that "Claims 1, 8-10, 17, 18, and 21 are being amended to correct for issues of indefiniteness. Claims 1, 8-10, 17, and 18 are being amended for the recitation of 'small' .... Claim 21 is being amended to indicate that the elements performed by the PACS are in reference to the elements of Claim 1."
EXAMINER'S RESPONSE: Examiner agrees. The rejection of Claims 1, 8-10, 17, 18, and 21 of the prior Office Action is withdrawn in light of arguments and/or amendments.
Applicant's arguments, see page 12-14, filed 20 January 2026, with respect to the rejections of Claims 1, 3-10, 12-18. 20, and 21 under 35 U.S.C. 101 have been fully considered but they are not persuasive.
APPLICANT'S ARGUMENT: Applicant argues (page 12, paragraph 5) that "Claims 1, 9, 10, and 18, as amended, recite transformation of an entirety natural language into a numerical representation thereof, that is, a latent semantic space. ... The recited element of 'transforming an entirety of the one or more sentences of natural language of impression portion into a latent semantic space' is therefore directed to a transformation from a human-based language space, which may carry embedded contextual meanings, to a machine-understandable numerical space, the numerical space comprising the contextual meanings embedded. Such a process, as further described throughout the Specification, is neither a mental process nor an abstract idea."
EXAMINER'S RESPONSE: Examiner respectfully disagrees. As currently claimed, amended Claim 1 recites the transformation of a sentence of natural language into a latent semantic space, which has been interpreted under BRI in light of the specification as possibly embodied by a transformation of the sentence form its plain language form to a numerical representation as a vector inhabiting a vector space. For example, see the instant specification at p. 9, lines 20-22, "The pretrained Bert model can further encode the sequence of words into a latent semantic space, such as a numeric vector." Examiner notes that, although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims.
The argued limitation of amended Claim 1, "generating an assessment ... including ... transforming ... into a latent semantic space," appears to recite a mental process step that is merely performed by means of computing machinery, but which could be performed, as recited, by a human in the mind or by means of a pen and paper. The additional element of the mental process, "by the trained language model," appears to recite application of a computer merely to perform the mental process step. In the absence of an additional element integrating any claimed mental process into a practical application or providing significantly more, the claim is directed to the mental process.
APPLICANT'S ARGUMENT: Applicant argues (page 13, paragraph 1) that "the recited claim elements are integrated into a practical application of handling diagnostic reports within a picture archiving and communication system (PACS). ... ¶As such, Claims 1, 9, 10, and 18 are directed to a specific improvement in a PACS by enabling said PACS to understand a context of natural language. It should be understood by one of skill in the art that PAC S are specialized computing and archiving systems configured to facilitate communication and storage of medical imaging files and documents in compliance with electronic health record standards. As disclosed at least in Paragraphs [0033]-[0034], diagnostic ambiguity in reports maintained by a PACS may affect patient care. Enabling the PACS to objectively understand medical records contained therein and take actions accordingly can therefore improve the functionality of the PACS in healthcare applications."
EXAMINER'S RESPONSE: Examiner respectfully disagrees that recitation of the PACS is sufficient to integrate the mental process steps into a practical application. Recitation of the PACS ("a picture archiving and communication system (PACS) implementing the trained language model" and "by the PACS, processing the diagnostic report ... including ... causing a computer-based system to render") does not currently appear to amount to more than application of a computer to perform the existing mental process steps of the claim.
APPLICANT'S ARGUMENT: Applicant argues (page 13, paragraph 3) that "the recited claim elements amount to significantly more than abstract ideas. Applicant respectfully submits that the present application is directed toward a technical problem and improvement that provides a technical solution to the technical problem."
Applicant argues (page 13, paragraph 4) that "existing approaches for evaluating uncertainty in diagnostic reporting may be focused on lexical interpretations as opposed to an understanding of contextualized semantics of said diagnostic reporting. The elements of Claims 1, 9, 10, and 18, as amended, can enable a PACS to understand complex human natural language through transforming said natural language into a latent semantic space, which may be a machine-interfaceable numerical representation. The PACS can perform subsequent manipulations on the latent semantic space of the natural language. As disclosed in Claims 1, 9, 10, and 18, such manipulations can include classifying uncertainty of diagnostic reporting, which can be useful for enabling the PACS to process the diagnostic reporting."
EXAMINER'S RESPONSE: Examiner respectfully disagrees. As previously indicated, the additional elements of amended Claim 1 reciting use of the PACS appear merely to recite use of a computer to perform mental process steps that are otherwise not further limited by additional elements integrating the steps into a practical application or providing significantly more.
Applicant's arguments, see pages 14-15, filed 20 January 2026, with respect to the rejections of Claims 1, 3-10, 12-18. 20, and 21 under 35 U.S.C. 103 have been fully considered but they are not persuasive.
APPLICANT'S ARGUMENT: Applicant argues (page 14, paragraph 3) that "in the examples provided in Peng1, Peng1 teaches identification of keywords (Methods and Table 2, Medical Findings Recognition) and dependency graphs (relations between words like 'not' and an adjective) as well as negation 'detection' based on specific words. ¶ The present application teaches against this. ... Table 1 and Paragraph [0066] may provide further contextualization and exemplification of semantic-level analysis (as recited in Claim 1 as a 'latent semantic space'). Neither Peng1 nor any combination of the cited prior art teach or suggest such a feature."
EXAMINER'S RESPONSE: Examiner notes that Applicant's argument pertains to newly claimed matter. In the rejection of amended Claim 1 below, Peng1 is relied on to teach the limitation "the impression portion having one or more sentences of natural language." Peng2 is relied on to teach the steps of "transforming an entirety of ... one or more sentences of natural language ... into a latent semantic space" and "classifying ... based on the latent semantic space."
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
The rejections of Claims 1, 8-10, 17, 18, and 21, and Claims 3-7, 12-16, and 20, which are dependent on them, of the previous Office Action have been withdrawn in view of arguments and/or amendments.
Claims 1, 4, 9, 10, 18, and 21, and Claims 3, 5-8, 12-17, and 20, which are dependent on them, are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Amended Claim 1 recites the limitation "the user" in "displaying a representation of the assessment of certainty to the user and to prompt the user for an input" (emphasis added). There is insufficient antecedent basis for this limitation in the claim. Claim 1 has previously recited a graphical user interface, but has previously recited no user of the GUI or the PACS. For the purposes of examination, amended Claim 1 has been interpreted to read "displaying a representation of the assessment of certainty to a user of the PACS and to prompt the user for an input." Claims 4, 9, 10, 18, and 21 are rejected under the same rationale as Claim 1.
Amended Claim 1 recites the limitation "the input" in "the assessment of certainty generated and the input" (emphasis added). Claim 1 has previously recited an input in "prompt the user for an input," but has not previously recited an input provided to the PACS by the user. For the purposes of examination, amended Claim 1 has been interpreted to read "based on one or more of the assessment of certainty generated and an input provided by the user." Claims 4, 9, 10, 18, and 21 are rejected under the same rationale as Claim 1.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-10, 12-18, 20, and 21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1
Step 1
Claim 1 recites a method for assessing diagnostic certainty in diagnostic reporting natural language, and thus the claimed process falls within a statutory category of invention.
Step 2A Prong 1
The claim recites identifying an impression portion of a diagnostic report submitted for certainty evaluation, the impression portion having one or more sentences of natural language, which is a mental process. The claim recites generating an assessment of certainty of the impression portion ... the generating the assessment of certainty including ... transforming an entirety of the one or more sentences of natural language of the impression portion into a latent semantic space, the transforming enabling understanding of semantics of the natural language, which is a mental process. The claim recites the generating further including ... classifying the impression portion into a certainty category of three or more certainty categories based on the latent semantic space, the three or more certainty categories including at least one definite certainty category and at least two uncertainty categories, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element identified using a trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by a picture archiving and communication system (PACS) implementing the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element trained in a pre-training stage in an unsupervised manner using artificial intelligence-based deep neural network learning by deep bidirectional reading invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element further trained in a fine-tuning stage for evaluating certainty of an amount of training impression portions of diagnostic reports specific to a task, the training impression portions including a plurality of one or more training sentences of natural language invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the evaluating certainty of the training impression portions including generating certainty data per training sentence indicative of a result of applying annotation rules specific to the task based on context provided by the training sentence as a whole invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the PACS, processing the diagnostic report, the processing the diagnostic report including, for an assessment of certainty that fails to satisfy a certainty criteria ... the processing the diagnostic report based on one or more of the assessment of certainty generated and ... input invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element causing a computer-based system to render a graphical user interface displaying a representation of the assessment of certainty to the user and to prompt the user for an input amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and outputting").
Step 2B
The additional element identified using a trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by a picture archiving and communication system (PACS) implementing the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element trained in a pre-training stage in an unsupervised manner using artificial intelligence-based deep neural network learning by deep bidirectional reading invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element further trained in a fine-tuning stage for evaluating certainty of an amount of training impression portions of diagnostic reports specific to a task, the training impression portions including a plurality of one or more training sentences of natural language invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the evaluating certainty of the training impression portions including generating certainty data per training sentence indicative of a result of applying annotation rules specific to the task based on context provided by the training sentence as a whole invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the PACS, processing the diagnostic report, the processing the diagnostic report including, for an assessment of certainty that fails to satisfy a certainty criteria ... the processing the diagnostic report based on one or more of the assessment of certainty generated and ... input invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element causing a computer-based system to render a graphical user interface displaying a representation of the assessment of certainty to the user and to prompt the user for an input is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 3
Step 1
Regarding Claim 3, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites wherein the generating the assessment of certainty further includes determining a probability that the assignment to the certainty category is correct, which is a mental process. Thus, the claim recites an abstract idea.
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 4
Step 1
Regarding Claim 4, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites wherein providing the input provided by the user includes providing an update to the impression portion, which is a mental process. The claim recites providing an override of the assessment of certainty that fails to satisfy the certainty criteria, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element resubmitting the impression portion for application of the one or more sentences of the updated impression portion to the trained language model amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering").
Step 2B
The additional element resubmitting the impression portion for application of the one or more sentences of the updated impression portion to the trained language model is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 5
Step 1
Regarding Claim 5, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
Claim 5 recites the abstract ideas recited by parent Claim 1.
Step 2A Prong 2, Step 2B
The additional element training the trained language model in the finetuning stage using a training set that is a subset of the certainty data invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 6
Step 1
Regarding Claim 6, the rejection of Claim 5 is incorporated.
Step 2A Prong 1
Claim 6 recites the abstract ideas recited by parent Claim 5.
Step 2A Prong 2, Step 2B
The additional element wherein training the trained language model in the fine-tuning stage includes at least one of validating using a validation set that is a subset of the certainty data invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element testing using a testing set that is a subset of the certainty data invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 7
Step 1
Regarding Claim 7, the rejection of Claim 5 is incorporated.
Step 2A Prong 1
The claim recites wherein training the trained language model in the fine-tuning stage includes iteratively adjusting at least one of the annotation rules and the certainty data by a plurality of reviewers, which is a mental process. The claim recites until evaluation of same training sentences by the plurality of reviewers results in the certainty data generated by different reviewers of the plurality of reviewers satisfying a criterion of consensus, which is a mental process. Thus, the claim recites an abstract idea.
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 8
Step 1
Regarding Claim 8, the rejection of Claim 5 is incorporated.
Step 2A Prong 1
The claim recites retraining the trained language model in the finetuning stage, based on a state of at least a portion of the trained language model after the pretraining stage and before the fine-tuning stage, which is a mental process. The claim recites evaluating certainty of the second training impression portions including generating second certainty data per second training sentence indicative of a result of applying annotation rules specific to the second task based on context provided by the second training sentence as a whole, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The additional element using another amount of second training impression portions of diagnostic reports specific to a second task, the second training impression portions including a plurality of one or more second training sentences of natural language invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element wherein the second task refers to a particular part of anatomy and/or a particular diagnostic modality does not amount to more than generally linking the use of a judicial exception to a particular field of use (see MPEP 2106.05(h), "limiting the abstract idea of collecting information").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 9
Step 1
Claim 9 recites a method for assessing diagnostic certainty in radiology reporting natural language, and thus the claimed process falls within a statutory category of invention.
Step 2A Prong 1
The claim recites identifying an impression portion of a diagnostic report submitted for certainty evaluation, the impression portion having one or more sentences of natural language, which is a mental process. The claim recites generating an assessment of certainty of the impression portion ... the generating the assessment of certainty including ... transforming an entirety of the one or more sentences of natural language of the impression portion into a latent semantic space, the transforming enabling understanding of semantics of the natural language, which is a mental process. The claim recites the generating further including ... classifying the impression portion into a certainty category of three or more certainty categories based on the latent semantic space, the three or more certainty categories including at least one definite certainty category and at least two uncertainty categories, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element identified using a trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by a picture archiving and communication system (PACS) implementing the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element trained in a pre-training stage in an unsupervised manner using artificial intelligence-based deep neural network learning by deep bidirectional reading invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element further trained in a fine-tuning stage for evaluating certainty of an amount of training impression portions of diagnostic reports specific to a task, the training impression portions including a plurality of one or more training sentences of natural language invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the evaluating certainty of the training impression portions including generating certainty data per training sentence indicative of a result of applying annotation rules specific to the task based on context provided by the training sentence as a whole invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the PACS, processing the diagnostic report, the processing the diagnostic report including, for an assessment of certainty that fails to satisfy a certainty criteria ... the processing the diagnostic report based on one or more of the assessment of certainty generated and ... input invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element causing a computer-based system to render a graphical user interface displaying a representation of the assessment of certainty to the user and to prompt the user for an input amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and outputting").
Step 2B
The additional element identified using a trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by a picture archiving and communication system (PACS) implementing the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element trained in a pre-training stage in an unsupervised manner using artificial intelligence-based deep neural network learning by deep bidirectional reading invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element further trained in a fine-tuning stage for evaluating certainty of an amount of training impression portions of diagnostic reports specific to a task, the training impression portions including a plurality of one or more training sentences of natural language invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the evaluating certainty of the training impression portions including generating certainty data per training sentence indicative of a result of applying annotation rules specific to the task based on context provided by the training sentence as a whole invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the PACS, processing the diagnostic report, the processing the diagnostic report including, for an assessment of certainty that fails to satisfy a certainty criteria ... the processing the diagnostic report based on one or more of the assessment of certainty generated and ... input invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element causing a computer-based system to render a graphical user interface displaying a representation of the assessment of certainty to the user and to prompt the user for an input is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 10
Step 1
Claim 10 recites a computer system, and thus the claimed machine falls within a statutory category of invention.
Step 2A Prong 1
The claim recites identify an impression portion of a diagnostic report submitted for certainty evaluation, the impression portion having one or more sentences of natural language, which is a mental process. The claim recites generate an assessment of certainty of the impression portion ... the generating the assessment of certainty including ... transforming an entirety of the one or more sentences of natural language of the impression portion into a latent semantic space, the transforming enabling understanding of semantics of the natural language, which is a mental process. The claim recites the generating further including ... classifying the impression portion into a certainty category of three or more certainty categories based on the latent semantic space, the three or more certainty categories including at least one definite certainty category and at least two uncertainty categories, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element a memory configured to store instructions; processor disposed in communication with said memory invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element identified using a trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by a picture archiving and communication system (PACS) implementing the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element trained in a pre-training stage in an unsupervised manner using artificial intelligence-based deep neural network learning by deep bidirectional reading invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element further trained in a fine-tuning stage for evaluating certainty of an amount of training impression portions of diagnostic reports specific to a task, the training impression portions including a plurality of one or more training sentences of natural language invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the evaluating certainty of the training impression portions including generating certainty data per training sentence indicative of a result of applying annotation rules specific to the task based on context provided by the training sentence as a whole invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the PACS, processing the diagnostic report, the processing the diagnostic report including, for an assessment of certainty that fails to satisfy a certainty criteria ... the processing the diagnostic report based on one or more of the assessment of certainty generated and ... input invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element causing a computer-based system to render a graphical user interface displaying a representation of the assessment of certainty to the user and to prompt the user for an input amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and outputting").
Step 2B
The additional element a memory configured to store instructions; processor disposed in communication with said memory invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element identified using a trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by a picture archiving and communication system (PACS) implementing the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element trained in a pre-training stage in an unsupervised manner using artificial intelligence-based deep neural network learning by deep bidirectional reading invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element further trained in a fine-tuning stage for evaluating certainty of an amount of training impression portions of diagnostic reports specific to a task, the training impression portions including a plurality of one or more training sentences of natural language invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the evaluating certainty of the training impression portions including generating certainty data per training sentence indicative of a result of applying annotation rules specific to the task based on context provided by the training sentence as a whole invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the PACS, processing the diagnostic report, the processing the diagnostic report including, for an assessment of certainty that fails to satisfy a certainty criteria ... the processing the diagnostic report based on one or more of the assessment of certainty generated and ... input invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element causing a computer-based system to render a graphical user interface displaying a representation of the assessment of certainty to the user and to prompt the user for an input is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claims 12-17, dependent on Claim 11, incorporate the rejection of Claim 11. Claims 12-17 incorporate substantively all the limitations of Claims 3-8, respectively, in system form and are rejected under the same rationales.
Regarding Claim 18
Step 1
Claim 18 recites a non-transitory computer readable storage medium and one or more computer programs embedded therein, and thus the claimed manufacture falls within a statutory category of invention.
Step 2A Prong 1
The claim recites identify an impression portion of a diagnostic report submitted for certainty evaluation, the impression portion having one or more sentences of natural language, which is a mental process. The claim recites generate an assessment of certainty of the impression portion ... the generating the assessment of certainty including ... transforming an entirety of the one or more sentences of natural language of the impression portion into a latent semantic space, the transforming enabling understanding of semantics of the natural language, which is a mental process. The claim recites the generating further including ... classifying the impression portion into a certainty category of three or more certainty categories based on the latent semantic space, the three or more certainty categories including at least one definite certainty category and at least two uncertainty categories, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element identified using a trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by a picture archiving and communication system (PACS) implementing the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element trained in a pre-training stage in an unsupervised manner using artificial intelligence-based deep neural network learning by deep bidirectional reading invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element further trained in a fine-tuning stage for evaluating certainty of an amount of training impression portions of diagnostic reports specific to a task, the training impression portions including a plurality of one or more training sentences of natural language invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the evaluating certainty of the training impression portions including generating certainty data per training sentence indicative of a result of applying annotation rules specific to the task based on context provided by the training sentence as a whole invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the PACS, processing the diagnostic report, the processing the diagnostic report including, for an assessment of certainty that fails to satisfy a certainty criteria ... the processing the diagnostic report based on one or more of the assessment of certainty generated and ... input invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element causing a computer-based system to render a graphical user interface displaying a representation of the assessment of certainty to the user and to prompt the user for an input amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and outputting").
Step 2B
The additional element identified using a trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by a picture archiving and communication system (PACS) implementing the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element trained in a pre-training stage in an unsupervised manner using artificial intelligence-based deep neural network learning by deep bidirectional reading invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element further trained in a fine-tuning stage for evaluating certainty of an amount of training impression portions of diagnostic reports specific to a task, the training impression portions including a plurality of one or more training sentences of natural language invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the evaluating certainty of the training impression portions including generating certainty data per training sentence indicative of a result of applying annotation rules specific to the task based on context provided by the training sentence as a whole invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element by the PACS, processing the diagnostic report, the processing the diagnostic report including, for an assessment of certainty that fails to satisfy a certainty criteria ... the processing the diagnostic report based on one or more of the assessment of certainty generated and ... input invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element causing a computer-based system to render a graphical user interface displaying a representation of the assessment of certainty to the user and to prompt the user for an input is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 20
Step 1
Regarding Claim 20, the rejection of Claim 18 is incorporated.
Step 2A Prong 1
The claim recites wherein training the trained language model in the fine-tuning stage includes iteratively adjusting at least one of the annotation rules and the certainty data by a plurality of reviewers, which is a mental process. The claim recites until evaluation of same training sentences by the plurality of reviewers results in the certainty data generated by different reviewers of the plurality of reviewers satisfying a criterion of consensus, which is a mental process. Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The additional element wherein the computer programs instructions that when executed by a computer system further cause the computer system train the trained language model in the finetuning stage using a training set that is a subset of the certainty data invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 21
Step 1
Claim 21 recites a picture archiving and communication system (PACS) screening for certainty in diagnostic reporting, and thus the claimed machine falls within a statutory category of invention.
Step 2A Prong 1
The claim recites the identifying the impression portion of the diagnostic report, which is a mental process. The claim recites generate the assessment certainty of the impression portion identified, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element a processor implementing the method of claim 1 invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the accessing the diagnostic report amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and outputting"). The additional element using the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the rendering, on an electronic display communicatively coupled to the processor, a graphical user interface displaying the assessment of certainty to the user, the graphical user interface, for an assessment of certainty filing the certainty criteria amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and outputting"). The additional element prompting the user for the input amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering and outputting"). The additional element handle the diagnostic report within the PACS based on the assessment of certainty generated and the input, the handling of the diagnostic report determining access to the diagnostic report by automated subroutines or users of the PACS invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
Step 2B
The additional element a processor implementing the method of claim 1 invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the accessing the diagnostic report amounts to insignificant extra-solution activity (see MPEP 2106.05(d), "storing and retrieving information in memory"). The additional element using the trained language model invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it"). The additional element the rendering, on an electronic display communicatively coupled to the processor, a graphical user interface displaying the assessment of certainty to the user, the graphical user interface, for an assessment of certainty filing the certainty criteria is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element prompting the user for the input is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element handle the diagnostic report within the PACS based on the assessment of certainty generated and the input, the handling of the diagnostic report determining access to the diagnostic report by automated subroutines or users of the PACS invokes a computer or other machinery merely as a tool to perform an existing process (see MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3-10, 12-18, 20, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Irvin, et al., "CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison" (hereinafter "Irvin") in view of Peng, et al., "NegBio: a high-performance tool for negation and uncertainty detection in radiology reports" (hereinafter "Peng1") in view of Reiner, "Quantitative Analysis of Uncertainty in Medical Reporting: Creating a Standardized and Objective Methodology" (hereinafter "Reiner") in view of Peng, et al., "Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets" (hereinafter "Peng2") in view of Devlin, et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (hereinafter "Devlin") in view of Kapit, et al., "System And Method For Medical Coding Of Vascular Interventional Radiology Procedures" (U.S. Patent Publication 2008/0004505 A1), hereinafter Kapit.
Regarding Claim 1, Irvin teaches:
a method for assessing diagnostic certainty in diagnostic reporting natural language (Irvin, p. 2, Label Extraction from Radiology Reports: "The labeler extracts mentions from a list of observations from the Impression section of radiology reports, which summarizes the key findings in the radiographic study. ... After extracting mentions of observations, we aim to classify them as negative ('no evidence of pulmonary edema, pleural effusions or pneumothorax'), uncertain ('diffuse reticular pattern may represent mild interstitial pulmonary edema'), or positive ('moderate bilateral effusions and bibasilar opacities'). The 'uncertain' label can capture both the uncertainty of a radiologist in the diagnosis as well as ambiguity inherent in the report ('heart size is stable')," where Irvin's classify corresponds to the instant assess, and uncertainty of a radiologist in the diagnosis corresponds to the instant diagnostic certainty), the method comprising:
identifying an impression portion of a diagnostic report submitted for certainty evaluation ... (Irvin, p. 2, Label Extraction from Radiology Reports: "We developed an automated rule-based labeler to extract observations from the free text radiology reports ... The labeler extracts mentions from a list of observations from the Impression section of radiology reports ... After extracting mentions of observations, we aim to classify them as negative ..., uncertain ..., or positive," where Irvin's extracting observations from the impression section corresponds to the instant identifying an impression portion);
generating an assessment of certainty of the impression portion (Irvin, p. 2, Fig. 2, "Labeler Output" column, and p. 3, Label Extraction from Radiology Reports, Mention Aggregation: "We use the classification for each mention of observations to arrive at a final label for 14 observations .... Observations with at least one mention that is positively classified in the report is assigned a positive (1) label. An observation is assigned an uncertain (u) label if it has no positively classified mentions and at least one uncertain mention, and a negative label if there is at least one negatively classified mention," where Irvin's classification labels correspond to the instant assessment of certainty) identified using a trained language model (Irvin, p. 2, Label Extraction from Radiology Reports: "We developed an automated rule-based labeler to extract observations from the free text radiology reports to be used as structured labels for the images. Our labeler is set up in three distinct stages: mention extraction, mention classification, and mention aggregation" and p. 2, Label Extraction from Radiology Reports, Mention Classification: "we follow a procedure similar to Peng et al.(2018): first, the report is split and tokenized into sentences ... then, each sentence is parsed using the Bllip parser trained using David McClosky's biomedical model," where Irvin's labeler with a biomedical sentence parser corresponds to the instant trained language model), the generating the assessment of certainty including, by the trained language model, transforming an entirety of the one or more sentences of natural language of the impression portion (Irvin, p. 2, Label Extraction from Radiology Reports: "To obtain the universal dependency parse ... the report is split and tokenized into sentences ... then, each sentence is parsed using the Bllip parser trained using David McClosky’s biomedical model ... finally, the universal dependency graph of each sentence is computed," where Irvin's dependency graph corresponds to the instant transformed sentence) ..., the transforming enabling understanding of semantics of the natural language by a ... system ... implementing the trained language model (Irvin, p. 6, Comparison of Uncertainty Approaches, Analysis: "In the detection of Atelectasis, the U-Ones approach performs the best, hinting that the uncertainty label for this observation is effectively utilized when treated as positive. We expect that phrases such as 'possible atelectasis' or 'may be atelectasis,' were meant to describe the most likely findings in the image, rather than convey uncertainty, which supports the good performance of U-Ones on this pathology" and p. 4, Uncertainty Approaches, Binary Mapping: "If the uncertainty label does convey semantically useful information to the classifier, then we expect that this approach can distort the decision making of classifiers and degrade their performance"), the generating further including, by the trained language model, classifying the impression portion to a certainty category of ... certainty categories ... (Irvin, p. 2, Label Extraction from Radiology Reports: "After extracting mentions of observations, we aim to classify them as negative ..., uncertain ..., or positive," where Irvin's uncertain and positive/negative classifications correspond to the instant certainty categories).
Irvin teaches a method using an automated rule-based and trained labeler to extract observations from free text radiology reports, including classification into certainty categories. Irvin teaches the method in the context of developing labeled datasets for deep learning methods used in tasks related to medical imaging.
Irvin does not explicitly teach the impression portion having one or more sentences of natural language.
However, Peng1 teaches:
the impression portion having one or more sentences of natural language (Peng1, p. 2, Methods: "NegBio tasks as inputting a sentence with pre-tagged mentions of medical findings, and checks whether a specific finding is negative or uncertain," where Peng1's findings sentence corresponds to the impression portion having one sentence, and where Peng1's method pertains to natural language, as in p. 188, Abstract: "we propose a new algorithm, NegBio, to detect negative and uncertain findings in radiology reports" and p. 88, "Even though many natural language processing applications have been developed in recent years that successfully extract findings mentioned in medical reports, discriminating between positive, negative, and uncertain findings remains challenging").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Irvin regarding identifying an impression portion of a diagnostic report submitted for certainty evaluation and generating an assessment of certainty using a trained language model with those of Peng1 regarding the impression portion having one or more sentences of natural language.
The motivation to do so would be to facilitate improved uncertainty detection by lowering detection false positives by basing results on syntactic context (Peng1, p. 190, Negation and uncertainty detection: "Since our patterns are defined on the graph, the negation/uncertainty scope is thus not limited to word distance. Instead, it is based on syntactic context" and p. 193, Discussion: "the use of negation and uncertainty detection on the syntactic level successfully removes false positive cases of 'positive' findings").
The Irvin/Peng1 combination does not explicitly teach three or more certainty categories ... the three or more certainty categories including at least one definite certainty category and at least two uncertainty categories.
However, Reiner teaches:
three or more certainty categories ... the three or more certainty categories including at least one definite certainty category and at least two uncertainty categories (Reiner, p. 148, Fig. 1, "Representative model for standardized quantification of report uncertainty," depicting four categories of certainty with categories including uncertainty: "0 = Definitive level of certainty (i.e. no uncertainty)," "1= High level of certainty," "3 = Intermediate level of certainty," and "5= Highest degree of uncertainty").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Irvin/Peng1 combination regarding the assessment of certainty including assigning the impression portion to a certainty category with those of Reiner regarding three or more certainty categories including at least one definite certainty category and at least two uncertainty categories.
The motivation to do so would be to facilitate supporting improved communication of diagnostic context for medical report data (Reiner, p. 145, Quantitative Analysis of Medical Data: "The communication and understanding of medical report data can in theory be improved upon when accompanied by quantitative measures of uncertainty. These uncertainty measures could provide the reader with context relating to the perceived level of diagnostic confidence and accuracy. ... ¶ One can draw a number of parallels between uncertainty measurements in laboratory medicine and medical imaging. ... In medical imaging, similar external factors affect data uncertainty including poor patient preparation, anatomic variation, suboptimal exam or protocol selection, reporting errors, image quality deficiencies, and patient-related variables (e.g., compliance, body habitus, age, comorbidities)").
The Irvin/Peng1/Reiner combination does not explicitly teach transforming an entirety of ... one or more sentences of natural language ... into a latent semantic space ... classifying ... based on the latent semantic space and wherein the trained language model was: trained in a pre-training stage in an unsupervised manner using artificial intelligence-based deep neural network learning by deep bidirectional reading further trained in a fine-tuning stage for evaluating ... training ... portions ... specific to a task, the evaluating ... of the training ... portions including generating ... data per training sentence indicative of a result of applying annotation rules specific to the task based on context provided by the training sentence as a whole.
However, Peng2 teaches:
transforming an entirety of ... one or more sentences of natural language ... into a latent semantic space (Peng2, p. 5, 4.2 Fine-tuning with ELMo: "For ... multi-label tasks, we followed the steps in fine-tuning with BERT but used the averaged ELMo embeddings of all words in each sentence as the sentence embedding" and p. 4, 4.1.2 Fine-tuning with BERT: "For multi-label tasks, we fine-tuned the model to predict multi-labels for each sentence in the document," where Peng2's sentence embedding correspond to the instant transforming into latent semantic space, as in the instant specification, p. 9, lines 20-23: "The pretrained Bert model can further encode the sequence of words into a latent semantic space, such as a numeric vector that represents the respective words based on words surrounding the word in either direction") ... classifying ... based on the latent semantic space (Peng2, p. 3, 3.4 Document multilabel classification: "The multilabel classification task predicts multiple labels from the texts. ... Annotation was performed at sentence level by an expert with 15+ years of experience in cancer research. We use ... abstracts for testing and the remaining abstracts for training" and p. 2, 2 Related work: "One established trend is a form of word embeddings that represent the semantic, using high dimensional vectors.... Similar methods also have been derived to improve embeddings of word sequences by introducing sentence embeddings") ...;
wherein the trained language model was: trained in a pre-training stage ... (Peng2, p. 4, 4.1.1 Pre-training BERT: "BERT ... is a contextualized word representation model that is pre-trained based on a masked language model ... [W]e pre-trained our own model BERT on PubMed abstracts and clinical notes ... We initialized BERT with pre-trained BERT provided by (Devlin et al., 2019)") using artificial intelligence-based deep neural network learning by deep bidirectional reading (Peng2, p. 4, 4.1.1 Pre-training BERT: "BERT ... is a contextualized word representation model that is pre-trained based on a masked language model, using bidirectional Transformers"); and
further trained in a fine-tuning stage (Peng2, p. 4, 4.1.2 Fine-tuning with BERT: "BERT is applied to various downstream text mining tasks while requiring only minimal architecture modification.... For multi-label tasks, we fine-tuned the model to predict multi-labels for each sentence in the document") for evaluating ... training ... portions ... specific to a task, the training ... portions including a plurality of one or more training sentences of natural language (Peng2, p. 3, 3.4 Document multilabel classification: "The multilabel classification task predicts multiple labels from the texts. ¶ HoC (the Hallmarks of Cancers corpus) consists of 1,580 PubMed abstracts annotated with ten currently known hallmarks of cancer... We use 315 (~20%) abstracts for testing and the remaining abstracts for training," where Peng2's multilabel abstract classification corresponds to evaluating portions), the evaluating ... of the training ... portions including generating ... data per training sentence indicative of a result of applying annotation rules specific to the task based on context provided by the training sentence as a whole (Peng2, p. 3, 3.4 Document multilabel classification: "HoC (the Hallmarks of Cancers corpus) consists of 1,580 PubMed abstracts annotated with ten currently known hallmarks of cancer... Annotation was performed at sentence level by an expert").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Irvin/Peng1/Reiner combination regarding generating an assessment of certainty of the impression portion identified using a trained
language model with those of Peng2 regarding transforming an entirety of one or more sentences of natural language into a latent semantic space classifying based on the latent semantic space and the trained language model being trained in a pre-training stage in an unsupervised manner using artificial intelligence-based deep neural network learning by deep bidirectional reading and further trained in a fine-tuning stage for evaluating training portions specific to a task, including generating data per training sentence indicative of a result of applying annotation rules specific to the task based on context provided by the training sentence as a whole.
The motivation to do so would be to facilitate model usage producing results in a clinical domain superior to those not pre-trained (Peng2, p. 1, Introduction: "we conduct experiments with two baselines: One makes use of the BERT model... Both are state-of-the-art language representation models and demonstrate promising results in NLP tasks of general purpose. We find that the BERT model pre-trained on PubMed abstracts ... and MIMIC-III clinical notes ... achieves the best results, and is significantly superior to other models in the clinical domain").
The Irvin/Peng1/Reiner/Peng2 combination does not explicitly teach the trained language model was: trained ... in an unsupervised manner.
However Devlin teaches:
the trained language model was: trained in a pre-training stage in an unsupervised manner (Devlin, p. 4, 3.1 Pre-training BERT: "we do not use traditional left-to-right or right-to-left language models to pre-train BERT. Instead, we pre-train BERT using two unsupervised tasks").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Irvin/Peng1/Reiner/Peng2 combination regarding wherein the trained language model was trained in a pre-training stage with those of Devlin regarding the trained language model was: trained in a pre-training stage in an unsupervised manner.
The motivation to do so would be to facilitate model usage taking advantage of improved language understanding through broader generalization (Devlin, p. 9, 6 Conclusion: "Recent empirical improvements due to transfer learning with language models have demonstrated that rich, unsupervised pre-training is an integral part of many language understanding systems. In particular, these results enable even low-resource tasks to benefit from deep unidirectional architectures. Our major contribution is further generalizing these findings to deep bidirectional architectures, allowing the same pre-trained model to successfully tackle a broad set of NLP tasks").
The Irvin/Peng1/Reiner/Peng2/Devlin combination does not explicitly teach enabling understanding of ... natural language by a picture archiving and communication system (PACS); by the PACS, processing the diagnostic report, the processing the diagnostic report including, for an assessment of certainty that fails to satisfy a certainty criteria, causing a computer-based system to render a graphical user interface displaying a representation of the assessment of certainty to the user and to prompt the user for an input, the processing the diagnostic report based on one or more of the assessment of certainty generated and the input.
However, Kapit teaches:
enabling understanding of ... natural language (Kapit, Fig. 2-6, depicting screenshots or wireframes of graphical application interfaces, and [0010]: "medical billing codes are electronically assigned to medical reports using natural language processing (NLP) process. The results of the NLP process are then displayed on a custom graphical user interface (GUI) for review by human medical coders") by a picture archiving and communication system (PACS) (Kapit, Fig. 4, depicting a diagnostic report and patient image, and [0069]: "FIG. 4 demonstrates a highlighting feature of the interface. ...[A]s shown in FIG. 4, vessels 31 in the graphic are highlighted and the view is centered on the highlighted vessel(s)" and [0053]: "using the above-mentioned chart-level confidence to route notes to different queues in order to optimize the human coder's experience, and optimize efficiency gains in the computer-assisted coding process," where Kapit's computer corresponds to the instant system) ...
by the PACS, processing the diagnostic report, the processing the diagnostic report including, for an assessment of certainty that fails to satisfy a certainty criteria (Kapit, [0063]: "the present invention uses the confidence assessment values to determine whether or not a chart is considered correct as coded ( and therefore appropriate for the 'Confident' queue, or whether there is some element (c,i) that fails to meet the confidence threshold and requires human review (in which case the chart belongs in the 'Review' queue)" and [0065]: "Another feature of one embodiment of the present invention, is providing a mechanism for outputting the resulting codes from the NLP process to a GUI user interface optimized for a human coder to review the results and make changes as appropriate"), causing a computer-based system to render a graphical user interface displaying a representation of the assessment of certainty to the user (Kapit, [0068]: "FIG. 3 shows an interface for checking the validity of the codes entered. An error message 20 is displaced in the event of a conflicts between the NLP engine's result and the human coder's result," where Fig. 3 depicts a user interface dialog, and where Kapit's human coder corresponds to the instant user) and to prompt the user for an input (Kapit, [0052]: "One of the features ... is a novel approach to confidence assessment that embeds the confidence assessment within a human-in-the-loop workflow that specifically involves human approval or lack of approval. In other words, a preferred aspect of the present invention assesses specifically the likelihood that a human coder will push the 'approve' button in response to the engine's coding," where Kapit's pushing the approve button corresponds to the instant prompting the user), the processing the diagnostic report based on one or more of the assessment of certainty generated and the input (Kapit, [0063]: "the present invention uses the confidence assessment values to determine whether or not a chart is considered correct as coded ( and therefore appropriate for the 'Confident' queue, or whether there is some element (c,i) that fails to meet the confidence threshold and requires human review (in which case the chart belongs in the 'Review' queue)").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Irvin/Peng1/Reiner/Peng2/Devlin combination regarding assessing diagnostic certainty in diagnostic reporting natural language with those of Kapit regarding enabling understanding of natural language by a picture archiving and communication system (PACS), processing the diagnostic report, the processing the diagnostic report including, for an assessment of certainty that fails to satisfy a certainty criteria, causing a computer-based system to render a graphical user interface displaying a representation of the assessment of certainty to the user and to prompt the user for an input, the processing the diagnostic report based on one or more of the assessment of certainty generated and the input.
The motivation to do so would be to enable a scenario where the user may provide feedback on automated evaluation results (Kapit, [0010]: "The results of the NLP process are then displayed ... for review by human medical coders. Human coders review and approve the codes, and provide feedback to the NLP engine if corrections are needed. This feedback process provides training for the NLP engine, allowing it to add to its knowledge base over time and to expand its ability to provide reliable coding").
Regarding Claim 9, Irvin teaches:
a method for assessing diagnostic certainty in radiology reporting natural language (Irvin, p. 2, Label Extraction from Radiology Reports: "The labeler extracts mentions from a list of observations from the Impression section of radiology reports, which summarizes the key findings in the radiographic study. ... After extracting mentions of observations, we aim to classify them as negative ('no evidence of pulmonary edema, pleural effusions or pneumothorax'), uncertain ('diffuse reticular pattern may represent mild interstitial pulmonary edema'), or positive ('moderate bilateral effusions and bibasilar opacities'). The 'uncertain' label can capture both the uncertainty of a radiologist in the diagnosis as well as ambiguity inherent in the report ('heart size is stable')," where Irvin's classify corresponds to the instant assess, and uncertainty of a radiologist in the diagnosis corresponds to the instant diagnostic certainty), the method comprising: precisely those steps recited in the rejection of Claim 1. Claim 9 is rejected under the same rationale as Claim 1.
Regarding Claim 10, Irvin teaches:
a computer system, comprising: a memory configured to store instructions; processor disposed in communication with said memory (Irvin, p. 1, Abstract: "We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. ... We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models [citing repository https://stanfordmlgroup.github.io/competitions/chexpert]," where a computer system comprising a processor and memory are inherent in execution of the source code of Irvin's repository), wherein the processor upon execution of the instructions is configured to: perform precisely those steps recited in the rejection of Claim 1. Claim 10 is rejected under the same rationale as Claim 1.
Regarding Claim 18, Irvin teaches:
a non-transitory computer readable storage medium and one or more computer programs embedded therein, the computer programs comprising instructions (Irvin, p. 1, Abstract: "We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. ... We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models [citing repository https://stanfordmlgroup.github.io/competitions/chexpert]," where a non-transitory computer readable storage medium is inherent in execution of the source code of Irvin's repository), which when executed by a computer system, cause the computer system to: perform precisely those steps recited in the rejection of Claim 1. Claim 18 is rejected under the same rationale as Claim 1.
Regarding Claim 3, the rejection of Claim 1 is incorporated.
The Irvin/Peng1/Reiner/Peng2/Devlin/Kapit combination teaches:
wherein the generating the assessment of certainty further includes determining a probability that the assignment to the certainty category is correct (Irvin, p. 1, Abstract: "We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs").
Claim 12 incorporates substantively all the limitations of Claim 3 in system form and is rejected under the same rationale.
Regarding Claim 4, the rejection of Claim 1 is incorporated.
Kapit further teaches:
wherein providing the input provided by the user includes providing an update to the impression portion and resubmitting the impression portion for application of the one or more sentences of the updated impression portion to the trained language model or providing an override of the assessment of certainty that fails to satisfy the certainty criteria (Kapit, [0071]: "a learning feedback loop ... allows changes made by the human coder to be identified specifically and sent back to the NLP engine. A human coder using a mouse can select text missed by the engine, identify the anatomy feature in the vascular path, and paste the resulting procedure code into the medical billing results. The text, location, and procedure code are sent back to the NLP engine server to allow the missed language to be learned by the technology and appropriate adjustments made to the engine").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Irvin/Peng1/Reiner/Peng2/Devlin/Kapit combination regarding rendering a graphical user interface displaying a representation of the assessment of certainty to a user and prompting the user for an input with the further teachings of Kapit regarding providing an update to the impression portion and resubmitting the impression portion for application of the one or more sentences of the updated impression portion to the trained language model.
The motivation to do so would be facilitate improvement in the functioning of the language model (Kapit, [0071]: "The text, location, and procedure code are sent back to the NLP engine server to allow the missed language to be learned by the technology and appropriate adjustments made to the engine").
Claim 13 incorporates substantively all the limitations of Claim 4 in system form and is rejected under the same rationale.
Regarding Claim 5, the rejection of Claim 1 is incorporated.
Peng2 further teaches:
training the trained language model in the fine tuning stage using a training set (Peng2, p. 4, 4.1.2 Fine-tuning with BERT: "BERT is applied to various downstream textmining tasks while requiring only minimal architecture modification. ... For multi-label tasks, we fine-tuned the model to predict multi-labels for each sentence in the document. We then combine the labels in one document and compare them with the gold-standard" and p. 3, 3.4 Document multilabel classification: "The multilabel classification task predicts multiple labels from the texts. HoC (the Hallmarks of Cancers corpus) consists of 1,580 PubMed abstracts annotated with ten currently known hallmarks of cancer") that is a subset of the certainty data (Peng2, p. 4, 4.1.1 Pre-training BERT: "we pre-trained our own model BERT on PubMed abstracts").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Irvin/Peng1/Reiner/Peng2/Devlin/Kapit combination regarding training a language model for evaluating certainty of diagnostic impressions in a fine-tuning stage with the further teachings of Peng2 regarding training the language model in a fine-tuning stage using a training set that is a subset of the certainty data.
The motivation to do so would be to take advantage of industry-standard, labeled data of high quality for model performance evaluation (Peng2 p. 2, 3 Tasks: "BLUE contains five tasks with ten corpora that cover a broad range of data quantities and difficulties (Table 1). Here, we rely on preexisting datasets because they have been widely used by the BioNLP community as shared tasks" and Peng2, p. 3, 3.4 Document multilabel classification: "HoC (the Hallmarks of Cancers corpus) consists of 1,580 PubMed abstracts annotated with ten currently known hallmarks of cancer.... Annotation was performed at sentence level by an expert with 15+ years of experience in cancer research").
Claim 14 incorporates substantively all the limitations of Claim 5 in system form and is rejected under the same rationale.
Regarding Claim 6, the rejection of Claim 5 is incorporated.
Peng2 further teaches:
wherein training the trained language model in the fine-tuning stage includes at least one of validating using a validation set that is a subset of the certainty data and testing using a testing set that is a subset of the certainty data (Peng2, p. 3, 3.4 Document multilabel classification: "The multilabel classification task predicts multiple labels from the texts. HoC (the Hallmarks of Cancers corpus) consists of 1,580 PubMed abstracts annotated with ten currently known hallmarks of cancer.... We use 315 (~20%) abstracts for testing and the remaining abstracts for training," with p. 3, Table 1: BLUE tasks, which lists the HoC corpus data as: HoC, documents: 1108 (train) 157 (dev) 315 (test), with the instant validation set corresponding to Peng2's Dev; and where p. 4, 4.1.1 Pre-training BERT: "we pre-trained our own model BERT on PubMed abstracts").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Irvin/Peng1/Reiner/Peng2/Devlin/Kapit combination regarding fine-tuning the language model using a subset of the pre-training data with the further teachings of Peng2 regarding fine-tuning the language model using validation set and test set that are subsets of the certainty data.
The motivation to do so would be to facilitate evaluating model performance in the face of scarce labeled data (Peng2, p. 5, 5 Benchmark results and discussion: "Because the BIOSSES dataset is very small (there are only 16 sentence pairs in the test set), all BERT models’ performance was unstable. ... Here, we obtained the best results by following the same strategy: selecting the best model on the development set after several runs").
Claim 15 incorporates substantively all the limitations of Claim 6 in system form and is rejected under the same rationale.
Regarding Claim 7, the rejection of Claim 5 is incorporated.
Kapit further teaches:
wherein training the trained language model in the fine-tuning stage includes iteratively adjusting at least one of the annotation rules and the certainty data (Kapit, [0070]: "Another preferred feature of the present invention, is a mechanism for the NLP engine and Coding-Workspace based GUI to interact. ... The process of changing medical facts and recalculating can then be repeated") by a plurality of reviewers (Kapit, [0055]: "The workflow of one embodiment of the presently preferred invention includes human coders who review notes and either approve them without changes, or modify the codes. Human action may also cause dictation to modified by the physician when the coder identifies bad language they can ask the physician for an addendum to clarify their document") until evaluation of same training sentences by the plurality of reviewers (Kapit, [0053]: "Another preferred feature of one embodiment of the present invention, is a system and method for combining code-level confidence assessments, in order to make a chart-level assessment. Yet another preferred feature of the present invention, is using the above-mentioned chart-level confidence to route notes to different queues in order to optimize the human coder's experience, and optimize efficiency gains in the computer-assisted coding process") results in the certainty data generated by different reviewers of the plurality of reviewers satisfying a criterion of consensus (Kapit, [0063]: "Another embodiment of the present invention uses the confidence assessment values to determine whether or not a chart is considered correct as coded ( and therefore appropriate for the 'Confident' queue), or whether there is some element ( c,i) that fails to meet the confidence threshold and requires human review (in which case the chart belongs in the 'Review' queue). The confidence threshold may be set manually in concert with the customer. Typical threshold values (>95%) are much higher than typical levels of intercoder agreement even between experienced human coders").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Irvin/Peng1/Reiner/Peng2/Devlin/Kapit combination regarding fine-tuning the language model using a subset of certainty data with the further teachings of Kapit regarding iteratively adjusting certainty data using the feedback of reviewers during fine-tuning of the language model.
The motivation to do so would be to improve the consistency of the performance of the language model (Kapit, [0065]: "The NLP-based coding engine does not always produce perfect results. However, the accuracy and consistency of the results may be greatly improved through the correction of the results by human coders. ... The end result of that collaboration may be used for training the engine to consistently produce more accurate results").
Claim 16 incorporates substantively all the limitations of Claim 7 in system form and is rejected under the same rationale.
Claim 20 incorporates substantively all the limitations of Claims 5 and 7 in non-transitory computer readable storage medium form and is rejected under the same rationales.
Regarding Claim 8, the rejection of Claim 5 is incorporated.
The Irvin/Peng1/Reiner/Peng2/Devlin/Kapit combination has been shown to teach:
retraining the trained language model in the fine-tuning stage, based on a state of at least a portion of the trained language model after the pretraining stage and before the fine-tuning stage (in the rejection of Claim 1, Peng2, p. 4, 4.1.1 Pre-training BERT: "[W]e pre-trained our own model BERT on PubMed abstracts and clinical notes" and p. 4, 4.1.2 Fine-tuning with BERT: "BERT is applied to various downstream text mining tasks while requiring only minimal architecture modification.... For multi-label tasks, we fine-tuned the model to predict multi-labels for each sentence in the document," where Peng2's pre-trained BERT model corresponds to the instant a state of the trained language model before fine-tuning), using another amount of second training impression portions of diagnostic reports specific to a second task, the second training impression portions including a plurality of one or more second training sentences of natural language (Peng2, p. 3, 3.4 Document multilabel classification: "The multilabel classification task predicts multiple labels from the texts. ¶ HoC (the Hallmarks of Cancers corpus) consists of 1,580 PubMed abstracts annotated with ten currently known hallmarks of cancer... We use 315 (~20%) abstracts for testing and the remaining abstracts for training," where Peng2's HoC abstracts and multilabel classification correspond to the instant second training portions and second task), the evaluating certainty of the second training impression portions including generating second certainty data per second training sentence indicative of a result of applying annotation rules specific to the second task based on context provided by the second training sentence as a whole (as recited in the rejection of Claim 1, Peng2, p. 3, 3.4 Document multilabel classification: "HoC (the Hallmarks of Cancers corpus) consists of 1,580 PubMed abstracts annotated with ten currently known hallmarks of cancer... Annotation was performed at sentence level by an expert").
The Irvin/Peng1/Reiner/Peng2/Devlin/Kapit combination teaches:
wherein the ... task refers to a particular part of anatomy and/or a particular diagnostic modality (Irvin, p. 2, Dataset: "CheXpert is a large public dataset for chest radiograph interpretation, consisting of 224,316 chest radiographs," where Irvin's chest corresponds to the instant part of anatomy).
Claim 17 incorporates substantively all the limitations of Claim 8 in system form and is rejected under the same rationale.
Regarding Claim 21, the Irvin/Peng1/Reiner/Peng2/Devlin/Kapit teaches:
... implementing the method of claim 1 ... to perform: the accessing the diagnostic report; the identifying the impression portion of the diagnostic report (Irvin, p. 2, Label Extraction from Radiology Reports: "We developed an automated rule-based labeler to extract observations from the free text radiology reports ... The labeler extracts mentions from a list of observations from the Impression section of radiology reports ... After extracting mentions of observations, we aim to classify them as negative ..., uncertain ..., or positive," where Irvin's extracting observations from the impression section corresponds to the instant identifying an impression portion);
the generating the assessment certainty of the impression portion (Irvin, p. 2, Fig. 2, "Labeler Output" column, and p. 3, Label Extraction from Radiology Reports, Mention Aggregation: "We use the classification for each mention of observations to arrive at a final label for 14 observations .... Observations with at least one mention that is positively classified in the report is assigned a positive (1) label. An observation is assigned an uncertain (u) label if it has no positively classified mentions and at least one uncertain mention, and a negative label if there is at least one negatively classified mention," where Irvin's classification labels correspond to the instant assessment of certainty) identified using a trained language model (as recited in the rejection of Claim 1, Irvin, p. 2, Label Extraction from Radiology Reports: "we follow a procedure similar to Peng et al.(2018): first, the report is split and tokenized into sentences ... then, each sentence is parsed using the Bllip parser trained using David McClosky's biomedical model," where Irvin's trained Bllip parser corresponds to the instant trained language model);
the rendering ... a graphical user interface displaying the assessment of certainty to the user (as recited in the rejection of Claim 1, Kapit, [0068]: "FIG. 3 shows an interface for checking the validity of the codes entered. An error message 20 is displaced in the event of a conflicts between the NLP engine's result and the human coder's result," where Fig. 3 depicts a user interface dialog, and where Kapit's human coder corresponds to the instant user), the graphical user interface, for an assessment of certainty filing the certainty criteria, prompting the user for the input (as recited in the rejection of Claim 1, Kapit, [0052]: "One of the features ... is a novel approach to confidence assessment that embeds the confidence assessment within a human-in-the-loop workflow that specifically involves human approval or lack of approval. In other words, a preferred aspect of the present invention assesses specifically the likelihood that a human coder will push the 'approve' button in response to the engine's coding," where Kapit's pushing the approve button corresponds to the instant prompting the user); and
the processing the diagnostic report ... based on the assessment of certainty generated and the input ... (as recited in the rejection of Claim 1, Kapit, [0063]: "the present invention uses the confidence assessment values to determine whether or not a chart is considered correct as coded ( and therefore appropriate for the 'Confident' queue, or whether there is some element (c,i) that fails to meet the confidence threshold and requires human review (in which case the chart belongs in the 'Review' queue)").
Kapit further teaches:
A picture archiving and communication system (PACS) screening ... diagnostic reporting (Kapit, Fig. 4, depicting a diagnostic report and patient image, and [0069]: "FIG. 4 demonstrates a highlighting feature of the interface. ...[A]s shown in FIG. 4, vessels 31 in the graphic are highlighted and the view is centered on the highlighted vessel(s)"), the PACS comprising: a processor (Kapit, [0053]: "Yet another preferred feature of the present invention, is using the above-mentioned chart-level confidence to route notes to different queues in order to optimize the human coder's experience, and optimize efficiency gains in the computer-assisted coding process," where Kapit's computer corresponds to the instant processor) ...
the processor configured to perform: ... the rendering, on an electronic display communicatively coupled to the processor (Kapit, Fig. 2-6, depicting screenshots or wireframes of graphical application interfaces, and [0010]: "medical billing codes are electronically assigned to medical reports using natural language processing (NLP) process. The results of the NLP process are then displayed on a custom graphical user interface (GUI) for review by human medical coders"); and
the processing the diagnostic report within the PACS ..., the handling of the diagnostic report determining access to the diagnostic report by automated subroutines or users of the PACS (Kapit, [0053]: "Another preferred feature of one embodiment of the present invention, is a system and method for combining code-level confidence assessments, in order to make a chart-level assessment. Yet another preferred feature of the present invention, is using the above-mentioned chart-level confidence to route notes to different queues in order to optimize the human coder's experience, and optimize efficiency gains in the computer-assisted coding process," where Kapit's routing into a queue corresponds to the instant handling determining access).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Irvin/Peng1/Reiner/Peng2/Devlin/Kapit combination regarding implementing the method of claim 1 with the further teachings of Kapit regarding a picture archiving and communication system (PACS) screening diagnostic reporting, the PACS comprising: a processor, the processor configured to render, on an electronic display communicatively coupled to the processor; handle the diagnostic report within the PACS, the handling of the diagnostic report determining access to the diagnostic report by automated subroutines or users of the PACS.
The motivation to do so would be to enable a scenario where the user may provide feedback on automated evaluation results (Kapit, [0010]: "The results of the NLP process are then displayed ... for review by human medical coders. Human coders review and approve the codes, and provide feedback to the NLP engine if corrections are needed. This feedback process provides training for the NLP engine, allowing it to add to its knowledge base over time and to expand its ability to provide reliable coding").
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT N DAY whose telephone number is (703)756-1519. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.N.D./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights