DETAILED ACTION
This is responsive to RCE filed on 02/25/2026 in which claims 1-39 are presented for examination; Claims 1, 2,12,15,19-21,25,32, 34, 36 and 37 have been amended.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/25/2026 has been entered.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-39 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claims 1, 32, 34, and 37, the amended claims recite:
“processing the unstructured commentary data using a multi-stage classifier architecture comprising at least one named entity recognition (NER) classifier and a commentary type classifier to identify for each item a commentary type and a list of medicines associated with the commentary.”
The specification doesn’t explicitly support multi-stage classifier architecture comprising NER classifier and commentary type classifier; the specification seems to be silent with regard to muti-stage classifier. Para 0195, 0203 does state stacked ensemble, or stacked embedding, which could be multi-stage classifiers; however, these paragraphs don’t teach where “named entity recognition (NER) classifier and a commentary type classifier to identify for each item a commentary type and a list of medicines associated with the commentary.” To the contrary, para 0203 states:
“[0203] When training the NER model and using it to tag posts with entities, the default configuration for spaCy's EntityRecognizer model was used. The model is backed by a neural network that uses a stacked embedding architecture. The details of spaCy's default EntityRecognizer are outlined in their video documentation (accessed via path watch?v=sqDHBH9IjRU on the www.youtube.com website).”
Also, Fig. 5, step S504 teaches applying commentary type classifier to identify commentary type, and then in next step S506 teaches applying NER classifier for entity identifier; thus, clearly using two distinct classifier for two different functions. At least this embodiment does not seem to be using the stacked embedding.
Simply put, the disclosure does teach using two different classifiers for commentary type, and entity identifier; however, it doesn’t teach multi-stage architecture such as stacking ensemble classifier; the paragraphs that do mentions use of stacking, provide no detail with regard to claimed function, rather simply state “The model is backed by a neural network that uses a stacked embedding architecture.”
Foe examining purposes, the examiner is interpreting the multi-stage classifier architecture to be spaCy’s EntityRecognizer model as described in para 0203, and described in admitted prior art (provided youtube link is over 8 years old).
Additionally, amended claims 1, 32, 34, and 37 recite: “processing the content analysis data to calculate a multi-dimensional feature vector indicative of the effectiveness or safety of the medicine.” The specifications is completely silent with regard to above claimed limitation. In fact, the specification is completely silent with regard to any type of vector. The specification supports original claim language of “processing the content analysis data to calculate an estimate indicative of the overall effectiveness or safety of the medicine.”
Regarding claim 12, the claim recites “wherein said at least one classifier includes the multi-stage classifier architecture further comprises a commentary type classifier configured to operate in sequence with the NER classifier.” As mention with regard to claim 1, the specification doesn’t disclose multi-stage classifier in the context of claim; furthermore, the specification doesn’t include term “sequence”. One potentially could interpret the steps S504, and S506 to be performed in sequence, however it is contrary to the claim language which requires multi-stage classifier; in case of Fig. 5, the steps being performed in sequence, are merely applying two different classifiers to achieve two distinct objectives (identifying commentary type, and generating entity identifier).
Dependent claims are rejected based on the rejected base claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-39 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1:
Step 1: Is the claim to a process, machine, manufacture or composition of matter?” Yes, it’s a method(process).
Step 2a Prong 1 (judicial exception)
Step 2A (1): “Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes , the claim comes under mental processes.
Claim 1 recites:
“A method of estimating the effectiveness or safety of a medicine, the method comprising: receiving from a remote source, unstructured commentary data encoding a plurality of items of commentary substantially related to medical subject-matter; processing the unstructured commentary data using a multi-stage classifier architecture comprising at least one named entity recognition(NER) classifier and a commentary type classifier to identify for each item a commentary type and a list of medicines associated with the commentary, the multi-stage classifier architecture employing machine learning to identify linguistic patterns in the unstructured commentary data and to associate the linguistic patterns with specific corresponding outputs to disambiguate medicine entities from common names based on context ; selecting a subset of items, from the plurality of items of commentary, by filtering the list of medicines against a medicine database encoding names associated with at least one jurisdiction and identified as referencing the medicine and whose commentary type has been identified as commentary from a patient who has used the medicine; processing the subset of items to generate content analysis data including, for each item, at least one estimate quantifying a respective at least one aspect of an effect of the medicine as described by the patient in the commentary; and processing the content analysis data to calculate a multi-dimensional feature vector indicative of the effectiveness or safety of the medicine.”
All the limitations above are abstract idea related to the mental process (concepts performed in the human mind (including an observation, evaluation, judgment, opinion)) with the exception of bold and underlined limitations. Claim language pertains to analyzing the opinions/comments about a medicine , received from a patient , who have used it. Relevant data/items can be selected from comments/opinions to estimate overall effectiveness of the medicine. All of this can be done on paper. one could look at the comments on paper, and determine the medicines being mentioned, and their effect. Also, a list of medicines having commentary can be analyzed for identifying a pattern (linguistic pattern) for understanding clinical outcome.
Step 2A(2): Prong Two: evaluate whether the claim recites additional elements that integrate the exception into a practical application of the exception. NO
The claim does recite additional elements; however they don’t integrate the exception into a practical application of the exception.
receiving from a remote source(Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g) )
multi-stage classifier architecture (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
named entity recognition(NER) classifier(Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
commentary type classifier(Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
machine learning(Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
Step 2B: evaluate whether the claim recites additional elements that amount to an inventive concept (aka “significantly more”) than the recited judicial exception? NO
As discussed previously with respect to Step 2A Prong Two, the additional element in the claim amounts to no more than mere instructions to apply the exception using a generic computer component.
Regarding claim limitation “receiving from a remote source,” the courts have recognized the computer functions as well‐understood, routine, and conventional functions when they are claimed in a merely generic manner (“i. Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information”); See, MPEP 2106.05 (d)(II)
The same analysis applies here in 2B, i.e., mere instructions to apply an exception using a generic computer component cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Dependent claims 2-31 and 39 further narrows the abstract idea defined in claim 1, and add the additional elements of “NER classifier”, “commentary type classifier”, “feature indicators”, “annotation data”, “training or retraining classifier”.
Under step 2A, prong two, the additional elements don’t integrate the exception into a practical application of the exception as merely adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f).
As discussed previously with respect to Step 2A Prong Two, the additional elements in the claim amounts to no more than mere instructions to apply the exception using a generic computer component.
The same analysis applies here in 2B, i.e., mere instructions to apply an exception using a generic computer component cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Regarding claim 32 , it is rejected under the same rationale as claim 1. In addition , the claim adds the additional elements of “computer system”, “processor”, “memory”, “computer program code”, “classifier”, “machine learning”, “training dataset”, “retraining the at least one classifier” .
Under step 2A, prong two, the additional elements don’t integrate the exception into a practical application of the exception as merely adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f).
As discussed previously with respect to Step 2A Prong Two, the additional elements in the claim amounts to no more than mere instructions to apply the exception using a generic computer component.
The same analysis applies here in 2B, i.e., mere instructions to apply an exception using a generic computer component cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Dependent claim 33 further narrows the abstract idea and add the additional elements of “training or retraining a classifier”, “system”, “classifier”.
Under step 2A, prong two, the additional elements don’t integrate the exception into a practical application of the exception as merely adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f).
As discussed previously with respect to Step 2A Prong Two, the additional elements in the claim amounts to no more than mere instructions to apply the exception using a generic computer component.
The same analysis applies here in 2B, i.e., mere instructions to apply an exception using a generic computer component cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Regarding claim 34:
Step 1: Is the claim to a process, machine, manufacture or composition of matter?” Yes, it’s a machine(computer system)
Step 2a Prong 1 (judicial exception)
Step 2A (1): “Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes , the claim comes under mental processes.
Claim 1 recites:
“ A computer system for estimating the effectiveness or safety of medicines, the computer system including: a commentary downloader module for downloading items of commentary from at least one remote source; a multi-stage classifier architecture comprising: a commentary type classifier module for identifying the type of commentary; a named entity recognition, NER, classifier module for identifying entities associated with each item of commentary; a medicine database encoding medicine data that encodes a plurality of medicine names associated with at least one jurisdiction; a commentary importer module which accesses and applies the commentary type classifier module, the NER classifier module and the medicine data in the medicine database, to select from the downloaded items of commentary a plurality of items of commentary that include at least one medicine entity, that include at least one appropriate medicine name, validated against the at least one jurisdiction in the medicine database, and that are identified as being a commentary type that is authored by a patient; a feature calculator module configured to calculate for each item of commentary a plurality of feature indicators selected from at least one of: at least one aspect of a personal experience, at least one measure of a tone of the item of commentary, at least one detected stance of the patient, a count of a number of times a medicine is mentioned, a count of a number of times a relevant symptom is mentioned, a sentiment estimate, and a count of the number of times a relevant feeling or experience is mentioned; and a summary score calculator module for calculating a summary score representative of the effectiveness or safety of a medicine in dependence on the feature indicators calculated by the feature calculator module for relevant items of the plurality of items of commentary. Wherein the summary score calculator module is further configured to weight the summary score based on a relevance indictor associated with the at least one jurisdiction”
All the limitations above are abstract idea related to the mental process (concepts performed in the human mind (including an observation, evaluation, judgment, opinion)) with the exception of bold and underlined limitations. Claim language pertains to Claim language pertains to analyzing the opinions/comments about a medicine , received from a patient , who have used it. Relevant data/items can be selected to estimate overall effectiveness of the medicine. All of this can be done on paper. Also, a specific name of medicine can be identified and the number of times the medicine is being used can easily be counted by recording on the paper. By analyzing all the data an effectiveness score for a medicine can easily be calculated .
Step 2A(2): Prong Two: evaluate whether the claim recites additional elements that integrate the exception into a practical application of the exception. NO
The claim does recite additional elements; however they don’t integrate the exception into a practical application of the exception.
computer system (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
commentary downloader(Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
remote source(Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
commentary type classifier(Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
named entity recognition(Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
NER classifier module (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
commentary importer module(Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
feature calculator module(Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
Step 2B: evaluate whether the claim recites additional elements that amount to an inventive concept (aka “significantly more”) than the recited judicial exception? NO
As discussed previously with respect to Step 2A Prong Two, the additional element in the claim amounts to no more than mere instructions to apply the exception using a generic computer component.
The same analysis applies here in 2B, i.e., mere instructions to apply an exception using a generic computer component cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Dependent claims 35-36 further narrows the abstract idea defined in claim 34 and add the additional elements of “classifier module”, “NER classifier”.
Under step 2A, prong two, the additional elements don’t integrate the exception into a practical application of the exception as merely adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f).
As discussed previously with respect to Step 2A Prong Two, the additional elements in the claim amounts to no more than mere instructions to apply the exception using a generic computer component.
The same analysis applies here in 2B, i.e., mere instructions to apply an exception using a generic computer component cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Regarding claim 37:
Step 1: Is the claim to a process, machine, manufacture or composition of matter?” Yes, it’s a method(process).
Step 2a Prong 1 (judicial exception)
Step 2A (1): “Does the claim recite an abstract idea, law of nature, or natural phenomenon? Yes , the claim comes under mental processes.
Claim 1 recites:
“A method of estimating the effectiveness or safety of medicines, comprising: downloading items of commentary from at least one remote source; applying a commentary type classifier and a named entity recognition, NER, classifier to select from the downloaded commentary a plurality of items of commentary that include at least one medicine entity and that are identified as being a commentary type that is authored by a patient; for each of the plurality of items of commentary, calculating a plurality of feature indicators selected from at least one of: at least one aspect of a personal experience, at least one measure of the tone of the item of commentary, at least one detected stance of the patient, a count of the number of times a medicine is mentioned, a count of the number of times a relevant symptom is mentioned, a sentiment estimate, and a count of the number of times a relevant feeling or experience is mentioned; and calculating a summary score representative of the effectiveness or safety of a medicine in dependence on the feature indicators calculated for relevant items of the plurality of items of commentary..”
All the limitations above are abstract idea related to the mental process (concepts performed in the human mind (including an observation, evaluation, judgment, opinion)) with the exception of bold and underlined limitations. Claim language pertains to analyzing the opinions/comments about a medicine , received from a patient , who have used it. Relevant data/items can be selected to estimate overall effectiveness of the medicine. All of this can be done on paper. Also, a specific name of medicine can be identified and the number of times the medicine is being used can easily be counted by recording on the paper. By analyzing all the data an effectiveness score for a medicine can easily be calculated .
Step 2A(2): Prong Two: evaluate whether the claim recites additional elements that integrate the exception into a practical application of the exception. NO
The claim does recite additional elements; however they don’t integrate the exception into a practical application of the exception.
classifier (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
remote source (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
NER, classifier (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f))
Step 2B: evaluate whether the claim recites additional elements that amount to an inventive concept (aka “significantly more”) than the recited judicial exception? NO
As discussed previously with respect to Step 2A Prong Two, the additional element in the claim amounts to no more than mere instructions to apply the exception using a generic computer component.
The same analysis applies here in 2B, i.e., mere instructions to apply an exception using a generic computer component cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Dependent claim 38, further narrows the abstract idea describes in claim 37, and adds the additional elements of “NER classifier”.
Under step 2A, prong two, the additional element don’t integrate the exception into a practical application of the exception as merely adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea - see MPEP 2106.05(f).
As discussed previously with respect to Step 2A Prong Two, the additional element in the claim amounts to no more than mere instructions to apply the exception using a generic computer component.
The same analysis applies here in 2B, i.e., mere instructions to apply an exception using a generic computer component cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1- 24, 32, 34-35 and 39 are rejected under 35 U.S.C. 103 as being unpatentable over TAKEDA et al. ( US 20180011977 A1 in view of Admitted Prior Art , herein after APA and further in view of Hu et al. (US 20170316175 A1)
Regarding claim 1, TAKEDA teaches a method of estimating the effectiveness or safety of a medicine, the method comprising:
Receiving from a remote source, unstructured commentary data encoding a plurality of items of commentary substantially related to medical subject-matter (para, “[0041] The acceptance unit 131 has a function that accesses a network (for example, the Internet or the intranet) via the communication unit 110, acquires data on that network, and records the web page information in the memory unit 140. In this example, data handled by the data analysis system 100 mainly indicate data at least partly including texts such as document data (for example, materials about drugs, materials in which side effects of the drugs are described, various kinds of comments exchanged over the web, e-mails, presentation materials, spreadsheet materials, meeting materials, contracts, organization charts, and business plans)……..” Note: para 00154 for unstructured data. );
processing the unstructured commentary data using [a multi-stage classifier architecture comprising at least one named entity recognition(NER) classifier and] a commentary type classifier to identify for each item a commentary type and a list of medicines associated with the commentary (Para, “[0111] Therefore, for example, when a doctor wants to know various opinions about “treatment of hay fever,” it is possible to pick up (or classify and select) comments which may highly possibly and truly discuss the “treatment of hay fever” from among a large amount of hay fever topics if there are a plurality of pieces of learning data such as learning data based on classification of being “related to hay fever” or “not related to hay fever” and learning data based on classification of being “related to treatment” or “not related to treatment.”
Also, para, “[0096] The case information 503 is case information about side effects of drug A as indicated in the drug information 501 and includes information such as doctors' opinions and a patient's impressions.” Note: here, commentary type refers to doctor or patient opinions. Also, see Fig. 6.
Also, para, “[0105] In a case of information like the bulletin board 600, the data analysis system 100 classifies whether each comment is related to the relevant topic or not.”
Also, para “[0106] The data analysis system 100 designates some comments related to the topic “XX” and some other comments not related to the topic “XX” with respect to each comment from the users. Then, the data analysis system 100 recognizes the designated comments as training data, extracts data elements, calculates weighted values in accordance with classification information indicating whether it is related to the topic “XX” or not, and stores them in the memory unit 140. As a result, learning data about the topic “XX” is generated.”)
the [multi-stage ]classifier [architecture] employing machine learning to identify [linguistic patterns in the] unstructured commentary data [and to associate the linguistic patterns with specific corresponding outputs to disambiguate medicine entities from common names based on context]( para, “[0106] The data analysis system 100 designates some comments related to the topic “XX” and some other comments not related to the topic “XX” with respect to each comment from the users. Then, the data analysis system 100 recognizes the designated comments as training data, extracts data elements, calculates weighted values in accordance with classification information indicating whether it is related to the topic “XX” or not, and stores them in the memory unit 140. As a result, learning data about the topic “XX” is generated.”)
selecting a subset of items, from the plurality of items of commentary,[ by filtering the list of medicines against a medicine database encoding names associated with at least one jurisdiction and] identified as referencing the medicine and whose commentary type has been identified as commentary from a patient who has used the medicine (para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. It should be noted that the comments may sometimes include comments which are not related to the drug information 701 at all.”)
processing the subset of items to generate content analysis data including, for each item, at least one estimate quantifying a respective at least one aspect of an effect of the medicine as described by the patient in the commentary(para, “[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.” Note: Also, see Fig. 7:
PNG
media_image1.png
700
651
media_image1.png
Greyscale
para, (“[0033] …..So, for example, in a case of drug side effect reports, the data analysis system can suggest a report which reports what appears to be a side effect that should highly possibly be actually identified as the side effect, from among a large number of listed reports. Furthermore, for example, in a case of medical portal sites, the data analysis system can suggest serious information from among a large number of posted comments.”
Para, “[0099] For example, if a word “fatigue” appears in the case information, there is a possibility that the word “fatigue” may be extracted as a data element and associated with a weighted value and that weighted value is stored as learning data. Then, when new unknown data is accepted and data elements are extracted from the unknown data and “fatigue” exists in the data elements, a high score will be presented as information indicating a high possibility that it may indicate a side effect of the relevant drug. Accordingly, when unknown data which appears to be related to side effects of the drug, scores of each piece of learning data for each of many side effects are presented and a score based on learning data of a side effect estimated to be highly related becomes a high value. So, highly related side effects will be found; and regarding any side effect which has not been identified (or discovered), if its score is high, that can be discovered as a new side effect. Furthermore, if scores of the unknown data are low, the unknown data can be classified as those lowly related to the side effects. So, it is possible to reduce time required to view unnecessary reports. Therefore, the data analysis system 100 can classify the unknown data on the basis of whether highly or lowly the unknown data is related to the side effects or what kind of side effects the unknown appears to be highly related, so that it is possible to support classification when reports are made about the side effects of a large number of drugs.”)
and processing the content analysis data [to calculate a multi-dimensional vector ] indicative of the effectiveness or safety of the medicine(para, “[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.”
Also, para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. It should be noted that the comments may sometimes include comments which are not related to the drug information 701 at all.”
Para, “[0120] For example, if there are learning data created based on classification of “relating to drug A” or “not relating to drug A” and learning data created based on classification of “relating to efficacy” or “not relating to efficacy,” it is possible to classify unknown data with both high scores as data which may highly possibly be related to the efficacy of drug A, from among a plurality of comments; and if there is further learning data created based on classification of “relating to users in their twenties” or “not relating to users in their twenties,” it is also possible to classify and select unknown data which may highly possibly be related to “the efficacy of drug A on users in their twenties.”)
TAKEDA does not explicitly teach:
[processing the unstructured commentary data using ] a multi-stage classifier architecture comprising at least one named entity recognition(NER) classifier [and a commentary type classifier to identify for each item a commentary type and a list of medicines associated with the commentary]
the multi-stage classifier architecture [employing machine learning to identify linguistic patterns in the unstructured commentary data and to associate the linguistic patterns with specific corresponding outputs to disambiguate medicine entities from common names based on context];
[selecting a subset of items, from the plurality of items of commentary, by] filtering the list of medicines against a medicine database encoding names associated with at least one jurisdiction [and identified as referencing the medicine and whose commentary type has been identified as commentary from a patient who has used the medicine]
[and processing the content analysis data to] calculate a multi-dimensional vector [indicative of the effectiveness or safety of the medicine]
APA teaches :
[processing the unstructured commentary data using ] a multi-stage classifier architecture comprising at least one named entity recognition(NER) classifier [and a commentary type classifier to identify for each item a commentary type and a list of medicines associated with the commentary] (para, “[0203] When training the NER model and using it to tag posts with entities, the default configuration for spaCy's EntityRecognizer model was used. The model is backed by a neural network that uses a stacked embedding architecture. The details of spaCy's default EntityRecognizer are outlined in their video documentation (accessed via path watch?v=sqDHBH9IjRU on the www.youtube.com website).” Note; here, the applicant admits using spaCy's EntityRecognizer model, which was described in th eprovided link, at least 8 years ago.)
the multi-stage classifier architecture [employing machine learning to identify linguistic patterns in the unstructured commentary data and to associate the linguistic patterns with specific corresponding outputs to disambiguate medicine entities from common names based on context] (para, “[0203] When training the NER model and using it to tag posts with entities, the default configuration for spaCy's EntityRecognizer model was used. The model is backed by a neural network that uses a stacked embedding architecture. The details of spaCy's default EntityRecognizer are outlined in their video documentation (accessed via path watch?v=sqDHBH9IjRU on the www.youtube.com website).” Note; here, the applicant admits using spaCy's EntityRecognizer model, which was described in th eprovided link, at least 8 years ago.)
[and processing the content analysis data to] calculate a multi-dimensional vector [indicative of the effectiveness or safety of the medicine] (para, “[0203] When training the NER model and using it to tag posts with entities, the default configuration for spaCy's EntityRecognizer model was used. The model is backed by a neural network that uses a stacked embedding architecture. The details of spaCy's default EntityRecognizer are outlined in their video documentation (accessed via path watch?v=sqDHBH9IjRU on the www.youtube.com website).” Note: the video at the provided link teaches spaCy's default EntityRecognizer model using multi-dimentional vector)
It would have been obvious for a person of ordinary skill in the art to incorporate multi-stage classifier teachings of admitted prior art into the teachings of TAKEDA at the time the application was filed in order to give a good balance of efficiency, accuracy and adaptability. (See, description of the video; note link/video is not being provided, as it is part of the spec, and available to applicant.)
TAKEDA as modified by APA does not explicitly teach :
[the multi-stage classifier architecture employing machine learning to] identify linguistic patterns in the [unstructured commentary ] data and to associate the linguistic patterns with specific corresponding outputs to disambiguate medicine entities from common names based on context.
[selecting a subset of items, from the plurality of items of commentary, by] filtering the list of medicines against a medicine database encoding names associated with at least one jurisdiction [and identified as referencing the medicine and whose commentary type has been identified as commentary from a patient who has used the medicine]
Hu teaches :
[the multi-stage classifier architecture employing machine learning to] identify linguistic patterns in the [unstructured commentary ] data and to associate the linguistic patterns with specific corresponding outputs to disambiguate medicine entities from common names based on context(para, “[0084] Social media monitoring in this embodiment can utilise established text analysis techniques (including named entity recognition and potentially also linguistic patterns) to detect drug names and key symptoms and complains. For instance, “Bactrim gives me headache” or “had Bactrim . . . very bad headache” can be a main message on social media. NER (named entity recognition) technology can help to identify “Bactrim” as the name of the medicine and “Headache” as the key complaint.”
Also, para, “[0090] When extracting drug names and symptoms, established linguistic patterns can be used to differentiate between negative and positive relationships. For instance “Headache after taking Bactrim” and “headache gone, after taking Bactrim” can be differentiated to indicate the connection between drug and symptoms.......”
Also, para “[0087] One or more medical-domain specific ontologies of drugs and more general medical interventions stored in one or more ontology databases can be used to define, disambiguate, and reconcile names. An ontology can also capture domain knowledge of symptoms and complaints. Such ontologies can come from existing ontology repositories such as OBO (Open Biomedical Ontologies) or are designed from scratch with help from medical experts.”
Also, para “[0105] Relations extracted from public data sources can sometimes be very unspecific and/or ambiguous. In this case ontology can be used to present more specific results for better end-user/expert response. For instance, from public data sources, a relation can be established between naproxen and ulcer. Using ontologies, relation can be refined as “naproxen, stomach ulcer”, etc. to allow better filtering and screening.”)
[selecting a subset of items, from the plurality of items of commentary, by] filtering the list of medicines against a medicine database encoding names associated with at least one jurisdiction [and identified as referencing the medicine and whose commentary type has been identified as commentary from a patient who has used the medicine] (para, “[0106] Identified symptoms and drugs (in either or both of user queries and extracted relations) can be subject to knowledge refinement. Ontologies are used to broaden and/or narrow the extracted topics/keywords. The rationale behind such rewriting is that drugs normally sold and mentioned by their brand name while different companies distribute the same drug with different brand names. By using semantic technology, new queries (internal queries) can be generated against different brands of the same drag and/or against the generic names. For instance, FIG. 4 shows the correspondence of generic names and brand names:
PNG
media_image2.png
502
848
media_image2.png
Greyscale
Also, para, “[0114] Furthermore, both the drug and symptom can be refined S100 based on domain ontology database 60, although the social media relations can remain unchanged, for traceability. For instance, drugs can be replaced with their generic name and other brand names to extract apparently irrelevant ADRs, using the query/relation expansion previously described. Symptoms can be replaced by synonyms or largely similar symptoms but with different descriptions.)
It would have been obvious for a person of ordinary skill in the art to apply linguistic pattern and jurisdiction teachings of Hu into the teachings of TAKEDA as modified by APA at the time the application was filed in order to provide a relation between a drug and adverse reactions. (Hu, Abstract “A system to produce validated weighted relations between drugs and adverse drug reactions (ADRs). At least one processor to monitor social media for links between drugs and ADRs; extract a relation between a drug and an ADR using named entity recognition to provide a weighted relation between the drug and the ADR, the weight based on confidence of the link between the drug and the ADR in social media. …”)
Regarding claim 2, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
TAKEDA further teaches wherein the multi-stage classifier architecture is configured to identify at least the list of medicines associated with the commentary [by identifying linguistic patterns to disambiguate medicine entities based on context]( para, [0099].... Accordingly, when unknown data which appears to be related to side effects of the drug, scores of each piece of learning data for each of many side effects are presented and a score based on learning data of a side effect estimated to be highly related becomes a high value. So, highly related side effects will be found; and regarding any side effect which has not been identified (or discovered), if its score is high, that can be discovered as a new side effect. Furthermore, if scores of the unknown data are low, the unknown data can be classified as those lowly related to the side effects. So, it is possible to reduce time required to view unnecessary reports. Therefore, the data analysis system 100 can classify the unknown data on the basis of whether highly or lowly the unknown data is related to the side effects or what kind of side effects the unknown appears to be highly related, so that it is possible to support classification when reports are made about the side effects of a large number of drugs.”)
TAKEDA as modified by APA does not explicitly teach [wherein the multi-stage classifier architecture is configured to identify at least the list of medicines associated with the commentary by] identifying linguistic patterns to disambiguate medicine entities based on context.
Hu teaches [wherein the multi-stage classifier architecture is configured to identify at least the list of medicines associated with the commentary by] identifying linguistic patterns to disambiguate medicine entities based on context(para, “[0084] Social media monitoring in this embodiment can utilise established text analysis techniques (including named entity recognition and potentially also linguistic patterns) to detect drug names and key symptoms and complains. For instance, “Bactrim gives me headache” or “had Bactrim . . . very bad headache” can be a main message on social media. NER (named entity recognition) technology can help to identify “Bactrim” as the name of the medicine and “Headache” as the key complaint.”
Also, para, “[0090] When extracting drug names and symptoms, established linguistic patterns can be used to differentiate between negative and positive relationships. For instance “Headache after taking Bactrim” and “headache gone, after taking Bactrim” can be differentiated to indicate the connection between drug and symptoms.......”
Also, para “[0087] One or more medical-domain specific ontologies of drugs and more general medical interventions stored in one or more ontology databases can be used to define, disambiguate, and reconcile names. An ontology can also capture domain knowledge of symptoms and complaints. Such ontologies can come from existing ontology repositories such as OBO (Open Biomedical Ontologies) or are designed from scratch with help from medical experts.”
Also, para “[0105] Relations extracted from public data sources can sometimes be very unspecific and/or ambiguous. In this case ontology can be used to present more specific results for better end-user/expert response. For instance, from public data sources, a relation can be established between naproxen and ulcer. Using ontologies, relation can be refined as “naproxen, stomach ulcer”, etc. to allow better filtering and screening.”)
It would have been obvious for a person of ordinary skill in the art to apply linguistic pattern teachings of Hu into the teachings of TAKEDA as modified by APA at the time the application was filed in order to provide a relation between a drug and adverse reactions. (Hu, Abstract “A system to produce validated weighted relations between drugs and adverse drug reactions (ADRs). At least one processor to monitor social media for links between drugs and ADRs; extract a relation between a drug and an ADR using named entity recognition to provide a weighted relation between the drug and the ADR, the weight based on confidence of the link between the drug and the ADR in social media. …”)
Regarding claim 3, TAKEDA as modified by APA and Hu teaches a method according to Claim 2.
Hu further teaches wherein the NER classifier additionally identifies references to at least one of: type of medicine, ailment, condition, symptom, potential side effect, possible medical outcome, and treatment type (para, “[0012] According to an embodiment of a first aspect there is provided a system to produce and validate weighted relations between drugs and adverse drug reactions ADRs, the system comprising: a public data monitoring module to monitor social media for links between drugs and ADRs; a knowledge extraction module to extract a relation between a drug and an ADR using named entity recognition and to provide a weighted relation between the drug and the ADR based on confidence of the link between the drug and the ADR in the social media; a local knowledge base to store the relation with its weight; a relation refinement module using domain knowledge in an ontology database to refine the weighted social media relation in accordance with one or more ontologies of drug names and of ADR symptoms;…..”
It would have been obvious for a person of ordinary skill in the art to apply NER classifier teachings of Hu into the teachings of TAKEDA as modified by APAand Hu at the time the application was filed in order to provide a relation between a drug and adverse reactions. (Hu, Abstract “A system to produce validated weighted relations between drugs and adverse drug reactions (ADRs). At least one processor to monitor social media for links between drugs and ADRs; extract a relation between a drug and an ADR using named entity recognition to provide a weighted relation between the drug and the ADR, the weight based on confidence of the link between the drug and the ADR in social media. …”)
Regarding claim 4, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
TAKEDA further teaches wherein at least one said classifier further identifies personal experiences or feelings described by the patient (para, “[0113] FIG. 7 is a diagram illustrating an example of a web page indicating impressions about the use of a drug by users who used that drug.”
Also, para, “[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.”
Also, see Fig. 7 below:
PNG
media_image1.png
700
651
media_image1.png
Greyscale
Para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”)
Regarding claim 5, TAKEDA as modified by APA and Hu teaches a method according to claim 4.
TAKEDA further teaches wherein processing the subset of items to generate content analysis data further comprises processing the personal experiences or feelings described by the patient (para, “[0103] FIG. 6 is a diagram illustrating an example of a web page such as a so-called online bulletin board where various kinds of users' opinions about a viewpoint questioned by a questioner are posted on the Web. The viewpoint in this example relates to medical care such as effects of drugs, chemicals which seem to be necessary to make a desired drug, and effective methods for treatment of a specified injury or disease.” Note: Also, see Fig. 7)
Regarding claim 6, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
TAKEDA further teaches wherein said at least one aspect of an effect of the medicine includes at least one of: a perceived effectiveness of the medicine, a perceived safety of the medicine, happiness or unhappiness associated with the medicine, and satisfaction or dissatisfaction associated with the medicine (para 0113-0114, “[0113] FIG. 7 is a diagram illustrating an example of a web page indicating impressions about the use of a drug by users who used that drug.
[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.”
Note: Also, see Fig. 7 below:
PNG
media_image1.png
700
651
media_image1.png
Greyscale
)
Regarding claim 7, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
TAKEDA further teaches wherein processing the subset of items to generate content analysis data further comprises processing the subset of items to estimate a degree of positivity or negativity expressed by the patient in relation to at least one of: the commentary considered as a whole, each mention of the medicine individually, and every mention of the medicine considered as a whole (para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”
Also, para, “[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.”
Also, para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. …”)
Regarding claim 8, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
TAKEDA further teaches wherein processing the subset of items to generate content analysis data preferably further comprises processing the subset of items to calculate a sentiment estimate, encoding a measure of at least one of: positivity, negativity and neutrality, expressed by the patient in relation to at least one of: the commentary considered as a whole, each mention of the medicine individually, and every mention of the medicine considered as a whole(para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”
Also, para, “[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.”
Also, para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. …”)
Regarding claim 9, TAKEDA as modified by APA and Hu teaches a method according to claim 8.
TAKEDA further teaches further comprising dividing each item into at least one part, and wherein processing the subset of items to calculate a sentiment estimate comprises processing each part separately and wherein each sentiment estimate relates to a respective part (para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”
Para, “[0067] The element evaluation unit 136 evaluates the respective data elements extracted by the element extraction unit 135 and calculates their weighted values (step S205). The element evaluation unit 136 transmits the calculated weighted values to the element evaluation unit 136.”)
Regarding claim 10, TAKEDA as modified by APA and Hu teaches a method according to claim 8.
TAKEDA further teaches further comprising processing each part to determine whether or not to exclude the part from further processing (para, “[0143] The element evaluation unit 136 for the data analysis system 100 firstly associates emotional evaluations with respect to data elements included in the training data (data elements including the user's emotional expressions, for example, morphemes such as “fun” and “sad”) and stores them. For example, the element evaluation unit 136 searches texts included in the training data to check whether predetermined keywords (such keywords are words relating to emotions in a case of texts) are included in the relevant texts or not. If the keywords are included, the element evaluation unit 136 associates the keywords with emotion scores calculated in accordance with a specified standard and stores them in the memory unit 140.”
Also, para, “[0104] A bulletin board 600 includes comments 601 to 605 from various users. Sorting out these comments to check whether they are really related to the relevant topic or not can be cumbersome work; however, if the data analysis system 100 is used, the index (score) for judging whether each comment is related to the relevant topic or not can be presented. Regarding the comments 601 to 605, some of the comments are related to the topic and some of them are not.”
Also, para “[0132] The unknown data evaluation unit 138 generates a vector indicating whether or not a specified data element (for example, a keyword) is included in each piece of the partial data, for each piece of the partial data. Then, the unknown data evaluation unit 138 executes scoring of the unknown data according to the following expression (5).”)
Regarding claim 11, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
TAKEDA further teaches further comprising processing each of the subset of items to identify at least one specific characteristic in the item and, if a said specific characteristic is identified, to exclude the item from further processing (para, “[0047] The data elements extracted by the element extraction unit 135 are selected by the data analysis system 100 in accordance with specified selection standards. As an example of a method for selecting the data elements under this circumstance, data elements which frequently appear in the relevant training data corresponding to the classification indicated by the classification information may be used. For example, when the classification information is managed by two values indicating that the relevant data is “related to” or “not related to” a specified event, the data elements may be selected by selecting remaining keywords, as the data elements, that are left after removing keywords extracted from training data, which is not related to the specified event, from keywords extracted from training data related to the specified event. Furthermore, the data elements may be designated by the user to the data analysis system 100 by using the input unit 120.”)
Regarding claim 12, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
APA further teaches wherein the multi-stage classifier architecture further comprises a commentary type classifier configured to operate in sequence with the NER classifier(para, “[0203] When training the NER model and using it to tag posts with entities, the default configuration for spaCy's EntityRecognizer model was used. The model is backed by a neural network that uses a stacked embedding architecture. The details of spaCy's default EntityRecognizer are outlined in their video documentation (accessed via path watch?v=sqDHBH9IjRU on the www.youtube.com website).” Note: para 0203 teaches stacked architechture, which operates in sequence. Also, see 112(a) rejection.)
It would have been obvious for a person of ordinary skill in the art to incorporate multi-stage classifier teachings of admitted prior art into the teachings of TAKEDA at the time the application was filed in order to give a good balance of efficiency, accuracy and adaptability. (See, description of the video; note link/video is not being provided, as it is part of the spec, and available to applicant.)
Regarding claim 13, TAKEDA as modified by APA and Hu teaches a method according to claim 12.
TAKEDA further teaches wherein the commentary type classifier is configured to identify at least one commentary type, selected from at least one of: patient opinion, medical reference data, medical professional opinion, scientific report, industry report, news report, and structured feedback (para, “[0103] FIG. 6 is a diagram illustrating an example of a web page such as a so-called online bulletin board where various kinds of users' opinions about a viewpoint questioned by a questioner are posted on the Web. The viewpoint in this example relates to medical care such as effects of drugs, chemicals which seem to be necessary to make a desired drug, and effective methods for treatment of a specified injury or disease.”
Para, “[0096] The case information 503 is case information about side effects of drug A as indicated in the drug information 501 and includes information such as doctors' opinions and a patient's impressions.”)
Regarding claim 14, TAKEDA as modified by APA and Hu teaches a method according to claim 12.
TAKEDA further teaches wherein the commentary type classifier is configured to identify at least one author type, selected from at least one of: patient, medical professional, medicine representative, research scientist, journalist; and other type(para, “[0096] The case information 503 is case information about side effects of drug A as indicated in the drug information 501 and includes information such as doctors' opinions and a patient's impressions.” Note: Also, see para 0111)
Regarding claim 15, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
TAKEDA further teaches wherein the [multi-stage] classifier [architecture] is configured in dependence on the medicine (para, “[0044] For example, the specified event herein used may be a “side effect of a drug,” “efficacy evaluation of a drug,” or a “specified topic on a web page” and various events may be relevant. Furthermore, regarding the classification information, for example, in a case of the “side effect of a drug,” it is possible to use the classification information indicating “related to the side effect” or “not related to the side effect”; and in a case of the “efficacy evaluation of a drug,” it is possible to use the classification information indicating “very good,” “good,” “average,” “bad,” and “very bad”; and in a case of the “specified topic on a web page,” it is possible to use the classification information indicating “related to the topic” or “not related to the topic.” The content of classification and the classification information are decided by the user. Furthermore, the number of pieces of classification information is not limited as long as there are at least two levels of classification information as mentioned in the examples above. “)
TAKEDA as modified by APA and Hu does not explicitly teach [wherein] the multi-stage classifier architecture [is configured in dependence on the medicine]
APA further teaches [wherein] the multi-stage classifier architecture [is configured in dependence on the medicine] para, “[0203] When training the NER model and using it to tag posts with entities, the default configuration for spaCy's EntityRecognizer model was used. The model is backed by a neural network that uses a stacked embedding architecture. The details of spaCy's default EntityRecognizer are outlined in their video documentation (accessed via path watch?v=sqDHBH9IjRU on the www.youtube.com website).” Note: the video at the provided link teaches spaCy's default EntityRecognizer model using multi-dimentional vector)
It would have been obvious for a person of ordinary skill in the art to incorporate multi-stage classifier teachings of admitted prior art into the teachings of TAKEDA at the time the application was filed in order to give a good balance of efficiency, accuracy and adaptability. (See, description of the video; note link/video is not being provided, as it is part of the spec, and available to applicant.)
Regarding claim 16, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
TAKEDA further teaches wherein selecting the subset of items further comprises filtering the subset of items by one of a plurality of cohorts, and calculating a respective estimate indicative of the overall effectiveness or safety of the medicine for each cohort (para, “[0031] Therefore, a data analysis system according to this embodiment analyzes to which event among a plurality of events the input data is highly related. In order to do so, the data analysis system: firstly extracts data elements from data related to one event among the plurality of events and from data which is not related to the event; calculates respective weighted values of these data elements; associates the corresponding weighted values with the respective data elements; and stores them as first learning data. This processing is executed for each event to generate as many pieces of learning data as the number of events.” Also, see Fig. 5.
Para, “[0120] For example, if there are learning data created based on classification of “relating to drug A” or “not relating to drug A” and learning data created based on classification of “relating to efficacy” or “not relating to efficacy,” it is possible to classify unknown data with both high scores as data which may highly possibly be related to the efficacy of drug A, from among a plurality of comments; and if there is further learning data created based on classification of “relating to users in their twenties” or “not relating to users in their twenties,” it is also possible to classify and select unknown data which may highly possibly be related to “the efficacy of drug A on users in their twenties.”)
Regarding claim 17, TAKEDA as modified by APA and Hu teaches a method according to Claim 1.
TAKEDA as modified by APA and Hu does not explicitly teach:
accessing a medicine database containing medicine data that encodes a plurality of medicine names associated with at least one jurisdiction;
retrieving data from the medicine database in accordance with a search query, the retrieved data including at least one medicine name, wherein selecting the subset of items further comprises processing items of commentary which include at least one said at least one medicine name.
Hu further teaches:
accessing a medicine database containing medicine data that encodes a plurality of medicine names associated with at least one jurisdiction (para, “[0106] Identified symptoms and drugs (in either or both of user queries and extracted relations) can be subject to knowledge refinement. Ontologies are used to broaden and/or narrow the extracted topics/keywords. The rationale behind such rewriting is that drugs normally sold and mentioned by their brand name while different companies distribute the same drug with different brand names. By using semantic technology, new queries (internal queries) can be generated against different brands of the same drag and/or against the generic names. For instance, FIG. 4 shows the correspondence of generic names and brand names:
PNG
media_image2.png
502
848
media_image2.png
Greyscale
Also, para, “[0114] Furthermore, both the drug and symptom can be refined S100 based on domain ontology database 60, although the social media relations can remain unchanged, for traceability. For instance, drugs can be replaced with their generic name and other brand names to extract apparently irrelevant ADRs, using the query/relation expansion previously described. Symptoms can be replaced by synonyms or largely similar symptoms but with different descriptions.);
retrieving data from the medicine database in accordance with a search query, the retrieved data including at least one medicine name, wherein selecting the subset of items further comprises processing items of commentary which include at least one said at least one medicine name (para, “[0012] According to an embodiment of a first aspect there is provided a system to produce and validate weighted relations between drugs and adverse drug reactions ADRs, the system comprising: a public data monitoring module to monitor social media for links between drugs and ADRs; a knowledge extraction module to extract a relation between a drug and an ADR using named entity recognition and to provide a weighted relation between the drug and the ADR based on confidence of the link between the drug and the ADR in the social media; a local knowledge base to store the relation with its weight; a relation refinement module using domain knowledge in an ontology database to refine the weighted social media relation in accordance with one or more ontologies of drug names and of ADR symptoms; a quantification ADR module to further quantify the weighted social media relation by using drug and ADR links extracted from research publications and/or from clinical trial reports and providing a research weight for the relation; and/or to quantify the weighted social media relations by using an internet search engine and searching for the drug and the ADR, numbers of hits quantifying an internet weight for the relation.”)
It would have been obvious for a person of ordinary skill in the art to apply medicine name teachings of Hu into the teachings of TAKEDA as modified by APA and Hu at the time the application was filed in order to provide a relation between a drug and adverse reactions. (Hu, Abstract “A system to produce validated weighted relations between drugs and adverse drug reactions (ADRs). At least one processor to monitor social media for links between drugs and ADRs; extract a relation between a drug and an ADR using named entity recognition to provide a weighted relation between the drug and the ADR, the weight based on confidence of the link between the drug and the ADR in social media. …”)
Regarding claim 18, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
TAKEDA further teaches wherein said plurality of items are initially selected by performing a plurality of searching or filtering operations to find a respective plurality of sets of items of commentary, and forming the plurality of items from the plurality of sets of items of commentary (para, “[0143] The element evaluation unit 136 for the data analysis system 100 firstly associates emotional evaluations with respect to data elements included in the training data (data elements including the user's emotional expressions, for example, morphemes such as “fun” and “sad”) and stores them. For example, the element evaluation unit 136 searches texts included in the training data to check whether predetermined keywords (such keywords are words relating to emotions in a case of texts) are included in the relevant texts or not. If the keywords are included, the element evaluation unit 136 associates the keywords with emotion scores calculated in accordance with a specified standard and stores them in the memory unit 140.”
Para, “[0033] As a result, the data analysis system can present an index for judging to which event the unknown data is highly related, depending on whether the score is high or low. Therefore, the data analysis system can present the index based on a plurality of standards (training data). So, for example, in a case of drug side effect reports, the data analysis system can suggest a report which reports what appears to be a side effect that should highly possibly be actually identified as the side effect, from among a large number of listed reports. Furthermore, for example, in a case of medical portal sites, the data analysis system can suggest serious information from among a large number of posted comments.”)
Regarding claim 19, TAKEDA as modified by APA and Hu teaches a method according to claim 17.
TAKEDA further teaches wherein the searching or filtering operations include searching or filtering by medicine name and additionally by at least one of: commentary type, author type, type of medicine, ailment, condition, symptom, potential side effect, possible medical outcome, treatment type, an aspect of personal experience, measure of the tone of the commentary, detected stance of the patient, and feeling or emotion (para, “[0182] (h) Regarding the data analysis system according to any one of (a) to (f) above, the information about the medicinal drug may be information about the medical personnel's opinion about a specified viewpoint about the medicinal drug.”
Also, para “[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.”
Also, para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. It should be noted that the comments may sometimes include comments which are not related to the drug information 701 at all.”)
Regarding claim 20, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
TAKEDA further teaches wherein processing the multi-dimensional feature vector further comprises:
processing a plurality of feature indicators selected from at least one of: at least one aspect of a personal experience; at least one measure of the commentary, at least one detected stance of the patient, a count of a number of times a medicine is mentioned, a count of the number of times a relevant symptom is mentioned, a sentiment estimate, and a count of the number of times a relevant feeling or experience is mentioned; and combining the indicators to generate the estimate indicative of the overall effectiveness or safety of the medicine (para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. …”
Also, para, “[0033]….. Therefore, the data analysis system can present the index based on a plurality of standards (training data). So, for example, in a case of drug side effect reports, the data analysis system can suggest a report which reports what appears to be a side effect that should highly possibly be actually identified as the side effect, from among a large number of listed reports. Furthermore, for example, in a case of medical portal sites, the data analysis system can suggest serious information from among a large number of posted comments.”
Also, para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”)
and combining the indicators to generate the estimate indicative of the overall effectiveness or safety of the medicine (para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”
Para, “[0167] (a) A data analysis system according to the present invention includes: a training data acquisition unit (132, 133) hat acquires a combination of training data including information about a medicinal drug and a plurality of pieces of classification information for classifying the training data on the basis of a plurality of classification standards; a learning unit (134 to 137) that learns a pattern of the information about the medicinal drug from distribution of data elements which constitute at least part of the training data and appear according to the classification information; an unknown data acquisition unit (131, 132) that acquires unknown data from a specified information source; a data evaluation unit (138) that evaluates the acquired unknown data on the basis of the learned pattern with respect to each of the plurality of classification standards; and a presentation unit (139) that presents the information about the medicinal drug included in the unknown data to a user according to evaluation by the data evaluation unit.”)
Regarding claim 21, TAKEDA as modified by APA and Hu teaches a method according to claim 20.
TAKEDA further teaches wherein calculating the estimate indicative of the effectiveness or safety of the medicine comprises selecting a set of the feature indicators, applying a respective weighting to each selected feature indicator [based on jurisdictional relevance indicators retrieved from the medicine database], and combining a weighted plurality of feature indicators into a single estimate indicative of the effectiveness or safety of the medicine (para, “[0015] Furthermore, the extraction unit may extract morphemes relating to an emotional expression as each of the data elements; the calculation unit may calculate a weighted value of the morpheme relating to the emotional expression; and the data evaluation unit may evaluate the unknown data on the basis of the morpheme relating to the emotional expression included in the unknown data with respect to each of the plurality of classification standards.”
Also, para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.” Note: Also, see para 0051.
Para, “[0068] The element evaluation unit 136 calculates weighted values by adding weighted values calculated for other data elements to the above-obtained weighted values of the data elements by using the expression (2) mentioned above (step S206). The element evaluation unit 136 transmits data elements corresponding to the calculated weighted values to the evaluation storage unit 137.”
“[0047] The data elements extracted by the element extraction unit 135 are selected by the data analysis system 100 in accordance with specified selection standards. As an example of a method for selecting the data elements under this circumstance, data elements which frequently appear in the relevant training data corresponding to the classification indicated by the classification information may be used. For example, when the classification information is managed by two values indicating that the relevant data is “related to” or “not related to” a specified event, the data elements may be selected by selecting remaining keywords, as the data elements, that are left after removing keywords extracted from training data, which is not related to the specified event, from keywords extracted from training data related to the specified event. Furthermore, the data elements may be designated by the user to the data analysis system 100 by using the input unit 120.”)
TAKEDA as modified by APA and Hu does not explicitly teach [selecting a set of the feature indicators, applying a respective weighting to each selected feature indicator based on] jurisdictional relevance indicators retrieved from the medicine database, [and combining a weighted plurality of feature indicators into a single estimate indicative of the effectiveness or safety of the medicine]
Regarding claim 22, TAKEDA as modified by APA and Hu teaches a method according to claim 21.
TAKEDA further teaches wherein the selected set of feature indicators includes at least one primary indicator of effectiveness or safety of the medicine and at least one relevance indicator, representing a measure of the amount of opinion expressed (para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”)
Regarding claim 23, TAKEDA as modified by APA and Hu teaches a method according to claim 21.
TAKEDA further teaches further comprising restricting at least one of the selected feature indicators to a sub-range within the output range of the single estimate (para, “[0123] (1) In the above-described embodiment, the unknown data evaluation unit 138 calculates the score of unknown data by calculating an inner product between the data element vector and the weight of each data element; however, this calculation method is just an example. The unknown data evaluation unit 138 may calculate the score of the unknown data by using other calculation methods. For example, the unknown data evaluation unit 138 may calculate score S of the unknown data by using the following expression (3) instead of the aforementioned expression (2).” Note: here, by attaching weight factor, one is ensuring that at least one feature indicator is restricted to range below 100%; also in second calculation metho presented, the frequency is used, so in that type of calculation as long there are more than one feature indicators, the indicator will be restricted to less than 100, as it will have to account for other factor contribution.)
Regarding claim 24, TAKEDA as modified by APA and Hu teaches a method according to claim 20.
TAKEDA as modified by APA and Hu does not explicitly teaches wherein processing the content analysis data is carried out with respect to a predetermined time period, wherein the subset of items is selected in respect of commentary falling within the predetermined time period.
Hu further teaches wherein processing the content analysis data is carried out with respect to a predetermined time period, wherein the subset of items is selected in respect of commentary falling within the predetermined time period (Para, “[0100] In this method, given a specific time window, the confidence is the ratio of number of mentioning to the total number of data items (e.g. tweets). The time window is split into different fragment and the overall confidence is the sum of the ratios of all fragments adjusted by an exponential decaying factor. Note that the overall confidence is not necessarily between 0-1. But this number can be normalised against a popular benchmark topic to bring the value into 0-1 as follows, where topic is an arbitrary popular topic to gauge the popularity of the joint topic of D and S.”
“[0143] Local cached data are associated with a time stamp indicating when the data was last updated. If the time elapse is too long (over a threshold S140), at check S130 the public data monitoring module will update the local data, for example including a check whether the external data have been updated since the time stamp. If yes, live data extraction will be carried out in step S150, before returning an answer to the user in step S160.”)
It would have been obvious for a person of ordinary skill in the art to apply time period teachings of Hu into the teachings of TAKEDA as modified by APA and Hu at the time the application was filed in order to use the recent/updated data for analysis. (Para, “[0143] Local cached data are associated with a time stamp indicating when the data was last updated. If the time elapse is too long (over a threshold S140), at check S130 the public data monitoring module will update the local data, ….”)
Regarding claim 34, TAKEDA teaches a computer system for estimating the effectiveness or safety of medicines, the computer system including:
a commentary downloader module for downloading items of commentary from at least one remote source(para, “[0041] The acceptance unit 131 has a function that accesses a network (for example, the Internet or the intranet) via the communication unit 110, acquires data on that network, and records the web page information in the memory unit 140. In this example, data handled by the data analysis system 100 mainly indicate data at least partly including texts such as document data (for example, materials about drugs, materials in which side effects of the drugs are described, various kinds of comments exchanged over the web, e-mails, presentation materials, spreadsheet materials, meeting materials, contracts, organization charts, and business plans)……..”);
a commentary type classifier module for identifying the type of commentary((Para, “[0111] Therefore, for example, when a doctor wants to know various opinions about “treatment of hay fever,” it is possible to pick up (or classify and select) comments which may highly possibly and truly discuss the “treatment of hay fever” from among a large amount of hay fever topics if there are a plurality of pieces of learning data such as learning data based on classification of being “related to hay fever” or “not related to hay fever” and learning data based on classification of being “related to treatment” or “not related to treatment.”
Also, para, “[0096] The case information 503 is case information about side effects of drug A as indicated in the drug information 501 and includes information such as doctors' opinions and a patient's impressions.” Note: here, commentary type refers to doctor or patient opinions. Also, see Fig. 6.);
a commentary importer module which accesses and applies the commentary type classifier module(para, “[0105] In a case of information like the bulletin board 600, the data analysis system 100 classifies whether each comment is related to the relevant topic or not.”
Also, para, “[0106] The data analysis system 100 designates some comments related to the topic “XX” and some other comments not related to the topic “XX” with respect to each comment from the users. Then, the data analysis system 100 recognizes the designated comments as training data, extracts data elements, calculates weighted values in accordance with classification information indicating whether it is related to the topic “XX” or not, and stores them in the memory unit 140. As a result, learning data about the topic “XX” is generated.”
Also, para, “[0111] Therefore, for example, when a doctor wants to know various opinions about “treatment of hay fever,” it is possible to pick up (or classify and select) comments which may highly possibly and truly discuss the “treatment of hay fever” from among a large amount of hay fever topics if there are a plurality of pieces of learning data such as learning data based on classification of being “related to hay fever” or “not related to hay fever” and learning data based on classification of being “related to treatment” or “not related to treatment.”)
[the NER classifier module and] the medicine data in the medicine database, to select from the downloaded items of commentary a plurality of items of commentary that include at least one medicine entity, that include at least one appropriate medicine name [validated against the at least one jurisdiction in the medicine database, ] and that are identified as being a commentary type that is authored by a patient(para, “[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.”)
Also, para, “0117] When handling such a web page 700, the data analysis system 100 designates some comments related to the drug indicated in the drug information 701 and some comments not related to that drug with respect to the comments and extracts data elements from these comments in the same manner as in (Example 2) described above. Then, the data analysis system 100 calculates weighted values of the extracted data elements and stores them as learning data about drug A in the memory unit 140.”);
a feature calculator module configured to calculate for each item of commentary a plurality of feature indicators selected from at least one of: at least one aspect of a personal experience, at least one measure of a tone of the item of commentary, at least one detected stance of the patient, a count of a number of times a medicine is mentioned, a count of the number of times a relevant symptom is mentioned, a sentiment estimate, and a count of the number of times a relevant feeling or experience is mentioned(para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”
para, “[0143] The element evaluation unit 136 for the data analysis system 100 firstly associates emotional evaluations with respect to data elements included in the training data (data elements including the user's emotional expressions, for example, morphemes such as “fun” and “sad”) and stores them. For example, the element evaluation unit 136 searches texts included in the training data to check whether predetermined keywords (such keywords are words relating to emotions in a case of texts) are included in the relevant texts or not. If the keywords are included, the element evaluation unit 136 associates the keywords with emotion scores calculated in accordance with a specified standard and stores them in the memory unit 140.” Note: Also, see Fig. 7);
and a summary score calculator module for calculating a summary score representative of the effectiveness or safety of a medicine in dependence on the feature indicators calculated by the feature calculator module for relevant items of the plurality of items of commentary(para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”).
Wherein the summary score calculator module is further configured to weight the summary score based on a relevance indicator [associated with the at least one jurisdiction] (para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”).
TAKEDA does not explicitly teach:
a multistage classifier architecture comprising:
a named entity recognition, NER, classifier module for identifying entities associated with each item of commentary
a medicine database encoding medicine data that encodes a plurality of medicine names associated with at least one jurisdiction;
the NER classifier module [and the medicine data in the medicine database, to select from the downloaded items of commentary a plurality of items of commentary that include at least one medicine entity, that include at least one appropriate medicine name] validated against the at least one jurisdiction in the medicine database, [and that are identified as being a commentary type that is authored by a patient.
[Wherein the summary score calculator module is further configured to weight the summary score based on a relevance indicator associated with the] at least one jurisdiction
APA teaches :
a multistage classifier architecture comprising(para, “[0203] When training the NER model and using it to tag posts with entities, the default configuration for spaCy's EntityRecognizer model was used. The model is backed by a neural network that uses a stacked embedding architecture. The details of spaCy's default EntityRecognizer are outlined in their video documentation (accessed via path watch?v=sqDHBH9IjRU on the www.youtube.com website).” Note: the video at the provided link teaches spaCy's default EntityRecognizer model using multi-dimentional vector)
It would have been obvious for a person of ordinary skill in the art to incorporate multi-stage classifier teachings of admitted prior art into the teachings of TAKEDA at the time the application was filed in order to give a good balance of efficiency, accuracy and adaptability. (See, description of the video; note link/video is not being provided, as it is part of the spec, and available to applicant.)
TAKEDA as modified by APA does not explicitly teach:
a named entity recognition, NER, classifier module for identifying entities associated with each item of commentary
a medicine database encoding medicine data that encodes a plurality of medicine names associated with at least one jurisdiction;
the NER classifier module [and the medicine data in the medicine database, to select from the downloaded items of commentary a plurality of items of commentary that include at least one medicine entity, that include at least one appropriate medicine name] validated against the at least one jurisdiction in the medicine database, [and that are identified as being a commentary type that is authored by a patient.
Hu teaches:
a named entity recognition, NER, classifier module for identifying entities associated with each item of commentary(para, “[0012] According to an embodiment of a first aspect there is provided a system to produce and validate weighted relations between drugs and adverse drug reactions ADRs, the system comprising: a public data monitoring module to monitor social media for links between drugs and ADRs; a knowledge extraction module to extract a relation between a drug and an ADR using named entity recognition and to provide a weighted relation between the drug and the ADR based on confidence of the link between the drug and the ADR in the social media; a local knowledge base to store the relation with its weight; a relation refinement module using domain knowledge in an ontology database to refine the weighted social media relation in accordance with one or more ontologies of drug names and of ADR symptoms;”)
a medicine database encoding medicine data that encodes a plurality of medicine names associated with at least one jurisdiction(para, “[0106] Identified symptoms and drugs (in either or both of user queries and extracted relations) can be subject to knowledge refinement. Ontologies are used to broaden and/or narrow the extracted topics/keywords. The rationale behind such rewriting is that drugs normally sold and mentioned by their brand name while different companies distribute the same drug with different brand names. By using semantic technology, new queries (internal queries) can be generated against different brands of the same drag and/or against the generic names. For instance, FIG. 4 shows the correspondence of generic names and brand names:
PNG
media_image2.png
502
848
media_image2.png
Greyscale
Also, para, “[0114] Furthermore, both the drug and symptom can be refined S100 based on domain ontology database 60, although the social media relations can remain unchanged, for traceability. For instance, drugs can be replaced with their generic name and other brand names to extract apparently irrelevant ADRs, using the query/relation expansion previously described. Symptoms can be replaced by synonyms or largely similar symptoms but with different descriptions.);
the NER classifier module [and the medicine data in the medicine database, to select from the downloaded items of commentary a plurality of items of commentary that include at least one medicine entity, that include at least one appropriate medicine name] validated against the at least one jurisdiction in the medicine database, [and that are identified as being a commentary type that is authored by a patient.(para, “[0084] Social media monitoring in this embodiment can utilise established text analysis techniques (including named entity recognition and potentially also linguistic patterns) to detect drug names and key symptoms and complains. For instance, “Bactrim gives me headache” or “had Bactrim . . . very bad headache” can be a main message on social media. NER (named entity recognition) technology can help to identify “Bactrim” as the name of the medicine and “Headache” as the key complaint.”
Also, para, “[0106] Identified symptoms and drugs (in either or both of user queries and extracted relations) can be subject to knowledge refinement. Ontologies are used to broaden and/or narrow the extracted topics/keywords. The rationale behind such rewriting is that drugs normally sold and mentioned by their brand name while different companies distribute the same drug with different brand names. By using semantic technology, new queries (internal queries) can be generated against different brands of the same drag and/or against the generic names. For instance, FIG. 4 shows the correspondence of generic names and brand names:
PNG
media_image2.png
502
848
media_image2.png
Greyscale
Also, para, “[0114] Furthermore, both the drug and symptom can be refined S100 based on domain ontology database 60, although the social media relations can remain unchanged, for traceability. For instance, drugs can be replaced with their generic name and other brand names to extract apparently irrelevant ADRs, using the query/relation expansion previously described. Symptoms can be replaced by synonyms or largely similar symptoms but with different descriptions.);
[Wherein the summary score calculator module is further configured to weight the summary score based on a relevance indicator associated with the] at least one jurisdiction, para, “[0106] Identified symptoms and drugs (in either or both of user queries and extracted relations) can be subject to knowledge refinement. Ontologies are used to broaden and/or narrow the extracted topics/keywords. The rationale behind such rewriting is that drugs normally sold and mentioned by their brand name while different companies distribute the same drug with different brand names. By using semantic technology, new queries (internal queries) can be generated against different brands of the same drag and/or against the generic names. For instance, FIG. 4 shows the correspondence of generic names and brand names:
PNG
media_image2.png
502
848
media_image2.png
Greyscale
Also, para, “[0114] Furthermore, both the drug and symptom can be refined S100 based on domain ontology database 60, although the social media relations can remain unchanged, for traceability. For instance, drugs can be replaced with their generic name and other brand names to extract apparently irrelevant ADRs, using the query/relation expansion previously described. Symptoms can be replaced by synonyms or largely similar symptoms but with different descriptions
It would have been obvious for a person of ordinary skill in the art to apply NER classifier teachings of Hu into the teachings of TAKEDA as modified by APA at the time the application was filed in order to provide a relation between a drug and adverse reactions. Hu, Abstract “A system to produce validated weighted relations between drugs and adverse drug reactions (ADRs). At least one processor to monitor social media for links between drugs and ADRs; extract a relation between a drug and an ADR using named entity recognition to provide a weighted relation between the drug and the ADR, the weight based on confidence of the link between the drug and the ADR in social media. …”);
Regarding claim 35, TAKEDA as modified by APA and Hu teaches a method according to claim 34.
TAKEDA further teaches wherein the sentiment estimate encodes a measure of at least one of: positivity, negativity and neutrality, expressed by the patient in relation to at least one of: the commentary considered as a whole, each mention of the medicine individually, and every mention of the medicine considered as a whole(para, “[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.”
Also, para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. It should be noted that the comments may sometimes include comments which are not related to the drug information 701 at all.” Note : Also, see Fig. 7)
Claims 37-38 are rejected under 35 U.S.C. 103 as being unpatentable over TAKEDA in view of Hu (US 20170316175 A1)
Regarding claim 37, TAKEDA teaches a method of estimating the effectiveness or safety of medicines, comprising:
downloading items of commentary from at least one remote source(para, “[0041] The acceptance unit 131 has a function that accesses a network (for example, the Internet or the intranet) via the communication unit 110, acquires data on that network, and records the web page information in the memory unit 140. In this example, data handled by the data analysis system 100 mainly indicate data at least partly including texts such as document data (for example, materials about drugs, materials in which side effects of the drugs are described, various kinds of comments exchanged over the web, e-mails, presentation materials, spreadsheet materials, meeting materials, contracts, organization charts, and business plans)……..”);
applying a commentary type classifier[ and a named entity recognition, NER, classifier] to select from the downloaded commentary a plurality of items of commentary that include at least one medicine entity and that are identified as being a commentary type that is authored by a patient((para, “[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.”)
Also, para, “0117] When handling such a web page 700, the data analysis system 100 designates some comments related to the drug indicated in the drug information 701 and some comments not related to that drug with respect to the comments and extracts data elements from these comments in the same manner as in (Example 2) described above. Then, the data analysis system 100 calculates weighted values of the extracted data elements and stores them as learning data about drug A in the memory unit 140.”);
for each of the plurality of items of commentary, calculating a plurality of feature indicators to generate a structured feature set, the feature indicators selected from at least one of:
at least one aspect of a personal experience, at least one measure of a tone of the item of commentary, at least one detected stance of the patient, a count of the number of times a medicine is mentioned, a count of a number of times a relevant symptom is mentioned, a sentiment estimate, and a count of the number of times a relevant feeling or experience is mentioned para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”
para, “[0143] The element evaluation unit 136 for the data analysis system 100 firstly associates emotional evaluations with respect to data elements included in the training data (data elements including the user's emotional expressions, for example, morphemes such as “fun” and “sad”) and stores them. For example, the element evaluation unit 136 searches texts included in the training data to check whether predetermined keywords (such keywords are words relating to emotions in a case of texts) are included in the relevant texts or not. If the keywords are included, the element evaluation unit 136 associates the keywords with emotion scores calculated in accordance with a specified standard and stores them in the memory unit 140.” Note: Also, see Fig. 7);
and calculating a summary score representative of the effectiveness or safety of a medicine by performing a weighted aggregation of the structured feature set in dependence on the feature indicators calculated for relevant items of the plurality of items of commentary(para, “[0145] For example, let us assume that a sentence reciting that “it is great that this drug was highly effective, but it is a little disappointing that it caused a state close to a manic state” is included in the text and “great” and “disappointing” are stored as keywords in the memory unit 140 in advance and they are associated with the emotion scores “+1.4” and “+0.1,” respectively. In this case, the unknown evaluation unit 136 calculates, for example, the emotion score “+1.5” as the emotion score of the relevant text by adding both the above-mentioned scores.”
Also, para “[0042] The data extraction unit 132 has a function that extracts data from data stored in the memory unit 140, as the need arises. The data extraction unit 132 transmits data used to calculate the weighted values of the data elements to the data classification unit 134. Furthermore, the data extraction unit 132 extracts the unknown data, whose score has not been calculated, from the memory unit 140 and transmits it to the unknown data evaluation unit 138.” Note: Also, see para 0056)
TAKEDA does not explicitly teach:
[applying a commentary type classifier and] a named entity recognition, NER, classifier [to select from the downloaded commentary a plurality of items of commentary that include at least one medicine entity and that are identified as being a commentary type that is authored by a patient]
Hu teaches:
[applying a commentary type classifier and] a named entity recognition, NER, classifier [to select from the downloaded commentary a plurality of items of commentary that include at least one medicine entity and that are identified as being a commentary type that is authored by a patient](para, “[0084] Social media monitoring in this embodiment can utilise established text analysis techniques (including named entity recognition and potentially also linguistic patterns) to detect drug names and key symptoms and complains. For instance, “Bactrim gives me headache” or “had Bactrim . . . very bad headache” can be a main message on social media. NER (named entity recognition) technology can help to identify “Bactrim” as the name of the medicine and “Headache” as the key complaint.”
It would have been obvious for a person of ordinary skill in the art to apply NER classifier teachings of Hu into the teachings of TAKEDA at the time the application was filed in order to provide a relation between a drug and adverse reactions. (Hu, Abstract “A system to produce validated weighted relations between drugs and adverse drug reactions (ADRs). At least one processor to monitor social media for links between drugs and ADRs; extract a relation between a drug and an ADR using named entity recognition to provide a weighted relation between the drug and the ADR, the weight based on confidence of the link between the drug and the ADR in social media. …”);
Regarding claim 39, TAKEDA as modified by APA teaches a non-transitory computer readable medium encoding computer program code which, when executed on at least one processor of a computer, causes the computer to carry out the method of Claim1 (see claim 10 of TAKEDA, and rejection with regard to claim 1 above.)
Claim 25, 27-29 and 31-32 and 34 are rejected under 35 U.S.C. 103 as being unpatentable over TAKEDA as modified by APA and Hu and in view of DESAI et al (US 20220179906 A1)
Regarding claim 25, TAKEDA as modified by APA and Hu teaches a method according to claim 1.
TAKEDA further teaches further comprising:
selecting one of the plurality of items of commentary (para, “[0041] The acceptance unit 131 has a function that accesses a network (for example, the Internet or the intranet) via the communication unit 110, acquires data on that network, and records the web page information in the memory unit 140. In this example, data handled by the data analysis system 100 mainly indicate data at least partly including texts such as document data (for example, materials about drugs, materials in which side effects of the drugs are described, various kinds of comments exchanged over the web, e-mails, presentation materials, spreadsheet materials, meeting materials, contracts, organization charts, and business plans) and also include a wide variety of arbitrary data such as image data, sound data, and video data. It should be noted that the acceptance unit 131 may be designed to accept data from a connected storage medium (such as a USB memory) via an interface (such as a USB port) of the data analysis system 100.” Note: here side effect/adverse event is selected.
Also, para, “[0115] The drug information 701 is information indicative of basic information about the drug. The basic information in this example may include information such as the name of the drug, its main ingredients, permission and authorization information, a manufacturer, and a prescription method.”);
TAKEDA as modified by APA and Hu does not explicitly teach :
providing annotation data associated with the selected item of commentary, the annotation data identifying at least one of: commentary type, author type, name of medicine, type of medicine, ailment, condition, symptom, potential side effect, possible medical outcome, treatment type, an aspect of a personal experience, measure of a tone of the commentary, detected stance of the patient, and feeling or emotion
and training or retraining the multi-stage classifier architecture using a combination of the selected item of commentary and the associated annotation data to update context-based linguistic patterns.
DESAI teaches:
providing annotation data associated with the selected item of commentary, the annotation data identifying at least one of: commentary type, author type, name of medicine, type of medicine, ailment, condition, symptom, potential side effect, possible medical outcome, treatment type, an aspect of a personal experience, measure of the tone of the commentary, detected stance of the patient, and feeling or emotion (para, “[0100] FIG. 4 illustrates an example document used for training the learning model, according to an example embodiment. Document 400 may be an ICSR document and can include critical words or phrases 402 needed to classify the ICSR document. Document 400 may be used to train a learning model (e.g., learning model 104, as shown in FIG. 1) to classify ICSR documents. Document 400 may include annotations 404 (e.g., labels 204, as shown in FIG. 2). Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document. For example, the word “Name” may be assigned a label “ReporterTypeHCP.” Annotations 204 may be used to validate the results of assigning NER labels to document 400.”
Also, para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”)
and training or retraining the [multi-stage classifier architecture] using a combination of the selected item of commentary and the associated annotation data to update context-based[ linguistic] patterns(para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”)
Also, para, “[0112] In operation 1002, a learning engine 102 receives a request to train learning model 104 to classify documents of a domain. The request includes a set of documents and metadata for each of the documents. The metadata may be annotations labeling certain words or phrases in each document. Learning model 104 may be a NLP framework that uses CNN and BiLSTM.”
Also, para “[0064] Training application 112 may build a statistical NER model. For example, training application 112 may embed 81,900 MedDRA entities to create a statistical NER model. Training application 112 may also use the Unified Medical Language System (UMLS) to build the statistical NER model. The statistical NER model may be incorporated as spaCy EntityRuler search pattern attributes. The statistical NER model may be a dictionary or ontology used by learning model 104 to recognize words or phrases in an ICSR document. Training application 112 may load the statistical NER model in learning model 104. The statistical NER model may be used in combination with a standard language (e.g., English, Spanish, French, etc.).”).
It would have been obvious for a person of ordinary skill in the art to apply annotation data teachings of DESAI into the teachings of TAKEDA as modified by APAat the time the application was filed in order to classify the document. (Para, “[0100] ….. Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document….”)
APA further teaches:
[and training or retraining] the multi-stage classifier architecture [using a combination of the selected item of commentary and the associated annotation data to update context-based linguistic patterns]( (para, “[0203] When training the NER model and using it to tag posts with entities, the default configuration for spaCy's EntityRecognizer model was used. The model is backed by a neural network that uses a stacked embedding architecture. The details of spaCy's default EntityRecognizer are outlined in their video documentation (accessed via path watch?v=sqDHBH9IjRU on the www.youtube.com website).” Note: the video at the provided link teaches spaCy's default EntityRecognizer model using multi-dimentional vector)
It would have been obvious for a person of ordinary skill in the art to incorporate multi-stage classifier teachings of admitted prior art into the teachings of TAKEDA at the time the application was filed in order to give a good balance of efficiency, accuracy and adaptability. (See, description of the video; note link/video is not being provided, as it is part of the spec, and available to applicant.)
Hu further teaches:
[and training or retraining the multi-stage classifier architecture using a combination of the selected item of commentary and the associated annotation data to update context-based] linguistic patterns(para, “[0084] Social media monitoring in this embodiment can utilise established text analysis techniques (including named entity recognition and potentially also linguistic patterns) to detect drug names and key symptoms and complains. For instance, “Bactrim gives me headache” or “had Bactrim . . . very bad headache” can be a main message on social media. NER (named entity recognition) technology can help to identify “Bactrim” as the name of the medicine and “Headache” as the key complaint.”)
It would have been obvious for a person of ordinary skill in the art to apply linguistic pattern and jurisdiction teachings of Hu into the teachings of TAKEDA as modified by APA , Hu and Desai at the time the application was filed in order to provide a relation between a drug and adverse reactions. (Hu, Abstract “A system to produce validated weighted relations between drugs and adverse drug reactions (ADRs). At least one processor to monitor social media for links between drugs and ADRs; extract a relation between a drug and an ADR using named entity recognition to provide a weighted relation between the drug and the ADR, the weight based on confidence of the link between the drug and the ADR in social media. …”)
Regarding claim 27, TAKEDA as modified by APA , Hu and Desai teaches a method according to claim 25.
Takeda further teaches further comprising processing the commentary data for the selected item using said at least one classifier (para. “[0108] Then, after generating the learning data, the data analysis system 100 calculates and presents an index (score) for judging whether each comment which has not been classified is related to the relevant topic or not.”);
Desai further teaches and wherein providing the annotation data includes providing, at least in part, the output of said at least one classifier (para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”)
It would have been obvious for a person of ordinary skill in the art to apply annotation data teachings of DESAI into the teachings of TAKEDA as modified by APA, Hu and DESAI at the time the application was filed in order to classify the document. (Para, “[0100] ….. Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document….”)
Regarding claim 28, TAKEDA as modified by APA, Hu and Desai teaches a method according to claim 25.
Takeda further teaches further comprising repeating at least one of the steps of: processing the commentary data using said at least one classifier, selecting a subset of items, processing the subset of items, and processing the content analysis data after training or retraining said at least one classifier with the new or modified annotation data (para, “[0050] The element evaluation unit 136 can repeatedly re-evaluate an evaluation value of each data element and recalculate the weight of the data element until a calculated score of training data, regarding which the user has judged that the data is related to the specified event, becomes superior to a score of training data regarding which the user has judged that the data is not related to the specified event. Specifically speaking, the element evaluation unit 136 firstly calculates scores of training data on the basis of the weights calculated once. The element evaluation unit 136 arranges the training data according to the scores. When this happens, it is desirable that regarding the evaluation by the data analysis system 100, the training data related to the specified event should be arranged in superior positions and the training data not related to the specified event should be arranged in inferior positions. So, for example, the element evaluation unit 136 executes the calculation until the scores of the training data related to the specified event are arranged in the superior positions and the scores of the training data not related to the specified event are arranged in positions inferior to the above-described scores.”)
Regarding claim 29, TAKEDA as modified by APA, and Hu teaches a method of training a selected classifier for use with a method as claimed in Claim 1.
Takeda further teaches:
wherein the selected classifier is configured to identify, for an item of commentary, at least one of a commentary type and a list of medicines associated with the commentary (Para, “[0111] Therefore, for example, when a doctor wants to know various opinions about “treatment of hay fever,” it is possible to pick up (or classify and select) comments which may highly possibly and truly discuss the “treatment of hay fever” from among a large amount of hay fever topics if there are a plurality of pieces of learning data such as learning data based on classification of being “related to hay fever” or “not related to hay fever” and learning data based on classification of being “related to treatment” or “not related to treatment.”
Also, para, “[0096] The case information 503 is case information about side effects of drug A as indicated in the drug information 501 and includes information such as doctors' opinions and a patient's impressions.” Note: here, commentary type refers to doctor or patient opinions. Also, see Fig. 6.),
and wherein the method comprises: processing the commentary data using at least one classifier, including said selected classifier, to identify for each item a commentary type and a list of medicines associated with by the commentary (Para, “[0111] Therefore, for example, when a doctor wants to know various opinions about “treatment of hay fever,” it is possible to pick up (or classify and select) comments which may highly possibly and truly discuss the “treatment of hay fever” from among a large amount of hay fever topics if there are a plurality of pieces of learning data such as learning data based on classification of being “related to hay fever” or “not related to hay fever” and learning data based on classification of being “related to treatment” or “not related to treatment.”
Also, para, “[0096] The case information 503 is case information about side effects of drug A as indicated in the drug information 501 and includes information such as doctors' opinions and a patient's impressions.” Note: here, commentary type refers to doctor or patient opinions. Also, see Fig. 6.);
selecting one of the plurality of items of commentary (para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. It should be noted that the comments may sometimes include comments which are not related to the drug information 701 at all.”);
TAKEDA as modified by APAdoes not explicitly teach:
providing annotation data associated with the selected item of commentary, the annotation data identifying at least one of: commentary type, author type, name of medicine, type of medicine, ailment, condition, symptom, potential side effect, possible medical outcome, treatment type, an aspect of a personal experience, measure of the tone of the commentary, detected stance of the patient, and feeling or emotion;
and training or retraining the selected classifier using the combination of the selected item of commentary and the associated annotation data.
DESAI teaches:
providing annotation data associated with the selected item of commentary, the annotation data identifying at least one of: commentary type, author type, name of medicine, type of medicine, ailment, condition, symptom, potential side effect, possible medical outcome, treatment type, an aspect of a personal experience, measure of the tone of the commentary, detected stance of the patient, and feeling or emotion (para, “[0100] FIG. 4 illustrates an example document used for training the learning model, according to an example embodiment. Document 400 may be an ICSR document and can include critical words or phrases 402 needed to classify the ICSR document. Document 400 may be used to train a learning model (e.g., learning model 104, as shown in FIG. 1) to classify ICSR documents. Document 400 may include annotations 404 (e.g., labels 204, as shown in FIG. 2). Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document. For example, the word “Name” may be assigned a label “ReporterTypeHCP.” Annotations 204 may be used to validate the results of assigning NER labels to document 400.”
Also, para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”)
and training or retraining the selected classifier using the combination of the selected item of commentary and the associated annotation data (para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”)
Also, para, “[0112] In operation 1002, a learning engine 102 receives a request to train learning model 104 to classify documents of a domain. The request includes a set of documents and metadata for each of the documents. The metadata may be annotations labeling certain words or phrases in each document. Learning model 104 may be a NLP framework that uses CNN and BiLSTM.”).
It would have been obvious for a person of ordinary skill in the art to apply annotation data teachings of DESAI into the teachings of TAKEDA as modified by APAat the time the application was filed in order to classify the document. (Para, “[0100] ….. Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document….”)
Regarding claim 31, TAKEDA as modified by APA, Hu and Desai teaches a method according to Claim 29.
Takeda further teaches:
further comprising processing the commentary data for the selected item using the selected classifier (para. “[0108] Then, after generating the learning data, the data analysis system 100 calculates and presents an index (score) for judging whether each comment which has not been classified is related to the relevant topic or not.”);
Desai further teaches:
and wherein providing the annotation data includes providing, at least in part, the output of the selected classifier (para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”)
It would have been obvious for a person of ordinary skill in the art to apply annotation data teachings of DESAI into the teachings of TAKEDA as modified by APA , Hu and DESAI at the time the application was filed in order to classify the document. (Para, “[0100] ….. Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document….”)
Regarding claim 32, TAKEDA teaches a computer system for estimating an effectiveness or safety of a medicine, the computer system comprising:
at least one processor and at least one associated memory store(see para 0039);
wherein said at least one memory store includes computer program code which, when executed by said at least one processor, causes the computer system to perform operations comprising(see para, 0039):
receiving commentary data encoding a plurality of items of commentary substantially related to medical subject-matter( para, “[0041] The acceptance unit 131 has a function that accesses a network (for example, the Internet or the intranet) via the communication unit 110, acquires data on that network, and records the web page information in the memory unit 140. In this example, data handled by the data analysis system 100 mainly indicate data at least partly including texts such as document data (for example, materials about drugs, materials in which side effects of the drugs are described, various kinds of comments exchanged over the web, e-mails, presentation materials, spreadsheet materials, meeting materials, contracts, organization charts, and business plans)……..”);
processing the commentary data using at least one classifier to identify for each item a commentary type and a list of medicines associated with the commentary (Para, “[0111] Therefore, for example, when a doctor wants to know various opinions about “treatment of hay fever,” it is possible to pick up (or classify and select) comments which may highly possibly and truly discuss the “treatment of hay fever” from among a large amount of hay fever topics if there are a plurality of pieces of learning data such as learning data based on classification of being “related to hay fever” or “not related to hay fever” and learning data based on classification of being “related to treatment” or “not related to treatment.”
Also, para, “[0096] The case information 503 is case information about side effects of drug A as indicated in the drug information 501 and includes information such as doctors' opinions and a patient's impressions.” Note: here, commentary type refers to doctor or patient opinions. Also, see Fig. 6.);
the classifier employing machine learning to identify patterns in the commentary data and to associate the patterns with specific corresponding outputs(para, “[0072] The above-described operation is the operation of the data analysis system 100 to determine evaluations of the respective data elements. The processing illustrated in FIG. 2 is also processing for acquiring the training data which has been classified (or associated with the classification information) as designated by the user in order to classify the unknown data and extracting a pattern (for example, keywords or conceptually distribution of the keywords or meanings and concepts represented by the training data) included in the relevant training data. As a result of the processing illustrated in FIG. 2, preprocessing for identifying whether the unknown data is related to the specified event or not is completed.”)
selecting a subset of items, from the plurality of items of commentary, identified as referencing the medicine and whose commentary type has been identified as commentary from a patient who has used the medicine(para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. It should be noted that the comments may sometimes include comments which are not related to the drug information 701 at all.”);
processing the subset of items to generate content analysis data including, for each item, at least one quantified estimate of at least one aspect of a patient experience described in the commentary(para, “[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.” Note: Also, see Fig. 7:
PNG
media_image1.png
700
651
media_image1.png
Greyscale
para, (“[0033] …..So, for example, in a case of drug side effect reports, the data analysis system can suggest a report which reports what appears to be a side effect that should highly possibly be actually identified as the side effect, from among a large number of listed reports. Furthermore, for example, in a case of medical portal sites, the data analysis system can suggest serious information from among a large number of posted comments.”) ;
and processing the content analysis data to calculate an estimate indicative of the effectiveness or safety of the medicine. (para, “[0114] Referring to FIG. 7, a web page 700 includes drug information 701 and comments 702 to 704 indicating impressions about the use of a drug indicated in the drug information 701 by users who used that drug.”
Also, para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. It should be noted that the comments may sometimes include comments which are not related to the drug information 701 at all.”)
TAKEDA as modified by APA and Hu does not explicitly teach :
providing at least a portion of the selected subset of items to an annotation user and receiving an updated training dataset based on annotations from the annotation user
retraining the at least one classifier using the updated training dataset to improve identification accuracy of the patterns
Desai teaches:
providing at least a portion of the selected subset of items to an annotation user and receiving an updated training dataset based on annotations from the annotation user( para, “[0033] In a given embodiment, a server may receive a request to train a learning model to classify documents and identify entities within the documents specific to a domain. For example, the learning model may identify the entities within the documents to automatically summarize the document. The content of the documents may include one or more strings. Furthermore, the documents may include corresponding metadata. The metadata may be annotations that label one or more strings in the document. The annotations may be specific to the domain.”
Also, para “[0034] The server may train a learning model to classify the documents specific to the domain by generating a word embedding for each document. The server may tokenize each word embedding into segments, including one or more words of each word embedding. The server may train the learning model by recursively breaking down each of the segments of each document into a set of features, assigning a part-of-speech tag to the one or more words corresponding to each respective segment of each respective document based on predetermined weights assigned to each feature of the set of features of the respective segment, and assigning a dependency label to the one or more words corresponding to each respective segment of each respective document based o the part-of-speech tag assigned to the respective one or more words and the predetermined weights assigned to each feature of the set of features of the respective segment.....”)
retraining the at least one classifier using the updated training dataset to improve identification accuracy of the patterns( Also, para “[0034] The server may train a learning model to classify the documents specific to the domain by generating a word embedding for each document. The server may tokenize each word embedding into segments, including one or more words of each word embedding. The server may train the learning model by recursively breaking down each of the segments of each document into a set of features, assigning a part-of-speech tag to the one or more words corresponding to each respective segment of each respective document based on predetermined weights assigned to each feature of the set of features of the respective segment, and assigning a dependency label to the one or more words corresponding to each respective segment of each respective document based on the part-of-speech tag assigned to the respective one or more words and the predetermined weights assigned to each feature of the set of features of the respective segment.....”)
It would have been obvious for a person of ordinary skill in the art to apply annotation data teachings of DESAI into the teachings of TAKEDA as modified by APAat the time the application was filed in order to classify the document. (Para, “[0100] ….. Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document….”)
Claims 26 , 30 and 33 are rejected under 35 U.S.C. 103 as being unpatentable over TAKEDA as modified by APA and Hu in view of DESAI and further in view of Baird et al.(US 20160321229 A1)
Regarding claim 26, TAKEDA as modified by APA , Hu and Desai teaches a method according to claim 25.
Desai further teaches further comprising:
outputting the selected item of commentary to an annotation user (Para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”);
outputting the annotation data to the annotation user (Para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”);
It would have been obvious for a person of ordinary skill in the art to apply annotation data teachings of DESAI into the teachings of TAKEDA as modified by APA, Hu and DESAI at the time the application was filed in order to classify the document. (Para, “[0100] ….. Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document….”)
TAKEDA as modified by APA , Hu and Desai does not explicitly teach:
receiving user input from the annotation user including a direction to create, modify or delete at least a portion of the annotation data; and creating, modifying or deleting at least a portion of the annotation data in accordance with the received direction.
Baird teaches:
receiving user input from the annotation user including a direction to create, modify or delete at least a portion of the annotation data; and creating, modifying or deleting at least a portion of the annotation data in accordance with the received direction (para, “[0035] Alternatively or additionally, the user may activate an editing icon in the clipped-content summary document. After receiving activation information for the editing icon (which specifies a clip selection), the computer system may provide an annotation modification feature (such as a user-interface feature with a virtual keypad and a text box that displays entered text) in the clipped-content summary document that allows a user to modify an annotation corresponding to a content item associated with the clip selection. Moreover, after receiving a modification to the annotation corresponding to the content item, the computer system may change the annotation in metadata associated with the content item.”)
It would have been obvious for a person of ordinary skill in the art to apply annotation data changing teachings of Baird into the teachings of TAKEDA as modified by APAand Desai at the time the application was filed in order to change the annotated data. (Para 0035, “…..the computer system may change the annotation in metadata associated with the content item.”)
Regarding claim 30, TAKEDA as modified by APA and Desai teaches a method of training a selected classifier for use with a method as claimed in Claim 29.
Desai further teaches further comprising::
outputting the selected item of commentary to an annotation user (Para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”);
outputting the annotation data to the annotation user (Para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”);
It would have been obvious for a person of ordinary skill in the art to apply annotation data teachings of DESAI into the teachings of TAKEDA as modified by APAand DESAI at the time the application was filed in order to classify the document. (Para, “[0100] ….. Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document….”);
TAKEDA as modified by as modified by APA , HU and Desai does not explicitly teach:
receiving user input from the annotation user including a direction to create, modify or delete at least a portion of the annotation data; and creating, modifying or deleting at least a portion of the annotation data in accordance with the received direction.
Baird teaches:
receiving user input from the annotation user including a direction to create, modify or delete at least a portion of the annotation data; and creating, modifying or deleting at least a portion of the annotation data in accordance with the received direction (para, “[0035] Alternatively or additionally, the user may activate an editing icon in the clipped-content summary document. After receiving activation information for the editing icon (which specifies a clip selection), the computer system may provide an annotation modification feature (such as a user-interface feature with a virtual keypad and a text box that displays entered text) in the clipped-content summary document that allows a user to modify an annotation corresponding to a content item associated with the clip selection. Moreover, after receiving a modification to the annotation corresponding to the content item, the computer system may change the annotation in metadata associated with the content item.”)
It would have been obvious for a person of ordinary skill in the art to apply annotation data changing teachings of Baird into the teachings of TAKEDA as modified by APA and Desai at the time the application was filed in order to change the annotated data. (Para 0035, “…..the computer system may change the annotation in metadata associated with the content item.”)
Regarding claim 33, TAKEDA as modified by APA teaches a computer system [for training or retraining a classifier for use with a computer system(see Desai reference below)] as claimed in Claim 32.
TAKEDA further teaches :
wherein the selected classifier is configured to identify, for an item of commentary, at least one of a commentary type and a list of medicines associated with the commentary(Para, “[0111] Therefore, for example, when a doctor wants to know various opinions about “treatment of hay fever,” it is possible to pick up (or classify and select) comments which may highly possibly and truly discuss the “treatment of hay fever” from among a large amount of hay fever topics if there are a plurality of pieces of learning data such as learning data based on classification of being “related to hay fever” or “not related to hay fever” and learning data based on classification of being “related to treatment” or “not related to treatment.”
Also, para, “[0096] The case information 503 is case information about side effects of drug A as indicated in the drug information 501 and includes information such as doctors' opinions and a patient's impressions.” Note: here, commentary type refers to doctor or patient opinions. Also, see Fig. 6.);
and wherein the system comprises: at least one processor and at least one associated memory store(see para 0039);
wherein said at least one memory store includes computer program code which, when executed by said at least one processor, causes the computer system to perform the method of: selecting one of a plurality of items of commentary; causing the commentary data for the selected item of commentary to be processed using the selected classifier(para. “[0108] Then, after generating the learning data, the data analysis system 100 calculates and presents an index (score) for judging whether each comment which has not been classified is related to the relevant topic or not.”
(para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. It should be noted that the comments may sometimes include comments which are not related to the drug information 701 at all.”);
TAKEDA as modified by APAdoes not explicitly teach:
outputting the selected item of commentary to an annotation user);
receiving annotation data associated with the selected item of commentary, the annotation data identifying at least one of: commentary type, author type, name of medicine, type of medicine, ailment, condition, symptom, potential side effect, possible medical outcome, treatment type, an aspect of a personal experience, measure of the tone of the commentary, detected stance of the patient, and feeling or emotion,
and the annotation data including, at least in part, the output of the selected classifier;
outputting the annotation data to the annotation user;
receiving user input from the annotation user including a direction to create, modify or delete at least a portion of the annotation data; creating, modifying or deleting at least a portion of the annotation data in accordance with the received direction;
and causing the selected classifier to be trained or retrained using the combination of the selected item of commentary and the associated annotation data.
Desai teaches :
outputting the selected item of commentary to an annotation user((Para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”);
receiving annotation data associated with the selected item of commentary, the annotation data identifying at least one of: commentary type, author type, name of medicine, type of medicine, ailment, condition, symptom, potential side effect, possible medical outcome, treatment type, an aspect of a personal experience, measure of the tone of the commentary, detected stance of the patient, and feeling or emotion(para, “[0100] FIG. 4 illustrates an example document used for training the learning model, according to an example embodiment. Document 400 may be an ICSR document and can include critical words or phrases 402 needed to classify the ICSR document. Document 400 may be used to train a learning model (e.g., learning model 104, as shown in FIG. 1) to classify ICSR documents. Document 400 may include annotations 404 (e.g., labels 204, as shown in FIG. 2). Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document. For example, the word “Name” may be assigned a label “ReporterTypeHCP.” Annotations 204 may be used to validate the results of assigning NER labels to document 400.”
Also, para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”),
and the annotation data including, at least in part, the output of the selected classifier(para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”)
outputting the annotation data to the annotation user(Para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”)
and causing the selected classifier to be trained or retrained using the combination of the selected item of commentary and the associated annotation data(para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”)
Also, para, “[0112] In operation 1002, a learning engine 102 receives a request to train learning model 104 to classify documents of a domain. The request includes a set of documents and metadata for each of the documents. The metadata may be annotations labeling certain words or phrases in each document. Learning model 104 may be a NLP framework that uses CNN and BiLSTM.”).
It would have been obvious for a person of ordinary skill in the art to apply annotation data teachings of DESAI into the teachings of TAKEDA as modified by APAat the time the application was filed in order to classify the document. (Para, “[0100] ….. Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document….”).
TAKEDA as modified by APADesai does not explicitly teach:
receiving user input from the annotation user including a direction to create, modify or delete at least a portion of the annotation data; creating, modifying or deleting at least a portion of the annotation data in accordance with the received direction;
Baird teaches:
receiving user input from the annotation user including a direction to create, modify or delete at least a portion of the annotation data; creating, modifying or deleting at least a portion of the annotation data in accordance with the received direction(para, “[0035] Alternatively or additionally, the user may activate an editing icon in the clipped-content summary document. After receiving activation information for the editing icon (which specifies a clip selection), the computer system may provide an annotation modification feature (such as a user-interface feature with a virtual keypad and a text box that displays entered text) in the clipped-content summary document that allows a user to modify an annotation corresponding to a content item associated with the clip selection. Moreover, after receiving a modification to the annotation corresponding to the content item, the computer system may change the annotation in metadata associated with the content item.”)
It would have been obvious for a person of ordinary skill in the art to apply annotation data changing teachings of Baird into the teachings of TAKEDA as modified by APAand Desai at the time the application was filed in order to change the annotated data. (Para 0035, “…..the computer system may change the annotation in metadata associated with the content item.”)
Claims 36 and 38 are rejected under 35 U.S.C. 103 as being unpatentable over TAKEDA as modified by Hu , in view of DESAI and further in view of Baird et al.(US 20160321229 A1)
Regarding claim 36, TAKEDA as modified by Hu teaches a computer system according to Claim 34.
TAKEDA further teaches further comprising an annotation entry system configured to:
receive an item of commentary (para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. It should be noted that the comments may sometimes include comments which are not related to the drug information 701 at all.”);
TAKEDA as modified by Hu does not explicitly teaches:
receive associated annotation data encoding output of the commentary type classifier module and the NER classifier module in respect of the item of commentary;
output the item of commentary and the associated annotation data;
receive adjustments or additions to the annotation data;
carry out the adjustments or additions to the annotation data;
transmit the adjusted annotation data and cause the multi-stage classifier architecture to be trained or retrained using the adjusted annotation data.
Desai teaches:
receive associated annotation data encoding the output of the commentary type classifier module and the NER classifier module in respect of the item of commentary (para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”
Para, “[0104] In operation 604, a learning model (e.g., learning model 104, as shown in FIG. 1) can be trained using the training corpus stored in the training data repository. A developer or SME may transmit the request to train the learning using the training corpus. The learning model may be trained using the training corpus as described above. Based on the validation of the assigned parts-of-speech and dependency tags and NER labels, the learning model may determine whether the accuracy of the learning model's assignment of the assigned parts-of-speech and dependency tags and NER labels meets a threshold. The threshold may be preprogrammed or may be provided in the request to train the learning model. If the accuracy of the learning model's threshold is not met, a request may be transmitted to retrain the learning model with the same set of documents or a different set of documents.”);
output the item of commentary and the associated annotation data (Para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”
Para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”)
transmit the adjusted annotation data and cause at least one of the commentary type classifier module and the NER classifier module to be trained or retrained using the adjusted annotation data (para, “[0104] In operation 604, a learning model (e.g., learning model 104, as shown in FIG. 1) can be trained using the training corpus stored in the training data repository. A developer or SME may transmit the request to train the learning using the training corpus. The learning model may be trained using the training corpus as described above. Based on the validation of the assigned parts-of-speech and dependency tags and NER labels, the learning model may determine whether the accuracy of the learning model's assignment of the assigned parts-of-speech and dependency tags and NER labels meets a threshold. The threshold may be preprogrammed or may be provided in the request to train the learning model. If the accuracy of the learning model's threshold is not met, a request may be transmitted to retrain the learning model with the same set of documents or a different set of documents.” Note: training corpus includes annotated data; also though Desai doesn’t explicitly teach adjusting the annotated data; the adjusted annotated data is merely annotated data. Baird reference below is introduce that teaches that one can adjust the annotated data.)
It would have been obvious for a person of ordinary skill in the art to apply annotation data teachings of DESAI into the teachings of TAKEDA as modified by Hu at the time the application was filed in order to classify the document. (Para, “[0100] ….. Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document….”)
TAKEDA as modified by Desai and Hu does not explicitly teach:
receive adjustments or additions to the annotation data;
carry out the adjustments or additions to the annotation data ;
Baird teaches:
receive adjustments or additions to the annotation data (para, “[0035] Alternatively or additionally, the user may activate an editing icon in the clipped-content summary document. After receiving activation information for the editing icon (which specifies a clip selection), the computer system may provide an annotation modification feature (such as a user-interface feature with a virtual keypad and a text box that displays entered text) in the clipped-content summary document that allows a user to modify an annotation corresponding to a content item associated with the clip selection. Moreover, after receiving a modification to the annotation corresponding to the content item, the computer system may change the annotation in metadata associated with the content item.”);
carry out the adjustments or additions to the annotation data (para, “[0035] Alternatively or additionally, the user may activate an editing icon in the clipped-content summary document. After receiving activation information for the editing icon (which specifies a clip selection), the computer system may provide an annotation modification feature (such as a user-interface feature with a virtual keypad and a text box that displays entered text) in the clipped-content summary document that allows a user to modify an annotation corresponding to a content item associated with the clip selection. Moreover, after receiving a modification to the annotation corresponding to the content item, the computer system may change the annotation in metadata associated with the content item.”);
It would have been obvious for a person of ordinary skill in the art to apply annotation data changing teachings of Baird into the teachings of TAKEDA as modified by Desai and Hu at the time the application was filed in order to change the annotated data. (Para 0035, “…..the computer system may change the annotation in metadata associated with the content item.”)
Regarding claim 38, TAKEDA as modified by Hu teaches a method according to Claim 37.
Takeda further teaches further comprising:
receiving an item of commentary (para, “[0116] The comments 702 to 704 include information such as impressions of the use of the drug by patients who used the drug information 701 and opinions about the relevant drug. It should be noted that the comments may sometimes include comments which are not related to the drug information 701 at all.”);
TAKEDA as modified by Hu does not explicitly teaches:
receiving associated annotation data encoding the output of the commentary type classifier and the NER classifier in respect of the item of commentary;
outputting the item of commentary and the associated annotation data;
receiving adjustments or additions to the annotation data;
carrying out the adjustments or additions to the annotation data;
transmitting the adjusted annotation data and cause at least one of the commentary type classifier and the NER classifier to be trained or retrained using the adjusted annotation data.
Desai teaches:
receiving associated annotation data encoding the output of the commentary type classifier and the NER classifier in respect of the item of commentary (para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”
Para, “[0104] In operation 604, a learning model (e.g., learning model 104, as shown in FIG. 1) can be trained using the training corpus stored in the training data repository. A developer or SME may transmit the request to train the learning using the training corpus. The learning model may be trained using the training corpus as described above. Based on the validation of the assigned parts-of-speech and dependency tags and NER labels, the learning model may determine whether the accuracy of the learning model's assignment of the assigned parts-of-speech and dependency tags and NER labels meets a threshold. The threshold may be preprogrammed or may be provided in the request to train the learning model. If the accuracy of the learning model's threshold is not met, a request may be transmitted to retrain the learning model with the same set of documents or a different set of documents.”);
outputting the item of commentary and the associated annotation data (Para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”
Para, “[0103] FIG. 6 is a block diagram of a model approval flow, according to an example embodiment. Model approval flow 600 may be used to determine whether a learning model assigns parts-of-speech and dependency tags and NER labels at an acceptable level of accuracy. In operation 602, a training corpus (e.g., training data) may be created. More specifically, a Subject Matter Expert (SME) may annotate a document with labels assigned to words or phrases of a document necessary to classify the document. The SME may perform a quality check of the annotations. The annotated document may be part of the training corpus, and the training corpus can be stored in a training data repository (e.g., database 120, as shown in FIG. 1).”)
transmitting the adjusted annotation data and cause at least one of the commentary type classifier and the NER classifier to be trained or retrained using the adjusted annotation data (para, “[0104] In operation 604, a learning model (e.g., learning model 104, as shown in FIG. 1) can be trained using the training corpus stored in the training data repository. A developer or SME may transmit the request to train the learning using the training corpus. The learning model may be trained using the training corpus as described above. Based on the validation of the assigned parts-of-speech and dependency tags and NER labels, the learning model may determine whether the accuracy of the learning model's assignment of the assigned parts-of-speech and dependency tags and NER labels meets a threshold. The threshold may be preprogrammed or may be provided in the request to train the learning model. If the accuracy of the learning model's threshold is not met, a request may be transmitted to retrain the learning model with the same set of documents or a different set of documents.” Note: training corpus includes annotated data; also though Desai doesn’t explicitly teach adjusting the annotated data; the adjusted annotated data is merely annotated data. Baird reference below is introduce that teaches that one can adjust the annotated data.)
It would have been obvious for a person of ordinary skill in the art to apply annotation data teachings of DESAI into the teachings of TAKEDA as modified by Hu at the time the application was filed in order to classify the document. (Para, “[0100] ….. Annotations 404 may be a label assigned to critical words or phrase 402 needed to classify the ICSR document….”)
TAKEDA as modified by Desai and Hu does not explicitly teach:
receive adjustments or additions to the annotation data;
carry out the adjustments or additions to the annotation data ;
Baird teaches:
receiving adjustments or additions to the annotation data (para, “[0035] Alternatively or additionally, the user may activate an editing icon in the clipped-content summary document. After receiving activation information for the editing icon (which specifies a clip selection), the computer system may provide an annotation modification feature (such as a user-interface feature with a virtual keypad and a text box that displays entered text) in the clipped-content summary document that allows a user to modify an annotation corresponding to a content item associated with the clip selection. Moreover, after receiving a modification to the annotation corresponding to the content item, the computer system may change the annotation in metadata associated with the content item.”);
carrying out the adjustments or additions to the annotation data (para, “[0035] Alternatively or additionally, the user may activate an editing icon in the clipped-content summary document. After receiving activation information for the editing icon (which specifies a clip selection), the computer system may provide an annotation modification feature (such as a user-interface feature with a virtual keypad and a text box that displays entered text) in the clipped-content summary document that allows a user to modify an annotation corresponding to a content item associated with the clip selection. Moreover, after receiving a modification to the annotation corresponding to the content item, the computer system may change the annotation in metadata associated with the content item.”);
It would have been obvious for a person of ordinary skill in the art to apply annotation data changing teachings of Baird into the teachings of TAKEDA as modified by Desai and Hu at the time the application was filed in order to change the annotated data. (Para 0035, “…..the computer system may change the annotation in metadata associated with the content item.”)
Response to Arguments
Applicant's arguments filed on 02/25/2026 have been fully considered but they are not persuasive.
Remarks - 35 USC § 101
In remarks, Pg. 6, applicant contends: “these amendments are fully supported by the originally filed specification and drawings as reflected, for example, in paragraphs [0016], [0023], [0031], [0044], [0045], [0052], [0054], [0075], [0083], [0129] - [0137] of the instant published application U.S. Patent Application No. 2024/0120144 Al to Balmforth ("published application"), and as further described below.”
Please refer to 35 U.S.C 112(a) rejection above; also note that paragraph discussed in the writeup for potential support for multi-stage classifier is not listed above. The examiner have reviewed the entire specification, and could not determine the explicit support for amendments. Also, note, in the response after final mailed on 03/04/2026, the examiner have indicated that “applicant have provided plethora of paragraphs in specification (remarks, Pg,. 16) that are indicated to provide the support for the claim amendments; however, it is not explicitly clear that how these paragraphs provide the support for amended claims. The applicant is advised to provide the specific citation, that clearly indicates the support for claimed language, thus avoiding any new matter issue upon the entry of amened claim language.”
The remaining arguments with regard to 35 U.S.C 101, already have been addressed in the advisory office action mailed on 03/04/2026. Please note, that no new arguments/remarks were presented with filing of the RCE.
Remarks - 35 USC § 103
Applicant’s arguments with respect to claims 1-39 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
The arguments are related to amended claim language, and the amended claim language have been addressed in the prior art rejection section. In addition, please see 35 U.S.C 112(a) rejection with regard to amended claim language.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUMA WASEEM whose telephone number is (571)272-1316. The examiner can normally be reached Monday-Friday(9:00am - 5:00 pm) EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason B. Dunham can be reached on (571) 272-8109. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUMA WASEEM/Examiner, Art Unit 3686
/JASON B DUNHAM/Supervisory Patent Examiner, Art Unit 3686
Prosecution Insights