DETAILED ACTION
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/10/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. GB2116139.3, filed on 11/10/2021.
Status of Claims
Claims were amended pursuant to a preliminary amendment filed together with the initial set on 05/10/2024. For the examination purpose, the claim set with amended claims have been used. Claims 2-7, 10-13, 17, 19-22, 28, 32, 33, 35, 36, 42-44 were amended, claims 8, 9, 14, 15, 18, 23-27, 29-31, 34, 37-41 and 45-52 were cancelled. Claims 1-7, 10-13, 16-17, 19-22, 28, 32-33, 35-36, and 42-44 are pending of which Claims 1 and 44 are independent.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-7, 10-13, 16-17, 19-22, 28, 32, 33, 35, 36, 42-44 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
The Independent claims 1 and 44 recite “(i) receiving text data and a list of classes that defines the sensitive information to be labelled”; “(ii) generating a set of synthetic sentences and using the set of synthetic sentences for training the machine learning engine”; “(iii) predicting labels for entities in a sample of the text data, selecting a subsample of labelled sentences from the sample of text data to provide to an annotator for reviewing, and updating the training data with the reviewed sentences”; “and (iv) training the machine learning engine with the updated training data and repeating step (iii) until the performance of the machine learning engine meets an end-user requirement”. The limitations above as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process, as this could be performed in the human mind or with the aid of pen and paper.
The limitation of " receiving ... ", "generating ... ", "predicting ... ", as drafted covers mental activities. More specifically, a person can receive text data containing sensitive information and a list of classes where those information belongs to and which will be labelled. A person can generate a set of synthetic/artificial sentences and use them as training or standard data. Choose a sample of original text data and predict label for entities, choose a small size of sample and send it to another person, who will select the most appropriate label. The person can update the training/standard data with the new label and can continue the whole process until he/she is satisfied with the labels. All the steps above are examples of observation and evaluation that could be performed in the human mind or with the aid of pencil and paper.
The claims recite the additional limitation of “ machine learning engine”, claim 44 recites “processor”, “ non transitory computer readable media” for performing the method. All those are recited at a high level of generality and are recited as performing generic computer functions routinely used in computer applications. Throughout the specification, machine learning engine is used broadly, in a generic way. It can be named entity recognition models or sequence classifiers ( according to para.[0045]) or sequence tagger ( according to para.[0127]), which is not sufficient to amount to significantly more than the judicial exception. “Processor”, and “ non transitory computer readable media” are only recited in claim 44 and the specification didn’t mention anything about them. They are performing generic computer functions that are well-understood, routine and conventional activities amount to no more than implementing the abstract idea with a computerized system. The claims as drafted, are not patent eligible
Thus, taken alone, the additional elements do not amount to significantly more than the above identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds
nothing that is not already present when looking at the elements taken individually. There is no indication
that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation. Claims 1
and 44 are therefore not drawn to eligible subject matter as they are directed to an abstract idea without
significantly more than the abstract idea.
Claim 2 recites the additional limitation of “the received text data includes unstructured text data, structured text data or a combination of unstructured and structured text data” , determining that the text data is structured or unstructured or combination of both, is an observation, evaluation, which could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 2 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 3 recites “the received text data does not include any annotations or labels”, to determine that the text data does not have any labels or annotation, is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 3 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 4 recites “the method includes the step of providing a confidence score for each labelled sentence or entity, and in which the confidence score is a value that corresponds to the probability or likelihood that the entity belongs to the one or more classes”, to find out that how likely the label is matching with the actual entity, is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 4 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 5 recites “each entity is mapped to multiple labels, with a confidence score being associated with each label that has been mapped to the entity”, to find out that each entity is marked with multiple label and have a ranking or score regarding each label, is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 5 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 6 recites “the method includes the step of outputting the annotated text data”, to find out that the output text data is annotated, is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 6 does not recite any additional limitations. The claims as drafted, are not patent eligible.
Claim 7 recites “the end-user requirement includes one or more of the following: a predefined number of iterations reached a predefined confidence score reached for labelled sentences, a predefined percentage of recall, precision level, class performance or confusion score”, to determine that the number of iterations reached a predefined confidence scores or some other factors, is an evaluation, observation and could be performed with the aid of pen and paper. The claim recites additional limitation of end-user requirement, which is generic and which is not sufficient to amount to significantly more than the judicial exception. The claim 7 as drafted, is not patent eligible.
Claim 10 recites “the sample of text data is selected based on a probability sampling approach, such as a stratified sampling approach”, determining that sample data is based on probability sampling approach is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 8 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 11 recites “the synthetic sentences are generated based on grammar rules or models to produce a sequence of words or tokens in context, and in which the grammar rules or models are automatically selected based on analysing the received text data”, generation of synthetic sentences based on grammar rule/context, could be performed with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 11 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 12 recites “the synthetic sentences contain one or more entities that belong to the one or more received classes, and in which the entities that are generated based on a regular expression and/or using lookup lists”, determining that the synthetic sentences contain entities which belongs to the original text/expression, is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 12 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 13 recites “the method includes the step of introducing noise in the synthetic sentences, such as including generating typos”, introducing typo in a synthetic sentence is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 13 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 16 recites “the method includes the step of generating a confusion matrix that represents a comparison of the predicted labels with the labels reviewed by the annotator”, generating confusion matrix, which is a table of predicted labels and the annotated table could be performed with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 16 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 17 recites “the selection of labelled sentences in step (iii) is based on the generated confusion matrix, and in which the method includes the step of providing a confusion score for each labelled entity of the selected sentences, in which the confusion score is a value that indicates how close the prediction for a given class is to the prediction for another class.”, to determine that the each labelled entity with a confusion score indicate how close they are with the predicted class, is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 17 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 19 recites “the confusion score is determined for each selected sentence, based on the confusion score determined for each entity in the sentence”, to determine that the confusion score is determine for each sentence based on the score of the entity is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 19 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 20 recites “the machine learning engine is configured to rank each selected sentence based on an analysis of the confusion matrix and/or the confusion scores”, ranking sentences based on scores is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim recites additional limitations of the machine learning engine. Throughout the specification, machine learning engine is used broadly, in a generic way. It can be named entity recognition models or sequence classifiers ( according to para.[0045]) or sequence tagger ( according to para.[0127]), which is not sufficient to amount to significantly more than the judicial exception. The claim 20 as drafted, is not patent eligible.
Claim 21 recites “each class or class pair is assigned a weight and in which the labelled sentences provided to the annotator are selected based on the assigned weights”, to assign certain value/weight to the sentence and assigning the sentences to the annotator based on the weight, could be performed with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 21 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 22 recites “the confusion matrix and/or the confusion scores are updated for each iteration of step (iii)”, to determine that the score is updated in each iteration is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 22 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 28 recites “the weights are updated based on the confusion matrix and/or the confusion scores”, to determine that the weights/values are updated based on the scores, is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 28 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 32 recites “the method includes the step of representing each entity of the selected sentences into a vector space, in which the entities belong to the one or more classes defining the sensitive information to be labelled”, representing each entity in a vector or numerical place based on the sensitive class where they belong to, could be performed with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 32 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 33 recites “the method includes the step of determining a support for each class, in which the support refers to the set of labelled sentences that contain that class, and in which the method includes the step of representing the support for each class into a vector space and determining a centre within the vector space.”, to represent a set of labelled sentences in a vector space or numerical space and determine a center could be performed with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 33 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 35 recites “an outlier detector is used to detect outliers in the reviewed sentences and in which the outlier detector analyses each entity of the selected sentences in relation to the centre for each class”, to detect am outlier in sentences, in relation to the center is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception, as claim 35 does not recite any additional limitations. The claim as drafted, is not patent eligible.
Claim 36 recites “the machine learning engine is configured to learn to represent complex classes into multiple sub-classes, and in which the machine learning engine is configured to identify a complex class by analysing its vector space representation”, to determine a complex class of classification into sub classes based on vector space, is an evaluation, observation and could be performed in the human mind or with the aid of pen and paper. The claim recites additional limitations of the machine learning engine. Throughout the specification, machine learning engine is used broadly, in a generic way. It can be named entity recognition models or sequence classifiers ( according to para.[0045]) or sequence tagger ( according to para.[0127]), which is not sufficient to amount to significantly more than the judicial exception. The claim 36 as drafted, is not patent eligible.
Claim 42 recites “the machine learning engine is trained to de-identify text data”, to deidentify or masking a sensitive text data could be performed in the human mind or with the aid of pen and paper. The claim recites additional limitations of the machine learning engine. Throughout the specification, machine learning engine is used broadly, in a generic way. It can be named entity recognition models or sequence classifiers ( according to para.[0045]) or sequence tagger ( according to para.[0127]), which is not sufficient to amount to significantly more than the judicial exception. The claim 42 as drafted, is not patent eligible.
Claim 43 recites “the machine learning engine is trained to de-identify text data within image or video-based data”, to deidentify or masking a sensitive text data from image or video could be performed in the human mind or with the aid of pen and paper. The claim recites additional limitations of the machine learning engine. Throughout the specification, machine learning engine is used broadly, in a generic way. It can be named entity recognition models or sequence classifiers ( according to para.[0045]) or sequence tagger ( according to para.[0127]), which is not sufficient to amount to significantly more than the judicial exception. The claim 43 as drafted, is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 6,7, 11, 12, 42, and 44 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ( US 11914630 B2), hereinafter referenced as Wu, in view of Feder et al. (Active deep learning to detect demographic traits in free-form clinical notes, Journal of Biomedical Informatics, 2020), hereinafter referenced as Feder.
Regarding Claim 1, Wu teaches a computer implemented method for training a machine learning engine to label sensitive information from text data ( Wu: Column 2, lines 51-63, column 5, lines 45-50, Fig. 1 illustrates a machine learning model to categorize sensitive information from text data),
the method comprising:
(i) receiving text data and a list of classes that defines the sensitive information to be labelled ( Wu: Column 6, lines 20-40, Fig. 2, labelling function generation module 110 receives text data from lexical database module 140 and receiving text data category from module 230, where text data categories include, gender, health, political, race, religion, and biometrics);
(ii) generating a set of synthetic sentences and using the set of synthetic sentences for training the machine learning engine ( Wu: Column 4, lines 10-19, column 5, lines 20-27,the lexical database may provide cognitive synonyms of words and example sentences ( synthetic sentences) associated with each category, which is used to train the transformer based machine learning algorithm);
(iii) predicting labels for entities in a sample of the text data, [selecting a subsample of labelled sentences from the sample of text data to provide to an annotator for reviewing], and updating the training data with the reviewed sentences ( Wu: Column 7, lines 27-32, Fig. 1, labelling function generation module 110 are provided to probabilistic label generation module 120, which may annotate ( e.g., label) unlabeled text data with probabilistic labels determined for the unlabeled text data based on the labelling functions); and
(iv) training the machine learning engine with the updated training data and repeating step (iii) [until the performance of the machine learning engine meets an end-user requirement ] ( Wu: Column 8, lines 36-67, Figs. 1,4, unlabeled text data with probabilistic labels are being input to the Machine Learning Training Module 130 and tunes (e.g., refines) itself to generate one or more trained classifiers for classifying unlabeled text data).
Wu while teaching the method of claim 1, fails to explicitly teach the claimed, selecting a subsample of labelled sentences from the sample of text data to provide to an annotator for reviewing, repeating step and updating the training data with the reviewed sentences, (iii) until the performance of the machine learning engine meets an end-user requirement.
However, Feder does teach the claimed, (iii) predicting labels for entities in a sample of the text data, selecting a subsample of labelled sentences from the sample of text data to provide to an annotator for reviewing, and updating the training data with the reviewed sentences ( Feder: Section 3.2, para. [1],[ 3], page 3, Table 1, 2, illustrates labeling of sentences on the demographic traits. Section 4.3.2, human annotator review and annotate two list of automatically selected sentences ( subsample). Labels are corrected ( updated) in the newly annotated sentences and classifier is retrained);
and [(iv) training the machine learning engine with the updated training data and repeating step] (iii) until the performance of the machine learning engine meets an end-user requirement ( Feder: Section 4.3.2, page 5, the improved model which is retrained with the updated labels can be used to produce better results. After first iteration, the annotator disagreed with the model on 28% of the proxy-positive subset and 65% of the proxy-negative subset. After the second iteration, disagreement dropped to 18% and 29%, respectively, which shows multiple iteration will optimized the performance of the model).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Feder’s teaching of active learning to detect demographic traits in clinical notes, into the system and method of training a machine learning algorithm to classify unlabeled text data in freeform format, taught by Wu, because, this active learning process of train and correct process would improve the accuracy of classifier to detect demographic trait.(Feder [ abstract, section 4.2, para.[1]).
Claim 44 is a system claim comprising: one or more processors ( Wu: Column 12, lines 55-62, Fig. 8, processing unit 850); and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors cause the computing system to perform operations ( Wu: Column 12, lines 63-67,column 13, lines 1-25, Fig. 8, non-transitory computer readable media that store information (e.g., program instructions) that instructs other circuitry (e.g., a processor) to perform specified operations), performing the steps in method claim 1 above and as such, claim 44 is similar in scope and content to claim 1 and therefore, claim 44 is rejected under similar rationale as presented against claim 1 above.
Regarding Claim 2, Wu in view of Feder teach the method of claim 1. Wu further teaches, in which the received text data includes unstructured text data, structured text data or a combination of unstructured and structured text data ( Wu: Column 2, lines 51-59, abstract, Text data may be produced by a wide variety of data sources including web pages (such as Wikipedia), social media applications (such as Twitter), text messaging, emails, or chat messages (such as customer service chats). Text data from these different sources may be in a variety of unorganized, freeform formats without any characterization or labelling of the text data).
(Additionally, Feder: Section 1, para.[12], page 2, the clinical notes contain semi structured text data, with sections for different types of patient-related information.)
Regarding Claim 3, Wu in view of Feder teach the method of claim 1. Wu further teaches, in which the received text data does not include any annotations or labels ( Wu: Column 7, lines 48-53, Fig. 1, Unlabeled text data provider module 150 provides unlabeled text data).
Regarding Claim 6, Wu in view of Feder teach the method of claim 1. Wu further teaches, in which the method includes the step of outputting the annotated text data ( Wu: Column 9, lines 46-49, Fig.5, machine learning algorithm module 510 outputs annotated text data as a result of the classification of the unlabeled text data. Column 8, lines 16-28, Fig. 3, In certain embodiments, unsupervised machine learning module 310 annotates the unlabeled text data with the probabilistic labels generated (e.g., the unsupervised machine learning module 310 outputs unlabeled text data with probabilistic labels annotated to the data). As described above, a probabilistic label defines probabilities of a portion of unlabeled text data (e.g., a sentence or sentence fragment) being placed into one of the text categories. Accordingly, with multiple text data categories, a probabilistic label for a portion of unlabeled text data includes the probabilities of the portion being placed into each of the multiple text data categories as determined by the labelling functions applied by unsupervised machine learning module 310).
Regarding Claim 7, Wu in view of Feder teach the method of claim 1. Feder further teaches, in which the end-user requirement includes one or more of the following: a predefined number of iterations reached a predefined confidence score reached for labelled sentences, a predefined percentage of recall, precision level, class performance or confusion score ( Feder: Section 5, Table 3 illustrates the results and performance of trained model with only first iteration and with both first and second iteration, which shows improved score F1, recall and precision).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Feder’s teaching of active learning to detect demographic traits in clinical notes, into the system and method of training a machine learning algorithm to classify unlabeled text data in freeform format, taught by Wu, because, this active learning process of train and correct process would improve the accuracy of classifier to detect demographic trait.(Feder [ abstract, section 4.2, para.[1]).
Regarding Claim 11, Wu in view of Feder teach the method of claim 1. Wu further teaches, in which the synthetic sentences are generated based on grammar rules or models to produce a sequence of words or tokens in context, and in which the grammar rules or models are automatically selected based on analysing the received text data ( Wu: Column 4, lines 7-12, column 6, lines 1-7, example sentences ( synthetic sentences) are provided from the lexical database which include nouns, verbs, adjectives, and adverbs that are grouped into sets of cognitive synonyms, semantic relations between words and contextual patterns and semantic meanings for different categories).
Regarding Claim 12, Wu in view of Feder teach the method of claim 1. Wu further teaches, in which the synthetic sentences contain one or more entities that belong to the one or more received classes, and in which the entities that are generated based on a regular expression and/or using lookup lists ( Wu: Column 3, lines 11-17, column 4, lines 5-12, example sentences ( synthetic sentences) contain one or more entities from six different categories of interest. Regular expression scanners (sometimes referred to as lexical analyzers) may apply a ruleset to text data in order to identify patterns of text data and categorize the data.).
Regarding Claim 42, Wu in view of Feder teach the method of claim 1. Wu further teaches, in which the machine learning engine is trained to de-identify text data ( Wu: Column 10, lines 3-9, Fig.5, masking module 520 masks or redacts a sequential text data portion based on the sequential text data portion being labelled with at least one of the text data categories).
Claims 4, 5 and 32 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ( US 11914630 B2), hereinafter referenced as Wu, in view of Feder et al. (Active deep learning to detect demographic traits in free-form clinical notes, Journal of Biomedical Informatics, 2020), hereinafter referenced as Feder, further in view of Doyle et al. ( US 20200126533 A1), hereinafter referenced as Doyle.
Regarding Claim 4, Wu in view of Feder teach the method of claim 1. Wu in view of Feder fail to explicitly teach the claimed, in which the method includes the step of providing a confidence score for each labelled sentence or entity, and in which the confidence score is a value that corresponds to the probability or likelihood that the entity belongs to the one or more classes.
However, Doyle does teach the claimed, in which the method includes the step of providing a confidence score for each labelled sentence or entity, and in which the confidence score is a value that corresponds to the probability or likelihood that the entity belongs to the one or more classes ( Doyle: Para.[0032], [0033], an offensiveness score may be generated, where the offensiveness score may indicate a likelihood that the n-gram ( which can be a single word, a series of words, a sentence, a plurality of sentences), belongs to a certain class ( offensive or not offensive)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Doyle’s teaching of training a machine learning model for identifying offensive or non-offensive computer-generated text or speech, into the system and method, taught by Wu in view of Fader, because, by detecting offensiveness in an interactive or non-interactive use cases, in which the computer-generated text is based upon and responsive to user input or some other type of inputs, where any types of adversarial inputs can generate undesirable outputs lacking appropriate cultural or temporal context for a given audience, communication can be improved. (Doyle, Para.[0003]-[0009]).
Regarding Claim 5, Wu in view of Feder teach the method of claim 1. Wu in view of Feder fail to explicitly teach the claimed, in which each entity is mapped to multiple labels, with a confidence score being associated with each label that has been mapped to the entity.
However, Doyle does teach the claimed, in which each entity is mapped to multiple labels, with a confidence score being associated with each label that has been mapped to the entity ( Doyle: Para.[0044], some n-grams may have mapped to multiple labels. Para.[0032], [0033], an offensiveness score may be generated, where the offensiveness score may indicate a likelihood that the n-gram ( which can be a single word, a series of words, a sentence, a plurality of sentences), belongs to a certain class ( offensive or not offensive)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Doyle’s teaching of training a machine learning model for identifying offensive or non-offensive computer-generated text or speech, into the system and method, taught by Wu in view of Fader, because, by detecting offensiveness in an interactive or non-interactive use cases, in which the computer-generated text is based upon and responsive to user input or some other type of inputs, where any types of adversarial inputs can generate undesirable outputs lacking appropriate cultural or temporal context for a given audience, communication can be improved. (Doyle, Para.[0003]-[0009]).
Regarding Claim 32, Wu in view of Feder teach the method of claim 1. Wu in view of Feder fail to explicitly teach the claimed, in which the method includes the step of representing each entity of the selected sentences into a vector space, in which the entities belong to the one or more classes defining the sensitive information to be labelled.
However, Doyle does teach the claimed, in which the method includes the step of representing each entity of the selected sentences into a vector space, in which the entities belong to the one or more classes defining the sensitive information to be labelled ( Doyle: Para.[0064], [0065], corpus of unstructured natural language text, received as an input and may output a set of feature vectors for the words ( entities) in the corpus. By doing so, some embodiments may map feature vectors into a vector space, thereby enabling similar words ( known offensive or not offensive) to be identified based on grouping of the feature vectors).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Doyle’s teaching of training a machine learning model for identifying offensive or non-offensive computer-generated text or speech, into the system and method, taught by Wu in view of Fader, because, by detecting offensiveness in an interactive or non-interactive use cases, in which the computer-generated text is based upon and responsive to user input or some other type of inputs, where any types of adversarial inputs can generate undesirable outputs lacking appropriate cultural or temporal context for a given audience, communication can be improved. (Doyle, Para.[0003]-[0009]).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ( US 11914630 B2), hereinafter referenced as Wu, in view of Feder et al. (Active deep learning to detect demographic traits in free-form clinical notes, Journal of Biomedical Informatics, 2020), hereinafter referenced as Feder, further in view of Curtin et al. ( US 20210287045 A1), hereinafter referenced as Curtin.
Regarding Claim 10, Wu in view of Feder teach the method of claim 1. Wu in view of Feder fail to explicitly teach the claimed, in which the sample of text data is selected based on a probability sampling approach, such as a stratified sampling approach.
However, Curtin does teach the claimed, in which the sample of text data is selected based on a probability sampling approach, such as a stratified sampling approach ( Curtin: Para.[0030], the system may select an advanced sampling method that may use stratified sampling methods to control what species are sampled for different data sets).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Curtin’s teaching of training a neural network using transfer learning to analyze medical image data, into the system and method, taught by Wu in view of Fader, because, by effectively combining different datasets for the training of the neural networks would allow the neural network to perform tasks with high degree of accuracy. (Curtin, Para.[0016]).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ( US 11914630 B2), hereinafter referenced as Wu, in view of Feder et al. (Active deep learning to detect demographic traits in free-form clinical notes, Journal of Biomedical Informatics, 2020), hereinafter referenced as Feder, further in view of Su et al. ( US 20200320289 A1), hereinafter referenced as Su.
Regarding Claim 13, Wu in view of Feder teach the method of claim 1. Wu in view of Feder fail to explicitly teach the claimed, in which the method includes the step of introducing noise in the synthetic sentences, such as including generating typos.
However, Su does teach the claimed, in which the method includes the step of introducing noise in the synthetic sentences, such as including generating typos ( Su: Para.[0077], training data generator can augment training input by performing one or more data augmentation techniques such as introduce noise into at least part of the particular electronic document(s) in training input).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Su’s teaching of training and utilizing an artificial neural network (ANN) to recognize text, into the system and method, taught by Wu in view of Fader, because, generation of these specific training data sets can be used to improve and/or increase performance of ANNs trained with these specific training data sets. (Su, Para.[0029]).
Claims 16, 17, 22 and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ( US 11914630 B2), hereinafter referenced as Wu, in view of Feder et al. (Active deep learning to detect demographic traits in free-form clinical notes, Journal of Biomedical Informatics, 2020), hereinafter referenced as Feder, further in view of Le et al. ( US 20220383150 A1), hereinafter referenced as Le.
Regarding Claim 16, Wu in view of Feder teach the method of claim 1. Wu in view of Feder fail to explicitly teach the claimed, in which the method includes the step of generating a confusion matrix that represents a comparison of the predicted labels with the labels reviewed by the annotator.
However, Le does teach the claimed, in which the method includes the step of generating a confusion matrix that represents a comparison of the predicted labels with the labels reviewed by the annotator ( Le: Para.[0095]-[0097], Fig. 4J, generating a confusion matrix 450 which represents a true label of each data sample and a predicted label of the data sample).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Le’s teaching of utilizing machine learning to perform tasks involving natural language understanding for customer support, question answering, entity recognition, or intention detection, into the system and method, taught by Wu in view of Fader, because, By providing on-demand and cloud-vendor-agnostic use of cloud-based computing devices for user-selected machine-learning models and datasets, the disclosed system improves the efficiency and accuracy of computing systems that learn parameters for machine-learning models. (Le, Para.[0002]-[0003]).
Regarding Claim 17, Wu in view of Feder, further in view of Le teach the method of claim 16. Le further teaches, in which the selection of labelled sentences in step (iii) is based on the generated confusion matrix ( Le: Para.[0097],Fig. 4J, labelled data are selected by selecting a cell in the confusion matrix),
and in which the method includes the step of providing a confusion score for each labelled entity of the selected sentences, in which the confusion score is a value that indicates how close the prediction for a given class is to the prediction for another class ( Le: Para.[0077],[0104], Fig.4D, when generating a dataset and/or a data sample within the dataset, a dataset creator inputs text according to the labeling format to indicate specific entity types for content. F1 scores are calculated for two different datasets).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Le’s teaching of utilizing machine learning to perform tasks involving natural language understanding for customer support, question answering, entity recognition, or intention detection, into the system and method, taught by Wu in view of Fader, because, By providing on-demand and cloud-vendor-agnostic use of cloud-based computing devices for user-selected machine-learning models and datasets, the disclosed system improves the efficiency and accuracy of computing systems that learn parameters for machine-learning models. (Le, Para.[0002]-[0003]).
Regarding Claim 22, Wu in view of Feder, further in view of Le teach the method of claim 16. Le further teaches, in which the confusion matrix and/or the confusion scores are updated for each iteration of step (iii) ( Le: Para.[0097], Fig.4J, in response to a selection of a different cell in the confusion matrix 450, the client device 400 updates the graphical user interface to display different mapped data samples corresponding to the different cell).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Le’s teaching of utilizing machine learning to perform tasks involving natural language understanding for customer support, question answering, entity recognition, or intention detection, into the system and method, taught by Wu in view of Fader, because, By providing on-demand and cloud-vendor-agnostic use of cloud-based computing devices for user-selected machine-learning models and datasets, the disclosed system improves the efficiency and accuracy of computing systems that learn parameters for machine-learning models. (Le, Para.[0002]-[0003]).
Regarding Claim 28, Wu in view of Feder, further in view of Le teach the method of claim 16. Le further teaches, in which the weights are updated based on the confusion matrix and/or the confusion scores ( Le: Para.[0099],[0100], scores are updates based on confusion matrix).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Le’s teaching of utilizing machine learning to perform tasks involving natural language understanding for customer support, question answering, entity recognition, or intention detection, into the system and method, taught by Wu in view of Fader, because, By providing on-demand and cloud-vendor-agnostic use of cloud-based computing devices for user-selected machine-learning models and datasets, the disclosed system improves the efficiency and accuracy of computing systems that learn parameters for machine-learning models. (Le, Para.[0002]-[0003]).
Claims 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ( US 11914630 B2), hereinafter referenced as Wu, in view of Feder et al. (Active deep learning to detect demographic traits in free-form clinical notes, Journal of Biomedical Informatics, 2020), hereinafter referenced as Feder, further in view of Le et al. ( US 20220383150 A1), hereinafter referenced as Le, further in view of Doyle et al. ( US 20200126533 A1), hereinafter referenced as Doyle.
Regarding Claim 19, Wu in view of Feder, further in view of Le teach the method of claim 16. Wu in view of Feder, further in view of Le fail to explicitly teach the claimed, in which the confusion score is determined for each selected sentence, based on the confusion score determined for each entity in the sentence.
However, Doyle does teach the claimed, in which the confusion score is determined for each selected sentence, based on the confusion score determined for each entity in the sentence ( Doyle: Para.[0032], [0033], an offensiveness score may be generated, where the offensiveness score may indicate a likelihood that the n-gram ( which can be a single word, a series of words, a sentence, a plurality of sentences), belongs to a certain class ( offensive or not offensive)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Doyle’s teaching of training a machine learning model for identifying offensive or non-offensive computer-generated text or speech, into the system and method, taught by Wu in view of Fader further in view of Le, because, by detecting offensiveness in an interactive or non-interactive use cases, in which the computer-generated text is based upon and responsive to user input or some other type of inputs, where any types of adversarial inputs can generate undesirable outputs lacking appropriate cultural or temporal context for a given audience, communication can be improved. (Doyle, Para.[0003]-[0009]).
Regarding Claim 20, Wu in view of Feder, further in view of Le teach the method of claim 16. Wu in view of Feder, further in view of Le fail to explicitly teach the claimed, in which the machine learning engine is configured to rank each selected sentence based on an analysis of the confusion matrix and/or the confusion scores.
However, Doyle does teach the claimed, in which the machine learning engine is configured to rank each selected sentence based on an analysis of the confusion matrix and/or the confusion scores ( Doyle: Para.[0093], ranking based on semantic similarity and scores).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Doyle’s teaching of training a machine learning model for identifying offensive or non-offensive computer-generated text or speech, into the system and method, taught by Wu in view of Fader further in view of Le, because, by detecting offensiveness in an interactive or non-interactive use cases, in which the computer-generated text is based upon and responsive to user input or some other type of inputs, where any types of adversarial inputs can generate undesirable outputs lacking appropriate cultural or temporal context for a given audience, communication can be improved. (Doyle, Para.[0003]-[0009]).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ( US 11914630 B2), hereinafter referenced as Wu, in view of Feder et al. (Active deep learning to detect demographic traits in free-form clinical notes, Journal of Biomedical Informatics, 2020), hereinafter referenced as Feder, further in view of Le et al. ( US 20220383150 A1), hereinafter referenced as Le, further in view of Larlus-Larrondo et al. ( US 20150235160 A1), hereinafter referenced as Larlus-Larrondo .
Regarding Claim 21, Wu in view of Feder, further in view of Le teach the method of claim 16. Wu in view of Feder, further in view of Le fail to explicitly teach the claimed, in which each class or class pair is assigned a weight and in which the labelled sentences provided to the annotator are selected based on the assigned weights. 0085,0085
However, Larlus-Larrondo does teach the claimed, in which each class or class pair is assigned a weight and in which the labelled sentences provided to the annotator are selected based on the assigned weights(Larlus-Larrondo: Para.[0084],[0085], the ranking or other popularity measure may be used to compute a weighting for each class, and the classes are then sampled in proportion to their class weightings. Para.[0054], Fig. 3, the responses to the standard questions may be discarded or otherwise treated differently (e.g., by weighting their relevance in assigning labels to the standard questions with a weight which is lower than for the answers provided by crowdworkers (human annotator) which answered the gold questions with greater accuracy ).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Larlus-Larrondo’s teaching of generating questions for labeling tasks, into the system and method, taught by Wu in view of Fader further in view of Le, because, this would improve the accuracy of the fully automatic classification component with a human in the loop who reviews the most probable classes according to the classification component and chooses the correct one. (Larlus-Larrondo, Para.[0099]).
Claims 33, 35 and 36 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ( US 11914630 B2), hereinafter referenced as Wu, in view of Feder et al. (Active deep learning to detect demographic traits in free-form clinical notes, Journal of Biomedical Informatics, 2020), hereinafter referenced as Feder, further in view of Acharya et al. ( US 20070271287 A1), hereinafter referenced as Acharya.
Regarding Claim 33, Wu in view of Feder teach the method of Claim 1. Wu in view of Feder fail to explicitly teach the claimed, in which the method includes the step of determining a support for each class, in which the support refers to the set of labelled sentences that contain that class, and in which the method includes the step of representing the support for each class into a vector space and determining a centre within the vector space.
However, Acharya does teach the claimed, in which the method includes the step of determining a support for each class, in which the support refers to the set of labelled sentences that contain that class, and in which the method includes the step of representing the support for each class into a vector space and determining a centre within the vector space ( Acharya: Para.[0021]-[0023], Fig. 1, category data 11 is grouped into clusters , and/or classified into folders by the clustering/classification module 12 and described in vector space. Para.[0069], labels are assigned to category).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Acharya’s teaching of clustering and classification of multimedia data, into the system and method, taught by Wu in view of Fader, because, it would be beneficial to provide a system and method capable of clustering and classifying a category dataset by generating a hierarchy of clusters based on the clusters. (Acharya, Para.[0003]-[0007]).
Regarding Claim 35, Wu in view of Feder, further in view of Acharya teach the method of claim 33. Acharya further teaches, in which an outlier detector is used to detect outliers in the reviewed sentences and in which the outlier detector analyses each entity of the selected sentences in relation to the centre for each class ( Acharya: Para.[0110], An additional feature of hard classification is the detection and separation of outliers. Accordingly, the user sets a threshold value as the effective field-radius of the folders. If the distance between a program datum and its designated folder centre is more than this threshold, then that datum is considered to be an outlier. All such outliers are assigned to a new folder ).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Acharya’s teaching of clustering and classification of multimedia data, into the system and method, taught by Wu in view of Fader, because, it would be beneficial to provide a system and method capable of clustering and classifying a category dataset by generating a hierarchy of clusters based on the clusters. (Acharya, Para.[0003]-[0007]).
Regarding Claim 36, Wu in view of Feder teach the method of claim 1. Wu in view of Feder fail to explicitly teach the claimed, in which the machine learning engine is configured to learn to represent complex classes into multiple sub-classes, and in which the machine learning engine is configured to identify a complex class by analysing its vector space representation.
However, Acharya does teach the claimed, in which the machine learning engine is configured to learn to represent complex classes into multiple sub-classes, and in which the machine learning engine is configured to identify a complex class by analysing its vector space representation (Acharya: Para.[0022], Category data 11 is described in a vector space comprising multiple attributes or categories. Often the categories are discrete and lack a natural similarity measure between them. The data system 10 includes an input processing module 9 to preprocess input data, which contains both unstructured and semi-structured information, into category data and load the category data 11 ).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Acharya’s teaching of clustering and classification of multimedia data, into the system and method, taught by Wu in view of Fader, because, it would be beneficial to provide a system and method capable of clustering and classifying a category dataset by generating a hierarchy of clusters based on the clusters. (Acharya, Para.[0003]-[0007]).
Claim 43 is rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. ( US 11914630 B2), hereinafter referenced as Wu, in view of Feder et al. (Active deep learning to detect demographic traits in free-form clinical notes, Journal of Biomedical Informatics, 2020), hereinafter referenced as Feder, further in view of Hachey et al. ( US 20210256160 A1), hereinafter referenced as Hachey.
Regarding Claim 43, Wu in view of Feder teach the method of claim 1. Wu in view of Feder fail to explicitly teach the claimed, in which the machine learning engine is trained to de-identify text data within image or video-based data.
However, Hachey does teach the claimed, in which the machine learning engine is trained to de-identify text data within image or video-based data (Hachey: Para.[0027], Fig. 1, The servers 120 of system 10 may connect to numerous devices that can transmit or upload clinical texts. These clinical texts may be in the form of image files ( e.g. JPEG, PNG, TIFF), formatted documents (e.g. XML, HTML, PDF), text files, or other digital formats for textual information. Para.[0080], de-identification process includes redaction which is a process that detects graphical text in the pixels of the document and removes it).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Hachey’s teaching of system and method for automated text anonymization, into the system and method, taught by Wu in view of Fader, because, it would improve and accurately automating text anonymization in an easy to use manner by a clinical domain expert (perhaps a non-software expert) on a variety of reports or text data that are likely to have protected health information present within them. (Hachey, Para.[0009]).
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant's disclosure.
Moynihan et al. (US 20220198321 A1) teaches a computer-implemented method for validating input data from a third-party vendor. The method includes receiving, by a computing device, a plurality of prospectuses from a plurality of third-party entities and generating, by the computing device, a trained machine learning model using the plurality of prospectuses. The method also includes applying, by the computing device, the trained machine learning model on the input data to predict a classification label for the input data and generate a confidence level for the prediction.
Ding et al. (US 20210334596 A1) teaches a system for automatically labeling data using conceptual descriptions. In one example, the system includes an electronic processor configured to generate unlabeled training data examples from one or more natural language documents and, for each of a plurality of categories, determine one or more concepts associated with a conceptual description of the category and generate a weak annotator for each of the one or more concepts. The electronic processor is also configured to apply each weak annotator to each training data example and, when a training data example satisfies a weak annotator, output a category associated with the weak annotator. For each training data example, the electronic processor determines a probabilistic distribution of the plurality of categories. For each training data example, the electronic processor labels the training data example with a category having the highest value in the probabilistic distribution determined for the training data example.
Shirani et al. (US 20210133279 A1) teaches utilizing a neural network to flexibly generate label distributions for modifying a segment of text to emphasize one or more words that accurately communicate the meaning of the segment of text. For example, the disclosed systems can utilize a neural network having a long short-term memory neural network architecture to analyze a segment of text and generate a plurality of label distributions corresponding to the words included therein. The label distribution for a given word can include probabilities across a plurality of labels from a text emphasis labeling scheme where a given probability represents the degree to which the corresponding label describes the word. The disclosed systems can modify the segment of text to emphasize one or more of the included words based on the generated label distributions.
Sewak et al. (US 12197486 B2 ) teaches a technology which determines whether a candidate text is in a requested class by using a generative model that may not be trained on the requested class. The present technology may use of a model trained primarily in an unsupervised mode, without requiring a large number of manual user-input examples of a label class. The may produce a semantically rich positive example of label text from a candidate text and label. Likewise, the technology may produce from the candidate text and the label a semantically rich negative example of label text. The labeling service makes use of a generative model to produce a generative result, which estimates the likelihood that the label properly applies to the candidate text. In another aspect, the technology is directed toward a method for obtaining a semantically rich example that is similar to a candidate text.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NADIRA SULTANA whose telephone number is (571)272-4048. The examiner can normally be reached M-F,7:30 am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras D. Shah can be reached on (571) 270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NADIRA SULTANA/Examiner, Art Unit 2653 /FARIBA SIRJANI/Primary Examiner, Art Unit 2659