DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1 to 9 are rejected under 35 U.S.C. 103 as being unpatentable over SEWAK; Mohit et al. (US 20220414137 A1), hereinafter SEWAK, in view of DE PAIVA; Daniel da Silva et al. (US 20220180863 A1), hereinafter DE PAIVA, in further view of SETHURAMAN; Prabhakaran et al. (US 20230259991 A1), hereinafter SETHURAMAN.
SEWAK is relied upon as the main reference since it teaches text classification and labeling using a generative model using computer implemented techniques that rely on the log-probabilities, and teaching capabilities such as receiving candidate text and label, generating label-specific examples, determine label probabilities based on log-probabilities and determine the correspondence between the text and the selected label. DE PAIVA is relied upon for teaching the use of a language model calculated perplexity to measure the predictability of a text in a sequence, further teaching concepts of perplexity-based ranking and selection based on perplexity. SETHURAMAN is relied upon for teaching the use of perplexity as indicator of correspondence between a text passage and the semantic topic (which is analogous to the correspondence between a text passage and the assigned/determined label).
Regarding claim 1, SEWAK teaches:
A processor implemented method or text classification, the method comprising: receiving, via one or more hardware processors, a text, predefined numbers of class labels, a set of key phrases associated with each of the predefined class labels, and a connector sentence, wherein the text is to be classified into one or more class labels from among predefined class labels;
“ The technology described herein determines whether a candidate text is in a requested class. The technology may perform this classification without any prior training data or model trained on the requested class. In fact, a user may specify the class as a natural language input, rather than selecting it from existing classes. The requested class does not need to follow a hierarchy or be predefined. The technology is effective even when the requested class is a concept, such as diversity, rather than a noun. The requested class may be described herein as a label.” (SEWAK [0029])
“ The technology described herein determines whether a candidate text is in a requested class by using a generative model that may not be trained on the requested class. The present technology may use of a model trained primarily in an unsupervised mode, without requiring a large number of manual user-input examples of a label class. The may produce a semantically rich positive example of label text from a candidate text and label. Likewise, the technology may produce from the candidate text and the label a semantically rich negative example of label text. The labeling service makes use of a generative model to produce a generative result, which estimates the likelihood that the label properly applies to the candidate text. In another aspect, the technology is directed toward a method for obtaining a semantically rich example that is similar to a candidate text. ” (SEWAK [ABSTRACT])
“ ... In aspects, the candidate text may be a portion of a larger document, such as a sentence, phrase, or paragraph of a document. ... ” (SEWAK [0171])
generating, via the one or more hardware processors, a plurality of label-specific augmentations for the text based on each key phrase among the set of key phrases associated with each of the predefined class labels, and the connector sentence;
“ The method 1300, at block 1314, includes receiving a generated text from the generative model, the generated text comprising a plurality of tokens and associated probabilities. Generated text broadly includes the actual stream of text tokens produced by the model, as well as an associated token probability reported for each token, and a vector of log probabilities where each log probability describes a set number likelihoods corresponding to tokens that the model might have chosen. ... ” (SEWAK [0168])
“ In another aspect, the technology is directed toward a method of providing a semantically rich set of keywords from candidate text in the context of a label. Other solutions have been semantically poor in representation, and so the number of returns from a search engine that must be received to obtain a certain number of relevant results was large. The present technology improves the state of the art, for example, by providing good performance while producing a semantically rich set of keywords, thus reducing the amount of data required for training. A set of candidate text priority keywords are obtained from candidate text. A set of label priority keywords are obtained from the label. Priority keywords are assigned embedding vectors using a transformer-based model. ... ” (SEWAK [0010])
“ ... The NC method of label scoring at step 380 uses a number to represent the label class and another number to represent the anti-label class. So for the binary case label=1, anti-label=0. In an exemplary embodiment, an NLG model is used in zero-shot mode. For example, the model prompt might be prepared by combining the examples with respective labels using a sentence-class-separator. Separate examples using a sentence mask break. Next, the prompt is continued with another sentence-mask, followed by ‘sentence-class-separator’, followed by ‘Prediction-Start’ prompt. For a case with one positive and one negative generated example, the prompt may be: [‘Positive Example’ ‘sentence-class-separator’ 1 ‘sentence-break’ ‘Negative Example’ ‘sentence-class-separator’ 0 ‘sentence-break’ ‘candidate text’] ” (SEWAK [0171])
Wherein SEWAK’s sentence-class-separators serve as a connector sentence. The purpose of the sentence-class-separator is to prepare the text by combining examples with respective labels, therefore connecting the separated components into a complete sentence.
deriving, by a Language Model (LM) executed by the one or more hardware processors,
“...The prediction probability is generated/computed at step 1630 by finding keywords or synonyms of keywords in the generative output, and determining from the token probabilities of the keywords or synonyms in the output. For the SL mode, instead of using the numerical label, the prompt is used by combining respective labels. For example, if the label class has prioritized keyword list ‘service harmony’ and the anti-label class has prioritized keyword list ‘disservice disharmony’, and the method processes one positive and one negative example, then the prompt might be: [‘Let me know if there's anything else I can do for you. I'm happy to help’ ‘sentence-class-separator’ ‘service harmony’ ‘sentence-break’ ‘This is your problem, not mine’ ‘sentence-class-separator’ ‘disservice disharmony’ ‘sentence-break’ ‘candidate text’] ” (SEWAK [0101])
and wherein each of the log-likelihood based key phrase level features captures an increase in a log-likelihood of the key phrase from the set of key phrases,
“ The method 1300, at block 1314, includes receiving a generated text from the generative model, the generated text comprising a plurality of tokens and associated probabilities. Generated text broadly includes the actual stream of text tokens produced by the model, as well as an associated token probability reported for each token, and a vector of log probabilities where each log probability describes a set number likelihoods corresponding to tokens that the model might have chosen. ... ” (SEWAK [0168])
“...The prediction probability is generated/computed at step 1630 by finding keywords or synonyms of keywords in the generative output, and determining from the token probabilities of the keywords or synonyms in the output. For the SL mode, instead of using the numerical label, the prompt is used by combining respective labels. For example, if the label class has prioritized keyword list ‘service harmony’ and the anti-label class has prioritized keyword list ‘disservice disharmony’, and the method processes one positive and one negative example, then the prompt might be: [‘Let me know if there's anything else I can do for you. I'm happy to help’ ‘sentence-class-separator’ ‘service harmony’ ‘sentence-break’ ‘This is your problem, not mine’ ‘sentence-class-separator’ ‘disservice disharmony’ ‘sentence-break’ ‘candidate text’] ” (SEWAK [0101])
wherein the increase in the log-likelihood is a difference between a conditional log-likelihood of the key phrase given the text to be classified,
“Embodiment 20. The system as in any one of embodiments 14, 15, 16, 17, 18, or 19, further comprising confirming that the text snippet is likely to represent the class label by using a label scoring method that receives the text snippet and the class label and returns an indication that the probability that the text snippet embodies the class label is above a threshold. ” (SEWAK [0224])
“ At step 520, the generative model is applied to candidate text (denoted CT) to obtain a corresponding generative output (denoted GR-CT). The method proceeds to step 530 to compute a search score of candidate-generated text (GR-CT) from the document set created by the generative examples created at step 510. Generally, the idea of the SS method is to use the generative output for the candidate (GR-CT) as a query in a search engine and to measure the search resulting rank as a metric to decide if the results generated from the label examples (GR-EX-L1 and GR-EX-L2) are closer to the query (GR-CT) than the results generated from the anti-label examples (GR-EX-AL1 and GR-EX-AL2). Generally, a trained structural/semantic similarity model search engine is preferable, which measures semantic distance of between query and result, (e.g. Microsoft® DSSM). Alternatively, a GPT-3 search rank could be used. ” (SEWAK [0105])
and the log-likelihood of the key phrase;
“ The method 1300, at block 1314, includes receiving a generated text from the generative model, the generated text comprising a plurality of tokens and associated probabilities. Generated text broadly includes the actual stream of text tokens produced by the model, as well as an associated token probability reported for each token, and a vector of log probabilities where each log probability describes a set number likelihoods corresponding to tokens that the model might have chosen. ... ” (SEWAK [0168])
and ii) a class level log-likelihood based feature for each of the predefined class labels as maximum of the log-likelihood based key phrase level features associated with the corresponding class label;
“ The technology described herein determines whether a candidate text is in a requested class by using a generative model that may not be trained on the requested class. The present technology may use of a model trained primarily in an unsupervised mode, without requiring a large number of manual user-input examples of a label class. The may produce a semantically rich positive example of label text from a candidate text and label. Likewise, the technology may produce from the candidate text and the label a semantically rich negative example of label text. The labeling service makes use of a generative model to produce a generative result, which estimates the likelihood that the label properly applies to the candidate text. In another aspect, the technology is directed toward a method for obtaining a semantically rich example that is similar to a candidate text. ” (SEWAK [ABSTRACT])
“ The present technology has excellent objectivity. The objective of the system is not just the prediction of arbitrary candidate-label levels for arbitrary candidate-text, but also the associated confidence as this is required or otherwise useful in many downstream applications and associated software features that this disclosure empowers. Humans' cognition is generally biased by virtue of the limited understanding of an individual, and so cannot produce any objectively defined and auditable confidence number for the accuracy of cognition or a specific candidate-label level. ” (SEWAK [0077])
and predicting for a zero shot classification,
“ Embodiment 10. The media as in any of the preceding embodiments, wherein the generative model is GPT3 run in zero shot mode. ” (SEWAK [0214])
and ii) value of class level log-likelihood based features lying above a maximum threshold value.
“ Embodiment 20. The system as in any one of embodiments 14, 15, 16, 17, 18, or 19, further comprising confirming that the text snippet is likely to represent the class label by using a label scoring method that receives the text snippet and the class label and returns an indication that the probability that the text snippet embodies the class label is above a threshold. ” (SEWAK [0224])
“ Labeling service 142 returns a candidate label-class prediction, such as 1 for true, 0 for false, to provide a binary classification output. Generally, all rendered result data is received by labelling application 110 from labeling service 142, which provides the rendered result data to be presented on display 120. Generally, a labelling result that is rendered is any label-related information item whose display or use is provided to a component of a labeling system, e.g. shown in operating environment 100 when the labeling service 142 has determined that the candidate text meets acceptably criteria. For example, acceptability criteria may be that an estimate of the probability of a label class is above an acceptable threshold level. The rendered results are capable of being provided in an unsupervised fashion, because the system does not require a user to necessarily provide any examples of text that are properly classified to a defined label. ” (SEWAK [0067])
SEWAK does not teach, but DE PAIVA teaches:
deriving, by a Language Model (LM) executed by the one or more hardware processors, perplexity based key phrase level features
“Embodiments of the present disclosure provide for improved data sequence validity processing, for example to determine validity of sentences or other language within a particular language domain. Such improved processing is useful at least for arranging data sequences based on determined validity, and/or making determinations and/or performing actions based on the determined validity. A determined probability (e.g., transformed into the perplexity space) of each token appearing in a data sequence is used in any of a myriad of manners to perform such data sequence validity processing. Example embodiments provide for generating a perplexity value set for each data sequence in a plurality of data sequences, generating a probabilistic ranking set for the plurality of data sequences based on the perplexity value sets and at least one sequence ranking metric, and generating an arrangement of the plurality of data sequences based on the probabilistic ranking set. ” (DE PAIVA [Abstract])
wherein each of the perplexity based key phrase level features captures a reduction in perplexity of a key phrase from the set of key phrases,
“A language model may then be used to generate a probability for each token of a data sequence, representing how probable the token is to exist in a valid data sequence of the data sequence based on the surrounding context for the token. The probability may be transformed into a space that enables comparison with the perplexity value(s) determined for the language model determined during testing.” (DE PAIVA [0098])
“Embodiments of the present disclosure provide probabilistic data sequence processing for such data generation and/or corresponding determinations and/or actions in a manner that provides various technical improvements over conventional implementations for identifying invalid data sequences. For example, in at least some embodiments, utilization of a language model to determine perplexity and subsequent ranking based on the perplexity enables data sequence ranking and/or arrangement in a manner that is both more accurate and efficient than conventional implementations. Upon providing such accurate data sequence ranking(s) and/or arrangements, at least some embodiments are configured to enable use of such ranking(s) and/or arrangement in any of a myriad of subsequent use cases, such as to provide such data to a user, further process such data, remove and/or otherwise manipulate the set of data sequence(s) based on such data, and/or the like. Additionally or alternatively, at least some embodiments utilize a language model that reduces the complexity of implementations for data sequence processing without negatively impacting efficiency and/or accuracy. Additionally or alternatively, embodiments utilize processing components and models not limited to any particular order of data sequences and/or language of data sequences.” (DE PAIVA [0041])
wherein the reduction in perplexity is a ratio of conditional perplexity of the key phrase given the text to be classified, to the perplexity of the key phrase,
“FIG. 12 illustrates a flowchart depicting example operations of an example process for removing invalid data sequences from a set of data sequences as part of data sequence validity processing in accordance with at least some example embodiments of the present disclosure. In some such embodiments, the example process 1200 embodies another example computer-driven action that is performable based on the data generated and/or otherwise calculated as described herein. ... ” (DE PAIVA [0166])
“ Each of the depicted token perplexity graphs 702A-702C are depicted together with an indicator of which of the token perplexity graphs 702A-702C represent higher perplexity and thus more problematic data sequences (e.g., more likely to be invalid) as opposed to lower perplexity and less problematic data sequences (e.g., less likely to be invalid). The overall perplexity for a particular data sequence is represented by a probabilistic ranking embodying an area violating threshold value determined from the various perplexity values plotted in the corresponding token perplexity graph of the token perplexity graphs 702A-702C. ” (DE PAIVA [0124])
“ In some embodiments, the token perplexities are normalized based on a determinable value. For example, in one or more embodiments, each token perplexity is normalized based on a mean token perplexity determined for a particular language model at a particular position in a data sequence. ” (DE PAIVA [0127])
“ ... In some embodiments where the token perplexities are normalized, an area violating threshold value is determined based on multiple unacceptable perplexity thresholds, such as a lower unacceptable perplexity threshold at a particular offset below the determinable value utilized to normalize the token perplexities and an upper unacceptable perplexity threshold at the particular offset above the determinable value utilized to normalize the token perplexities. In this regard, the area violating threshold value may be determined based on an area below the lower unacceptable perplexity threshold and a second area above the upper unacceptable perplexity threshold. ” (DE PAIVA [0127])
the one or more class labels for the text based on one of : i) value of class level perplexity based features lying below a minimum threshold value;
“ Each of the depicted token perplexity graphs 602A-602C are similarly depicted together with an indicator of which of the token perplexity graphs 602A-602C represent higher perplexity and thus more problematic data sequences (e.g., more likely to be invalid) as opposed to lower perplexity and less problematic data sequences (e.g., less likely to be invalid). ... ” (DE PAIVA [0118])
“ In some embodiments, the token perplexities are normalized based on a determinable value. For example, in one or more embodiments, each token perplexity is normalized based on a mean token perplexity determined for a particular language model at a particular position in a data sequence. Alternatively or additionally, in one or more embodiments, each token perplexity is normalized based on a median token perplexity determined for a particular language model at a particular position in a data sequence. In some embodiments where the token perplexities are normalized, an area violating threshold value is determined based on multiple unacceptable perplexity thresholds, such as a lower unacceptable perplexity threshold at a particular offset below the determinable value utilized to normalize the token perplexities and an upper unacceptable perplexity threshold at the particular offset above the determinable value utilized to normalize the token perplexities. In this regard, the area violating threshold value may be determined based on an area below the lower unacceptable perplexity threshold and a second area above the upper unacceptable perplexity threshold. ” (DE PAIVA [0127])
It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of SEWAK the capability to include the phrase-level perplexity analysis as described by DE PAIVA. The benefit and motivation of such modification is discussed by DE PAIVA in the following portion: “ ... Such improved processing is useful at least for arranging data sequences based on determined validity, and/or making determinations and/or performing actions based on the determined validity. A determined probability (e.g., transformed into the perplexity space) of each token appearing in a data sequence is used in any of a myriad of manners to perform such data sequence validity processing. ... ” (DE PAIVA [ABSTRACT])
SEWAK in view of DE PAIVA does not teach, but SETHURAMAN teaches:
determining, by the one or more hardware processors, i) a class level perplexity based feature for each of the predefined class labels as a minimum of perplexity based key phrase level features associated with the corresponding class label,
“ Methods, systems and computer program products are provided for a machine learning text interpretation model. Text (e.g., customer comments) may be processed by a text interpretation model to identify topics. Text features may be processed together or separately to identify one or more sets of topic categories. Text features (e.g., customer pain point comments and customer blocker comments) may be optionally preprocessed, summarized, vectorized, topic modeled, clustered, and analyzed to select topic categories. A topic modeler may be selected from multiple topic modelers based on perplexity scores. ” (SETHURAMAN [0003])
“ Automation of review and interpretation of comments may reduce the delay between comments and improvements. An (e.g., unsupervised) intent (e.g., described issue) classification model may automatically identify issues (e.g., topics, subjects, intents, objectives, purposes) and/or issue priorities expressed by partners, consumers, enterprise customers, etc. An intent classification model may be unsupervised, for example, if there is no ground truth of issue areas for a large dataset to train the model. ” (SETHURAMAN [0024])
It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of SEWAK in view of DE PAIVA the capability to perform the classification scoring and ranking of the labels with more than just the log-likelihood by adding the class-level perplexity to the analysis. The benefit and motivation of such modification is discussed by SETHURAMAN in the following portion: “ … Customer and/or partner comments may be voluminous, which may be extremely time-consuming for engineering teams to review, interpret, correlate with comments by others, determine, and schedule product improvements. Embodiments described herein enable such comments to be categorized in an efficient manner such that they can readily acted upon. ” (SETHURAMAN [0019])
Regarding claim 2, the rejection of claim 1 is incorporated, furthermore SEWAK teaches:
The processor implemented method of claim 1, further comprises: enhancing an accuracy of prediction of text classification of the text into one or more class labels using a pretrained supervised machine learning classifier
“ The technology of the present system is not only more accurate than what is possible by above human heuristic or unbiased model classification (trained on unrepresentative data). Results of some experiments have demonstrated better accuracy, and an established ‘recall and/at-defined False Positive Rate (FPR)’ which gives the decision-maker more objective information about the utility of the model in a real life scenario. ” (SEWAK [0063])
“ The disclosed method augments training data for large, advanced NLP data with which the underlying model could deliver better recall/FPR/accuracy, and due to the richness and variation of ideas of data it could augment, the model could learn the context better, and more holistically, which means that the model could perform reasonable better for new data/domain. ” (SEWAK [0155])
“ Other text labelling systems have required some predefined minimum number of manual user-input examples and, in addition, have required a great deal of computer processing to perform supervised training of a label classifier. The present technology improves the state of the art by providing good performance while, for example, making use of a model trained primarily in an unsupervised mode, without requiring a large number of manual user-input examples of a label class. ... ” (SEWAK [0031])
log-likelihood based key phrase level features,
“ The method 1300, at block 1314, includes receiving a generated text from the generative model, the generated text comprising a plurality of tokens and associated probabilities. Generated text broadly includes the actual stream of text tokens produced by the model, as well as an associated token probability reported for each token, and a vector of log probabilities where each log probability describes a set number likelihoods corresponding to tokens that the model might have chosen. ” (SEWAK [0168])
and the class level log-likelihood based feature per class label,
“ The disclosure describes systems and methods to train text classification models without the need of representative labelled data or human grader's assistance to create representations that could otherwise be used directly or indirectly to create representative data conducive to train a Natural Language Processing (NLP) or a text classification model that could map/classify a candidate input text across one or more classes (class-labels) of interest. ” (SEWAK [0061])
log-likelihood based key phrase level features,
“ The method 1300, at block 1314, includes receiving a generated text from the generative model, the generated text comprising a plurality of tokens and associated probabilities. Generated text broadly includes the actual stream of text tokens produced by the model, as well as an associated token probability reported for each token, and a vector of log probabilities where each log probability describes a set number likelihoods corresponding to tokens that the model might have chosen. ... ” (SEWAK [0168])
“ The technology described herein provides this improved efficiency by receiving the candidate text and the label, and may produce from them a semantically rich positive example of label text. ... ” (SEWAK [0032])
and the class level log-likelihood based feature per class label obtained for a training data.
“ The method 1500, at block 1502, includes receiving, for a classifier, a training data instance comprising example text associated with a class label. The training data instance may be provided by a user through an interface. In another aspect, the training data is pulled from a collection of training data. ” (SEWAK [0180])
“ The technology described herein provides this improved efficiency by receiving the candidate text and the label, and may produce from them a semantically rich positive example of label text. Likewise, a labelling service may produce from the candidate text and the label a semantically rich negative example of label text. The labeling service makes use of a generative model to produce a generative result, which estimates the likelihood that the label properly applies to the candidate text. The success rate of the classification can be improved, while maintaining this improved efficiency, by obtaining a second generative result from a generative model and estimating label probability using the second generative result. ” (SEWAK [0032])
SEWAK does not teach, but DE PAIVA teaches:
that utilizes the perplexity based key phrase level features,
“ A language model may then be used to generate a probability for each token of a data sequence, representing how probable the token is to exist in a valid data sequence of the data sequence based on the surrounding context for the token. The probability may be transformed into a space that enables comparison with the perplexity value(s) determined for the language model determined during testing. ” (DE PAIVA [0098])
“ The term “token” refers to a string of characters (e.g., word, set of words (e.g., a phrase)), or character, within a larger data set. ... ” (DE PAIVA [0048])
wherein the supervised machine learning classifier is trained on the perplexity based key phrase level features,
“ Additionally or alternatively, in some embodiments multiple language models may be trained and tested based on a shared corpus. In some such embodiments, the language model associated with the most preferable perplexity value (e.g., the lowest perplexity value) may be selected for use. For example, in this regard, different language models may be chosen for different domains, languages, and/or the like. ” (DE PAIVA [0097])
It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of SEWAK the capability to include the phrase-level perplexity analysis as described by DE PAIVA. The benefit and motivation of such modification is discussed by DE PAIVA in the following portion: “ ... Such improved processing is useful at least for arranging data sequences based on determined validity, and/or making determinations and/or performing actions based on the determined validity. A determined probability (e.g., transformed into the perplexity space) of each token appearing in a data sequence is used in any of a myriad of manners to perform such data sequence validity processing. ... ” (DE PAIVA [ABSTRACT])
SEWAK in view of DE PAIVA does not teach, but SETHURAMAN teaches:
the class level perplexity based feature per class label,
“ Methods, systems and computer program products are provided for a machine learning text interpretation model. Text (e.g., customer comments) may be processed by a text interpretation model to identify topics. Text features may be processed together or separately to identify one or more sets of topic categories. Text features (e.g., customer pain point comments and customer blocker comments) may be optionally preprocessed, summarized, vectorized, topic modeled, clustered, and analyzed to select topic categories. A topic modeler may be selected from multiple topic modelers based on perplexity scores. ” (SETHURAMAN [0003])
“ Automation of review and interpretation of comments may reduce the delay between comments and improvements. An (e.g., unsupervised) intent (e.g., described issue) classification model may automatically identify issues (e.g., topics, subjects, intents, objectives, purposes) and/or issue priorities expressed by partners, consumers, enterprise customers, etc. An intent classification model may be unsupervised, for example, if there is no ground truth of issue areas for a large dataset to train the model. ” (SETHURAMAN [0024])
the class level perplexity based feature per class label ,
“ Methods, systems and computer program products are provided for a machine learning text interpretation model. Text (e.g., customer comments) may be processed by a text interpretation model to identify topics. Text features may be processed together or separately to identify one or more sets of topic categories. Text features (e.g., customer pain point comments and customer blocker comments) may be optionally preprocessed, summarized, vectorized, topic modeled, clustered, and analyzed to select topic categories. A topic modeler may be selected from multiple topic modelers based on perplexity scores. ” (SETHURAMAN [0003])
“ Automation of review and interpretation of comments may reduce the delay between comments and improvements. An (e.g., unsupervised) intent (e.g., described issue) classification model may automatically identify issues (e.g., topics, subjects, intents, objectives, purposes) and/or issue priorities expressed by partners, consumers, enterprise customers, etc. An intent classification model may be unsupervised, for example, if there is no ground truth of issue areas for a large dataset to train the model. ” (SETHURAMAN [0024])
It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of SEWAK in view of DE PAIVA the capability to perform the classification scoring and ranking of the labels with more than just the log-likelihood by adding the class-level perplexity to the analysis. The benefit and motivation of such modification is discussed by SETHURAMAN in the following portion: “ … Customer and/or partner comments may be voluminous, which may be extremely time-consuming for engineering teams to review, interpret, correlate with comments by others, determine, and schedule product improvements. Embodiments described herein enable such comments to be categorized in an efficient manner such that they can readily acted upon. ” (SETHURAMAN [0019])
Regarding claim 3, the rejection of claim 4 is incorporated, furthermore SEWAK teaches:
The processor implemented method as claimed in claim 1, wherein for each predicted class label of the text, an explanation is generated in the form of a ranked list of key phrases sorted using values of
“ ...The method also comprising determining a first ranked score of the positive example result based on a response from submitting the candidate result to a search engine as a query over a corpus comprising the positive example result and the negative example result. The method also comprising determining a second ranked score of the negative example result based on the response from submitting the candidate result to the search engine as the query over a corpus comprising the positive example result and the negative example result. The method also comprising determining a label probability estimate by comparing the first ranked score of the positive example result to the second ranked score of the negative example result. The method also comprising outputting an indication whether the candidate text corresponds to the label description based on the label probability estimate.” (SEWAK [0212])
“ A prioritized keyword extraction service (e.g. FIG. 9) is generally a service that takes a text string, extracts keywords and orders them, e.g. in a label structure such as a list of keywords ordered in descending importance order. ” (SEWAK [0041])
SEWAK doesn’t teach but DE PAIVA teaches the processor implemented method as claimed in claim 1, wherein for each predicted class label of the text, an explanation is generated in the form of a ranked list of key phrases sorted using values of the perplexity based key phrase level features
“ The term “arrangement of data sequences” refers to one or more data objects structured to embody an ordered set of data sequences based on the probabilistic rankings for the data sequences. Non-limited examples of an arrangement of data sequences include an ordered data object structured to include the data sequences in an order from highest probabilistic ranking to lowest probabilistic ranking (or from lowest probabilistic ranking to highest probabilistic ranking), including without limitation an ordered list, a linked list, a sorted array or vector, and/or other sorted data structure. ” (DE PAIVA [0063])
“ In accordance with a first aspect of the disclosure, a computer-implemented method for arranging a plurality of data sequences is provided. The computer-implemented method may be executed via one or more computing devices embodied in hardware, software, firmware, and/or a combination thereof. An example embodiment computer-implemented method includes, for each data sequence of a plurality of data sequences, each data sequence comprising a token sequence: calculating, utilizing a language model, a perplexity value set associated with the data sequence, wherein the perplexity value set comprises a perplexity value for each data token in token sequence of the data sequence; and generating a probabilistic ranking set for the plurality of data sequences, the probabilistic ranking set including a probabilistic ranking for each data sequence in the plurality of data sequences, and the probabilistic ranking set generated based on at least one sequence arrangement metric and the perplexity value set for each data sequence of the plurality of data sequences. The example computer-implemented method further includes generating an arrangement of the plurality of data sequences based on the probabilistic ranking set. ” (DE PAIVA [0004])
It would have been obvious to someone of ordinary skill in the art before the effective filling date of the claimed invention to include in the teachings of SEWAK the capability to include the phrase-level perplexity analysis as described by DE PAIVA. The benefit and motivation of such modification is discussed by DE PAIVA in the following portion: “ ... Such improved processing is useful at least for arranging data sequences based on determined validity, and/or making determinations and/or performing actions based on the determined validity. A determined probability (e.g., transformed into the perplexity space) of each token appearing in a data sequence is used in any of a myriad of manners to perform such data sequence validity processing. ... ” (DE PAIVA [ABSTRACT])
Regarding claim 4, arguments analogous to claim 1 are applicable, furthermore SEWAK teaches:
A system for text classification, the system comprising: a memory storing instructions; one or more Input/Output (I/O) interfaces; and one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to: …
“ Computing device 800 typically includes a variety of computer-readable media. Computer-readable media may be any available media that may be accessed by computing device 800 and includes both volatile and nonvolatile, removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. ” (SEWAK [0193])
“ Memory 812 includes computer storage media in the form of volatile and/or nonvolatile memory. The memory 812 may be removable, non-removable, or a combination thereof. Exemplary memory includes solid-state memory, hard drives, optical-disc drives, etc. Computing device 800 includes one or more processors 814 that read data from various entities such as bus 810, memory 812, or I/O components 820. Presentation component(s) 816 present data indications to a user or other device. Exemplary presentation components 816 include a display device, speaker, printing component, vibrating component, etc. I/O ports 818 allow computing device 800 to be logically coupled to other devices, including I/O components 820, some of which may be built in. ” (SEWAK [0196])
Regarding claim 5, the rejection of claim 4 is incorporated, furthermore arguments analogous to claim 2 are applicable.
Regarding claim 6, the rejection of claim 4 is incorporated, furthermore arguments analogous to claim 3 are applicable.
Regarding claim 7, arguments analogous to claim 1 are applicable, furthermore SEWAK teaches:
One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause: …
“ The technology described herein may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions, such as program components, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program components, including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks or implements particular abstract data types. The technology described herein may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, specialty computing devices, etc. Aspects of the technology described herein may also be practiced in distributed computing environments where tasks are performed by remote-processing devices that are linked through a communications network. ” (SEWAK [0191])
“ Computing device 800 typically includes a variety of computer-readable media. Computer-readable media may be any available media that may be accessed by computing device 800 and includes both volatile and nonvolatile, removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. ” (SEWAK [0193])
Regarding claim 8, the rejection of claim 7 is incorporated, furthermore arguments analogous to claim 2 are applicable.
Regarding claim 9, the rejection of claim 7 is incorporated, furthermore arguments analogous to claim 3 are applicable.
Pertinent Art Statement
Kumar Choubey; Prafulla et al. (US 20230334245 A1) is not relied on for this rejection, but it is listed as pertinent art because it teaches zero-shot text classification, the use of pre-trained language models for classification labels, label scoring/ranking, and threshold-based prediction for those labels.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HECTOR J. CRESPO FEBLES whose telephone number is (571)272-4512. The examiner can normally be reached Mon - Fri 7:30 - 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at (571) 272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HECTOR J. CRESPO FEBLES/Examiner, Art Unit 2657
/DANIEL C WASHBURN/Supervisory Patent Examiner, Art Unit 2657
Prosecution Insights