Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Introduction
This office action is in response to Applicant’s submission filed on 2/22/2024. As such, claims 1-20 have been examined.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, 5-11, and 14-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 recites a system that, under the broadest reasonable interpretation, claims limitations that cover performance of the limitations in the human mind with the assistance of physical aids (e.g., pen and paper), but for the recitation of generic or well-known or conventional computer components. That is, other than reciting “processor and computer readable medium,” nothing in these claim limitations precludes the steps from practically being performed in the mind. As a whole, claim 1 pertains to evaluating AI generated summary, which is a mental process that a human can do. Individually, each of the limitations also pertains to a mental process, for example:
accessing, for a precision evaluation of an AI generated text summary using the evaluation framework, a data structure including the AI generated text summary and an original text summarized in the AI generated text summary by an AI summarization engine; (e.g., a human receiving AI generated summary and then reading the original text content, like a news article for instance.)
calculating a plurality of summarization evaluation metrics that evaluate a summarization of the original text in the AI generated text summary using the evaluation framework with the AI generated text summary and the original text, wherein the plurality of summarization evaluation metrics are selected for the evaluation framework based on a relevance assessment and a significance assessment of each of the plurality of summarization evaluation metrics when performing the precision evaluation of the summarization; (e.g., the human assessing how well the summary captures main points/theme of the original text/article/content to generate a qualitative judgement on its accuracy, coherence and completeness.) [this step can also be perform by the human using a generic computer component, like entering the summary into some well-known and conventional evaluation tool like ROUGE or METEOR]
weighting the plurality of summarization evaluation metrics based on a plurality of weights applied by the evaluation framework; (e.g., the human weighting or putting more or less emphasis on the different evaluation metrics, for example, conciseness could be weighted as more important than a detailed summary.)
computing a final evaluation score of the summarization of the original text in the AI generated text summary by the AI summarization engine based on an aggregation of the weighted plurality of summarization evaluation metrics; (e.g., the human combining the weights of various evaluation metrics to come up with a final/overall evaluation score. This could be done with assistance of pen and paper.)
outputting the precision evaluation of the AI generated text summary based on the computed final evaluation score; (e.g., the human providing feedback or assessment of the AI generated summary on a paper.)
and updating the data structure with the computed final evaluation score for the precision evaluation. (e.g., the human remembers this evaluation for future reference when evaluating summaries from the same AI algorithm or write down the evaluation score on a paper and note that it is related to or associated with specific AI generated summary.)
The judicial exception is not integrated into a practical application. In particular, the claims only recites generic computing components. Such generic computing components are recited at a high-level of generality (i.e., as a generic processor performing a generic computer function of receiving, determining, or outputting information) such that they amount to no more than mere instructions to apply the exception using generic computer components. Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
Claim 1 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional limitations of using generic computer components amount to no more than mere instructions to apply the exception using generic computer components. Mere instructions to apply an exception using generic computer components cannot provide an inventive concept. Claim 1 is not patent eligible.
The examiner further notes that the use of claimed generic computer components (“processor and computer readable medium,”) invokes such generic computer components “merely as a tool to perform an existing process”. MPEP 2106.05(f). MPEP 2106.05(f) further explains:
Use of a computer or other machinery in its ordinary capacity for economic or other tasks (e.g., to receive, store, or transmit data) or simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. See Affinity Labs v. DirecTV, 838 F.3d 1253, 1262, 120 USPQ2d 1201, 1207 (Fed. Cir. 2016) (cellular telephone); TLI Communications LLC v. AV Auto, LLC, 823 F.3d 607, 613, 118 USPQ2d 1744, 1748 (Fed. Cir. 2016) (computer server and telephone unit). Similarly, "claiming the improved speed or efficiency inherent with applying the abstract idea on a computer" does not integrate a judicial exception into a practical application or provide an inventive concept. Intellectual Ventures I LLC v. Capital One Bank (USA), 792 F.3d 1363, 1367, 115 USPQ2d 1636, 1639 (Fed. Cir. 2015).
Claim 1 recites generic computer components (“processor and computer readable medium,”), with respect to performing tasks. MPEP 2106.05(d) and (f) further provides examples of court decisions where the courts found generic computing components to be mere instructions to apply a judicial exception, and further explains “increased speed” (e.g., using a computer to increase the speed of an otherwise mental process) does not provide an inventive concept. For example:
A commonplace business method or mathematical algorithm being applied on a general purpose computer, Alice Corp. Pty. Ltd. V. CLS Bank Int’l, 573 U.S. 208, 223, 110 USPQ2d 1976, 1983 (2014); Gottschalk v. Benson, 409 U.S. 63, 64, 175 USPQ 673, 674 (1972); Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015).
A process for monitoring audit log data that is executed on a general-purpose computer where the increased speed in the process comes solely from the capabilities of the general-purpose computer, FairWarning IP, LLC v. Iatric Sys., 839 F.3d 1089, 1095, 120 USPQ2d 1293, 1296 (Fed. Cir. 2016) (emphasis added).
Performing repetitive calculations. Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) ("The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims.")
Claim 10 recites method claim that corresponds to the system of claim 1 and is therefore rejected under the same grounds as claim 1 above.
Claim 19 recites non-transitory CRM claim that corresponds to the system of claim 1 and is therefore rejected under the same grounds as claim 1 above. While claim 19 further recites “computer-readable medium having stored thereon computer-readable instructions”, these are merely generic computer components recited at a high-level of generality such that they amount to no more than mere instructions to apply the exception using a generic computer component. Therefore, none of these limitations (a) integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea or (b) amount to significantly more than the judicial exception, because in either case the additional limitations merely utilize generic computer components that amounts to no more than mere instructions to apply the exception using generic computer function. Claim 19 is not patent eligible.
Claims 2, 5-9, 11, 14-18, and 20 depend from independent claims 1, 10 and 19, do not remedy any of the deficiencies of claims 1, 10 and 19, and therefore are rejected on the same grounds as claim 1, 10 and 19 from above.
Claim 2 further recites: wherein one of the plurality of summarization evaluation metrics comprises a global coherence metric configured to assess a topic similarity between the original text and the AI generated text summary. (e.g., the human assess the topic similarity between the original text and the AI generated summary to determine if the AI generated summary accurately reflects the main topics or themes of the original text.)
Claim 5, further recites: wherein the plurality of summarization evaluation metrics further comprise at least one of a recall-oriented understudy for Gisting evaluation (ROUGE)-2 score, a ROUGE-L score, a bilingual evaluation understudy (BLEU) score, a metric for evaluation of translation with explicit ordering (METEOR) score, a bidirectional encoder representations from transformers (BERT) score, an entity preservation score, a semantic similarity score using BERT or robustly optimized BERT pretraining approach (RoBERTa), a length ratio, a precision-at-K score, a word error rate, a normalized cross entropy, or an overlap coefficient. (e.g., the human can evaluate a word error rate or a length ratio.) [the claim only requires one of the recited features]
Claim 6, further recites: wherein the plurality of summarization evaluation metrics are split into four categories each having a subset of the plurality of summarization evaluation metrics, and wherein the plurality of weights comprise one of a same weight or a different configurable weight applied to each of the four categories when performing the weighting the plurality of summarization evaluation metrics. (e.g., the human can define the categories, assigning weights to each category and perform the evaluation which is a math calculation that can be done using pen and paper.)
Claim 7, further recites: wherein the four categories comprise a content quality metrics category, a coherence and structure metrics category, a semantic similarity metrics category, and an entity preservation metrics category. (e.g., the human can evaluate the various metrics listed, including content quality metric (how well is the summary written), coherence and structure metric (does the summary make sense), semantic similarity metric (is the meaning still the same) and entity preservation metric (keeping entity in the summary).)
Claim 8, further recites: wherein calculating the plurality of summarization evaluation metrics include separately calculating each of the four categories using the subset of the plurality of summarization evaluation metrics for a corresponding one of the four categories. (e.g., the human separately evaluating how well the AI generated summary performed in each specific metric category, like pointing out or rating higher relevance score while rating a lower coherence score.)
Claim 9, further recites: wherein, prior to the calculating the plurality of summarization evaluation metrics, the summary evaluation operations further comprise: preprocessing the AI generated text summary and the original text; (e.g., the human doing some preprocessing task, like removing punctuations and/or stopwords like “the” on the AI generated summary and original text.)
and tokenizing preprocessed text in the AI generated text summary and the original text, wherein the calculating the plurality of summarization evaluation metrics using the evaluation framework is with the tokenized preprocessed text from the AI generated text summary and the original text. (e.g., the human preprocessing the AI generated summary and original text into a list of words for instance. This could be done with assistance of paper and pen.)
Claim 11 recites limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Claim 14 recites limitations similar to the limitations of Claim 5 and is rejected under similar rationale.
Claim 15 recites limitations similar to the limitations of Claim 6 and is rejected under similar rationale.
Claim 16 recites limitations similar to the limitations of Claim 7 and is rejected under similar rationale.
Claim 17 recites limitations similar to the limitations of Claim 8 and is rejected under similar rationale.
Claim 18 recites limitations similar to the limitations of Claim 9 and is rejected under similar rationale.
Claim 20 recites limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
In sum, claims 2, 5-9, 11, 14-18 and 20 depend from claim 1, 10, and 19, and further recite mental processes as explained above. None of the additional limitations recited in claims 2, 5-9, 11, 14-18 and 20 amount to anything more than the same or a similar abstract idea as recited in claims 1, 10 and 19 respectively. Nor do any limitations in claims 2, 5-9, 11, 14-18 and 20: (a) integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea or (b) amount to significantly more than the judicial exception because the additional limitations of using generic computer components amounts to no more than mere instructions to apply the exception using generic computer components. Claims 2, 5-9, 11, 14-18 and 20 are not patent eligible.
Claims 3-4 and 12-13 are directed toward specific way of training machine learning model to analyze and compare a summary to the original text. These claims are not considered abstract ideas or even if they contain abstract idea, they contain sufficient technical detail that could be considered a practical applications, therefore they are deem patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 5-11, and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mukherjee (US 20230122609), in view of Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021). Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9, 391-409, further in view of Krishna, K., Bransom, E., Kuehl, B., Iyyer, M., Dasigi, P., Cohan, A., & Lo, K. (2023). LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. arXiv preprint arXiv:2301.13298, and furthermore in view of Bowers (US 20200143301).
Regarding Claim 1, Mukherjee discloses: 1. A machine learning (ML) system ([0008] a system) configured to evaluate artificial intelligence (AI) generated text summaries using an evaluation framework for a weighting strategy of a plurality of metrics, the ML system comprising: ([0037] the evaluation metrics for the candidate summarizers are determined using a single document or over a set of documents. For example, when computing evaluation metrics using a set of documents, the average, running average, or another technique such as a statistical or sampling technique can be used to determine an overall evaluation metric for each summarizer from multiple document evaluation metrics. In some embodiments, a running average associates different weights for each evaluation metric. For example, the more recently determined evaluation metrics can be weighted more heavily than evaluation metrics determined in the past.)
a processor and a computer readable medium operably coupled thereto, the computer readable medium comprising a plurality of instructions stored in association therewith that are accessible to, and executable by, the processor, to perform summary evaluation operations which comprise: ([0008] The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor.)
accessing, for a precision evaluation of an AI generated text summary using the evaluation framework, ([0037] In some embodiments, the evaluation metrics for the candidate summarizers are determined using a single document or over a set of documents.)[evaluation metric reads on evaluation framework]
a data structure including the AI generated text summary and an original text summarized in the AI generated text summary by an AI summarization engine; ([0035] the generated candidate summaries can be stored along with the original document’s paragraph and corresponding base summary of the paragraph. [0037] In some embodiments, the evaluation metrics for the candidate summarizers are determined using a single document or over a set of documents.)
calculating a plurality of summarization evaluation metrics that evaluate a summarization of the original text in the AI generated text summary using the evaluation framework with the AI generated text summary and the original text, ([0037] In some embodiments, the evaluation metrics for the candidate summarizers are determined using a single document or over a set of documents. For example, a Rouge-L f-measure metric can be determined for each candidate summarizer to evaluate how close the summarizer’s generated summary is to the base summary.)
weighting the plurality of summarization evaluation metrics based on a plurality of weights applied by the evaluation framework; ([0037] a running average associates different weights for each evaluation metric. For example, the more recently determined evaluation metrics can be weighted more heavily than evaluation metrics determined in the past.)
and updating the data structure with the computed final evaluation score for the precision evaluation. ([0040] the process can compute a running evaluation metric of the different summarizers that is updated as additional documents are summarized. … the data for evaluating summaries and/or summarizers including evaluation metrics is stored in a database such as database 123 of FIG. 1.)
Murkherjee does not explicitly disclose the following features.
Fabbri discloses: wherein the plurality of summarization evaluation metrics are selected for the evaluation framework based on a relevance assessment and a significance assessment of each of the plurality of summarization evaluation metrics when performing the precision evaluation of the summarization; ([sect 3.1 Evaluation metrics] Our selection of evaluation methods includes several recently introduced metrics that have been applied to both text generation and summarization, standard machine translation metrics, and other miscellaneous performance statistics. ROUGE, BLEU, BERTscore, SummaQA reads on relevance assessment. METEOR, ROUGE, BERTscore and BLEU reads on significance assessment.)
Mukherjee and Fabbri are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Mukherjee to combine the teaching of Fabbri for the above-mentioned feature, because SummEval is a rich resource framework for evaluating AI generated summaries (Fabbri, [Conclusion]).
Mukherjee and Fabbri does not disclose the following features.
Krishna discloses: computing a final evaluation score of the summarization of the original text in the AI generated text summary by the AI summarization engine based on an aggregation of the weighted plurality of summarization evaluation metrics; ([sect 3.1, pg. 6 right col] In our setup we assume all FINE units are equally weighted while aggregating them to the final summary score.)
Mukherjee/Fabbri/Krishna are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Mukherjee and Fabbri to combine the teaching of Krishna for the above-mentioned feature, because LongEval can provide a guideline for standardized evaluation for summarization (Krishna, [Conclusion]).
Mukherjee/Fabbri/Krishna does not disclose the following features.
Bowers discloses: outputting the precision evaluation of the AI generated text summary based on the computed final evaluation score; ([0262] a summary and/or overall assessment scores may be displayed, e.g., in residual risk header panel 5810.)
Mukherjee also discloses: and updating the data structure with the computed final evaluation score for the precision evaluation. ([0197] (Upon receipt of comments from a given co-worker and/or expert, the system 100 may label the request as being complete. The system 100 may also update the Exam Prep workspace 1100 with the received solicited comments. To this end, the system 100 may provide an organized and efficient framework to request for comments from internal and external collaborators, to track such requests, and to review and utilize such comments in the examination-preparation process.) [updating a data structure within a system after receiving feedback.]
Mukherjee/Fabbri/Krishna/Bowers are considered analogous art. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of teachings to combine with the teaching of Bowers for the above-mentioned feature, because improve user experience can be provided by displaying a summary and overall assessment score (Bowers, [0262]).
Regarding Claim 2, Mukherjee/Fabbri/Krishna/Bowers disclose all the limitation of Claim 1.
Fabbri further discloses: wherein one of the plurality of summarization evaluation metrics comprises a global coherence metric configured to assess a topic similarity between the original text and the AI generated text summary. ([sect 4.3, pg. 6] Coherence - We align this dimension with the DUC quality question (Dang, 2005) of structure and coherence whereby ‘‘the summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic.’’)
Where the rationale for the combination would be similar to the one provided above.
Regarding Claim 5, Mukherjee/Fabbri/Krishna/Bowers disclose all the limitation of Claim 2.
Mukherjee further discloses: wherein the plurality of summarization evaluation metrics further comprise at least one of a recall-oriented understudy for Gisting evaluation ([0011] a Rouge-L f-measure metric is computed for each of the abstractive summaries,)[the claim only requires one of the features recited]
Fabbri further discloses: ROUGE-2 score, ROUGE-L score, BLEU score, METEOR score, and BERTscore (see table 2. For ROUGE-2 and ROUGE-L, BLEU(also, sect 3.1, and table 4b), METEOR (table 2, 4b and sect 3.1 and sect 6.), BERTscore (sect 3.1 and table 2 and 4a)
The rationale for the combination would be similar to the one provided.
Regarding Claim 6, Mukherjee/Fabbri/Krishna/Bowers disclose all the limitation of Claim 1.
Fabbri further discloses: wherein the plurality of summarization evaluation metrics are split into four categories each having a subset of the plurality of summarization evaluation metrics, (see table 2, copy and pasted for reference below)
PNG
media_image1.png
725
480
media_image1.png
Greyscale
and wherein the plurality of weights comprise one of a same weight or a different configurable weight applied to each of the four categories when performing the weighting the plurality of summarization evaluation metrics. ([sect 3.1, pg. 6 right col] In our setup we assume all FINE units are equally weighted while aggregating them to the final summary score. Sect 5, Our experiments in Section 3.1 assigned an equal weight to each FINE unit while calculating the overall score of the summary. However, the faithfulness of some FINE units may be more important than others. A non-uniform weighing of FINE units may be a good strategy if there is a notion of how critical a particular unit is for a summary’s correctness) [Although this is with regard to faithfulness, the same concept of choosing same weight or different weights can be applied to other categories as well]
The rationale for the combination would be similar to the one already provided.
Regarding Claim 7, Mukherjee/Fabbri/Krishna/Bowers disclose all the limitation of Claim 6.
Fabbri further discloses: wherein the four categories comprise a content quality metrics category ([sect 3.1] ROUGE), a coherence and structure metrics category ([sect 4.3, pg. 6] Coherence - We align this dimension with the DUC quality question (Dang, 2005) of structure and coherence whereby ‘‘the summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic.’’), a semantic similarity metrics category ([sect 3.1] BERTscore and MoverScore), and an entity preservation metrics category ([sect 4.3] Consistency – the factual alignment between the summary and the summarized source. A factually consistent summary contains only statements that are entailed by the source document. Annotators were also asked to penalized summaries that contained hallucinated facts) [this reads on entity preservation because this ensures that key people, organizations, or locations from the source text are retained and accurately represented].
The rationale for the combination would be similar to the one already provided.
Regarding Claim 8, Mukherjee/Fabbri/Krishna/Bowers disclose all the limitation of Claim 7.
Fabbri further discloses: wherein calculating the plurality of summarization evaluation metrics include separately calculating each of the four categories using the subset of the plurality of summarization evaluation metrics for a corresponding one of the four categories. (see table 2 which shows how the four categories using subset of the evaluation metrics are calculated separately)
Regarding Claim 9, Mukherjee/Fabbri/Krishna/Bowers disclose all the limitation of Claim 1.
Mukherjee further discloses: prior to the calculating the plurality of summarization evaluation metrics, the summary evaluation operations further comprise: preprocessing the AI generated text summary and the original text; ([0022] In some embodiments, summary service and evaluation engine 203 is further utilized to preprocess the content before summarization. For example, the provided content can be stripped of non-text data such as images and formatting. In some embodiments, the content is filtered by removing non-alphanumeric characters other than punctuation. In various embodiments, the pre-processing is performed to provide the text summarizers with content that is text-based.)
Fabbri further discloses: and tokenizing preprocessed text in the AI generated text summary and the original text, wherein the calculating the plurality of summarization evaluation metrics using the evaluation framework is with the tokenized preprocessed text from the AI generated text summary and the original text. ([sect 3.1] BertScore (Zhang et al., 2020) computes similarity scores by aligning generated and reference summaries on a token-level. Token alignments are computed greedily to maximize the cosine similarity between contextualized token embeddings from BERT.)
The rationale for the combination would be similar to the one already provided.
Claim 10-11, and 14-18 recites limitations similar to the limitations of Claim 1-2, and 5-9 respectively, and are rejected under similar rationale.
Regarding Claim 19, Mukherjee discloses: 19. A non-transitory computer-readable medium having stored thereon computer-readable instructions executable to evaluate artificial intelligence (AI) generated text summaries using an evaluation framework for a weighting strategy of a plurality of metrics by a machine learning (ML) system, the computer-readable instructions executable to perform summary evaluation operations which comprise: ([0008] The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor,)
As for the rest of the claim, they recite the system of claim 1, and therefore the rationale applied in the rejection of claim 1 is equally applicable.
Claim 20 recites limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Allowable Subject Matter
Claims 3-4, and 12-13 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 3, the claim discloses a specific technique for measuring topic similarity between original document/text and the AI generated summary using LDA topic modeling.
The closest prior arts are: Ramsl US 20230096118 – which discloses generating a quality score for topics of datasets using topic model such as LDA. However, it does not disclose training of the LDA. Kumar, J., & Vashistha, R. (2023, July). Automatic Assessment System Using Topic Modelling and TF-IDF Approach. In 2023 IEEE World Conference on Applied Intelligence and Computing (AIC) (pp. 81-85). IEEE. – which discuss topic modeling using LDA, but it does not discuss summarization or comparing topics of original text to a machine or AI generated summary. Further, none of them do so in the manner as specifically claimed.
Claim 12, although in different statutory category, nevertheless they recited similar language as claim 3, therefore is subject to the condition mentioned from above. Claims 4 and 13 depends and further limits claims 3 and 12, therefore they are also objected to but otherwise allowable for the same rationale as the claim on which they depended from.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Kryscinski (US 20210124876) – discloses evaluating a summary to verify for factual consistencies from the source text. See Abstract for details.
Madhusudhan (US 20240078380) – discloses a evaluation framework for evaluation of AI summarization models. See para 0137, 0139, 0162 and 0180 for additional details.
Rosamma, K. S., & Patil, N. (2023, December). Measuring the Quality of Text Summarization: A Survey of Evaluation Approaches. In 2023 OITS International Conference on Information Technology (OCIT) (pp. 290-296). IEEE. – discloses various benchmark for summarization evaluation. See Abstract and sects 2-3 and table 1 for additional details.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Philip H Lam whose telephone number is (571)272-1721. The examiner can normally be reached 9 AM-3 PM Pacific time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bhavesh Mehta can be reached on 571-272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHILIP H LAM/ Examiner, Art Unit 2656