DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement filed 06/15/2023 fails to comply with the provisions of 37 CFR 1.97, 1.98 and MPEP § 609 because the following US Patent Numbers don’t match the inventor as provided in the table; therefore, the following publications cited in the Information Disclosure Statement will not be considered:
(4) 108966702 2021-01-19 EISENZOPF
(5) 113035872 2022-04-12 HUANG
(6) 115396502 2022-12-17 WAYNE
(7) 110304122 2019-10-10 SHANMUGAN
Applicant is advised that the date of any re-submission of any item of information contained in this information disclosure statement or the submission of any missing element(s) will be the date of submission for purposes of determining compliance with the requirements based on the time of filing the statement, including all certification requirements for statements under 37 CFR 1.97(e). See MPEP § 609.05(a).
Specification objections
The disclosure is objected to because of the following informalities:
Specification at page 7, paragraph [0029] recites “answer references 126”, and at paragraph [0031] recites “answers references 128”. They seem both to designate the “answers references 128” at Fig. 1; therefore, paragraph [0029] should recite “answers references 128” to be consistent with Fig. 1 and [0031]. Appropriate correction is required.
Claim Objections
Claims 1, 9 and 15 are objected to because of the following informalities: these claims recite grammatical error “a user a user” in the limitation “generating an answer report indicating whether the answer is valid or invalid to transmit to a user a user that submitted the question to render”. Appropriate correction is required.
Claims 8 is also objected to because of the following informalities: the limitation “determining a domain database applicable to the domain of the question and the answer, wherein the domain database is searched to the obtain reference” has a grammatical error. It should read: “determining a domain database applicable to the domain of the question and the answer, wherein the domain database is searched to obtain the reference”. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception without significantly more.
Regarding Claim 1:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
Calculating a similarity score between the answer and the reference; — This limitation is directed to the abstract idea of a mathematical concepts, as calculating a score is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.).
Determining whether the similarity score exceeds a threshold value; — This limitation is directed to the abstract idea of a mental process as determining if a score exceeds a threshold is analogous to an observation step to compare it with a mathematical value which is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Indicating, in answer information, that the answer is valid in response to the similarity score exceeding the threshold value or that the answer is invalid in response to the similarity score not exceeding the threshold value; and— This limitation is directed to the abstract idea of a mental process as indicating validity of an answer according to a value is analogous to an observation, evaluation and judgment steps, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
A computer program product for verifying answers produced from an artificial intelligence chatbot in response to questions inputted to the artificial intelligence chatbot, wherein the computer program product comprises a computer readable storage medium having computer readable program instructions that when executed perform operations. – This limitation invokes a computer merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to integrate the exception into a practical application.
Searching a database using keywords from the answer to obtain a reference for the answer; — This limitation is reasonable considered as an insignificant extra solution activity, as searching a database to obtain a reference, under BRI, is mere data gathering per MPEP 2106.05(g)(3).
The operations comprising: receiving an answer to a question submitted to the artificial intelligence chatbot; – This limitation is directed to mere data gathering, which is an insignificant extra-solution activity [see MPEP 2106.05(g)(3)] and therefore fails to integrate the judicial exception into a practical application.
Generating an answer report indicating whether the answer is valid or invalid to transmit to a user a user that submitted the question to render” – This limitation is directed to mere data outputting and/or transmitting the result, which is an insignificant extra-solution activity [see MPEP 2106.05(g)(3)] and therefore fails to integrate the judicial exception into a practical application.
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
A computer program product for verifying answers produced from an artificial intelligence chatbot in response to questions inputted to the artificial intelligence chatbot, wherein the computer program product comprises a computer readable storage medium having computer readable program instructions that when executed perform operations– This limitation invokes a computer merely as a tool for performing an existing process [see MPEP 2106.05(f)(2)] and therefore fails to amount to significantly more than the judicial exception.
Searching a database using keywords from the answer to obtain a reference for the answer; — This limitation is proper to state it as well-understood, routine and conventional (WURC), since searching a database under BRI is further considered as retrieving information in memory, being one of the examples that the courts have recognized as computer functions considered WURC. This is stated in MPEP 2106.05 (d) II iv: Storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc.
The operations comprising: receiving an answer to a question submitted to the artificial intelligence chatbot. – This limitation is directed to receiving or transmitting data over a network, which the courts have recognized as well-understood, routine, conventional activity when they are claimed at a high level of generality or as insignificant extra-solution activity [see MPEP 2106.05(d) II. i] and therefore fails to amount to significantly more than the judicial exception.
Generating an answer report indicating whether the answer is valid or invalid to transmit to a user a user that submitted the question to render – This limitation is directed to receiving or transmitting data over a network, which the courts have recognized as well-understood, routine, conventional activity when they are claimed at a high level of generality or as insignificant extra-solution activity [see MPEP 2106.05(d) II. i] and therefore fails to amount to significantly more than the judicial exception.
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 2:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
Calculating the similarity score between the answer and the reference comprises: This limitation is directed to the abstract idea of a mathematical concepts, as calculating a score is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.).
Determining word type similarity scores between words in the answer, of at least one specified word type, to words in the reference; — This limitation is directed to the abstract idea of a mental process as determining the similarity score between the words in the answer and words in the reference with a mathematical value which is analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Aggregating the word type similarity scores to produce the similarity score between the answer and the reference. — This limitation is directed to the abstract idea of mathematical concepts, as calculating the scores to create another score is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.).
Regarding Claim 3:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
Determining a domain of the question. — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Determining specified word types for the domain, wherein the at least one specified word type comprises the determined specified word types for the domain. — This limitation is directed to the abstract idea of a mental process (including an observation evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Regarding Claim 4:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
Wherein the specified word type is part of plurality of specified word types, wherein there is a weight for each of the specified word types, wherein the calculating the word type similarity scores comprises. — this limitation is directed to the abstract idea of a mathematical concepts, introducing the word type, the weight and the similarity score is analogous to a mathematical relationship (see MPEP 2106.04(a)(2) I. A.). Also calculating the similarity score is analogous to a mathematical calculating (see MPEP 2106.04(a)(2) I. C.).
For each word type of the word types, calculating a weighted average similarity score as an average of the similarity scores for the word type multiplied by a weight for the word type; — this limitation is directed to the abstract idea of a mathematical concepts, as calculating a score is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.).
Summing weighted average similarity scores for the word types to produce the similarity score between the answer and the reference.” — this limitation is directed to the abstract idea of a mathematical concepts, as summing weighted average similarity scores is analogous to a mathematical calculation (see MPEP 2106.04(a)(2) I. C.).
Regarding Claim 5:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
Determining the similarity score, and determining whether the similarity score exceeds the threshold value are performed for the answers, wherein the answer report renders information indicating whether the answers are valid or invalid. —This limitation is directed to the abstract idea of mathematical concepts as determining the score is analogous to the mathematical calculation (see MPEP 2106.04(a)(2) I. C.). Further to mental processes such as determining if the score exceeds a value and determining validity, which are analogous to evaluation and judgment, which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III.
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
Receiving an answer to a question submitted to the artificial intelligence chatbot, – This limitation is directed to mere data gathering, which is an insignificant extra-solution activity [see MPEP 2106.05(g)(3)] and therefore fails to integrate the judicial exception into a practical application.
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
Receiving an answer to a question submitted to the artificial intelligence chatbot, – This limitation is directed to receiving or transmitting data over a network, which the courts have recognized as well-understood, routine, conventional activity when they are claimed at a high level of generality or as insignificant extra-solution activity [see MPEP 2106.05(d) II. i] and therefore fails to amount to significantly more than the judicial exception.
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 6:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon? No.
Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application? – No, there are no additional elements that integrate the judicial exception into a practical application.
Updating the answer information to indicate the reference and information to locate the reference; – This limitation recites the ‘updating’ of data, which under broadest reasonable interpretation, is considered as selecting a particular data source to be manipulated, being insignificant extra solution activity under 2106.05 (g) (3).
Providing the answer information to use to train the artificial intelligence chatbot to produce an invalid answer with a low confidence level. – This limitation recites the training of a chatbot to produce an answer, however, it is recited at a high level of generality further used output an answer, therefore it amounts to mere instructions to implement an abstract idea (producing an answer is considered a judgment, being mental processes) a computer [see MPEP 2106.05(f)(2)] and therefore fails to integrate the exception into a practical application.
Step 2B – Does the claim recite any additional elements that amount to significantly more than the judicial exception? – No, there are no additional elements that amount to significantly more than the judicial exception.
Updating the answer information to indicate the reference and information to locate the reference; – This limitation recites the ‘updating’ of data, which under broadest reasonable interpretation, is considered as electronic recordkeeping, being well understood, routine and conventional (see MPEP 2106.05 (d) II iii).
Providing the answer information to use to train the artificial intelligence chatbot to produce an invalid answer with a low confidence level. – This limitation recites the training of a chatbot to produce an answer, however, it is recited at a high level of generality further used output an answer, therefore it amounts to mere instructions to implement an abstract idea (producing an answer is considered a judgment, being mental processes) a computer [see MPEP 2106.05(f)(2)] and therefore fails to amount to significantly more than the judicial exception.
Step 2A Prong Two and Step 2B:
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d) I.), failing step 2A prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B. The claim is ineligible.
Regarding Claim 7:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
Wherein the answer report juxtaposes the answer with respect to content of the reference and visual indication of whether the answer is valid or invalid. — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.)
Regarding Claim 8:
Step 1 – Is the claim to a process, machine, manufacture, or composition of matter?
Yes
Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites the abstract ideas of:
Determining a domain of the question and the answer; — This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Determining a domain database applicable to the domain of the question and the answer, wherein the domain databased is searched to the obtain reference— This limitation is directed to the abstract idea of a mental process (including an observation, evaluation, judgement, opinion) which can be performed by the human mind, or by a human using pen and paper (see MPEP 2106.04(a)(2) III. C.).
Regarding claims 9-11:
Claims 9-11 recite analogous limitations to claims 1-3 (respectively) and therefore they are rejected on the same grounds as claims 1-3.
Regarding claims 12-13:
Claims 12-13 recite analogous limitations to claims 5-6 (respectively) and therefore they are rejected on the same grounds as claims 5-6.
Regarding claims 14 and 20
Claims 14 and 20 recite analogous limitations to claims 8 (respectively) and therefore they are rejected on the same grounds as claims 8.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION. —The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 4 is rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor regards as the invention. Claim 4 recites that “there is a weight for each of the specified word types” and further recites “calculating a weighted average similarity score… multiplied by a weight for the word type.” However, the claim does not clearly define the relationship between the recited “weight” and the specified word types. It is unclear whether each specified word type is associated with a unique weight, whether the same weight is applied to multiple word types, or whether the weight is dynamically determined during calculation. Additionally, the claim does not provide objective boundaries for the value, scope, or determination of the recited weight. As a result, one of ordinary skill in the art would not be reasonably apprised of the scope of claim 4, rendering the claim indefinite.
Claim 5 is rejected under 35 U.S.C 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor regards as the invention. Claim 1 recites “receiving an answer to a question,” which reasonably indicates receiving a single answer corresponding to the question. However, dependent claim 5 recites that “receiving the answer comprises receiving a plurality of answers to the question.” It is unclear how a single “answer” recited in claim 1 can comprise a plurality of answers as recited in claim 5. The claim language does not clarify whether the plurality of answers replaces the single answer, represents multiple candidate answers, or otherwise modifies the meaning of “answer” in claim 1. As a result, the scope of claim 5 is unclear, rendering the claim indefinite.
Claim 12 and claim 18 are rejected under the same reason as claim 5.
Claim 6 is rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor regards as the invention. The limitation “…an invalid answer with a low confidence level” is considered unclear as the term “low” is subjective and indefinite. The specification fails to establish the metes and bounds of this term as there is no established threshold or limit to state that a confidence level below a threshold is considered “low”. The claim and the specification do not have a definite measurement to determine how “low” a confidence level is. Clarification is required.
Claim 13 and claim 19 are rejected under the same reason as claim 6.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 5, 7, 8, 9, 12, 14, 15, 18, and 20 is/are rejected under 35 U.S.C. 102(a) (1) as being anticipated by Patel et al (US Pub. No. 2022/0004715- hereinafter Patel).
Referring to Claim 1, Patel teaches a computer program product for verifying answers produced from an artificial intelligence chatbot in response to questions inputted to the artificial intelligence chatbot, wherein the computer program product comprises a computer readable storage medium having computer readable program instructions that when executed perform operations (see Patel at Paragraphs [0044]: “Knowledge gap program 300 operates to identify knowledge gaps the processed data of a chatbot by comparing question-answer pairs generated by the chatbot with content of useful document portions of a document. The knowledge of the chatbot may be based on the document”. Examiner interprets the knowledge of the chatbot by comparing the question-answer pairs to the claimed verification of answers produced from an AI chatbot. Further, see [0036]: “Program 300 is stored in persistent storage 210 for access and/or execution by one or more of the respective computer processors 204, usually through one or more memories of memory 208”. Examiner interprets this persistent storage as the claimed computer readable storage medium),
Receiving an answer to a question submitted to the artificial intelligence chatbot (see Patel at paragraph [0044]: “Knowledge gap program 300 operates to identify knowledge gaps the processed data of a chatbot by comparing question-answer pairs generated by the chatbot with content of useful document portions of a document. The knowledge of the chatbot may be based on the document”. Examiner interprets the answer generated by the chatbot from the question-answer pair as the claimed answer to a question submitted to an AI chatbot);
Searching a database using keywords from the answer to obtain a reference for the answer. (See Patel at paragraph [0057]: “Processing proceeds to step S290, where comparison mod 390 compares entities of text portions having a first label with entities of QA pairs formed by the chatbot. In this example, groups of paragraphs labeled as “useful” serve as the basis from which corresponding entities are collected. The entities associated with groups labeled as useful are compared with entities included in the QA pairs.”) Examiner interprets using the entities of QA pairs to compare with the entities in the text portions as equivalent to using the keywords of the answer to search the database. Further, (See Patel at paragraph [0058]: “Processing proceeds to step S295, where entity module (“mod”) 395 identifies a set of entities not present in the QA pairs. In this example, for any entities of labeled paragraphs that do not match an entity of the QA pairs in QA pairs store 113 (FIG. 1) are recorded to labeled corpus store 302 for later use.”) Examiner interprets recording the unmatched entities for later use as equivalent obtaining the reference. Thus, Patel teaches using the keywords and searching the database to obtain the answers).
Calculating a similarity score between the answer and the reference (see Patel at [0092]: “Comparison of the processed document text with the QA pairs output by the chatbot is performed in some embodiments by comparing all text portions that are among the “useful” labeled clusters to all answers from the chatbot-produced QA pairs using fuzzy similarity metrics weighted by the inverse of the relevance score of the text portions. The comparison results in comparison metric scores for each text portion.”) Examiner interprets using fuzzy similarity metrics to compare the document text with the QA pair as equivalent to calculating the similarity score between the answer and the reference.)
Determining whether the similarity score exceeds a threshold value (see Patel at [0092]: “The comparison results in comparison metric scores for each text portion.”) Examiner interprets using fuzzy similarity metrics to compare the document text with the QA pair as equivalent to calculating the similarity score between the answer and the reference. The text portions having a comparison metric score greater than a pre-defined score threshold are identified as containing knowledge gap data not covered by knowledge of the chatbot.” Examiner interprets this comparison between fuzzy similarity metric score and threshold as equivalent as the claimed similarity score compared/exceeding a threshold.)
Indicating, in answer information, that the answer is valid in response to the similarity score exceeding the threshold value or that the answer is invalid in response to the similarity score not exceeding the threshold value (see Patel at [0092]: “The text portions having a comparison metric score greater than a pre-defined score threshold are identified as containing knowledge gap data not covered by knowledge of the chatbot. Examiner interprets this comparison whether fuzzy metric score greater than the threshold to identify the knowledge gap data covered or not covered by chatbot as equivalent as indicating the valid or invalid answer as claimed.)
“Generating an answer report indicating whether the answer is valid or invalid to transmit to a user a user that submitted the question to render.” (See Patel at [0097]: “Processing ends at step S620, where program 300 presents to a user the knowledge gaps identified in “useful” clusters with a corresponding relevance score.” Examiner interprets this presenting the knowledge gaps to the user as equivalent as generating an answer report.) Also, (See Patel at [0062]:” Further, some embodiments of the present invention allow an expert to provide inputs on what information is important to a particular use case so that experts may find text in a document for which no QA pairs are generated by the chatbot, thus identifying QA system failures.” Examiner interprets the expert providing inputs to identifying QA system failures as the user submitted the question. Thus, Patel teaches generating the report with valid or invalid to the user as claimed).
Referring to Claim 5, Patel teaches: The computer program product of claim 1, wherein the receiving the answer comprises receiving a plurality of answers to the question, (See Patel at [0092]: “Comparison of the processed document text with the QA pairs output by the chatbot is performed in some embodiments by comparing all text portions that are among the “useful” labeled clusters to all answers from the chatbot-produced QA pairs using fuzzy similarity metrics weighted by the inverse of the relevance score of the text portions. The comparison results in comparison metric scores for each text portion.” Examiner interprets comparison of the document text with the QA pairs output by the chatbot to equivalent as the computer program receiving answers to a question.)
Wherein the operations of searching the database; (See Patel at [0101]: (iii) receiving by the computing device documents associated with a similar topic as the chatbot; (iv) extracting by the computing device document entities from the documents associated with the similar topic;” Examiner interprets receiving documents and extracting the entities from the document equivalent the operation of searching the database.)
Determining the similarity score; (See Patel at [0092]: “Comparison of the processed document text with the QA pairs output by the chatbot is performed in some embodiments by comparing all text portions that are among the “useful” labeled clusters to all answers from the chatbot-produced QA pairs using fuzzy similarity metrics weighted by the inverse of the relevance score of the text portions. The comparison results in comparison metric scores for each text portion.” Examiner interprets comparison of the processed document text with the QA pairs by the chatbot, using fuzzy similarity metrics, and the comparison results in comparison metric scores for each text portion 100equivalent to claimed determining the similarity score.)
Determining whether the similarity score exceeds the threshold value are performed for the answers, wherein the answer report renders information indicating whether the answers are valid or invalid. (See Patel at [0092]: “The text portions having a comparison metric score greater than a pre-defined score threshold are identified as containing knowledge gap data not covered by knowledge of the chatbot.” Examiner interprets comparison metric score with a pre-defined score threshold equivalents as claimed determining whether the similarity score exceeds the threshold value. (See Patel at [0064]: “Once the gaps are identified as paragraphs within the document corpus, some embodiments of the present invention present the gaps to the SME via a graphical use interface (GUI). The chatbot approval system may further provide a relevance ranking to the SME for both he generated QA pairs and the knowledge gaps. The relevance ranking of the knowledge gaps is indicative of the quality of the automatically generated QA pairs. The relevance ranking or relevance score indicates how useful the QA pair is for a given instance of a chatbot targeting a particular domain of knowledge.” Examiner interprets this ranking provided to the user via a GUI to be equivalent as the answer report renders information indicating whether the answers are valid or not.)
Referring to Claim 7, Patel teaches the computer program product of claim 1, wherein the answer report juxtaposes the answer with respect to content of the reference and visual indication of whether the answer is valid or invalid. (See Patel, Paragraphs [0044]: “Knowledge gap program 300 operates to identify knowledge gaps the processed data of a chatbot by comparing question-answer pairs generated by the chatbot with content of useful document portions of a document.” Examiner interprets comparing question-answer pair with the content of the document as equivalent as claimed juxtaposing the answer to content of the reference.) (See Patel at [0064]: “Once the gaps are identified as paragraphs within the document corpus, some embodiments of the present invention present the gaps to the SME via a graphical use interface (GUI).” Examiner interprets presenting the gaps to SME (subject matter experts) via a GUI as equivalent as generating report to the user in a visual way.) Thus, Patel teaches claimed the answer report juxtaposes the answer with respect to content of the reference and visual indication of whether the answer is valid or invalid).
Referring to Claim 8, Patel teaches the computer program product of claim 1, wherein the searching the database comprises (See Patel at [0044]: “Knowledge gap program 300 operates to identify knowledge gaps the processed data of a chatbot by comparing question-answer pairs generated by the chatbot with content of useful document portions of a document.” Examiner interprets the useful knowledge document portions as equivalent to the claimed searching the database).
Determining a domain of the question and the answer;(See Patel at [0022]: “Knowledge gaps in a chatbot are identified with reference to a domain-specific document and a set of QA pairs of the chatbot. Entities and/or entity values associated with the document are compared to the entities and/or entity values of the QA pairs. Entities of the document not associated with the QA pairs are identified as knowledge gaps. The QA pairs and knowledge gaps are ranked by relevance to the domain.” Examiner interprets that the QA pairs and ranked by relevance to the domain as equivalent to determining a domain of the QA.)
Determining a domain database applicable to the domain of the question and the answer, wherein the domain database is searched to the obtain reference. (See Patel at [0044]: “The knowledge of the chatbot may be based on the document. The document may be a domain-specific document covering the same domain in which the chatbot is trained.” Examiner interprets the domain-specific document as equivalent to the claimed determining a domain of the database).
Referring to independent Claims 9 and 15, these claims are rejected on the same basis as independent claim 1, mutatis mutandis, since they are analogous claims.
Referring to dependents Claim 12 and 18, these claims are rejected on the same basis as dependent claim 5, mutatis mutandis, since they are analogous claims.
Referring to dependents Claim 14 and 20, these claims are rejected on the same basis as dependent claim 8, mutatis mutandis, since they are analogous claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 2, 4,10,16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Patel in view of Miao (EP 3690676 A1- hereinafter Miao).
Referring to Claim 2, Patel teaches the computer program product of claim 1, however, it fails to teach wherein the calculating the similarity score between the answer and the reference comprises:
Determining word type similarity scores between words in the answer, of at least one specified word type, to words in the reference;
Aggregating the word type similarity scores to produce the similarity score between the answer and the reference.
Miao teaches, in an analogous system:
Determining word type similarity scores between words in the answer, of at least one specified word type, to words in the reference; (See Miao at [0038 - 0039]: “S240, calculating, in at least one target dimension, a score of a similarity between the to-be-verified answer and each piece of the target authoritative data respectively. Specifically, the target dimension may include: a keyword dimension, a sentence dimension, and a semantic dimension. The keyword dimension may refer to similarity comparison between the keywords extracted from the to-be-verified answer and keywords extracted from the respective target authoritative data; the sentence dimension may refer to a similarity comparison between a sentence included in the to-be-verified answer and sentences included in the respective target authoritative data; and the semantic dimension may refer to a similarity comparison between the semantics of the to-be-verified answer and the semantics of respective target authoritative data.” Examiner interprets those keyword dimension, sentence dimension and semantic dimension to be equivalent as the word types, and the authoritative data to be equivalent as the reference. Thus, Art Miao teaches claimed determining the word type similarity score between the answer and reference.)
Aggregating the word type similarity scores to produce the similarity score between the answer and the reference. (See Miao at [0054]: “It should be noted that, based on the scores of similarity in any one or any two dimensions described above, the mean value of the weighted sum of the scores of similarity may be calculated as the authority score of the to-be-verified answer, thereby performing authority verification on the to-be-verified answers.” Examiner interprets calculating a weighted sum of the similarity score of the mean to be equivalent as claimed aggregating the word type similarity scores to produce the similarity score.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Patel with the above teachings of Miao by calculating the similarity score between the answer and the reference, as taught by Patel, and determining word type similarity scores between words in the answer, of at least one specified word type, to words in the reference and aggregating the word type similarity scores to produce the similarity score between the answer and the reference, as taught by Miao. The modification would have been obvious because one of ordinary skill in the art would be motivated to verify the answers before delivering them to the user, thereby providing authoritative answer, as suggested by Miao at [0056]: “The degree of similarity between the to-be verified answer and the authoritative data may be evaluated at a plurality of dimensions, implementing accurately evaluating whether the to-be-verified answer is authoritative”.
Referring to Claim 4, Patel teaches the computer program product of claim 2, however, it fails to teach calculating the similarity score between the answer and the reference comprises: wherein the specified word type is part of plurality of specified word types, wherein there is a weight for each of the specified word types, wherein the calculating the word type similarity scores comprises:
For each word type of the word types, calculating a weighted average similarity score as an average of the similarity scores for the word type multiplied by a weight for the word type.
Summing weighted average similarity scores for the word types to produce the similarity score between the answer and the reference.
Miao teaches, in an analogous system:
Wherein the specified word type is part of plurality of specified word types (See Miao at [0038 - 0039]: “S240, calculating, in at least one target dimension, a score of a similarity between the to-be-verified answer and each piece of the target authoritative data respectively. Specifically, the target dimension may include: a keyword dimension, a sentence dimension, and a semantic dimension. The keyword dimension may refer to similarity comparison between the keywords extracted from the to-be-verified answer and keywords extracted from the respective target authoritative data; the sentence dimension may refer to a similarity comparison between a sentence included in the to-be-verified answer and sentences included in the respective target authoritative data; and the semantic dimension may refer to a similarity comparison between the semantics of the to-be-verified answer and the semantics of respective target authoritative data.” Examiner interprets these keyword dimension, a sentence dimension, and a semantic dimension to be equivalent as the plurality of specified word types.)
Wherein there is a weight for each of the specified word types, wherein the calculating the word type similarity scores comprises: (See Miao at [0052]: “α, β and γ are preset ratio factors that may be set by the user as needed, and α, β and γ are respectively used to indicate the weight value of the first similarity score, the weight value of the second similarity score and the weight value of the third similarity score. The authority threshold may be set by the user as needed. In addition, the size of the authority threshold may be controlled subsequently to effectively screen the authoritative to-be-verified answer.” Examiner interprets coefficient α, β, and γ to be equivalent as the weights for word types, and calculating the authority of the answer score be equivalent as claimed calculating the word type similarity score.)
For each word type of the word types, calculating a weighted average similarity score as an average of the similarity scores for the word type multiplied by a weight for the word type (See Miao at [0051 - 0052]: “calculating an authority score of the to-be-verified answer according to the formula:
PNG
media_image1.png
58
360
media_image1.png
Greyscale
Where N is the number of target authoritative data satisfying the correlation condition screened out from the authoritative data set; Word(Ans, Pi) is a score of a similarity between the to-be-verified answer Ans and the ith target authoritative data Pi in the keyword dimension, Word(Ans, Pi) is a score of a similarity between Ans and Pi in the sentence dimension, Sim(Ans, Pi) is a score of a similarity between Ans and Pi in the semantic dimension, and α, β and γ are preset ratio factors; and in response to the authority score being less than a set authority threshold, filtering and removing the to-be-verified answer from the community question answer data set. Word(Ans, Pi) is used to indicate the degree of the to-be-verified answer coinciding with the ith target authoritative data in the keyword dimension; Sent(Ans, Pi) is used to indicate the degree of the to-be-verified answer coinciding with the ith target authoritative data in the sentence dimension; Sim(Ans, Pi) is used to indicate the degree of the to-be-verified answer coinciding with the ith target authoritative data in the semantic dimension; and α, β and γ are preset ratio factors that may be set by the user as needed, and α, β and γ are respectively used to indicate the weight value of the first similarity score, the weight value of the second similarity score and the weight value of the third similarity score. The authority threshold may be set by the user as needed. In addition, the size of the authority threshold may be controlled subsequently to effectively screen the authoritative to-be-verified answer.” Examiner interprets the functions of Word (Ans, Pi), Sent (Ans, Pi) and Sim (Ans, Pi) to be equivalent to the similarity scores for each word type, the coefficient α, β, and γ to be equivalent as the weight for each word type, dividing the total sum by N represents the operation of calculating the average. Thus, Miao teaches claimed calculating the weighted average of the similarity score for the word type multiplied by a weight for the word type.)
Summing weighted average similarity scores for the word types to produce the similarity score between the answer and the reference (See Miao at [0051 - 0052]: “calculating an authority score of the to-be-verified answer according to the formula:
PNG
media_image1.png
58
360
media_image1.png
Greyscale
Where N is the number of target authoritative data satisfying the correlation condition screened out from the authoritative data set; Word(Ans, Pi) is a score of a similarity between the to-be-verified answer Ans and the ith target authoritative data Pi in the keyword dimension, Word(Ans, Pi) is a score of a similarity between Ans and Pi in the sentence dimension, Sim(Ans, Pi) is a score of a similarity between Ans and Pi in the semantic dimension, and α, β and γ are preset ratio factors; and in response to the authority score being less than a set authority threshold, filtering and removing the to-be-verified answer from the community question answer data set. Word(Ans, Pi) is used to indicate the degree of the to-be-verified answer coinciding with the ith target authoritative data in the keyword dimension; Sent(Ans, Pi) is used to indicate the degree of the to-be-verified answer coinciding with the ith target authoritative data in the sentence dimension; Sim(Ans, Pi) is used to indicate the degree of the to-be-verified answer coinciding with the ith target authoritative data in the semantic dimension; and α, β and γ are preset ratio factors that may be set by the user as needed, and α, β and γ are respectively used to indicate the weight value of the first similarity score, the weight value of the second similarity score and the weight value of the third similarity score. The authority threshold may be set by the user as needed. In addition, the size of the authority threshold may be controlled subsequently to effectively screen the authoritative to-be-verified answer.” Examiner interprets the functions of Word (Ans, Pi), Sent (Ans, Pi) and Sim (Ans, Pi) to be equivalent to the similarity scores for each word type, the coefficient α, β, and γ to be equivalent as the weight for each word type, dividing the total sum by N represents the operation of calculating the average, and the mathematical summation symbol in the formula represents the summing operation. In this claim, the calculation steps involve calculating the mean first and then the sum, whereas in the prior art, the cumulative sum is calculated before the mean is determined. According to the mathematical theorems, the results of these two calculation methods are identical. Thus, Miao teaches claimed summing weighted average similarity scores to produce the similarity score).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Patel with the above teachings of Miao by calculating a weighted average similarity score and summing weighted average similarity scores with the word types and the weights. The modification would have been obvious because one of ordinary skill in the art would be motivated to verify the answers before delivering them to the user, thereby providing authoritative answer, as suggested by Miao at [0056]: “The degree of similarity between the to-be verified answer and the authoritative data may be evaluated at a plurality of dimensions, implementing accurately evaluating whether the to-be-verified answer is authoritative”.
Referring to dependent Claims 10 and 16, these claims are rejected on the same basis as dependent claim 2, mutatis mutandis, since they are analogous claims.
Claim(s) 3, 11, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Patel in view of Miao and further in view of Zhang (TW I746690- hereinafter Zhang).
Referring to Claim 3, the combination of Patel and Miao teaches the computer program product of claim 2, however, it fails to teach
Determining a domain of the question, and
Determining specified word types for the domain, wherein the at least one specified word type comprises the determined specified word types for the domain.
Zhang teaches, in an analogous system:
Determining a domain of the question (See Zhang [claim 1]: “performing word segmentation on the question sentence to obtain the word segmentation set of the question sentence; based on the domain dictionary corresponding to the domain to which the question sentence belongs, performing keyword tagging on each word segmentation in the word segmentation set to obtain the first marking result; ” Examiner interprets corresponding performing keyword tagging based on the domain dictionary corresponding to the domain to the question as equivalent as determining a domain of the question); and
Determining specified word types for the domain, wherein the at least one specified word type comprises the determined specified word types for the domain. (See Zhang at [claim 1]: “the domain dictionary is used to store the correspondence between each domain word and its corresponding word type; according to this The first marking result and the second marking result determine the word marking result of each word segmentation in the word segmentation set, where the word marking result includes: whether each participle of the question sentence is a keyword and the word type of each keyword; ” Examiner interprets two marking operations resulting key words in the domain and their word types as equivalent as determining specified word types for the domain.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Patel and Miao with the above teachings of Zhang by determining a similarity score between words in the answer, as taught by Patel and Miao, and further determining a domain of the question, and determining specified word types for the domain, wherein the at least one specified word type comprises the determined specified word types for the domain, as taught by Zhang. The modification would have been obvious because one of ordinary skill in the art would be motivated to classify the intent of the question and answers in order to ultimately provide the most accurate answer, as suggested by Zhang (See Zhang at Pages [50 – 51]: “Therefore, the keyword extraction model and intent recognition model trained are more accurate in extracting keywords and classifying intent, thereby making the answers generated in this embodiment more accurate”.
Referring to dependent Claims 11 and 17, these claims are rejected on the same basis as dependent claim 3, mutatis mutandis, since they are analogous claims.
Claim(s) 6, 13, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Patel in view of Patel and further in view of Liu (NPL, Label distribution for learning with noisy labels, by Liu, hereinafter Liu).
Referring to Claim 6, Patel teaches the computer program product of claim 1, and “updating the answer information to indicate the reference and information to locate the reference;” (See at Patel [0047]:” Some embodiments of the present invention evaluation the knowledge gap is performed by identifying paragraphs within a document using clustering techniques to obtain expert annotations for a sample of paragraphs. In some examples, the expert annotations include tags of either “Useful” or “Not Useful” on which a useful QA pair is based. The expert annotations serve as seeds for identifying a group, or cluster, of paragraphs that are likely useful in responding to queries with a particular domain of knowledge. Knowledge gaps are identified by comparing the QA pairs ingested into the chatbot with the paragraphs obtained from the clustering activity.” Examiner interprets that the marking the expert annotations on QA pair as equivalent to updating the answer information, and using the annotations to identifying a group or a cluster of paragraphs, then identifying the knowledge gaps as equivalent to indicate the reference and information to locate the reference).
However, Patel fails to teach: providing the answer information to use to train the artificial intelligence chatbot to produce an invalid answer with a low confidence level.
Liu teaches, in an analogous system: providing the answer information to use to train the artificial intelligence chatbot to produce an invalid answer with a low confidence level. (See Liu at [Introduction]: “we make a comparison between the model trained with forward correction loss a classical label correction method, on samples containing noisy labels and the model trained with cross entropy loss on filtered samples with clean labels. Forward correction loss has good performances in asymmetric noise pattern but performs poorly under symmetric noise cases. In contrast, the model learned with only clean labels have stable performances on both noise patterns. It shows that the samples with clean labels are more important than the correcting operations under specific noise patterns. However, the uncertainty of labels makes it difficult to identify the samples with clean labels. To reduce the uncertainty of labels, a metric named label confidence is proposed in this paper for measuring the reliability of each label, in which clean labels get high confidence scores while noisy labels achieve low confidence scores.” Examiner interprets that noisy labels achieve low confidence scores as equivalent to using answer information to train chat bot to produce answer with low confidence level.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Patel with the above teachings of Liu by updating the answer information to indicate the reference and information to locate the reference, as taught by Patel, and providing the answer information to use to train the artificial intelligence chatbot to produce an invalid answer with a low confidence level, as taught by Liu. The modification would have been obvious because one of ordinary skill in the art would be motivated to better differentiate good date and bad data for AI model training as suggested by Liu at [Introduction]: “To reduce the uncertainty of labels, a metric named label confidence is proposed in this paper for measuring the reliability of each label, in which clean labels get high confidence scores while noisy labels achieve low confidence scores.”.
Referring to dependent Claims 13 and 19, these claims are rejected on the same basis as dependent claim 6, mutatis mutandis, since they are analogous claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIAYUE MA whose telephone number is (571)272-9658. The examiner can normally be reached between 9 am to 5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Jiayue Ma/
Examiner, Art Unit 2126
/DAVID YI/ Supervisory Patent Examiner, Art Unit 2126