DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-15 are rejected under 35 U.S.C. 101 because the claims are directed toward an abstract idea without significantly more.
Claim 1, is rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (an abstract idea) and does not include additional elements that amount significantly more than the judicial exception.
Step 1
Claim 1 is directed toward a “method”, which is a method and thus falls within a statutory category under the most recent guidelines of 35 U.S.C. 101.
Step 2A, Prong 1
Claim 1 recites a method for determining a hallucination score of an artificial intelligence (AI) model in a language processing system, comprising steps of “receiving a prompt and an answer”; “inserting a keyword into the answer”; “generating a first word set by using words present in the prompt”; “generating a second word set by using words present in the answer with the inserted keyword”; “generating embedding vectors of the first word set and the second word set”; and “calculating a hallucination score based on the embedding vectors.”
These limitations collectively recite the collection, mathematical evaluation of information, including language evaluation. As characterized by the USPTO guidance and case law, such activities fall within the abstract-idea groupings of mental processes (e.g. observations, evaluations, and judgments that could be performed in the human mind or with pen and paper) and organizing /transmitting information. Reference can be made to latest patent eligibility guidelines. Accordingly, claim 1 recites an abstract idea.
Step 2A, Prong 2
The claim is implemented on a “language processing system”, which is a generic computer components performing their well-understood, routine, and conventional functions of storing and executing instructions, receiving requests, and sending content.
The claim does not recite any specific improvement to computer functionality (e.g., a particular translation algorithm, model architecture, data structure, memory organization, caching mechanism, latency-reduction technique, or network protocol that improves the operation of the computer or network). Nor does it effect a transformation of a physical article or use the abstract idea in any other manner that imposes a meaningful limit on the claim’s scope. Therefore, the claim does not integrate the abstract idea into a practical application under Step 2A, Prong 2.
Step 2B
Beyond the abstract idea, the additional elements are the generic “system” components performing their conventional functions. Implementing the abstract idea on generic computer components does not amount to significantly more. Alice, 573 U.S. at 223–24).
The ordered combination of limitations mirrors the abstract idea itself performed using routine computer operations. There is no recited unconventional hardware, no technical improvement to the functioning of the computer itself, and no nonconventional arrangement of known components etc.
Accordingly, claim 1 does not include an “inventive concept” sufficient to transform the abstract idea into a patent-eligible application.
Therefore , claim 1 is directed to an abstract idea and does not recite additional elements that integrate the exception into a practical application or amount to significantly more than the exception itself. Claim 1 is therefore rejected under 35 U.S.C. § 101. Dependent claims 2-11 do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea.
The Independent claim 12 recite(s) an apparatus for determining a hallucination score of an artificial intelligence model in a language processing system, the apparatus comprising: “a storage unit configured to store information necessary for operation of the apparatus”; and “a processor connected to the storage unit, wherein the processor is configured to: receive a prompt and an answer, insert a keyword into the answer, generate a first word set by using words present in the prompt, generate a second word set by using words present in the answer with the inserted keyword, generate embedding vectors of the first word set and the second word set, and calculate a hallucination score based on the embedding vectors.”
All the limitations can be performed by a human being including applying a service algorithm. These limitations collectively recite the collection, evaluation and translation of information, including language evaluation. As characterized by the USPTO guidance and case law, such activities fall within the abstract-idea groupings of mental processes (e.g. observations, evaluations, and judgments that could be performed in the human mind or with pen and paper) and organizing /transmitting information. Reference can be made to latest patent eligibility guidelines. Accordingly, claim 12 recites an abstract idea.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. With respect to integration of the abstract idea into a practical application, the additional element of using a generic computing device the determining and data gathering steps amount to no more than mere instructions to apply the exception using a generic computer. The current specification on paragraph 0038, clearly specifies that “… the platform may refer to an operating system constituting a system that provides a safety information detection method according to the present disclosure. The user devices 110a and 110b may obtain input data (e.g., e-mail, user input, electronic documents, etc.), transmit the input data to the server 120 through the communication network, and interact with the server 120. Each of the user devices 110a and 110b may include a communication unit for communication, a storage unit for storing data and programs, a display unit for displaying information, an input unit for user input, and a processor for control. For example, each of the user devices 110a and 110b may be a general-purpose device (e.g., a smartphone, a tablet, a laptop computer, a desktop computer) or a platform-specific access terminal in which an application or program for platform access is installed.”
The additional elements have been considered both individually and as an ordered combination in the significantly more consideration. The inclusion of the computer or memory and controller to perform the selecting and generating steps amount to no more than mere instructions to apply the exception using generic computer components. Mere instructions to apply an exception using a generic computing device cannot provide an inventive concept. Therefore, claim 12 as drafted is not patent eligible. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements when considered both individually and as an ordered combination do not amount to significantly more than the abstract idea.
Thus, taken alone, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation. Independent claim 12 is therefore not drawn to eligible subject matter as they are directed to an abstract idea without significantly more. Claims 13-15 are dependent claims and do not contain subject matter that can be overcome the rejection of independent claim 12.
All dependent claims when analyzed as a whole are held to be patent ineligible under 35 U.S.C. §101 because any additional recited limitations fail to establish that the claims are not directed to an abstract idea for the same reasons already recited for the independent claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-15 are rejected under 35 U.S.C. 103 as being unpatentable over Aberle (US 2024/0062019 A1) in view of Tunstall-Pedoe et al., (US 2023/0316006 A1).
As per claim 1, Aberle teaches a method for determining a hallucination score of an artificial intelligence model in a language processing system, the method comprising:
receiving a prompt and an answer (0047, 0085-0086);
inserting a keyword into the answer (0046);
generating a first word set by using words present in the prompt (0047, 0085-0086);
generating a second word set by using words present in the answer with the inserted keyword (0047, 0085-0086);
generating embedding vectors of the first word set and the second word set (0046-0047).
Aberle, however while teaching scoring (0080, 0086) does not specifically teach calculating a hallucination score based on the embedding vectors. Tunstall-Pedoe et al., do teach calculating a hallucination score based on the embedding vector (0863-0887). Therefore it would have been obvious to one of ordinary skill in the art to incorporate the teaching of Tunstall-Pedoe to calculate a hallucination score, in the method of Aberle , because this would provide the end user with a result that is reliably replicated and replacing parts of the result that change, with language that expresses that the answer is unknown (Tunstall-Pedoe 0863).
As per claim 2, Aberle in view of Tunstall-Pedoe et al., teach the method of claim 1, wherein an embedding vector of the second word set is an embedding vector for one or more words combining a keyword and an answer (0046 – 0047).
As per claim 3, Aberle in view of Tunstall-Pedoe et al., teach the method of claim 2, wherein the first word set is generated based on a number of words of the answer (0046-0047).
As per claim 4, Aberle in view of Tunstall-Pedoe et al., teach the method of claim 2, wherein the calculating of the hallucination score comprises: calculating a similarity score between an embedding vector of the first word set and the embedding vector of the second word set; and calculating the hallucination score based on the similarity score (Aberle: 0010, 0046, 0066, Tunstall-Pedoe: 0863-0867).
As per claim 5, Aberle in view of Tunstall-Pedoe et al., teach the method of claim 4, further comprising determining reliability of the answer based on comparison between the hallucination score and a threshold value (Tunstall-Pedoe: 0863-0867).
As per claim 6, Aberle in view of Tunstall-Pedoe et al., teach the method of claim 5, wherein based on the hallucination score be equal to or greater than the threshold value, the reliability of the answer is determined to be high, and wherein based on the hallucination score be smaller than the threshold value, the reliability of the answer is determined to be low (Tunstall-Pedoe: 0863-0867).
As per claim 7, Aberle in view of Tunstall-Pedoe et al., teach the method of claim 4, wherein the similarity score is calculated using one method among mean squared difference similarity, cosine similarity, Pearson similarity, or L2 (Aberle, 0010, 0046, 0066).
As per claim 8, Aberle in view of Tunstall-Pedoe et al., teach the method of claim 1, wherein the generating of the first word set further comprises: based on the prompt including a plurality of sentences, dividing the prompt into sentence units; generating a first embedding vector of the plurality of the divided sentences; generating a second embedding vector of the second word set; calculating a similarity score based on the first embedding vector and the second embedding vector; selecting a sentence with the similarity score being highest; and generating word sets by using words present in the selected sentence and the answer with the inserted keyword (Tunstall-Pedoe: 0217, 0238, 0401).
As per claim 9, Aberle in view of Tunstall-Pedoe et al., teach the method of claim 1, wherein the receiving of the prompt and the answer comprises: receiving a question and the prompt from a user; inputting the question and the prompt into an artificial intelligence (AI) system; obtaining an answer based on the prompt and the question that are input into the AI system; and receiving the obtained answer (Aberle, 0050).
As per claim 10, Aberle in view of Tunstall-Pedoe et al., teach the method of claim 1, wherein the receiving of the prompt and the answer comprises: identifying a question input from a user (Aberle, 0045-0046); receiving a prompt from an external database based on the question (Aberle, 0045-0046); inputting the question and the prompt into an AI system (Aberle, 0050); obtaining an answer based on the prompt and the question that are input into the AI system (Aberle, 0050); and receiving the obtained answer(Aberle, 0045-0046).
As per claim 11, Aberle in view of Tunstall-Pedoe et al., teach the method of claim 1, wherein the embedding vector is generated by using an AI model (Aberle, abstract), and wherein the AI model includes at least one of a sentence-transformer (Aberle, abstract), a transformer (Aberle, abstract), an LLM embedding model (Aberle, 0050, 0086), or an OpenAI embedding model (Aberle, 0050).
As per claim 12, Aberle et al., teach an apparatus for determining a hallucination score of an artificial intelligence model in a language processing system, the apparatus comprising:
a storage unit configured to store information necessary for operation of the apparatus (0120);
and a processor connected to the storage unit, wherein the processor is configured to (0034-0035, 0119-0120):
receive a prompt and an answer, insert a keyword into the answer, generate a first word set by using words present in the prompt, generate a second word set by using words present in the answer with the inserted keyword, generate embedding vectors of the first word set and the second word set (0046-0047, 0085-0086).
Aberle, however, does not specifically teach calculating a hallucination score based on the embedding vectors. Tunstall-Pedoe et al., do teach calculating a hallucination score based on the embedding vector (0863-0887). Therefore it would have been obvious to one of ordinary skill in the art to incorporate the teaching of Tunstall-Pedoe to calculate a hallucination score, in the method of Aberle , because this would provide the end user with a result that is reliably replicated and replacing parts of the result that change, with language that expresses that the answer is unknown (Tunstall-Pedoe 0863).
As per claim 13, Aberle in view of Tunstall-Pedoe et al., teach the apparatus of claim 12, wherein the processor is further configured to: based on the prompt including a plurality of sentences, divide the prompt into sentence units, generate a first embedding vector of the plurality of the divided sentences, generate a second embedding vector of the answer with the inserted keyword, calculate a similarity score based on the first embedding vector and the second embedding vector, select a sentence with the similarity score being highest, and generate word sets by using words present in the selected sentence and the answer with the inserted keyword (Tunstall-Pedoe: 0217, 0238, 0401).
As per claim 14, Aberle in view of Tunstall-Pedoe et al., teach the apparatus of claim 12, wherein the processor is further configured to: receive a question and the prompt from a user, input the question and the prompt into an artificial intelligence (AI) system, obtain an answer based on the prompt and the question that are input into the AI system, and receive the obtained answer (Aberle: 0045-0046, 0050).
As per claim 15, Aberle in view of Tunstall-Pedoe et al., teach the apparatus of claim 12, wherein the processor is further configured to: receive a question from a user, receive a prompt from an external database based on the question, input the question and the prompt into an AI system, obtain an answer based on the prompt and the question that are input into the AI system, and receive the obtained answer (Aberle: 0045-0047, 0050).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Please see attached form PTO-892.
The following is related prior art applicable to Applicant invention.
Cowburn et al., (US 10,074,381 B1) teach an augmented reality system to generate and cause display of an augmented reality interface at a client device. Various embodiments may detect speech, identify a source of the speech, transcribe the speech to a text string, generate a speech bubble based on properties of the speech and that includes a presentation of the text string, and cause display of the speech bubble at a location in the augmented reality interface based on the source of the speech.
McNamara et al., (US 2025/0259007 A1) teach a method for facilitating customizable communications with users. The method may include receiving at least one input associated with at least one user from at least one user device associated with the at least one user, analyzing the at least one input, identifying at least one instruction from a plurality of instructions based on the analyzing of the at least one input, and obtaining at least one output using at least one language model based on the at least one input and the at least one instruction. Further, the method may include transmitting the at least one output to the at least one user device. Further, the method may include storing the at least one input and the at least one output.
Park et al., (US 2024/0420491 A1) teach network infrastructure for user-specific generative intelligence. Providing user-specific context to a generically trained LLM introduces a variety of complications (privacy, resource utilization, training costs, etc.). Various aspects of the present disclosure provide novel user-specific data structures, privacy and access control, layers of data, and session management, within a network infrastructure for generative intelligence. For example, user-specific embedding vectors may be used to provide user context to a generically trained foundation model. In some variants, edge devices capture multiple modalities of user context (images, audio; not just text). Privacy and access control mechanisms also allow a user to control information that is captured and sent to the foundation model. Session management further decouples a user's conversational state from the foundation model's session state. These concepts and others may be used to emulate e.g., a chatbot based virtual assistant that responds based on user context.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VIJAY B CHAWAN whose telephone number is (571)272-7601. The examiner can normally be reached 7-5 Monday thru Thursday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Richemond Dorvil can be reached at 571-272-7602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VIJAY B CHAWAN/Primary Examiner, Art Unit 2658
Prosecution Insights