DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a mental process without significantly more.
Independent claims 1, 12, and 19 recite substantially the same operations in method, non-transitory machine readable medium, and computing device forms. Under their broadest reasonable interpretation, the claims are directed to searching source information and using that information to formulate and present a multiple-choice question and corresponding answers. For example, under the BRI the claims relate to:
in response to receiving a user input query, generating similarity scores corresponding to similarities between the user input query and chunks of content within a repository (a person can mentally receive/read a question from another person then compare a question with portions of available source material and mentally judge or rank/score the degree to which each portion relates to the question); selecting one or more chunks from the repository based upon similarity scores between the user input query and the one or more chunks (A person can mentally or with a pen and paper select the passages considered most relevant to the query); generating a pregeneration prompt based upon the user input query, the one or more chunks, and instructions for a model (a person can mentally formulate instructions based upon a question, relevant source material, and a desired task); inputting the pregeneration prompt into the model to generate an initial question (A person can mentally or with a pen and paper draft an initial question from the selected information); generating a prompt based upon the initial question, the one or more chunks, and the instructions for the model (A person can mentally refine or formulate additional instructions based upon the initial question and source material); inputting the prompt into the model to generate question and answer content in a multiple choice format (A person can mentally or with a pen and paper prepare a multiple choice question having one correct answer and one or more incorrect answers); and providing the question and answer content in the multiple choice format through a user interface for user engagement (A person can present the completed multiple choice question to another person).
As described above, these limitations can be carried out as a series of mental steps. The judicial exception is not integrated into a practical application because the only additional elements recited are a non-transitory computer medium and system comprising a computer processor, memory, and LM models, which are general-purpose hardware and software tools being used as a tool to implement the mental process. The computer-executable instructions and the user device are conventional components that utilize the basic functions of a computer to automate the abstract gathering and summarizing of data.
The claim(s) do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the only additional elements recited are the recitation of generic processors, storage devices, and AI models used to perform the mental steps of query/answer comparison. These amount to nothing more than a "computerized" version of a standard human cognitive task.
The remaining dependent claims fail to add patent-eligible subject matter to the independent claims:
Claim 2 simply adds populating the pregeneration prompt with a situation, date, task, and format tag. A person preparing instructions can organize instructions into sections describing the circumstances, relevant date, requested task, and desired answer format. The tags merely organize informational content supplied to the model and do not improve the operation of the computer or model.
Claim 3 simply adds defining the situation tag to describe a question and answer generation persona and including the user query as the topic. A person can mentally adopt the role of a quiz writer and use a supplied topic to formulate a question. Assigning a persona and topic to a model merely specifies the informational content and role described by the prompt.
Claim 4 simply adds defining an ordered set of tasks that constrains the model to produce a multiple choice question having one correct answer and other incorrect answers. A person can following ordered instructions to write a question, identify one correct response, and draft distractors. This limitation describes the desired intellectual task and informational output rather than a technological improvement.
Claim 5 simply adds defining an expected response format and discarding outputs that do not conform to that format. A person can compare a drafted question with formatting instructions and reject the draft when it fails to conform. The model and computer merely automate that evaluation.
Claim 6 adds assigning one of two confidence values to a potential question, disqualifying a question assigned the first value, and retaining a question assigned the second value. A person can judge whether a proposed question is sufficient supported or reliable, mark if it is acceptable or unacceptable, discard the unacceptable question, and retain the acceptable question. The binary confidence designations are evaluative labels and do not improve computer technology.
Claim 7 adds altering potential initial questions that do not exist withing the repository. A person can compare a proposed question with available source material and discard the question when it is not supported by that material. This remains an abstract evaluation of information.
Claim 8 simply adds retrieving a trusted document, dividing it into chunks, generating vector embeddings for the chunks, and storing the chunks, embeddings, and metadata. Retrieving, dividing, labeling, and storing source information are data gathering and data organization activities. Generative vector embeddings additionally recites a mathematical representation of textual information. The claim fails to specify an improved embedding algorithm, storage architecture, or computer operation.
Claim 9 simply adds that the metadata includes title, published date, and updated date. A person can record a document’s title and dates as descriptive information. The limitation merely specifies informational content of the metadata.
Claim 10 simply adds selecting first and second document chunk s having an allowed percentage of overlap. A person can copy or divide portions of a document such that part of one portion is repeated in the next portion. To the extent a percentage is calculated, the limitation additionally reciting a mathematical calculation. The claim doesn’t recite a technological improvement in document storage or processing.
Claim 11 simply adds selecting a chunk containing material that preserves contextual information. A person can group related sentences or passages together to preserve their context. The limitation recites a judgement concerning the meaning and organization of information.
Claim 13 simply adds generating a query vector embedding, using a similarity function to compare that embedding with chunk embeddings, assigning similarity scores, and selecting higher scoring chunks. These limitations expressly recite mathematical representations, comparisons, and scoring relationships. The claim does not identify a particular improved embedding or similarity algorithm and merely uses the calculations to perform the abstract task of determining which information is most relevant.
Claim 14 simply adds applying a similarity score threshold to disqualify chunks below the threshold. Applying a numerical threshold is a mathematical comparison used to implement the abstract judgement that some source material is insufficiently relevant and this limitation doesn’t improve the functioning of the computer.
Claim 15 simply adds populating the later prompt with a situation tag, date tag, task tag, and format tag. As with claim 2, a person can organize written instructions into sections describing time, context, task, and desired format. The tags organize informational content and do not supply a technological improvement.
Claim 16 simply adds defining a question and answer generation persona and using the initial question as the topic for generating multiple choice content. A person acting as a quiz writer can use an initial question as the subject for preparing a final question and answer choices. The limitation specifies the model’s informational role and task.
Claim 17 simple adds an ordered set of tasks constraining the model to output multiple choice content having one correct answer and other incorrect answers which a person can follow the same ordered instructions when preparing a quiz.
Claim 18 simply adds defining an expected multiple-choice response format and discarding outputs that fail to conform. A person can review a drafted quiz against formatting requirements and reject a nonconforming draft. Automating that review through a model does not add significantly more.
Claim 20 simply adds populating the interface with a question and multiple answers, receiving the user’s answer selection, and displaying the correct answer and reason it is correct. A person administering a quiz can present a question and answer choices, receive a selected answer, identify the correct answer, and explain why it is correct the user interface merely performs the generic functions of displaying information and receiving a selection and does not transform the abstract educational interaction into a practical technological application.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 6-9, 11-14, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Berglund et al. (hereinafter Berglund) (US 20240403341 A1) in view of Khan et al. (hereinafter Khan) (US 20220415203 A1).
Regarding claim 1, Berglund discloses:
A method, comprising:
in response to receiving a user input query (Berglund, P[0032]: "a user inputs a search query at the user interface 202, which can be displayed on a user device. The search query is received by the query service 112" (teaches receiving a user entered query and initiating the disclosed query processing operations in response)), generating similarity scores corresponding to similarities between the user input query and chunks of content within a repository (Berglund, P[0038]: "the search engine 114 generates an embedding similarity score between a query embedding that represents the search query and a respective text embedding associated with a text chunk." (the "query embedding" corresponds to the claimed use input query, the "text embedding associated with a text chunk" corresponds to the claimed chunk of content, and the generated "embedded similarity score" corresponds to the claims similarity score between the query and the chunk));
selecting one or more chunks from the repository based upon similarity scores between the user input query and the one or more chunks (Berglund, P[0038]: "A text chunk can be identified as relevant to the query when its embedding similarity score satisfies a similarity threshold." (the "text chunk" corresponds to the claimed chunk of content, and identifying the chunk as relevant when its embedding similarity score satisfies the threshold corresponds to selecting one or more chunks based upon the similarity scores.));
generating a pregeneration prompt based upon the user input query, the one or more chunks, and instructions for a model (Berglund, P[0046]: "the query service 112 sends the LLM 140 a prompt that includes the search query and the text chunks matched to the search query, instructing the LLM 140 to generate an answer to the search query based on the matching text chunks." (query service invokes the LLM to generate additional queries and sends LLM 'a prompt that includes the search query and the text chunks matched to the search query' together with instructions directing the LLM's generation operation, teaches forming an LLM prompt including the search query, the text chunks matched to the search query, and instructions directing LLM generation. Berglund uses this prompt to generate and answer and using the same retrieved chunk in the earlier question-generation prompt would have been an obvious modification);
inputting the pregeneration prompt into the model to generate an initial question (Berglund, P[0045]: "based on the query received from the user. In some implementations, the query service 112 identifies other search queries submitted by other users that are similar to the search query being processed. Additionally or alternatively, the query service 112 invokes the LLM" (the generated "additional questions" correspond to the claimed initial question generated before the subsequent answer generation prompt));
generating a prompt based upon the initial question, the one or more chunks, and the instructions for the model (Berglund, P[0046]: "the text chunks matched to the search query, instructing the LLM 140 to generate an answer to the search query based on the matching text chunks. The prompt can further include the additional questions generated based on the query to improve the query answer generated by the LLM. When the prompt includes the additional questions, the prompt can instruct the LLM to generate a combined answer" (the "additional questions" correspond to the claimed initial question, the "text chunks matched to the search query" correspond to the claimed chunks and the instruction to "generated a combined answer' corresponds to the claimed model instructions.));
Berglund does not explicitly disclose:
inputting the prompt into the model to generate question and answer content in a multiple choice format; and
providing the question and answer content in the multiple choice format through a user interface for user engagement
However, Khan discloses:
inputting the prompt into the model to generate question and answer content in a multiple choice format (Khan, P[0090]: "the NLG interface 110 presents a knowledge assessment item to a user after the knowledge assessment item is generated by a NLG model based on a conditioning input. ", "The knowledge assessment item may be presented to the user broken up into the question stem and answer choices. " (the "conditioning input" corresponds to the claimed prompt, generating the item "by a NLG model" corresponds to inputting the prompt into the model to generate content and the generated "question stem and answer choices" correspond to question and answer content in multiple choice format.)); and
providing the question and answer content in the multiple choice format through a user interface for user engagement (Khan, P[0090]: "The knowledge assessment item may be presented to the user after initial checks by item verification 122 via a user interface 126 to the NLG interface 110. The knowledge assessment item may be presented to the user broken up into the question stem and answer choices. In some implementations, the user interface 126 may provide features allowing a user to select a key (correct answer) and distractors from among the generated answer choices." (presenting the question stem and answer choices "via user interface" corresponds to providing multiple choice question and answer content through the claimed user interface and allowing the user to select a correct key and distractors teaches user engagement with this content)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Berglund in view of Khan. Doing so would have provided the ability to generate and present multiple choice assessment content of Khan (Khan, Abstract, P[0090]) with the retrieval-augmented LLM system of Berglund (Berglund, Abstract, P[0046]-P[0049]), thus, predictably providing user interaction and assessment functionality while retaining Berglund’s retrieval grounding via the use of Khan’s assessment-item generation and presentation combined with Berglund’s retrieval augmented LLM system.
Regarding claim 6, the combination of Berglund and Khan discloses the method of claim 1.
The combination further discloses:
wherein the generating the pregeneration prompt further comprises:
populating the pregeneration prompt with task instructions for the model to assign a confidence having a first value or a second value for a potential initial question (Berglund, P[0045]: "if one of the answers has a significantly lower confidence score than other answers" (the generated candidates are assigned distinguishable confidence values, representing lower and higher confidence as first and second values would have been an obvious binary implementation));
in response to the confidence being set to the first value, disqualifying the potential initial question (Berglund, P[0045]: "the query service 112 may remove the lower-scored answer from the set of answers output to the user." (lower-confidence candidate is disqualified, while the remaining higher confidence candidates are retained int eh output set.)); and
in response to the confidence being set to the second value, retaining the potential initial question for further consideration (Berglund, see mapping above).
Regarding claim 7, the combination of Berglund and Khan discloses the method of claim 1.
The combination further discloses:
comprising:
inputting the pregeneration prompt into the model for filtering potential initial questions that do not exist within the repository (Berglund, P[0049]: "Inputs the received answer and the set of content back into the LLM to ask the LLM to confirm that the answer was generated based on or is consistent with the set of content." (the generated candidate and repository content are input into the model to filter content that is not grounded in or consistent with the repository)).
Regarding claim 8, the combination of Berglund and Khan discloses the method of claim 1.
The combination further discloses:
comprising:
retrieving a document from a trusted content source (Berglund, P[0026]: "the content management system 110 retrieves a content item from the content repository 150" (the retrieved repository content item corresponds to the claimed document from a managed source));
parsing the document into a plurality of chunks (Berglund, P[0027]: "the content management system 110 splits text of the content item into sentence-length text", P[0029]: "the system 110 can group the similar sentence-length portions into text chunks." (document is divided into portions and grouped into multiple chunks));
generating vector embeddings for each chunk of the plurality of chunks (Berglund, P[0029]: "a new embedding can be generated for the text chunk" (a vector embedding is generated for each resulting chunk)); and
storing the vector embeddings, the plurality of chunks, and metadata information into the repository (Berglund, P[0030]: "stores the text embeddings associated with the text chunks of the content items in the content repository 150.", P[0039]: "content metadata that can include, for example, an author of a content item, a time stamp indicating when the content item was created or most recently updated, tags or categorization labels applied to the content item, or a description" (the chunks and their associated embeddings are stored in the repository together with metadata associated with the underlying content items)).
Regarding claim 9, the combination of Berglund and Khan discloses the method of claim 8.
The combination further discloses:
wherein the metadata information includes a title, a published date, and an updated date of the document (Berglund, P[0039]: "metadata that can include, for example, an author of a content item, a time stamp indicating when the content item was created or most recently updated, tags or categorization labels applied to the content item, or a description for the content item." (the created timestamp corresponds most closely to the claimed publication date, and the most recently-updated timestamp corresponds to the updated date, content item reads on title)).
Regarding claim 11, the combination of Berglund and Khan discloses the method of claim 8.
The combination further discloses:
comprising:
selecting a first chunk to include content that preserves contextual information of the content (Berglund, P[0029]: "If one sentence is sufficiently similar to the next sentence, the sentences can be combined into the same text chunk because the sentences are likely to be related to the same topic and/or to one another" (grouping sentences related to the same topic or to one another into the same chunk preserves their contextual relationship)).
Regarding claim 12, claim 12 recites the non-transitory machine-readable medium corresponding to the method described in claim 1 and is rejected for the same reasons as above.
Berglund further discloses:
A non-transitory machine-readable medium having stored thereon processor-executable instructions that when executed cause performance of operations, the operations comprising (Berglund, P[0088], Claim 10):
Regarding claim 13, the combination of Berglund and Khan discloses the non-transitory machine-readable medium of claim 12.
The combination further discloses:
the operations comprising:
generating a user input vector embedding for the user input query (Berglund, Claim 1: "a query embedding corresponding to the user query" (the query embedding is the claimed vector embedding representing the user input query));
utilizing a similarity function to compare the user input vector embedding to vector embeddings of the chunks to assign the similarity scores (Berglund, Claim 1: "an embedding similarity score between a query embedding corresponding to the user query and the text embedding associated with each text chunk" (the query embedding is compared with each chunk embedding, and the resulting embedding-similarity value is the claimed assigned similarity score)); and
selecting the one or more chunks based upon the one or more chunks having higher similarity scores than other chunks (Berglund, Claim 1: "selecting, by the computer system, at least one relevant text chunk from the set of text chunks based on the embedding similarity score" (selection of relevant chunk according to its similarity score corresponds to selecting chunks whose scores indicate greater relevance than other chunks)).
Regarding claim 14, the combination of Berglund and Khan discloses the non-transitory machine-readable medium of claim 12.
The combination further discloses:
the operations comprising:
applying a similarity score threshold to the similarity scores to disqualify chunks with similarities scores below the similarity score threshold (Berglund, P[0038]: "chunk can be identified as relevant to the query when its embedding similarity score satisfies a similarity threshold." (chunks satisfying the threshold are retained as relevant, necessarily excluding or disqualifying chunks whose scores fall below the threshold)).
Regarding claim 19, claim 12 recites the computing device corresponding to the method described in claim 1 and is rejected for the same reasons as above.
Berglund further discloses:
A computing device comprising:
a processor (Berglund, P[0083], P[0088]); and
memory comprising processor-executable instructions that when executed by the processor cause performance of operations, the operations comprising (Berglund, P[0083], P[0088]):
Claims 2-5 and 15-18 are rejected under 35 U.S.C. 103 as being unpatentable over Berglund et al. (hereinafter Berglund) (US 20240403341 A1) in view of Khan et al. (hereinafter Khan) (US 20220415203 A1) and in further view of Attali et al. (hereinafter Attali) (US 20230080674 A1).
Regarding claim 2, the combination of Berglund and Khan discloses the method of claim 1.
Berglund, in combination with Khan, further discloses:
wherein the generating the pregeneration prompt further comprises:
populating the pregeneration prompt with a situation tag (Berglund, P[0047]: "The prompt to the LLM 14 can further include context information of the query. For example, the query service 112 retrieves contextual information associated with the user who submitted the search query or the context in which the search query was generated, providing the contextual information to the LLM with the search query and the related content to enable the LLM to generate more customized answers" ("context information" describing the user and circumstances in which query was generated corresponds to information contained in claimed situation tag)), a date tag (Berglund, P[0040]: "The context information can include, for example, an identity of a user who submitted the query, a history of queries recently submitted by the user, or a context in which the query was submitted (e.g., while viewing a particular content item, after accessing a set of content items, after receiving a message that links to or attaches a particular content item, at a certain time of day or time of year, during certain events, etc.).")
The combination does not explicitly disclose:
A task tag, and a format tag;
However, Attali discloses:
a task tag (Attali, P[0127]: "Define one or more item generation templates, with each template consisting of a set of Instructions and Examples for generating questions and/or answers that can be used as input for a Transformer-Based Language Model" (the "set of instructions" defining the generation of questions and answers corresponds to the model task contained in the claimed task tag)), and a format tag (Attali, P[0087]: "“Generate short paragraphs from high school textbooks on the specified topic.” Note that this Instruction provides the desired format (short paragraphs), level (high-school textbook), and subject matter for the output (the specific topic)." (the instruction identifies and supplies a "desired format" for the model output, corresponding to the information contained in the claimed format tag)) It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Berglund in view of Khan and Attali. Doing so would have provided the model instructions, output formation controls, conditioning, and ordered generation of correct/incorrect multiple choice responses of Attali (Attali, Abstract, P[0087], P[0127]) with the ability to generate and present multiple choice assessment content of Khan (Khan, Abstract, P[0090]) with the retrieval-augmented LLM system of Berglund (Berglund, Abstract, P[0046]-P[0049]), thus, predictably improving the generation of accurate, structured, multiple-choice assessment content.
Regarding claim 3, the combination of Berglund, Khan, and Attali discloses the method of claim 2.
The combination further discloses:
comprising:
defining the situation tag to describe a persona as a question answer generation agent (Berglund, P[0047]: "The prompt to the LLM 14 can further include context information of the query. For example, the query service 112 retrieves contextual information associated with the user who submitted the search query or the context in which the search query was generated, providing the contextual information to the LLM with the search query and the related content to enable the LLM to generate more customized answers. For example, the query service 112 retrieves a job title of the user and instructs the LLM to “generate an answer for a beginner” or to “generate an answer for a busy executive.” " (the instructions defining how the LLM should act for ap articular audience corresponds to defining a model persona and applying that role to question generation corresponds to the claimed question answer generation agent persona) and includes the user input query as a topic for the model to generate the initial question as the question answer generation agent (Berglund, P[0045]: "At 226, the query service 112 generates a set of additional queries, based on the query received from the user. In some implementations, the query service 112 identifies other search queries submitted by other users that are similar to the search query being processed. Additionally or alternatively, the query service 112 invokes the LLM 140 or other trained models to generate a list of similar queries." (the query received from the user supplies the topic and invoking the LLM to generate similar queries corresponds to generating the claimed initial question based on that topic)).
Regarding claim 4, the combination of Berglund, Khan, and Attali discloses the method of claim 2.
The combination further discloses:
comprising:
defining the task tag to describe an ordered set of tasks to be performed by the model (Attali, P[0127]: "Define one or more item generation templates, with each template consisting of a set of Instructions and Examples for generating questions and/or answers that can be used as input for a Transformer-Based Language Model", P[0030]: "Based on the selected Source Passage(s), for each such passage, generate a set of possible correct responses to one or more multiple-choice questions using a Transformer-Based Language Model", P[0036]: "Using the set of Alternative Passages generated or corresponding to each selected Source Passage, generate a set of possible incorrect responses to each of the multiple-choice questions referred to in the previous step" (in instructions define successive model operations for generating the question answers, including first generating correct responses and then generating incorrect responses, thus, describing an ordered set of tasks)), wherein the ordered set of tasks include one or more tasks constraining the model to output the initial question having the multiple choice format (Attali, P[0128]: "generate a set of possible correct responses to one or more multiple-choice questions using a Transformer-Based Language Model", "generate a set of possible incorrect responses to each of the multiple-choice questions referred to in the previous step using a Transformer-based Language Model" (the instructions constrain the transformer model to generate responses belonging to a multiple choice question)) with multiple answers where a single answer is correct and other answers are not correct (Attali, P[0056]: "Construct a set of test items using the selected correct response (or responses) generated for each selected Source Passage as the correct answer(s) and the selected incorrect responses generated for the Corresponding Alternative Passage(s) as the incorrect answer(s) for each of the multiple-choice questions for which answers were generated." (the selected response from the source passage is used as the correct answer, while responses generated from alternative passages are used as incorrect answers. Selecting one correct response provides the claimed single correct answer embodiment)).
Regarding claim 5, the combination of Berglund, Khan, and Attali discloses the method of claim 2.
The combination further discloses:
comprising:
defining the format tag to describe an expected response format for the initial question (Attali, P[0087]: "Generate short paragraphs from high school textbooks on the specified topic.” Note that this Instruction provides the desired format (short paragraphs), level (high-school textbook), and subject matter for the output (the specific topic)." (the instruction defines the expected format of the model-generated output, corresponding to the information provided by the claimed format tag)); and
discarding outputs by the model that do not conform to the expected response format described by the format tag (Khan, P[0089]: "Where the one or more features of the raw knowledge assessment item do not match the specifications for knowledge assessment items, the NLG interface 110 executes block 510 and discards or re-generates the raw knowledge assessment item." (the generated output is compared with expected specifications and discarded or regenerated when it does not conform)).
Regarding claim 15, the combination of Berglund and Khan discloses the non-transitory machine-readable medium of claim 12.
Berglund, in combination with Khan, further discloses:
the operations comprising:
populating the prompt with a situation tag (Berglund, P[0047]: "The prompt to the LLM 14 can further include context information of the query. For example, the query service 112 retrieves contextual information associated with the user who submitted the search query or the context in which the search query was generated" (user and query-context information corresponds to the claimed situation information placed in the prompt)), a date tag (Berglund, P[0040]: "at a certain time of day or time of year, during certain events, etc.", "can send the context information to the LLM 140 as part of a prompt" (temporal information corresponds to date information, and Berglund places that contextual information it the model prompt)),
The combination of Berglund and Khan does not explicitly disclose:
a task tag, and a format tag;
However, Attali discloses:
a task tag (Attali, P[0031]: "set of Instructions for the model" (the instructions specify the model operation to be performed and correspond to the task information in the claimed task tag)), and a format tag (Attali, P[0013]: "goals or general characteristics of the desired output", P[0015]: "Format of a generated passage" (prompt instructions specify the desired output format, corresponding to the claimed format tag)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Berglund in view of Khan and Attali. Doing so would have provided the model instructions, output formation controls, conditioning, and ordered generation of correct/incorrect multiple choice responses of Attali (Attali, Abstract, P[0087], P[0127]) with the ability to generate and present multiple choice assessment content of Khan (Khan, Abstract, P[0090]) with the retrieval-augmented LLM system of Berglund (Berglund, Abstract, P[0046]-P[0049]), thus, predictably improving the generation of accurate, structured, multiple-choice assessment content.
Regarding claim 16, the combination of Berglund, Khan, and Attali discloses the non-transitory machine-readable medium of claim 15.
The combination further discloses:
the operations comprising:
defining the situation tag to describe a persona as a question answer generation agent (Berglund, P[0047]: "“generate an answer for a beginner” or to “generate an answer for a busy executive.” " (the prompt defines the role, audience, or manner in which the model is to operate, teaching persona-based model instructions.)) and includes the initial question as a topic for the model to generate the question and answer content in the multiple choice format as the question answer generation agent (Berglund, P[0046]: "The prompt can further include the additional questions generated based on the query" (the previously generated questions is included in the later model prompt as a subject of the requested response)).
Regarding claim 17, the combination of Berglund, Khan, and Attali discloses the non-transitory machine-readable medium of claim 15.
the operations comprising:
defining the task tag to describe an ordered set of tasks to be performed by the model (Attali, P[0305]: "generate a multiple-choice question", "generate one or more correct responses to the multiple-choice question", "generate one or more incorrect responses to the multiple-choice question" (quoted sequence defines an ordered set of generation tasks)), wherein the ordered set of tasks include one or more tasks constraining the model to output the question and answer content in the multiple choice format (Attali, P[0309]: "construct a test item using the selected correct and incorrect answers for the multiple-choice question" (the instructed output is a multiple choice question and answer item)) with multiple answers where a single answer is correct and other answers are not correct (Attali, P[0207]: "the selected response generated for the Source Passage as the correct answer and the selected responses generated for the Alternative Passages as the incorrect answer(s)" (one selected response serves as the correct answer and the remaining selected responses serve as incorrect alternatives)).
Regarding claim 18, the combination of Berglund, Khan, and Attali discloses the non-transitory machine-readable medium of claim 15.
the operations comprising:
defining the format tag to describe an expected response format for the question and answer content in the multiple choice format (Khan, P[0086]: "The raw knowledge assessment item may include, in various embodiments, a question stem, answer choices, diagrams, and/or passages or other material provided in conjunction with the item. The raw knowledge assessment item may be provided in text format including formatting. " (the expected generated item structure includes a question stem and formatted answer choices, corresponding to the claimed multiple choice response format)); and
discarding outputs by the model that do not conform to the expected response format described by the format tag (Khan, P[0089]: "do not match the specifications for knowledge assessment items, the NLG interface 110 executes block 510 and discards or re-generates the raw knowledge assessment item." (the generated item is tested against the expected specifications and discarded or regenerated when it fails to conform)).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Berglund et al. (hereinafter Berglund) (US 20240403341 A1) in view of Khan et al. (hereinafter Khan) (US 20220415203 A1) and in further view of Glass et al. (hereinafter Glass) (US 20050060643 A1).
Regarding claim 10, the combination of Berglund and Khan disclose the method of claim 8.
The combination of Berglund and Khan does not explicitly disclose:
comprising:
selecting a first chunk to include a first portion of the document; and
selecting a second chunk to include a second portion of the document, wherein the second portion overlaps the first portion within a percentage of allowed overlapHowever, Glass discloses:
comprising:
selecting a first chunk to include a first portion of the document (Glass, see mapping below); and
selecting a second chunk to include a second portion of the document, wherein the second portion overlaps the first portion within a percentage of allowed overlap (Glass, P[0118]: "Interrelated chunk attributes include chunk boundary definitions, chunk size, including fixed or variable length, and chunk overlap, if any. One method of selecting document substrings or chunks is to extract all substrings of a fixed character length (n-grams) or a fixed number of words, sentences or paragraphs in length. The prior art suggests that accurately detecting sentences can be difficult. In some cases the substrings may be padded to make them all of equal length. These techniques may be configured to extract either overlapping or contiguous substrings" (teaches selecting multiple document portions as chunks and configuring successive chunks to overlap. Expressing the allowed overlap amount as a percentage would have been an obvious numerical implementation of the disclosed overlap attribute.)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing dating of the claimed invention to combine the teachings of Berglund, Khan, and Glass. Doing so would have provided the selection of document chunks having configurable chunk boundaries and overlapping portions provided by Glass (Glass, Abstract, P[0118]) with the ability to generate and present multiple choice assessment content of Khan (Khan, Abstract, P[0090]) with the retrieval-augmented LLM system of Berglund (Berglund, Abstract, P[0046]-P[0049]), thus, predictably reducing the loss of contextual information at chunk boundaries and predictably improving retrieval quality for subsequent question generation.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Berglund et al. (hereinafter Berglund) (US 20240403341 A1) in view of Khan et al. (hereinafter Khan) (US 20220415203 A1) and in further view of Mitalski et al. (hereinafter Mitalski) (US 20140106331 A1).
Regarding claim 20, The combination of Berglund and Khan disclose the computing device of claim 19.
The combination of Berglund and Khan does not explicitly disclose:
the operations comprising:
populating the user interface with a question specified by the question and answer content;
populating the user interface with a plurality of answers specified by the question and answer content, wherein the plurality of answers includes a correct answer and one or more incorrect answers; and
in response to a user selecting an answer from the plurality of answers through the user interface, displaying the correct answer and a reason that the correct answer is correct.
However, Mitalski discloses:
the operations comprising:
populating the user interface with a question specified by the question and answer content (Mitalski, P[0078]: "a sample question 522 is illustrated in screenshot 520" (the question is populated and displayed through the disclosed graphical user interface));
populating the user interface with a plurality of answers specified by the question and answer content (Mitalski, P[0078]: "a question with a plurality of answers, represented by choices (a)-(e)" (the interface presents the question together in multiple answer choices)), wherein the plurality of answers includes a correct answer and one or more incorrect answers (Mitalski, P[0051]: "Field 124 allows the instructor to specify the correct answer", P[0078]: "The user may select what he or she considers the correct answer" (the answer set necessarily includes a predetermined correct answer and alternatives that are incorrect)); and
in response to a user selecting an answer from the plurality of answers through the user interface (Mitalski, P[0078]: "The user may select what he or she considers the correct answer by selecting the circle next to answer", "check the answer by selecting the check mark" (the user selects an answer and submits or checks that selection through the interface)), displaying the correct answer and a reason that the correct answer is correct (Mitalski, P[0078]: "a detailed explanation may be provided to the user explaining whether the user selected the correct answer and providing pertinent information on the correct answer." (after the answer is selected and checked, the interface identifies correctness and displays explanatory information concerning the correct answer)).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing dating of the claimed invention to combine the teachings of Berglund, Khan, and Mitalski. Doing so would have provided the system with a presentation of answer choices to a user, reception of a selected answer, and display of correct answer with explanatory feedback after user selection of Mitalski (Mitalski, Abstract, P[0078]) with the ability to generate and present multiple choice assessment content of Khan (Khan, Abstract, P[0090]) with the retrieval-augmented LLM system of Berglund (Berglund, Abstract, P[0046]-P[0049]), thus, predictably improving the educational value and user engagement of the presented assessment content through explanatory feedback following user interaction.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHASHIDHAR S MANOHARAN whose telephone number is (571)272-6772. The examiner can normally be reached M-F 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHASHIDHAR SHANKAR MANOHARAN/ Examiner, Art Unit 2655
/ANDREW C FLANDERS/ Supervisory Patent Examiner, Art Unit 2655
Prosecution Insights