DETAILED ACTION
This action is in response to the initial filing of application no. 18/752,964 on 06/25/2024.
Claims 1- 20 are still pending in this application, with claims 1, 13 and 20 being independent.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 6, 13, 16 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mukherjee et al. (US 2024/0354436) (“Mukherjee”) in view of Java et al. (US 2025/0005048) (“Java”).
For claim 1, Mukherjee discloses a method of evaluating context-specific content generated by a generative artificial intelligence model (Abstract), comprising: obtaining user data that is specific to a user (Context data comprising conversation history, user role, user information, etc. is obtained from a context model, Fig.3B, 8, Fig.4, 408 and 410; [0066] [0092] [0114] [0115] [0128] [0129]) of a document search system ( [0072 – 0075]), the user data indicative of a contextual situation of the user (Conversation history of a user session provides a situational context in which a user’s query is received., [0047]); providing an initial prompt (first prompt) to the generative artificial intelligence model (LLM, Fig.1A, 130; [0059] [0060] [0072]) based on the user data (The first prompt is generated based on a user query and the context data comprising conversation history, user role, user information, etc. The first prompt is transmitted to the LLM., Fig.3B, 8 and 9, Fig.4, 412 and 414; [0117] [0118] [0130] [0131]), the initial prompt instructing the generative artificial intelligence model to automatically generate initial content that is specific to the contextual situation of the user (The prompt comprises instructions to an LLM and user context information. The user context information is used to by the LLM to understand the meaning of the instructions to generate an output/content. Therefore, the LLM is being instructed to generate an output/content based on the user context information., [0064] [0066] [0115]); generating feedback data on the initial content according to one or more quality metrics (Feedback provided by the system or a user indicates correctness or acceptability of the LLM’s output, [0049] [0139] [0140] [0157] [0160]); and performing one or more actions based on the feedback data (An updated prompt is generated and provided to the LLM to generate an output that fulfills a user’s expectation, [0141] [0160]).
Yet, Mukherjee fails to teach the following: the document search system is a software application.
However, Java discloses a system and method for the purpose of performing a document snippet search (Abstract), wherein a document search system (which comprises functionality to search a document based on a query, [0022]) is implemented as part of a software application (e.g. desktop, mobile or sever application comprising software) ([0022] [0067] [0070 - 0073]).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve Mukherjee’s invention in the same way that Java’s invention has been improved to achieve the predictable results of the document search system (which comprises functionality to search a document based on a user query) (Mukherjee, [0072] [0075]) is implemented as a part of a software application for the purpose of providing and utilizing computer based models to search a corpus of digital documents (Mukherjee, [0003]) since documents are no longer physical documents stored in the real world (Java, [0001]).
For claims 6 and 16, Mukherjee further discloses, wherein the initial content comprises a plurality of answers generated by the generative artificial intelligence model in response to a question included in the initial prompt (Mukherjee, Fig.9, 906 and 908; [0154]).
For claim 13, Mukherjee discloses a system for evaluating context-specific content generated by a generative artificial intelligence model (Abstract), comprising: a memory (Fig.12, 1206) including computer executable instructions ([0175]); and a processor (Fig.12, 1204) configured to execute the computer executable instructions and cause the system ([0174] [0175]) to: obtain user data that is specific to a user (Context data comprising conversation history, user role, user information, etc. is obtained from a context model, Fig.3B, 8, Fig.4, 408 and 410; [0066] [0092] [0114] [0115] [0128] [0129]) of a document search system ( [0072 – 0075]), the user data indicative of a contextual situation of the user (Conversation history of a user session provides a situational context in which a user’s query is received., [0047]); provide an initial prompt (first prompt) to the generative artificial intelligence model (LLM, Fig.1A, 130; [0059] [0060] [0072]) based on the user data (The first prompt is generated based on a user query and the context data comprising conversation history, user role, user information, etc. The first prompt is transmitted to the LLM., Fig.3B, 8 and 9, Fig.4, 412 and 414; [0117] [0118] [0130] [0131]), the initial prompt instructing the generative artificial intelligence model to automatically generate initial content that is specific to the contextual situation of the user (The prompt comprises instructions to an LLM and user context information. The user context information is used to by the LLM to understand the meaning of the instructions to generate an output/content. Therefore, the LLM is being instructed to generate an output/content based on the user context information., [0064] [0066] [0115]); generate feedback data on the initial content according to one or more quality metrics (Feedback provided by the system or a user indicates correctness or acceptability of the LLM’s output, [0049] [0139] [0140] [0157] [0160]); and perform.one or more actions based on the feedback data (An updated prompt is generated and provided to the LLM to generate an output that fulfills a user’s expectation, [0141] [0160]).
Yet, Mukherjee fails to teach the following: the document search system is a software application.
However, Java discloses a system and method for the purpose of performing a document snippet search (Abstract), wherein a document search system (which comprises functionality to search a document based on a query, [0022]) is implemented as part of a software application (e.g. desktop, mobile or sever application comprising software) ([0022] [0067] [0070 - 0073]).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve Mukherjee’s invention in the same way that Java’s invention has been improved to achieve the predictable results of the document search system (which comprises functionality to search a document based on a user query) (Mukherjee, [0072] [0075]) is implemented as a part of a software application for the purpose of providing and utilizing computer based models to search a corpus of digital documents (Mukherjee, [0003]) since documents are no longer physical documents stored in the real world (Java, [0001]).
For claim 20, Mukherjee discloses a non-transitory computer-readable medium comprising instructions to be executed in a computer system to evaluate context-specific content generated by a generative artificial intelligence model (Abstract; Fig.12, ,1206; [0174] [0175]), wherein the instructions when executed in the computer system cause the computer system ([0175]) to: obtain user data that is specific to a user (Context data comprising conversation history, user role, user information, etc. is obtained from a context model, Fig.3B, 8, Fig.4, 408 and 410; [0066] [0092] [0114] [0115] [0128] [0129]) of a document search system ( [0072 – 0075]), the user data indicative of a contextual situation of the user (Conversation history of a user session provides a situational context in which a user’s query is received., [0047]); provide an initial prompt (first prompt) to the generative artificial intelligence model (LLM, Fig.1A, 130; [0059] [0060] [0072]) based on the user data (The first prompt is generated based on a user query and the context data comprising conversation history, user role, user information, etc. The first prompt is transmitted to the LLM., Fig.3B, 8 and 9, Fig.4, 412 and 414; [0117] [0118] [0130] [0131]), the initial prompt instructing the generative artificial intelligence model to automatically generate initial content that is specific to the contextual situation of the user (The prompt comprises instructions to an LLM and user context information. The user context information is used to by the LLM to understand the meaning of the instructions to generate an output/content. Therefore, the LLM is being instructed to generate an output/content based on the user context information., [0064] [0066] [0115]); generate feedback data on the initial content according to one or more quality metrics (Feedback provided by the system or a user indicates correctness or acceptability of the LLM’s output, [0049] [0139] [0140] [0157] [0160]); and perform.one or more actions based on the feedback data (An updated prompt is generated and provided to the LLM to generate an output that fulfills a user’s expectation, [0141] [0160]).
Yet, Mukherjee fails to teach the following: the document search system is a software application.
However, Java discloses a system and method for the purpose of performing a document snippet search (Abstract), wherein a document search system (which comprises functionality to search a document based on a query, [0022]) is implemented as part of a software application (e.g. desktop, mobile or sever application comprising software) ([0022] [0067] [0070 - 0073]).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve Mukherjee’s invention in the same way that Java’s invention has been improved to achieve the predictable results of the document search system (which comprises functionality to search a document based on a user query) (Mukherjee, [0072] [0075]) is implemented as a part of a software application for the purpose of providing and utilizing computer based models to search a corpus of digital documents (Mukherjee, [0003]) since documents are no longer physical documents stored in the real world (Java, [0001]).
Claim(s) 2, 3, 9, 10, 11, 14, 17 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mukherjee et al. (US 2024/0354436) (“Mukherjee”) in view of Java et al. (US 2025/0005048) (“Java”) and further in view of Peng et al. (US 2024/0362418) (“Peng”).
For claims 2 and 14, the combination of Mukherjee and Java further discloses
wherein the one or more actions comprise: modifying the initial prompt provided to the generative artificial intelligence model based, at least in part, on the feedback data to generate a modified prompt (Mukherjee, [0049]).
Yet, the combination of Mukherjee and Java fails to teach the following: providing the modified prompt to the generative artificial intelligence model; and obtaining updated content from the generative artificial intelligence model, the updated content being improved compared to the initial content according to the one or more quality metrics.
However, Peng discloses a system and method for interacting with a language model (Abstract), comprising the following: modifying a prompt provided to a generative artificial intelligence model ([0022] [0028 – 0030]) based on feedback data (Fig.2, 210, 214, 222, Fig.12, 1212, 1214, Fig.13, 1302; [0037] [0039] [0040] [0042 – 0046] [0117]); providing the modified prompt the generative artificial intelligence model (Fig.2, 212 and Fig.13, 1304; [0117]); and obtaining updated content from the generative artificial intelligence model (Fig.2, 212 and Fig.13, 1306; [0042] [0117]), the updated content being improved compared to the initial content according to the one or more quality metrics (The revised response satisfies the usefulness measure and/or the user deems the response sufficient, [0043 – 0045] [0091 – 0100] [0117]).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Mukherjee and Java in the same way that Peng’s invention has been improved to achieve the following, predictable results for the purpose of increasing user satisfaction by providing content which fulfills a user’s expectation (Mukherjee, [0141]): further providing the modified prompt to the generative artificial intelligence model; and further obtaining updated content from the generative artificial intelligence model, the updated content being improved compared to the initial content according to the one or more quality metrics.
For claim 3, Mukherjee and Peng further disclose, wherein modifying the initial prompt includes: adding information to the initial prompt based on the feedback data (Mukherjee, [0141]) (Peng, The feedback information is added to the initial input information., [0024] [0037] [0039] [0040] [0044] [0117]); or removing information from the initial prompt based on the feedback data.
For claim 9, the combination of Mukherjee and Peng fails to teach, wherein generating the feedback data comprises: providing the initial content and at least a portion of the initial prompt to an additional generative artificial intelligence model trained to determine quality of the initial content.
However, Peng discloses a system and method for interacting with a language model (Abstract), comprising the following: providing the initial content and at least a portion of the initial prompt (knowledge information retrieved from a KAC, [0035 – 0037]) to an additional generative artificial intelligence model trained to determine quality of the initial content (A utility system comprises a scoring component that computes a usefulness score for a response generated by a language model. The scoring component uses a machine-trained model to compute the usefulness measure, [0042] [0043] [0091 – 0099]).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Mukherjee and Java in the same way that Peng’s invention has been improved to achieve the following, predictable results for the purpose of increasing user satisfaction by providing content which fulfills a user’s expectation (Mukherjee, [0141]), wherein generating the feedback data further comprise automatically generating feedback data (Muhkerjee, [0049]) by: providing the initial content and at least a portion of the initial prompt (Mukherjee, documents/knowledge information, [0114 – 0117]) to an additional generative artificial intelligence model trained to determine quality of the initial content.
For claims 10 and 17, the combination of Mukherjee and Peng fails to teach, wherein generating the feedback data comprises: providing the initial content to an additional generative artificial intelligence model configured to evaluate the initial content; determining whether additional evaluation of the initial content is needed based, at least in part, on a confidence score output by the additional generative artificial intelligence model and associated with the initial content.
However, Peng discloses a system and method for interacting with a language model (Abstract), comprising the following: providing the initial content to an additional generative artificial intelligence model configured to evaluate the initial content (A utility system comprises a scoring component that computes a usefulness score for a response generated by a language model. The scoring component uses a machine-trained model to compute the usefulness measure, [0042] [0043] [0091 – 0099]); and determining whether additional evaluation of the initial content is needed based, at least in part, on a confidence score output by the additional generative artificial intelligence model and associated with the initial content (In an alternative manner of operation, the RAS 104 uses the user interface component 118 to inform the user whenever a response generated by the language model 106 is deemed deficient based on analysis performed by the utility system 138. A response is deemed deficient if a usefulness measure does not satisfy a threshold value, [0043] [0044] [0091 – 0094]).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Mukherjee and Java in the same way that Peng’s invention has been improved to achieve the following, predictable results for the purpose of increasing user satisfaction by providing content which fulfills a user’s expectation (Mukherjee, [0141]), wherein generating the feedback data further comprise automatically generating feedback data (Muhkerjee, [0049]) by: providing the initial content to an additional generative artificial intelligence model configured to evaluate the initial content; determining whether additional evaluation of the initial content is needed based, at least in part, on a confidence score output by the additional generative artificial intelligence model and associated with the initial content.
For claims 11 and 18, Mukherjee and Peng further disclose, wherein determining whether additional evaluation of the initial content is needed comprises: determining whether the confidence score output by the additional generative artificial intelligence model exceeds a threshold confidence score (Mukherjee, [0141]) (Peng, In an alternative manner of operation, the RAS 104 uses the user interface component 118 to inform the user whenever a response generated by the language model 106 is deemed deficient based on analysis performed by the utility system 138. A response is deemed deficient if a usefulness measure does not satisfy a threshold value, [0043] [0044] [0091 – 0094]); and in response to determining the confidence score does not exceed the threshold confidence score, providing the initial content for additional evaluation e (Mukherjee, [0141]) (Peng, In an alternative manner of operation, the RAS 104 uses the user interface component 118 to inform the user whenever a response generated by the language model 106 is deemed deficient based on analysis performed by the utility system 138. A response is deemed deficient if a usefulness measure does not satisfy a threshold value, [0043] [0044] [0091 – 0094]).
Claim(s) 4, 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mukherjee et al. (US 2024/0354436) (“Mukherjee”) in view of Java et al. (US 2025/0005048) (“Java”), and further in view of Leslie et al. (US 2025/0061116) (“Leslie”) and further in view of Fox et al. (US 2021/0174016) (“Fox”).
For claims 4 and 15, the combination of Mukherjee and Java further discloses the following: providing a user interface displaying at least a portion of the initial prompt, the user interface further displaying the initial content generated by the generative artificial intelligence model (Mukherjee, Fig.11, 1100 and 1130; [0157] [0158] ); and receiving feedback data comprising user input with respect to the initial content from an expert (Mukherjee, A user is broadly interpreted to include all human beings including both novices and experts, [0081]) via one or more user interface elements of the user interface (Mukherjee, [0160]).
Yet, the combination of Mukherjee and Java fails to tech the following: generating a file comprising at least a portion of the user data and the initial content generated by the generative artificial intelligence model; and updating the file with the feedback data.
However, Leslie discloses a system and method for generating nature language responses to user queries (Abstract), comprising the following: a database is generated comprising at least a portion of user data (user information) and content (model’s response) generated by a generative artificial intelligence model ([0043 – 0047] [0125 - 0129] [0132 – 0135] [0150]); and the database is updated with feedback (user ratings) ([0114] [0151 - 0152]).
Additionally, Fox discloses a system and method for the purpose of generating questions and answers (Abstract), wherein question and answer data is stored in either a file or a database ([0079])
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Mukherjee and Java in the same way that Leslie’s invention has been improved to achieve the following, predictable results for the purpose of effectively assessing the performance of a generative model to improve response generation (Leslie, [0150- 0152]): further generating a database comprising at least a portion of the user data and the initial content generated by the generative artificial intelligence model; and further updating the database with the feedback data.
Furthermore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Mukherjee, Java and Leslie in the same way that Fox’s invention has been improved to achieve the following, predictable results for the purpose of effectively assessing the performance of a generative model to improve response generation (Leslie, [0150- 0152]): the user data, initial content and feedback data (question and answer data) are stored in either a file of database.
For claim 5, Mukherjee, Leslie and Fox further disclose that the file comprises a comma separated value (CSV) file (Mukherjee, [0046] [0047] [0049] [0066] [0130] [0132]) (Leslie, [0150 – 0152])(Fox, [0079]).
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mukherjee et al. (US 2024/0354436) (“Mukherjee”) in view of Java et al. (US 2025/0005048) (“Java”), and further in view of Bodegas Martinez et al. (US 2019/0303610) (“Bodegas”) and further in view of Sharma et al. (US 2022/0129369) (“Sharma”).
For claim 7, the combination of Mukherjee and Java fails to teach, wherein obtaining the user data comprises: obtaining data for the user based, at least in part, on a unique identifier for the user, the data comprising personal information about the user; removing or anonymizing the personal information to generate anonymized data; and generating a test account based on the anonymized data.
However, Bodegas discloses a system and method for on-demand de-identification of data in a computer storage system (Abstract), comprising the following: obtaining data for a user based, at least on part on a unique identifier for the user ([0054] and claim 1), the data comprising personal information about the user (name and phone number, Fig.2A, 115); and removing or anonymizing the personal information to generate anonymized data (Fig.2C, Retention Table; [0054] and claim 1).
Additionally, Sharma discloses a system and method for generating test accounts(Abstract), wherein the test accounts are generated based on anonymized data ([0022 -0025]).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Mukherjee and Java in the same way that Bodegas’s invention has been improved to achieve the following, predictable results for the purpose of cost-effectively storing user data in accordance with data privacy laws and requirements (Bodegas, [0002 – 0005]): further obtaining data for the user based, at least in part, on a unique identifier for the user, the data comprising personal information about the user; and removing or anonymizing the personal information to generate anonymized data.
Additionally, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Mukherjee, Java and Bodegas in the same way that Sharma’s invention has been improved to achieve the following, predictable results for the purpose of improving the accuracy and reliability of the system by further testing the system using user data which satisfies privacy requirements (Sharma, [0001 – 0004]): further generating a test account based on the anonymized data.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mukherjee et al. (US 2024/0354436) (“Mukherjee”) in view of Java et al. (US 2025/0005048) (“Java”), and further in view of Bodegas Martinez et al. (US 2019/0303610) (“Bodegas”), and further in view of Sharma et al. (US 2022/0129369) (“Sharma”) and further in view of Ganesan et al. (US 10,698,794) (“Ganesan”).
For claim 8, the combination of Mukherjee, Java, Bodegas and Sharma fails to teach the following: providing an initial prompt to the generative artificial intelligence model comprises: obtaining credentials associated with the test account; accessing the test account via the credentials to establish a session associated with the test account; in response to establishing the session, obtaining the anonymized data; and generating the initial prompt based, at least in part, on the anonymized data.
However, Ganesan discloses a system and method for servicing application requests (Abstract), comprising the following: obtaining credentials associated with a test account (column 3 lines 32 – 50; column 8 lines 15 – 26; column 12 lines 7 -22); accessing the test account via the credentials to establish a session associated with the test account (column 8 lines 35 – column 9 line 5; column 12 lines 36 - 42); and in response to establish the session, obtaining test data and executing the software application with the test data (column 5 lines 10 -25; column 9 lines 17 – 40; column 12 lines 43 – column 13 line 15).
Therefore, it would have been obvious to one of ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Mukherjee, Java, Bodegas and Sharma in the same way that Ganesan’s invention has been improved to achieve the following, predictable results for the purpose of improving the accuracy and reliability of the system by anonymously testing the system (Sharma, [0001 – 0004]) (Ganesan, column 3 lines 40 – 50), wherein providing an initial prompt to the generative artificial intelligence model further comprises: obtaining credentials associated with the test account; accessing the test account via the credentials to establish a session associated with the test account; in response to establishing the session, obtaining test data, e.g. anonymized data; and executing the software application by generating the initial prompt based, at least in part, on the test data, e.g. anonymized data.
Claim(s) 12 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mukherjee et al. (US 2024/0354436) (“Mukherjee”) in view of Java et al. (US 2025/0005048) (“Java”), and further in view of Peng et al. (US 2024/0362418) (“Peng”) and further in view of Zhang et al. (US 2025/0190449) (“Zhang”).
For claims 12 and 19, the combination of Mukherjee, Java and Peng further discloses, wherein providing the initial content for additional evaluation comprises generating a user interface displaying the initial content (Mukherjee, [0141]) (Peng, [0043] [0045]), the user interface comprising one or more user interface elements configured to receive input from one or more user’s (Mukherjee, [0141])(Peng, [0032] [0033 [0043] [0045]), the input indicative of the one or more user’s evaluation of the initial content (Mukherjee, [0141])(Peng, User [0032] [0033 [0043] [0045]).
Yet, the combination of Mukherjee, Java and Peng fails to teach that the users are experts.
However, Zhang discloses a system and method for the purpose of supporting generative artificial intelligence assisted analytics of structured data sets (Abstract), wherein either users or experts provide feedback on answers generated by a generative artificial intelligence model ([0056] [0091 – 0094]).
Therefore, it would have been obvious to one or ordinary skill in the art at the time of applicant’s filing to improve the invention disclosed by the combination of Mukherjee, Java and Peng in the same way that Zhang’s invention has been improved to achieve the following, predictable results for the purpose of improving the accuracy of responses generated by a generative artificial intelligence model using feedback data (Zhang, [0091]): the evaluation of the initial content is further provided by users or experts.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SONIA L GAY whose telephone number is (571)270-1951. The examiner can normally be reached Monday-Friday 9-5 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at 571-272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SONIA L GAY/Primary Examiner, Art Unit 2657