Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-14 rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claims do not fall within at least one of the four categories of patent eligible subject matter because claim 1 recites a “system” comprising only an “LLM engine” configured to perform software functions, without reciting a processor, memory, computer, circuitry, or other physical structure. Thus, the claimed LLM engine encompasses software per se and does not necessarily fall within a statutory category. The dependent claims (2-14) are likewise rejected under 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-8, 13-18, 22, 23 are rejected under 35 U.S.C. 103 as being unpatentable over Brennan (US 20180068221 A1), in view of Aberle (US 11748577 B1).
Regarding claim 1, Brennan discloses “accept as input one or more of a plurality of elements associated with an incorrect classification of an object by a supervised learning system;” (See [0043], [0044]; Brennan discloses using a NLP system to cluster elements with an incorrect classification of training objects by a supervised learning system)
“collate one or more of the plurality of elements using one or more LLMs to complete preparatory analysis on the object that has been incorrectly classified;” (See [0043], [0053]; Brennan discloses using a cognitive system, which uses a NLP model (simpler version of a LLM), to cluster the plurality elements on incorrectly classified objects, including false positive labels)
“present the document to a human analyst by inserting the document via a labeling interface to correct labeling and/or one or more models used to classify the object by the supervised learning system” (See [0037], [0039]; Brennan discloses presenting the document to a human analyst to correct labeling using a browser-based interface)
Brennan fails to explicitly disclose, “generate a document about the incorrectly classified object in accordance with the preparatory analysis of the one or more of the plurality of elements;”.
Aberle teaches “generate a document about the incorrectly classified object in accordance with the preparatory analysis of the one or more of the plurality of elements;” (See [Col 15, Lines 60-65]; Aberle discloses generating a response document based on given information).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Brennan and Aberle before them to modify Brennan to generate a document about the incorrectly classified object. One would be motivated to generate a document that identifies and explains the incorrectly classified object for a human analyst, see e.g., [Col 7, Lines 8-18] and [Col 15, Lines 60-65], where Aberle teaches that a document is generated for the purpose of identifying and marking natural language text.
Regarding claim 3, Brennan discloses “the incorrect classification is a either a false positive or a false negative classification of the object by the supervised learning system” (See [0043]; Brennan discloses that erroneous training examples (incorrect classifications) can be classified as false positives, and potentially as false negatives).
Regarding claim 4, Brennan discloses “the plurality of elements include the object that has been incorrect classified by the supervised learning system” (See [0043]; Brennan discloses incorrectly classified objects as part of the training examples (plurality of elements)).
Regarding claim 5, Brennan discloses “the plurality of elements further include a specific feedback associated with the object, wherein such feedback is used to determine the classification of the object” (See [0042]; Brennan discloses using computed entropy scores from objects as derived feedback from an automated action to determine the classification of an object).
Regarding claim 6, Brennan fails to explicitly disclose, “the plurality of elements further include a taxonomy of a set of criteria and/or the definitions of labels used for classifying the object”.
Aberle teaches “the plurality of elements further include a taxonomy of a set of criteria and/or the definitions of labels used for classifying the object.” (See [Col 20, Lines 4-9]; Aberle discloses using previously generated documents as a taxonomy of a set of criteria according to requirement specifications).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Brennan and Aberle before them to modify Brennan to use a taxonomy for object classification. One would be motivated to classify objects using a taxonomy to determine how objects should be classified, see e.g., [Col 20, Lines 4-9], where Aberle uses relevant content from a database of previously generated documents to identify similar content to objects.
Regarding claim 7, Brennan discloses “the LLM engine is configured to combine the incorrectly classified object with the taxonomy and/or the feedback associated with the object for preparatory analysis of misclassification of the object based on the one or more LLMs” (See [0043]; Brennan discloses using feedback associated with the object for preparatory analysis of the misclassification of the objects)
Regarding claim 8, Brennan discloses “each of the one or more LLMs is a type of artificial intelligence (AI) algorithm that uses deep learning techniques and large datasets to perform natural language processing (NLP) tasks by recognizing natural language content of the one or more of the plurality of elements.” (See [0053]; Brennan discloses using natural language processing (NLP) to analyze natural language content of the elements)
Regarding claim 13, Brennan discloses “the document includes the object with annotations explaining why the object is incorrectly classified and/or a suggestion on how to correct the classification.” (See [0043], [0044]; Brennan discloses annotating objects with labels for classification, and then discloses that objects that are annotated with incorrect classifications can be provided with verification suggestions for human evaluation)
Regarding claim 14, Brennan fails to explicitly disclose, “the document includes a synthesized report covering combination of the one or more of the plurality of elements using the one or more LLMs”.
Aberle teaches “the document includes a synthesized report covering combination of the one or more of the plurality of elements using the one or more LLMs” (See [Col 18, Lines 26-33]; Aberle discloses that a synthesized document includes a report of the plurality of content used for generating documents and the edits made by a human for classification).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Brennan and Aberle before them to modify Brennan to include a generated report that covers all the work done by the models. One would be motivated to generate a report to summarize all changes that were made during the process, see e.g., [Col 18, Lines 26-33], where Aberle teaches generating a document that includes the changes that were made during the process.
Regarding claim 17, Brennan discloses “the plurality of elements include the object that has been incorrect classified by the supervised learning system” (See [0043]; Brennan discloses incorrectly classified objects as part of the training examples (plurality of elements))
“the plurality of elements further include a specific feedback associated with the object, wherein such feedback is used to determine the classification of the object” (See [0042]; Brennan discloses using computed entropy scores from objects as derived feedback from an automated action to determine the classification of an object)
Brennan fails to explicitly disclose, “the plurality of elements further include a taxonomy of a set of criteria and/or the definitions of labels used for classifying the object”.
Aberle teaches “the plurality of elements further include a taxonomy of a set of criteria and/or the definitions of labels used for classifying the object.” (See [Col 20, Lines 4-9]; Aberle discloses using previously generated documents as a taxonomy of a set of criteria according to requirement specifications).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Brennan and Aberle before them to modify Brennan to use a taxonomy for object classification. One would be motivated to classify objects using a taxonomy to determine how objects should be classified, see e.g., [Col 20, Lines 4-9], where Aberle uses relevant content from a database of previously generated documents to identify similar content to objects.
Regarding claim 22, Brennan discloses “the document includes the object with annotations explaining why the object is incorrectly classified and/or a suggestion on how to correct the classification.” (See [0043], [0044]; Brennan discloses annotating objects with labels for classification, and then discloses that objects that are annotated with incorrect classifications can be provided with verification suggestions for human evaluation)
Brennan fails to explicitly disclose, “the document includes a synthesized report covering combination of the one or more of the plurality of elements using the one or more LLMs”.
Aberle teaches “the document includes a synthesized report covering combination of the one or more of the plurality of elements using the one or more LLMs” (See [Col 18, Lines 26-33]; Aberle discloses that a synthesized document includes a report of the plurality of content used for generating documents and the edits made by a human for classification).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Brennan and Aberle before them to modify Brennan to include a generated report that covers all the work done by the models. One would be motivated to generate a report to summarize all changes that were made during the process, see e.g., [Col 18, Lines 26-33], where Aberle teaches generating a document that includes the changes that were made during the process.
Regarding claims 15 and 23, these claims are similar in scope to claim 1.
Regarding claim 16, this claim is similar in scope to claim 3.
Regarding claim 18, this claim is similar in scope to claim 7.
Claim Rejections - 35 USC § 103
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Brennan (US 20180068221 A1), in view of Aberle (US 11748577 B1), and further in view of Beaver (US 20220374609 A1).
Regarding claim 2, Brennan-Aberle fails to explicitly disclose, “the object is a an electronic message or a line in a log file”.
Beaver teaches “the object is a an electronic message or a line in a log file” (See [0023], [0024]; Beaver discloses that the data sources used for the object come from conversation logs).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Brennan-Aberle and Beaver before them to modify Brennan-Aberle to clarify that the object is an electronic message or line in a log file. One would be motivated to define where the objects can be obtained from, see e.g., [0024], where Beaver teaches that the source for data can be messages from conversation logs and live chat logs.
Claim Rejections - 35 USC § 103
Claims 9, 10, 11, 19, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Brennan (US 20180068221 A1), in view of Aberle (US 11748577 B1), and further in view of Ouyang (US 20230296923 A1).
Regarding claim 9, Brennan discloses “the LLM engine is configured to utilize one or more… models to collate the one or more of the plurality of elements.” (See [0053]; Brennan discloses collating the plurality of elements using a NLP system and neural network language model)
Brennan-Aberle fails to explicitly disclose, “one or more multimodal models”.
Ouyang teaches “one or more multimodal models” (See [0003], [0004]; Ouyang discloses using multimodal models using artificial intelligence algorithms, such as machine learning, to interpret the multimodal data).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Brennan-Aberle and Ouyang before them to modify Brennan-Aberle to use multimodal models. One would be motivated to collate elements using multimodal models to use various types of data for machine learning, see e.g., [0004], where Ouyang teaches using data from different sensors for more precise.
Regarding claim 10, Brennan-Aberle fails to explicitly disclose, “each of the one or more multimodal models is an ML model that includes one or more neural networks each specialized in analyzing a particular modality”.
Ouyang teaches “each of the one or more multimodal models is an ML model that includes one or more neural networks each specialized in analyzing a particular modality.” (See [0004]; Ouyang discloses that the multimodal models used can be modeled using machine learning (ML) models for analyzing modal data).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Brennan-Aberle and Ouyang before them to modify Brennan-Aberle to specify a multimodal model for analyzing a particular modality. One would be motivated to allocate a multimodal model to a certain modality for the purpose of modeling that specific modality’s data, see e.g., [0004], where Ouyang teaches using a machine learning algorithm for modeling multimodal data.
Regarding claim 11, Brennan discloses “each of the one or more… models processes information from one of a plurality of sources to understand content of the one or more of the plurality of elements and unlock insights into the incorrect classification of the object.” (See [0043]; Brennan discloses using information associated with the object for analysis of the incorrect classification of the objects)
Brennan-Aberle fails to explicitly disclose, “the one or more multimodal models”.
Ouyang teaches “the one or more multimodal models” (See [0003], [0004]; Ouyang discloses that processing multimodal information helps with understanding how different forms of data can impact machine learning).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Brennan-Aberle and Ouyang before them to modify Brennan-Aberle to use multimodal models with a plurality of sources. One would be motivated to take other forms of data into consideration for understanding incorrect classifications using a multimodal model, see e.g., [0004], where Ouyang describes using multimodal data to obtain a better understanding of the environment that the data is obtained from.
Regarding claim 19, this claim is similar in scope to claim 10.
Regarding claim 20, this claim is similar in scope to claim 11.
Claim Rejections - 35 USC § 103
Claims 12, 21 are rejected under 35 U.S.C. 103 as being unpatentable over Brennan (US 20180068221 A1), in view of Aberle (US 11748577 B1), and further in view of Agarwal (US 20240005640 A1).
Regarding claim 12, Brennan-Aberle fails to explicitly disclose, “the document is a HTML document code-generated by the one or more LLMs”.
Agarwal teaches “the document is a HTML document code-generated by the one or more LLMs.” (See [0113]; Agarwal discloses that a synthesized document can be generated as a HTML document).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Brennan-Aberle and Agarwal before them to modify Brennan-Aberle to generate a document as a HTML document. One would be motivated to generate a document as a HTML format as one of multiple different formats depending on the required format, see e.g., [0113], where Agarwal teaches using multiple different formats to generate a document for different use cases based on what format a machine learning model may accept.
Regarding claim 21, this claim is similar in scope to claim 12.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID KIM whose telephone number is (571)272-4331. The examiner can normally be reached 7:30 AM - 4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.K./Examiner, Art Unit 2141
/TAN H TRAN/Primary Examiner, Art Unit 2141
Prosecution Insights