Prosecution Insights
Last updated: October 02, 2026
Application No. 18/584,344

TRAINING OF LARGE LANGUAGE MODELS USING AUTOMATED REFERENCE AUGMENTATION

Non-Final OA §101§103§112
Filed
Feb 22, 2024
Examiner
QAYYUM, ZESHAN
Art Unit
Tech Center
Assignee
ServiceNow Inc.
OA Round
1 (Non-Final)
40%
Grant Probability
Moderate
1-2
OA Rounds
2y 7m
Est. Remaining
71%
With Interview

Examiner Intelligence

Grants 40% of resolved cases
40%
Career Allowance Rate
180 granted / 444 resolved
-19.5% vs TC avg
Strong +31% interview lift
Without
With
+30.8%
Interview Lift
resolved cases with interview
Typical timeline
5y 2m
Avg Prosecution
25 currently pending
Career history
475
Total Applications
across all art units

Statute-Specific Performance

§101
26.1%
-13.9% vs TC avg
§103
34.0%
-6.0% vs TC avg
§102
6.9%
-33.1% vs TC avg
§112
30.5%
-9.5% vs TC avg
Black line = Tech Center average estimate • Based on career data from 444 resolved cases

Office Action

§101 §103 §112
DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Status of Claims Claims 1-20 have been examined. Claim Objections Claims 12-19 are objected to because of the following informalities: Claim 12 recites “A system, comprising: a processor configured to…” The claim is silent with respect to any memory storing instructions and executed by the processor to perform the claimed functions. Examiner suggested to write the claim as: “A system, comprising a processor and a memory, wherein the memory storing computer executable instruction that when executed by the processor causes the processor to perform the functions of….”. Appropriate correction is required. Claim Rejections - 35 USC § 101 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more. In the instant case, claims 1-11 are directed to a method, claims 12-19 are directed to a system comprising a processor and claim 20 is directed to a computer program product embodied in a non-transitory computer readable medium. Therefore, these claims fall within the four statutory categories of invention. The claims are directed to collecting, analyzing and outputting the data which is an abstract idea. Specifically, the claims recite “receiving training data item….; segmenting the source content….; identifying at least one entailment pair…; annotating the training data…; and providing the annotated training data…” which is grouped within the “mental process” grouping of abstract ideas in prong one of step 2A of the Alice/Mayo test (See MPEP 2106) because the claims involve a series of steps of receiving a piece of text (i.e. source) and a piece of text responding to it, breaking each piece of text into smaller pieces, finding a pair of pieces, one from the response and one from the source, where the source piece support the response piece, labeling the data with that pairing and sending the labeled data which is a process that deals with mental process where human annotators manually reading passages and labeling entailment relationships. Accordingly, the claims recite an abstract idea (See MPEP 2106). This judicial exception is not integrated into a practical application because, when analyzed under prong two of step 2A of the Alice/Mayo test (See MPEP 2106), the additional elements of the claims such as, a large language model, processor, non- transitory computer readable medium merely use a computer as a tool to perform an abstract idea. Specifically, a large language model, processor, non- transitory computer readable medium perform the steps of receiving a piece of text (i.e. source) and a piece of text responding to it, breaking each piece of text into smaller pieces, finding a pair of pieces, one from the response and one from the source, where the source piece support the response piece, labeling the data with that pairing and sending the labeled data. The use of a processor/computer as a tool to implement the abstract idea does not integrate the abstract idea into a practical application because it requires no more than a computer performing functions that correspond to acts required to carry out the abstract idea. The additional elements do not involve improvements to the functioning of a computer, or to any other technology or technical field (MPEP 2106.05(a)), the claims do not apply or use the abstract idea to effect a particular treatment or prophylaxis for a disease or medical condition (Vanda Memo), the claims do not apply the abstract idea with, or by use of, a particular machine (MPEP 2106.05(b)), the claims do not effect a transformation or reduction of a particular article to a different state or thing (MPEP 2106.05(c)), and the claims do not apply or use the abstract idea in some other meaningful way beyond generally linking the use of the abstract idea to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception (MPEP 2106.05(e) and Vanda Memo). Therefore, the claims do not, for example, purport to improve the functioning of a computer. Nor do they effect an improvement in any other technology or technical field. Accordingly, the additional elements do not impose any meaningful limits on practicing the abstract idea, and the claims are directed to an abstract idea. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when analyzed under step 2B of the Alice/Mayo test (See MPEP 2106), the additional elements of a large language model, processor, non- transitory computer readable medium, to perform the steps amounts to no more than using a computer or processor to automate and/or implement the abstract idea of collecting, analyzing and outputting the data. As discussed above, taking the claim elements separately, a large language model, processor, non- transitory computer readable medium perform the steps of receiving a piece of text (i.e. source) and a piece of text responding to it, breaking each piece of text into smaller pieces, finding a pair of pieces, one from the response and one from the source, where the source piece support the response piece, labeling the data with that pairing and sending the labeled data. These functions correspond to the actions required to perform the abstract idea. Viewed as a whole, the combination of elements recited in the claims merely recite the concept of collecting, analyzing and outputting the data. Therefore, the use of these additional elements does no more than employ the computer as a tool to automate and/or implement the abstract idea. The use of a computer or processor to merely automate and/or implement the abstract idea cannot provide significantly more than the abstract idea itself (MPEP 2106.05(I)(A)(f) & (h)). Therefore, the claim is not patent eligible. Dependent claims further describe the abstract idea of collecting, analyzing and outputting the data. Specifically, claims 2, 7, 13 and 18 further recite generate a response to new content and within that response, including a marker pointing back to the specific part of the source material the response relies on, which is part of the abstract idea. Claims 3 and 14 recite determination of entailment score which is part of the abstract idea. Claims 4-6 and 15-17 further describing the annotation, which is part of the abstract idea of collecting, analyzing and outputting the data. Claims 8-11 and 19 further describing duplication of claims 1 and 12 respectively, which is the abstract idea. The dependent claims do not include additional elements that integrate the abstract idea into a practical application or that provide significantly more than the abstract idea. Therefore, the dependent claims are also not patent eligible. Claim Rejections - 35 USC § 112 The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. Claim 20 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 20 recites “a computer program product embodied in a non-transitory computer-readable medium and comprising computer instructions for: receiving…; segmenting…; identifying…; annotating…; and providing…” The claim is unclear to one of the ordinary skills in art because it is unclear if the instructions functionally related to claimed non-transitory computer readable medium. Further instructions are not executed by any processor or device; therefore, these instructions are only stored and not functionally related to claimed non-transitory computer readable storage medium. (In re Zletz, 893 F.2d 319, 13USPQ2d 1320 (Fed. Cir. 1989), MPEP 2173.02 (III)(B)). Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over MORGOLIN (US20250238615) in view of Dingliwal (US 20250005298). With respect to claims 1, 12 and 20 MORGOLIN discloses: receiving a training data item including source content and a response associated with the source content (See paragraphs 0010, 0067 and 0090); segmenting the source content into a plurality of source content segments, and segmenting the response into a plurality of target segments (See paragraphs 0048, 0067 and 0073); annotating the training data item using the at least one entailment pair (See paragraph 0010, 0037, 0073 and 0093); providing the annotated training data item to a large language model (See paragraphs 0040, 0096). MORGOLIN does not explicitly disclose: identifying at least one entailment pair that includes a target segment included in the plurality of target segments and a source content segment included in the plurality of source content segments. Dingliwal discloses: identifying at least one entailment pair that includes a target segment included in the plurality of target segments and a source content segment included in the plurality of source content segments (See paragraph 0076). Therefore, it would have been obvious to one of the ordinary skill in the art at the time invention was filed to modify the MORGOLIN reference with Dingliwal reference in order to reduce probability if hallucinations in the output produced by LLM). With respect to claims 2 and 13 MORGOLIN in view of Dingliwal discloses all the limitations as described above. MORGOLIN further discloses: training the large language model using the annotated training data item. (See paragraphs 0040, 0096). With respect to claims 3 and 14 MORGOLIN in view of Dingliwal discloses all the limitations as described above. Dingliwal further discloses: determining an entailment score associated with the at least one entailment pair, wherein the entailment score comprises a numerical value indicating a degree of entailment; and determining the at least one entailment pair by at least comparing the entailment score to entailment scores associated with other entailment pairs and comparing the entailment score to a predetermined entailment score threshold.(See paragraph 0076). Therefore, it would have been obvious to one of the ordinary skill in the art at the time invention was filed to modify the MORGOLIN reference with Dingliwal reference in order to reduce probability if hallucinations in the output produced by LLM (Dingliwal ¶ 0019). With respect to claims 4 and 15 MORGOLIN in view of Dingliwal discloses all the limitations as described above. MORGOLIN further discloses: annotating the response corresponding to the source content by including a reference to the source content segment of the at least one entailment pair, wherein the reference is positioned adjacent to the target segment of the at least one entailment pair (See paragraphs 0010, 0037, 0073 and 0093). With respect to claims 5 and 16 MORGOLIN in view of Dingliwal discloses all the limitations as described above. MORGOLIN further discloses: wherein the reference is positioned at a beginning of the target segment of the at least one entailment pair. (See paragraphs 0010, 0037, 0073 and 0093). With respect to claims 6 and 17 MORGOLIN in view of Dingliwal discloses all the limitations as described above. MORGOLIN further discloses: wherein annotating the training data item comprises using one or more of the following: a line number, a timestamp, or a speaker label. (See paragraphs 0010, 0037, 0073 and 0093). With respect to claims 7 and 18 MORGOLIN in view of Dingliwal discloses all the limitations as described above. MORGOLIN further discloses: training the large language model to generate a new response for additional source content and generate at least one reference embedded in the new response, wherein the at least one reference refers to at least one portion of the additional source content. (See paragraphs 0040, 0055-0056, 0096). With respect to claims 8 and 19 MORGOLIN in view of Dingliwal discloses all the limitations as described above. MORGOLIN further discloses: receiving additional source content to summarize; annotating the additional source content to summarize; and providing the annotated additional source content to the large language model to generate an annotated response of the additional source content to summarize. (See paragraphs 0048, 0055-0056, 0067 and 0073). With respect to claim 9 MORGOLIN in view of Dingliwal discloses all the limitations as described above. MORGOLIN further discloses: wherein the source content comprises text content, and the method further comprising: segmenting the source content into the plurality of source content segments by segmenting the source content into one or more of the following: a plurality of sentences or a plurality of paragraphs. (See paragraphs 0010, 0037, 0073 and 0093). With respect to claim 10 MORGOLIN in view of Dingliwal discloses all the limitations as described above. MORGOLIN further discloses: wherein the source content comprises a chat transcript, and the method further comprising: segmenting the source content into the plurality of source content segments by segmenting the source content into a plurality of chat messages. (See paragraphs 0048, 0067 and 0073). With respect to claim 11 MORGOLIN in view of Dingliwal discloses all the limitations as described above. MORGOLIN further discloses: segmenting the source content into the plurality of source content segments based on a source content type. (See paragraphs 0048, 0067 and 0073). Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZESHAN QAYYUM whose telephone number is (571)270-3323. The examiner can normally be reached Monday-Friday 9:00AM-6:00PM EST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John W Hayes can be reached at (571) 272-6708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /ZESHAN QAYYUM/Primary Examiner, Art Unit 3697
Read full office action

Prosecution Timeline

Feb 22, 2024
Application Filed
May 08, 2024
Response after Non-Final Action
Sep 10, 2026
Non-Final Rejection mailed — §101, §103, §112 (current)

Precedent Cases

Applications granted by this same examiner with similar technology

Patent 12737737
SYSTEMS AND METHOD FOR AUTOMATIC TRANSACTION ROUTING AND EXECUTION
4y 5m to grant Granted Sep 15, 2026
Patent 12725148
Techniques For Securely Communicating Sensitive Data
5y 6m to grant Granted Sep 01, 2026
Patent 12711495
WEB3 TRANSFER PROTOCOL
3y 5m to grant Granted Aug 18, 2026
Patent 12711306
ELECTRONIC FORM AUTOMATION
2y 11m to grant Granted Aug 18, 2026
Patent 12705639
METHOD OF PROCESSING NON-FUNGIBLE TOKEN
3y 4m to grant Granted Aug 11, 2026
Study what changed to get past this examiner. Based on 5 most recent grants.

Strategy Recommendation AI-generated — please review before filing

Get a prosecution strategy drawn from examiner precedents, rejection analysis, and claim mapping.
Typically takes 5-10 seconds — AI-generated, attorney review required before filing

Prosecution Projections

1-2
Expected OA Rounds
40%
Grant Probability
71%
With Interview (+30.8%)
5y 2m (~2y 7m remaining)
Median Time to Grant
Low
PTA Risk
Based on 444 resolved cases by this examiner. Grant probability derived from career allowance rate.

Sign in with your work email

Enter your email to receive a magic link. No password needed.

Personal email addresses (Gmail, Yahoo, etc.) are not accepted.

Free tier: 3 strategy analyses per month