DETAILED ACTION
This communication is in response to the Amendments and Arguments filed on 05/15/2026.
Claim(s) 1-20 are pending and have been examined. Hence, this action has been made FINAL.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments and Amendments
Amendments to the claims by the Applicant have been considered and addressed below.
With respect to the 35 USC § 112, 101, 102, and 103 rejections, the Applicant provides several arguments in which the Examiner will respond accordingly, below.
35 USC § 112 rejection(s)
Arguments on page 5 of the Remarks filed on 05/15/2026
Examiner’s Response to Arguments:
Applicant’s arguments with respect to 35 U.S.C. § 112 have been fully considered and are persuasive. The 35 U.S.C. § 112 of claim 6 has been withdrawn.
35 USC § 101 rejection(s)
Arguments on page 5-7 of the Remarks filed on 05/15/2026
Examiner response to Arguments:
Applicant’s arguments, with respect to the rejection(s) of independent claim(s) 1, 9, and 16 under 35 USC 101 have been fully considered but are not persuasive.
The Applicant argues that:
The claims, rather, provide a technical process that improves the functioning in a specific technical field, namely, minimizing positional bias in large language models (LLMs).
As described in the specification, "[p]ermutation self-consistency may improve the quality, consistency, and prompt-order invariance of a blackbox LLM." […] These steps work together to reduce the effects of positional bias inherent in LLMs by ensuring that "[e]ach permuted LLM input prompt may experience any positional bias differently because each permuted list may provide the list in a different order."
See e.g., the Office Action at 4. MPEP § 2106.04(d)(1) provides that a claim reciting a judicial exception is not directed to the judicial exception if it also recites additional elements demonstrating that the claim as a whole integrates the exception into a practical application," and that "one way to demonstrate such integration is when the claimed invention improves the functioning of a computer or improves another technology or technical field."
… in SRI Int'l, Inc. v. Cisco Systems, Inc., 930 F.3d 1295, 1303 (Fed. Cir. 2019), "claims to detecting suspicious activity by using network monitors and analyzing network packets were found to be an improvement in computer network technology and not directed to an abstract idea." MPEP 2106.04(d)(1). Similarly, here, the present claims recite specific technical steps that improve LLM technology by minimizing positional bias through permutation self-consistency.
… the claimed steps cannot be practically performed in the human mind. Generating a plurality of LLM outputs from a plurality of LLM inputs and determining a final LLM output based on aggregating those outputs requires the use of an LLM, a machine learning model that cannot be replicated by mental processes.
The Examiner respectfully disagrees with these arguments and notes:
Regarding the invention improving functioning in a technical field points 1-4, from above), the Examiner notes and refers the Applicant to the MPEP 2106.05(a):
“It is important to note that in order for a method claim to improve computer functionality, the broadest reasonable interpretation of the claim must be limited to computer implementation. That is, a claim whose entire scope can be performed mentally, cannot be said to improve computer technology. Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 120 USPQ2d 1473 (Fed. Cir. 2016) (a method of translating a logic circuit into a hardware component description of a logic circuit was found to be ineligible because the method did not employ a computer and a skilled artisan could perform all the steps mentally). Similarly, a claimed process covering embodiments that can be performed on a computer, as well as embodiments that can be practiced verbally or with a telephone, cannot improve computer technology. See RecogniCorp, LLC v. Nintendo Co., 855 F.3d 1322, 1328, 122 USPQ2d 1377, 1381 (Fed. Cir. 2017) (process for encoding/decoding facial data using image codes assigned to particular facial features held ineligible because the process did not require a computer).” (Emphasis added)
Also, regarding cited case law (i.e., SRI Int'l, Inc. v. Cisco Systems, Inc., 930 F.3d 1295, 1303 (Fed. Cir. 2019)), these are all deemed rooted in computer technology. However, the Instant Application does not comprise similar recitations (i.e., detecting suspicious activity by using network monitors and analyzing network packets).
For example, the Instant Application recites:
receiving, by a device, an original large language model (LLM) input prompt comprising a list of items in a first order, wherein the items, of the list of items, are listed;
generating a plurality of items, wherein for each list, of the plurality of the lists of the items, the items of the list are listed in orders different from the first order and each othe
generating a plurality of LLM outputs from a plurality of LLM inputs, each of the plurality of LLM inputs comprising one of the plurality of lists of the items;
determining a final LLM output based on aggregating the plurality of LLM outputs; and
causing a response to the original LLM input prompt using the final LLM output.
Therefore, the Instant Application is not rooted in computer technology, but rather on implementing an abstract idea in natural language processing, more specifically in the field of responding to large language model input prompts. More details on the rationale used to examine the claims rejected under 35 U.S.C. § 101 of the Instant Application are provided below for clarification.
Regarding the steps not being practically performed in the human mind, the Examiner notes that while the claims recite LLM-related limitations and additional elements, such as first and/or second devices, the claims do not recite language regarding features or processes a human is incapable of performing in the mind and/or with the assistance of pen and paper.
Please see detailed analysis below (Prong Two) for more details on how the Examiner understands the independent claims do not recite additional elements that integrate the judicial exception into a practical application. Hence, not qualifying as patent eligible subject matter under 35 U.S.C. § 101.
Please refer to MPEP 2106.04(II): Eligibility Step 2A: Whether a Claim is Directed to a Judicial Exception: (A) Step 2A is a Two-Prong Inquiry:
(1) Prong One:
Prong One asks does the claim recite an abstract idea, law of nature, or natural phenomenon? In Prong One examiners evaluate whether the claim recites a judicial exception, i.e. whether a law of nature, natural phenomenon, or abstract idea is set forth or described in the claim. While the terms "set forth" and "described" are thus both equated with "recite", their different language is intended to indicate that there are two ways in which an exception can be recited in a claim. For instance, the claims in Diehr, 450 U.S. at 178 n. 2, 179 n.5, 191-92, 209 USPQ at 4-5 (1981), clearly stated a mathematical equation in the repetitively calculating step, and the claims in Mayo, 566 U.S. 66, 75-77, 101 USPQ2d 1961, 1967-68 (2012), clearly stated laws of nature in the wherein clause, such that the claims "set forth" an identifiable judicial exception. Alternatively, the claims in Alice Corp., 573 U.S. at 218, 110 USPQ2d at 1982, described the concept of intermediated settlement without ever explicitly using the words "intermediated" or "settlement." […]
An example of a claim that recites a judicial exception is "A machine comprising elements that operate in accordance with F=ma." This claim sets forth the principle that force equals mass times acceleration (F=ma) and therefore recites a law of nature exception. Because F=ma represents a mathematical formula, the claim could alternatively be considered as reciting an abstract idea. Because this claim recites a judicial exception, it requires further analysis in Prong Two in order to answer the Step 2A inquiry. An example of a claim that merely involves, or is based on, an exception is a claim to "A teeter-totter comprising an elongated member pivotably attached to a base member, having seats and handles attached at opposing sides of the elongated member." This claim is based on the concept of a lever pivoting on a fulcrum, which involves the natural principles of mechanical advantage and the law of the lever. However, this claim does not recite these natural principles and therefore is not directed to a judicial exception (Step 2A: NO). Thus, the claim is eligible at Pathway B without further analysis.
From this analysis, in Step 2A, Prong One, the Examiner has evaluated the independent claims accordingly and determined that the amended independent claims as drafted indeed describe a judicial exception (i.e., an abstract idea), which represent a mental process (which can be performed by a human with pen and paper).
Similar to what was discussed in the Non-Final Rejection mailed on 12/17/2025, the limitations as drafted cover a human (mental process).
More specifically, the independent claim(s) recite(s):
1. (Currently Amended) A method comprising:
receiving, by a device, an original large language model (LLM) input prompt comprising a list of items in a first order, wherein the items, of the list of items, are listed;
generating a plurality of lists of the items, wherein for each list, of the plurality of the lists of the items, the items of the list are listed in orders different from the first order and each other;
generating a plurality of LLM outputs from a plurality LLM inputs, each of the plurality of LLM inputs comprising one of the plurality of lists of the items;
determining a final LLM output based on aggregating the plurality of LLM outputs; and
causing a response to the original LLM input prompt using the final LLM output.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving a request (e.g., written) with a list of items in a first order;
Re-ordering said items in a different order, wherein for each re-ordered list the items are ordered differently from the first order;
Using predetermined set of steps/rules (i.e., LLM) to generate or write down a response to the received requests comprising a list of items;
Using predetermined set of steps/rules (i.e., aggregating (e.g., mathematical concept)) to select a final response;
Writing down said response.
9. (Currently Amended) A method comprising:
receiving, by a first device, an original large language model (LLM) input comprising instructions and a list of items having a first order;
generating a plurality of lists of items, wherein for each list, of the plurality of the lists of the items, the items of the list are listed in orders different from the first order;
generating a plurality of LLM inputs each of the plurality of LLM inputs comprising the instructions and one of the plurality of the lists of the items;
generating a final LLM output by aggregating a plurality of LLM outputs to the plurality of LLM inputs; and
sending, to a second device, the final LLM output.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving a request (e.g., written) with instructions and a list of items in a first order;
Re-ordering said items in a different order, wherein for each re-ordered list the items are ordered differently from the first order;
Using predetermined set of steps/rules (i.e., LLM) to generate or write down a response to the received requests comprising the instructions and a list of items;
Using predetermined set of steps/rules (i.e., aggregating (e.g., mathematical concept)) to select a final response;
Writing down said response.
16. (Currently Amended) A method comprising:
receiving, by a first device, a first large language model (LLM) input comprising a list of items listed in an original order;
generating a plurality of lists of the items each list comprising the items listed in an order different from the original order, wherein the order is determined randomly;
sending, to a second device, a plurality of LLM inputs each comprising one of the plurality of lists;
receiving a plurality of LLM outputs based on the plurality of LLM inputs;
generating a final LLM output by aggregating the plurality of LLM outputs; and
causing a response to the first LLM input using the final LLM output.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving a request (e.g., written) with a list of items in a first order;
Re-ordering said items in a different order, wherein for each re-ordered list the items are ordered differently (and randomly) from the first order;
Using predetermined set of steps/rules (i.e., LLM) to generate or write down a response to the received requests comprising a list of items;
Using predetermined set of steps/rules (i.e., aggregating (e.g., mathematical concept)) to select a final response;
Writing down said response.
Please also refer to MPEP 2106.05(f)(2): Whether the claim invokes computers or other machinery merely as a tool to perform an existing process, and MPEP 2106.06(b): Clear Improvement to a Technology or to Computer Functionality.
Please refer to MPEP 2106.04(II): Eligibility Step 2A: Whether a Claim is Directed to a Judicial Exception: (A) Step 2A is a Two-Prong Inquiry:
(2) Prong Two:
Prong Two asks does the claim recite additional elements that integrate the judicial exception into a practical application? In Prong Two, examiners evaluate whether the claim as a whole integrates the exception into a practical application of that exception. If the additional elements in the claim integrate the recited exception into a practical application of the exception, then the claim is not directed to the judicial exception (Step 2A: NO) and thus is eligible at Pathway B. This concludes the eligibility analysis. If, however, the additional elements do not integrate the exception into a practical application, then the claim is directed to the recited judicial exception (Step 2A: YES), and requires further analysis under Step 2B (where it may still be eligible if it amounts to an ‘‘inventive concept’’). For more information on how to evaluate whether a judicial exception is integrated into a practical application, see MPEP § 2106.04(d)(2).
From this analysis, in Step 2A, Prong Two, the Examiner has evaluated the independent claims accordingly and determined that the amended independent claims as drafted that the claims as a whole do not include additional elements that integrate the exception into a practical application of that exception. (i.e., an abstract idea). As discussed in the Non-Final Rejection mailed on 12/17/2025:
This judicial exception is not integrated into a practical application because for example: claims 1, 9, and 16 recite “a device,” “a first device” and/or “a second device”. As an example, in ¶ [0020 and 0022] of the as filed specification, disclose: “[0020] The gateway 111 may also comprise one or more local network interfaces to communicate, via one or more local networks, with devices in the premises 102a. Such devices may comprise, e.g., display devices 112 (e.g., televisions), other devices 113 (e.g., a DVR or STB), personal computers 114, laptop computers 115, wireless devices 116 (e.g., wireless routers, wireless laptops, notebooks, tablets and netbooks, cordless phones (e.g., Digital Enhanced Cordless Telephone—DECT phones), mobile phones, mobile televisions, personal digital assistants (PDA)), landline phones 117 (e.g., Voice over Internet Protocol—VoIP phones), and any other desired devices… [0022] FIG. 2 shows hardware elements of a computing device 200 that may be used to implement any of the computing devices shown in FIG. 1 (e.g., the mobile devices 125, any of the devices shown in the premises 102a, any of the devices shown in the local office 103, any of the wireless access points 127, any devices with the external network 109) and any other computing devices discussed herein (e.g., a content server 106, an LLM server 122, a mobile device 125, a wireless device 116, a personal computer 114, a laptop computer 115, etc.).”. Therefore, a general-purpose computer or computing device is described and mainly used as an application thereof. Accordingly, these additional elements do not integrate the abstract idea into a practical idea because it does not impose any meaningful limits on practicing the abstract idea.
Please also refer to MPEP 2106.05(f)(2): Whether the claim invokes computers or other machinery merely as a tool to perform an existing process.
Finally, please refer to MPEP 2106.05(A): Relevant Considerations For Evaluating Whether Additional Elements Amount To An Inventive Concept
Limitations that the courts have found not to be enough to qualify as "significantly more" when recited in a claim with a judicial exception include:
i. Adding the words "apply it" (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, e.g., a limitation indicating that a particular function such as creating and maintaining electronic records is performed by a computer, as discussed in Alice Corp., 573 U.S. at 225-26, 110 USPQ2d at 1984 (see MPEP § 2106.05(f));
ii. Simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry, as discussed in Alice Corp., 573 U.S. at 225, 110 USPQ2d at 1984 (see MPEP § 2106.05(d));
From this analysis, in Step 2B, the Examiner has evaluated the independent claims accordingly and determined that the independent claims as drafted have limitations that the courts have found not to be enough to qualify as "significantly more" when recited in a claim with a judicial exception. Similar to what was discussed in the Non-Final Rejection mailed on 12/17/2025:
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of using a computer is listed as a general computing device as noted. The claim is not patent eligible.
In summary, the Examiner respectfully disagrees with the arguments above.
For more details, please refer to updated 35 U.S.C. § 101 rejections for claims 1-20, below.
35 USC § 102/103 rejection(s)
Arguments on pages 7-9 of the Remarks filed on 05/15/2026
Examiner’s Response to Arguments:
Arguments have been considered but these are not persuasive. The Examiner respectfully disagrees with the arguments of Qin et al. not teaching the aggregating step as recited in the independent claims. The Examiner notes that the claims recite: “determining a final LLM output based on aggregating the plurality of LLM outputs.”
This language claim as drafted is considered to, under the broadest reasonable interpretation, be disclosed in Qin et al. at least in Figs. 2 and 3A-B paragraphs [0007, 0041, 0047, and 0049] as cited in the office action mailed on 12/17/2025 and also in additional citations such as paragraphs [0033, 0044, and 0063] (all citations incorporated below for reference). Here, Qin et al. discloses generating an output by using generative sequence processing model based on pairwise comparisons and also generates aggregate scores for set of texts (e.g., passages 1 to N) and each comparison to generate an aggregate score for the respective texts. Hence, the Examiner considers the limitation of “determining a final LLM output based on aggregating the plurality of LLM outputs” is taught by Qin et al. and suggests incorporating more details regarding the aggregation.
(as previously cited):
Fig. 2 (212: plurality of passages (i.e., Passage 1 [Wingdings font/0xE0] Passage N) and plurality of outputs (304 306 and 308, 310))
Figs. 3A-B (300: machine-learned model, 302: LLM, 308: ordered list, 310: final ranking)
¶ [0007, 0041, 0047, and 0049]: [0007]: “According to another example embodiment of the present disclosure, a computer-implemented method for prompt-based ranking can be performed by one or more computing devices and can include generating a prompt comprising a query, a first set of text associated with a first candidate result, and a second set of text associated with a second candidate result…” [0041]: “FIG. 2 depicts a block diagram of an example machine-learned model 200 according to example embodiments of the present disclosure. In some implementations, a ranking system 214 (e.g., software or a component of a computing device) can receive a set of input data 204 comprising a query and input data 212 comprising a plurality of sets of text (e.g., passage 1 through passage N), such as documents, and, as a result of receipt of the input data 204 and 212, initiate the machine-learned model 200. Thus, in some implementations, the machine-learned 200 can include a generative sequence processing model 202 (e.g., a large language model) that is operable to be prompted with a query (e.g., input data 204) and pairs of sets of text of the plurality of sets of text (e.g., input data 212), each set of text associated with a candidate result of the generative sequence processing model 202, and provide output data 216 comprising generated text and/or output data 218 comprising a score…” ¶ [0049]: “FIG. 3B depicts a block diagram of an example machine-learned model 300 according to example embodiments of the present disclosure. The machine-learned model 300 is similar to the machine-learned model 200 of FIG. 2 except that machine-learned model 300 further includes pairwise ranking prompting with the machine-learned model 300. Thus, in some implementations, a ranking system 314 (e.g., software or a component of a computing device) can receive a query and a plurality of sets of text (e.g., passage 1 through passage N), such as documents, and the machine-learned model 300 can include a generative sequence processing model 302 (e.g., a large language model) that is operable to obtain an ordered list 308 of the plurality of sets of text and compare the entries by starting at the bottom of the ordered list 308 (e.g., the passage on the right side) and comparing and swapping the entry to the entry above it on the list (e.g., the passage to its left) with a stride of 1, so one pass requires O(N) complexity where N is the number of documents or passages. For instance, the final entry (e.g., passage 1 on the right side) in the ordered list is compared to the entry above the final entry in the list (e.g., passage 1 is compared with passage 5, which is to the left of passage 1). Next, the entry above the final entry (e.g., passage 1 after the swap) can be compared and swapped with the entry above it in the list (e.g., passage 1 is compared with passage 4, which is to the left of passage 1) with a stride of 1. The comparing and swapping can be performed for each entry in the ordered list until the first entry of the list is compared and swapped to generate a final ranking 310.”
Fig. 4 and ¶ [0057]: “At 408, the computing system generates, by the generative sequence processing model based on the one or more pairwise comparisons, an output comprising generated text identifying the first set of text or the second set of text as a higher ranked set of text in response to the query. In some examples, the computing system generates an output comprising a first score for the first set of text and a second score for the second set of text in response to the query and determines, based on the first score and the second score, that the first set of text or the second set of text is a higher ranked set of text in response to the query, and the first score identifies a probability of the generative sequence processing model generating the first set of text in response to the query and the second score identifies a probability of the generative sequence processing model generating the second set of text in response to the query.”
(additionally):
¶ [0033]: “In an implementation of pairwise ranking prompting, all pairs of documents can be enumerated, and a global aggregation can be performed to generate a score for each document.”
¶ [0044]: “… the generative sequence processing model 202 can perform comparisons of a set of text (e.g., passage 1) to each remaining set of text (e.g., passage 2 through passage N), generate a score for the set of text (e.g., passage 1) for each of the pairwise comparisons, and add the scores together to generate an aggregate score for the set of text (e.g., passage 1).”
¶ [0063]: “… for each respective set of text in the plurality of sets of text, performs comparisons of the respective set of text to each set of text in the plurality of sets of text, generates a score for the respective set of text for each comparison, and aggregates each score for the respective set of text for each comparison to generate an aggregate score for the respective set of text.”
For more details, please refer to updated 35 U.S.C. § 102/103 rejections for claims1-20, below.
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is equested in correcting any errors of which applicant may become aware in the specification.
Claim Objections
Claim 11 objected to because of the following informalities: “wherein aggregating…” should read “wherein the aggregating…”. Appropriate correction is required.
Claim 12 objected to because of the following informalities: “… the final LLM output outputs …” should read “… the final LLM output . Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. More specifically directed to the abstract idea grouping of: mental process and/or mathematical concept.
The independent claim(s) recite(s):
1. (Currently Amended) A method comprising:
receiving, by a device, an original large language model (LLM) input prompt comprising a list of items in a first order, wherein the items, of the list of items, are listed;
generating a plurality of lists of the items, wherein for each list, of the plurality of the lists of the items, the items of the list are listed in orders different from the first order and each other;
generating a plurality of LLM outputs from a plurality LLM inputs, each of the plurality of LLM inputs comprising one of the plurality of lists of the items;
determining a final LLM output based on aggregating the plurality of LLM outputs; and
causing a response to the original LLM input prompt using the final LLM output.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving a request (e.g., written) with a list of items in a first order;
Re-ordering said items in a different order, wherein for each re-ordered list the items are ordered differently from the first order;
Using predetermined set of steps/rules (i.e., LLM) to generate or write down a response to the received requests comprising a list of items;
Using predetermined set of steps/rules (i.e., aggregating (e.g., mathematical concept)) to select a final response;
Writing down said response.
9. (Currently Amended) A method comprising:
receiving, by a first device, an original large language model (LLM) input comprising instructions and a list of items having a first order;
generating a plurality of lists of items, wherein for each list, of the plurality of the lists of the items, the items of the list are listed in orders different from the first order;
generating a plurality of LLM inputs each of the plurality of LLM inputs comprising the instructions and one of the plurality of the lists of the items;
generating a final LLM output by aggregating a plurality of LLM outputs to the plurality of LLM inputs; and
sending, to a second device, the final LLM output.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving a request (e.g., written) with instructions and a list of items in a first order;
Re-ordering said items in a different order, wherein for each re-ordered list the items are ordered differently from the first order;
Using predetermined set of steps/rules (i.e., LLM) to generate or write down a response to the received requests comprising the instructions and a list of items;
Using predetermined set of steps/rules (i.e., aggregating (e.g., mathematical concept)) to select a final response;
Writing down said response.
16. (Currently Amended) A method comprising:
receiving, by a first device, a first large language model (LLM) input comprising a list of items listed in an original order;
generating a plurality of lists of the items each list comprising the items listed in an order different from the original order, wherein the order is determined randomly;
sending, to a second device, a plurality of LLM inputs each comprising one of the plurality of lists;
receiving a plurality of LLM outputs based on the plurality of LLM inputs;
generating a final LLM output by aggregating the plurality of LLM outputs; and
causing a response to the first LLM input using the final LLM output.
This reads on a human (e.g., mentally and/or using pen and paper):
Receiving a request (e.g., written) with a list of items in a first order;
Re-ordering said items in a different order, wherein for each re-ordered list the items are ordered differently (and randomly) from the first order;
Using predetermined set of steps/rules (i.e., LLM) to generate or write down a response to the received requests comprising a list of items;
Using predetermined set of steps/rules (i.e., aggregating (e.g., mathematical concept)) to select a final response;
Writing down said response.
This judicial exception is not integrated into a practical application because for example: claims 1, 9, and 16 recite “a device,” “a first device” and/or “a second device”. As an example, in ¶ [0020 and 0022] of the as filed specification, disclose: “[0020] The gateway 111 may also comprise one or more local network interfaces to communicate, via one or more local networks, with devices in the premises 102a. Such devices may comprise, e.g., display devices 112 (e.g., televisions), other devices 113 (e.g., a DVR or STB), personal computers 114, laptop computers 115, wireless devices 116 (e.g., wireless routers, wireless laptops, notebooks, tablets and netbooks, cordless phones (e.g., Digital Enhanced Cordless Telephone—DECT phones), mobile phones, mobile televisions, personal digital assistants (PDA)), landline phones 117 (e.g., Voice over Internet Protocol—VoIP phones), and any other desired devices… [0022] FIG. 2 shows hardware elements of a computing device 200 that may be used to implement any of the computing devices shown in FIG. 1 (e.g., the mobile devices 125, any of the devices shown in the premises 102a, any of the devices shown in the local office 103, any of the wireless access points 127, any devices with the external network 109) and any other computing devices discussed herein (e.g., a content server 106, an LLM server 122, a mobile device 125, a wireless device 116, a personal computer 114, a laptop computer 115, etc.).”. Therefore, a general-purpose computer or computing device is described and mainly used as an application thereof. Accordingly, these additional elements do not integrate the abstract idea into a practical idea because it does not impose any meaningful limits on practicing the abstract idea.
The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of using a computer is listed as a general computing device as noted. The claim is not patent eligible.
With respect to claims 2 and 17, the claim(s) recite:
2. The method of claim 1, further comprising sending the final LLM output to a second device.
17. The method of claim 16, further comprising sending the final LLM output to a third device.
This reads on a human (e.g., mentally and/or using pen and paper):
Writing down the response.
Additional limitations of “second/third device” present and analysis as described in independent claims, above.
With respect to claim 3, the claim(s) recite:
3. (Currently Amended) The method of claim 1, wherein the original LLM input prompt further comprises instructions; and wherein the plurality of the LLM inputs further comprise the instructions.
This reads on a human (e.g., mentally and/or using pen and paper):
The request (e.g., written) comprising instructions
No additional limitations are present.
With respect to claims 4, 11, and 19, the claim(s) recite:
4. (Currently Amended) The method of claim 1, wherein the aggregating the plurality of LLM outputs comprises determining a Kendall tau distance between each of the plurality of LLM outputs; and wherein the final LLM output is determined based on the Kendall tau distance.
11. The method of claim 9, wherein aggregating the plurality of LLM outputs comprises determining a distance between each of the plurality of LLM outputs.
19. (Currently Amended) The method of claim 16, wherein generating the final LLM output comprises determining a distance between each of the plurality of LLM outputs, wherein the distances are determined based on a Kendall tau distance; and wherein the aggregation of the plurality of LLM outputs is based on the distances.
This reads on a human (e.g., mentally and/or using pen and paper):
Using predetermined set of steps/rules (i.e., aggregating according to a distance such as Kendall tau distance (e.g., mathematical concept)) to select a final response
No additional limitations are present.
With respect to claims 5, 12, and 18, the claim(s) recite:
5. The method of claim 1, wherein determining the final LLM output further comprises determining a similarity between each of the plurality of LLM outputs.
12. (Currently Amended) The method of claim 9, wherein the generating the final LLM output outputs comprises determining a similarity between each of the plurality of LLM outputs.
18. The method of claim 16, further comprising determining a similarity between each of the plurality of LLM outputs; and wherein the aggregation of the plurality of LLM outputs is based on the similarity between the LLM outputs.
This reads on a human (e.g., mentally and/or using pen and paper):Using predetermined set of steps/rules (i.e., aggregating / similarity (e.g., mathematical concept)) to select a final response
No additional limitations are present.
With respect to claims 6 and 13, the claim(s) recite:
6. (Currently Amended) The method of claim 1, wherein the orders different from the first order of the plurality of lists of items are determined randomly.
13. (Currently Amended) The method of claim 9, wherein, for each list of the plurality of the lists of the items, an order of the items is determined randomly.
This reads on a human (e.g., mentally and/or using pen and paper):
Re-ordering said items in a different order in a random manner.
No additional limitations are present.
With respect to claim 7, the claim(s) recite:
7. (Currently Amended) The method of claim 1, wherein aggregating the plurality of LLM outputs comprises determining a number of swaps between the plurality of LLM outputs; and wherein determining the final LLM output is based on the number of swaps.
This reads on a human (e.g., mentally and/or using pen and paper):
Using predetermined set of steps/rules (i.e., swapping) to select a final response
No additional limitations are present.
With respect to claims 8, 15, and 20, the claim(s) recite:
8. (Currently Amended) The method of claim 1, wherein the device comprises a server.
15. (Currently Amended) The method of claim 9, wherein the first device is a server and the second device is a mobile device or a server.
20. The method of claim 16, wherein the first device comprises a wireless device and the second device comprises a server.
This reads on a human (e.g., mentally and/or using pen and paper):
Writing down the response
The additional limitations present of “server”, “first device”, or “wireless device” follow the same discussion as applied to independent claims above.
With respect to claim 10, the claim(s) recite:
10. (Currently Amended) The method of claim 9, wherein the instructions comprise to sort an order of the list of items.
This reads on a human (e.g., mentally and/or using pen and paper):
Following instructions of sorting a list.
No additional limitations are present.
With respect to claim 14, the claim(s) recite:
14. (Currently Amended) The method of claim 9, wherein a quantity of the plurality of inputs is based on a quantity number of items in the list of items.
This reads on a human (e.g., mentally and/or using pen and paper):
Identifying inputs based on a number of items in a list.
No additional limitations are present.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 3, and 6-8 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Qin et al. (US 20250124067 A1).
As to independent claim 1, Qin et al. teaches:
1. A method (see ¶ [0007]: “According to another example embodiment of the present disclosure, a computer-implemented method for prompt-based ranking can be performed by one or more computing devices and can include generating a prompt comprising a query, a first set of text associated with a first candidate result, and a second set of text associated with a second candidate result…”) comprising:
receiving, by a device, an original large language model (LLM) input prompt comprising a list of items in a first order, wherein the items, of the list of items, are listed (see ¶ [0007] citation as in preamble above and further: “…generating a prompt comprising a query, a first set of text associated with a first candidate result, and a second set of text associated with a second candidate result. The computer-implemented method can further include prompting a generative sequence processing model with the prompt…”
and ¶ [0041]: “FIG. 2 depicts a block diagram of an example machine-learned model 200 according to example embodiments of the present disclosure. In some implementations, a ranking system 214 (e.g., software or a component of a computing device) can receive a set of input data 204 comprising a query and input data 212 comprising a plurality of sets of text (e.g., passage 1 through passage N), such as documents, and, as a result of receipt of the input data 204 and 212, initiate the machine-learned model 200. Thus, in some implementations, the machine-learned 200 can include a generative sequence processing model 202 (e.g., a large language model) that is operable to be prompted with a query (e.g., input data 204) and pairs of sets of text of the plurality of sets of text (e.g., input data 212), each set of text associated with a candidate result of the generative sequence processing model 202, and provide output data 216 comprising generated text and/or output data 218 comprising a score…”);
PNG
media_image1.png
686
364
media_image1.png
Greyscale
generating a plurality of lists of the items, wherein for each list, of the plurality of the lists of the items, the items of the list are listed in orders different from the first order and each other (see ¶ [0007 and 0041] citations as in limitation(s) above and further Fig. 3B (300: machine-learned model, 302: LLM, 308: ordered list, 310: final ranking) and ¶ [0049]: “FIG. 3B depicts a block diagram of an example machine-learned model 300 according to example embodiments of the present disclosure. The machine-learned model 300 is similar to the machine-learned model 200 of FIG. 2 except that machine-learned model 300 further includes pairwise ranking prompting with the machine-learned model 300. Thus, in some implementations, a ranking system 314 (e.g., software or a component of a computing device) can receive a query and a plurality of sets of text (e.g., passage 1 through passage N), such as documents, and the machine-learned model 300 can include a generative sequence processing model 302 (e.g., a large language model) that is operable to obtain an ordered list 308 of the plurality of sets of text and compare the entries by starting at the bottom of the ordered list 308 (e.g., the passage on the right side) and comparing and swapping the entry to the entry above it on the list (e.g., the passage to its left) with a stride of 1, so one pass requires O(N) complexity where N is the number of documents or passages. For instance, the final entry (e.g., passage 1 on the right side) in the ordered list is compared to the entry above the final entry in the list (e.g., passage 1 is compared with passage 5, which is to the left of passage 1). Next, the entry above the final entry (e.g., passage 1 after the swap) can be compared and swapped with the entry above it in the list (e.g., passage 1 is compared with passage 4, which is to the left of passage 1) with a stride of 1. The comparing and swapping can be performed for each entry in the ordered list until the first entry of the list is compared and swapped to generate a final ranking 310.”);
generating a plurality of LLM outputs from a plurality LLM inputs, each of the plurality of LLM inputs comprising one of the plurality of lists of the items (see Fig. 3B (300: machine-learned model, 302: LLM, 308: ordered list, 310: final ranking) and ¶ [0007, 0041, and 0049] citations as in limitation(s) above, more specifically ¶ [0049]: “… The comparing and swapping can be performed for each entry in the ordered list until the first entry of the list is compared and swapped to generate a final ranking 310.” and further Fig. 2 (212: plurality of passages (i.e., Passage 1 [Wingdings font/0xE0] Passage N) and plurality of outputs (304 306 and 308, 310)) and ¶ [0047]: “ FIG. 3A depicts a block diagram of an example machine-learned model 300 according to example embodiments of the present disclosure. The machine-learned model 300 is similar to the machine-learned model 200 of FIG. 2 except that machine-learned model 300 further includes pairwise ranking prompting with the machine-learned model 300. Thus, in some implementations, the machine-learned model 300 can include a generative sequence processing model 302 (e.g., a large language model) that is operable to perform the one or more pairwise comparisons between the first set of text (e.g., passage 1) and the second set of text (e.g., passage 2) based on the query by obtaining an initial ranking 304 of the sets of text (e.g., input data 206), such as a local ordering, in the form of a list. For example, the first entry in the list may be the second set of data (e.g., passage 2) which is to the left and the second entry in the list may be the first set of data (e.g., passage 1) which is to the right and is also the final entry in the list in this example because there are two passages input into the generative sequence processing model 202.”);
PNG
media_image2.png
440
246
media_image2.png
Greyscale
PNG
media_image3.png
358
364
media_image3.png
Greyscale
determining a final LLM output based on aggregating the plurality of LLM outputs (see Figs. 2 and 3A-B and ¶ [0007, 0041, 0047, and 0049] citations as in limitation(s) above and further Fig. 4 and ¶ [0057]: “At 408, the computing system generates, by the generative sequence processing model based on the one or more pairwise comparisons, an output comprising generated text identifying the first set of text or the second set of text as a higher ranked set of text in response to the query. In some examples, the computing system generates an output comprising a first score for the first set of text and a second score for the second set of text in response to the query and determines, based on the first score and the second score, that the first set of text or the second set of text is a higher ranked set of text in response to the query, and the first score identifies a probability of the generative sequence processing model generating the first set of text in response to the query and the second score identifies a probability of the generative sequence processing model generating the second set of text in response to the query.”
additionally, see ¶ [0033, 0044, 0063]: “[0033] In an implementation of pairwise ranking prompting, all pairs of documents can be enumerated, and a global aggregation can be performed to generate a score for each document. [0044] … ), the generative sequence processing model 202 can perform comparisons of a set of text (e.g., passage 1) to each remaining set of text (e.g., passage 2 through passage N), generate a score for the set of text (e.g., passage 1) for each of the pairwise comparisons, and add the scores together to generate an aggregate score for the set of text (e.g., passage 1). [0063] … for each respective set of text in the plurality of sets of text, performs comparisons of the respective set of text to each set of text in the plurality of sets of text, generates a score for the respective set of text for each comparison, and aggregates each score for the respective set of text for each comparison to generate an aggregate score for the respective set of text.”); and
causing a response to the original LLM input prompt using the final LLM output (see Figs. 2, 3A-B, and 4 and ¶ [0007, 0041, 0047, 0049, and 0057] citations as in limitation(s) above, more specifically: Fig. 3B (300: machine-learned model, 302: LLM, 308: ordered list, 310: final ranking) and ¶ [0049]: “… The comparing and swapping can be performed for each entry in the ordered list until the first entry of the list is compared and swapped to generate a final ranking 310.” and further ¶ [0057]: “At 408, the computing system generates, by the generative sequence processing model based on the one or more pairwise comparisons, an output comprising generated text identifying the first set of text or the second set of text as a higher ranked set of text in response to the query. In some examples, the computing system generates an output comprising a first score for the first set of text and a second score for the second set of text in response to the query and determines, based on the first score and the second score, that the first set of text or the second set of text is a higher ranked set of text in response to the query, and the first score identifies a probability of the generative sequence processing model generating the first set of text in response to the query and the second score identifies a probability of the generative sequence processing model generating the second set of text in response to the query.”).
Regarding claim 3, Qin et al. further teaches:
3. (Currently Amended) The method of claim 1, wherein the original LLM input prompt further comprises instructions (see ¶ [0007] citation as in claim 1 above and further: “…generating a prompt comprising a query, a first set of text associated with a first candidate result, and a second set of text associated with a second candidate result. The computer-implemented method can further include prompting a generative sequence processing model with the prompt…”
and ¶ [0041]: “FIG. 2 depicts a block diagram of an example machine-learned model 200 according to example embodiments of the present disclosure. In some implementations, a ranking system 214 (e.g., software or a component of a computing device) can receive a set of input data 204 comprising a query and input data 212 comprising a plurality of sets of text (e.g., passage 1 through passage N),…”); and wherein the plurality of the LLM inputs further comprise the instructions (see ¶ [0007 and 0041] citations as in claim 1 and/or limitation above and further:
¶ [0074]: “FIG. 7 is a block diagram of an example processing flow for using machine-learned model(s) 1 to process input(s) 2 to generate output(s) 3.”
¶ [0078]: “Input(s) 2 can generally include or otherwise represent various types of data. Input(s) 2 can include one type or many different types of data…”
¶ [0079]: “Example data types for input(s) 2 or output(s) 3 include natural language text data, … Data can be raw or processed and can be in any format or schema.”
¶ [0144]: “Model host 31 can execute machine-learned model(s) 1 to perform inference for various tasks using various types of data. For example, various different input(s) 2 and output(s) 3 can be used for various different tasks...”
and ¶ [0146]: “In some implementations, input(s) 2 can be or otherwise represent natural language data…”).
Regarding claim 6, Qin et al. further teaches:
6. (Currently Amended) The method of claim 1, wherein the orders different from the first order of the plurality of lists of items are determined randomly (see ¶ [0007, 0041, and 0049] citations as in claim 1 above and further Fig. 3B (300: machine-learned model, 302: LLM, 308: ordered list, 310: final ranking)).
Regarding claim 7, Qin et al. further teaches:
7. The method of claim 1, wherein aggregating the plurality of LLM outputs comprises determining a number of swaps between the plurality of LLM outputs (see Figs. 2 and 3A-B and ¶ [0007, 0041, 0047, and 0049] citations as in claim 1 above. More specifically: “[0049]… Thus, in some implementations, a ranking system 314 (e.g., software or a component of a computing device) can receive a query and a plurality of sets of text (e.g., passage 1 through passage N), such as documents, and the machine-learned model 300 can include a generative sequence processing model 302 (e.g., a large language model) that is operable to obtain an ordered list 308 of the plurality of sets of text and compare the entries by starting at the bottom of the ordered list 308 (e.g., the passage on the right side) and comparing and swapping the entry to the entry above it on the list (e.g., the passage to its left) with a stride of 1, so one pass requires O(N) complexity where N is the number of documents or passages. For instance, the final entry (e.g., passage 1 on the right side) in the ordered list is compared to the entry above the final entry in the list (e.g., passage 1 is compared with passage 5, which is to the left of passage 1). Next, the entry above the final entry (e.g., passage 1 after the swap) can be compared and swapped with the entry above it in the list (e.g., passage 1 is compared with passage 4, which is to the left of passage 1) with a stride of 1. The comparing and swapping can be performed for each entry in the ordered list until the first entry of the list is compared and swapped to generate a final ranking 310.” and further Fig. 4 and ¶ [0057]: “At 408, the computing system generates, by the generative sequence processing model based on the one or more pairwise comparisons, an output comprising generated text identifying the first set of text or the second set of text as a higher ranked set of text in response to the query. In some examples, the computing system generates an output comprising a first score for the first set of text and a second score for the second set of text in response to the query and determines, based on the first score and the second score, that the first set of text or the second set of text is a higher ranked set of text in response to the query, and the first score identifies a probability of the generative sequence processing model generating the first set of text in response to the query and the second score identifies a probability of the generative sequence processing model generating the second set of text in response to the query.”); and
wherein determining the final LLM output is based on the number of swaps (see Figs. 2 and 3A-B and ¶ [0007, 0041, 0047, and 0049] citations as in claim 1 above. More specifically: [0049]: “…the machine-learned model 300 can include a generative sequence processing model 302 (e.g., a large language model) that is operable to obtain an ordered list 308 of the plurality of sets of text and compare the entries by starting at the bottom of the ordered list 308 (e.g., the passage on the right side) and comparing and swapping the entry to the entry above it on the list (e.g., the passage to its left) with a stride of 1, so one pass requires O(N) complexity where N is the number of documents or passages. …The comparing and swapping can be performed for each entry in the ordered list until the first entry of the list is compared and swapped to generate a final ranking 310.”).
Regarding claim 8, Qin et al. further teaches:
8. The method of claim 1, wherein the device comprises a server (see ¶ [0136]: “For example, model host 31 can operate on a server system that provides a machine-learning service to client device(s) that operate client(s) 32 (e.g., over a local or wide-area network). Client device(s) can be end-user devices used by individuals. Client device(s) can be server systems that operate client(s) 32 to provide various functionality as a service to downstream end-user devices.” and ¶ [0164]: “…Machine-learned model(s) 55 can be received from server computing system(s) 60, model development platform system 70, third party system(s) 80 (e.g., an application distribution platform), or developed locally on computing device 50…”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 2 is/are rejected under 35 U.S.C. 103 as being unpatentable Qin et al. (US 20250124067 A1) as applied to claim 1 above, and further in view of Sharpe et al. (US 20250259012 A1).
Regarding claim 2, Qin et al. teaches the limitations as in claim 1, above.
However, Qin et al. does not explicitly teach, but Sharpe et al. does teach:
2. The method of claim 1, further comprising sending the final LLM output to a second device (see ¶ [0174-0177]: “[0174] Embodiment #3: The method of embodiment #1 further comprising: [0175] determining, by the one or more computing devices, a second sequential data token for the first event based on inserting the first data value and the second data value into a natural language template for the first event data; [0176] providing, by the one or more computing devices, the second sequential data token as input to a second language model that outputs generative text; and [0177] based on the second sequential data token, receiving, by the one or more computing devices, second generative text from the second language model.”).
Qin et al. and Sharpe et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Qin et al. to incorporate the teachings of Sharpe et al. of sending the final LLM output to a second device which provides the benefit of improving the quality of output provided by the large language model ([0039] of Sharpe et al.).
Claim 4-5 is/are rejected under 35 U.S.C. 103 as being unpatentable Qin et al. (US 20250124067 A1) as applied to claim 1 above, and further in view of Nitsure et al. ("Risk aware benchmarking of large language models." arXiv preprint arXiv:2310.07132 (2023)).
Regarding claim 4, Qin et al. teaches the limitations as in claim 1, above.
However, Qin et al. does not explicitly teach, but Nitsure et al. does teach:
4. (Currently Amended) The method of claim 1, wherein the aggregating the plurality of LLM outputs comprises determining a Kendall tau distance between each of the plurality of LLM outputs (see ¶ 5 of 1. Introduction: “Our main contributions are: 1. Interpretable Metrics-Portfolio (Section 4). Drawing inspiration from econometrics and mathematical finance, we define a metrics-portfolio for aggregating metrics. This portfolio normalizes and aggregates metrics, yielding a single interpretable number assessing each output of a LLM…”
¶ A. Ablation Studies Metrics Aggregation Versus Portfolio: “For portoflio, computing ranking using FSD and SSD including the portfolio computation on 5K samples for 5 bootstrap samples , we have mean execution time of 32.01 ± 4.51 s. For FSd and SSD ranking computation for all metrics, followed by rank using pearson distance the execution time is of 254.99 ± 16.76 s. On the other hand, we observe on the mix-instruct dataset a consistency of ranks between these two approaches (FSD or SDD on portfolio & FSD or SSD on all metrics followed by rank aggregation) as quantified by the kendall-tau similarity between the ranks: 1. Kendall Tau(R-SSD@P, RA(R-SSD@M)) = 0.848, 2. Kendall Tau(R-FSD@P, RA(R-FSD@M)) =0.878. We see that these two approaches lead to similar ranks while portfolio approach leads to 7x speedups.”
¶ F.3 Rank Aggregation: “Given N ranks πi , i = 1 . . . N represented as permutations in Sk, the rank aggregation in [Pihur et al., 2009] solves the following problem : (
PNG
media_image4.png
58
140
media_image4.png
Greyscale
) where αi ≥ 0, PN i=1 αi = 1 represent importance of each ranking and d is a distance between permutations. [Pihur et al., 2009] have multiple choices of distance such as Pearson or Kendall’s-Tau…”); and the final LLM output is determined based on the Kendall tau distance (see ¶ 5 Intro, ¶ A. Ablation Studies Metrics Aggregation Versus Portfolio, and ¶ F.3 Rank Aggregation citations as in limitation above. More specifically, ¶ 5 of 1. Introduction: “…This portfolio normalizes and aggregates metrics, yielding a single interpretable number assessing each output of a LLM…”).
Qin et al. and Nitsure et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in large language model. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Qin et al. to incorporate the teachings of Nitsure et al. of wherein aggregating the plurality of LLM outputs comprises determining a Kendall tau distance between each of the plurality of LLM outputs; and the final LLM output is determined based on the Kendall tau distance which provides the benefit of yielding a single interpretable number assessing each output of a LLM (¶ 5 of 1. Introduction of Nitsure et al.).
Regarding claim 5, Qin et al. teaches the limitations as in claim 1, above.
However, Qin et al. does not explicitly teach, but Nitsure et al. does teach:
5. The method of claim 1, wherein determining the final LLM output further comprises determining a similarity between each of the plurality of LLM outputs (see ¶ 5 Intro, ¶ A. Ablation Studies Metrics Aggregation Versus Portfolio, and ¶ F.3 Rank Aggregation citations as in limitation above. “kendall-tau similarity ”).
Qin et al. and Nitsure et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in large language model. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Qin et al. to incorporate the teachings of Nitsure et al. of wherein determining the final LLM output further comprises determining a similarity between each of the plurality of LLM outputs which provides the benefit of yielding a single interpretable number assessing each output of a LLM (¶ 5 of 1. Introduction of Nitsure et al.).
Claims 9-10, 13-17, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable Qin et al. (US 20250124067 A1) further in view of Sharpe et al. (US 20250259012 A1).
As to independent claim 9, Qin et al. teaches:
9. A method (see ¶ [0007] as in claim 1, above.) comprising:
receiving, by a first device, an original large language model (LLM) input comprising instructions and a list of items having a first order (see ¶ [0007] citation as in preamble above and further: “…generating a prompt comprising a query, a first set of text associated with a first candidate result, and a second set of text associated with a second candidate result. The computer-implemented method can further include prompting a generative sequence processing model with the prompt…”
and ¶ [0041]: “FIG. 2 depicts a block diagram of an example machine-learned model 200 according to example embodiments of the present disclosure. In some implementations, a ranking system 214 (e.g., software or a component of a computing device) can receive a set of input data 204 comprising a query and input data 212 comprising a plurality of sets of text (e.g., passage 1 through passage N),…);
generating a plurality of lists of items, wherein for each list, of the plurality of the lists of the items, the items of the list are listed in orders different from the first order (see ¶ [0007 and 0041] citations as in claim 1 above and further Fig. 3A-B (300: machine-learned model, 302: LLM, 308: ordered list, 310: final ranking) and ¶ [0049] citations as in claim 1 above.);
generating a plurality of LLM inputs each of the plurality of LLM inputs comprising the instructions and one of the plurality of the lists of the items (see ¶ [0007] citation as in preamble above and further: “…generating a prompt comprising a query, a first set of text associated with a first candidate result, and a second set of text associated with a second candidate result…”);
generating a final LLM output by aggregating a plurality of LLM outputs to the plurality of LLM inputs (see Fig. 3B and ¶ [0007, 0041, and 0049] citations as in limitation(s) above and further Fig. 2 (212: plurality of passages (i.e., Passage 1 [Wingdings font/0xE0] Passage N) and plurality of outputs (304 306 and 308, 310)) and ¶ [0047]: “ FIG. 3A depicts a block diagram of an example machine-learned model 300 according to example embodiments of the present disclosure. The machine-learned model 300 is similar to the machine-learned model 200 of FIG. 2 except that machine-learned model 300 further includes pairwise ranking prompting with the machine-learned model 300. Thus, in some implementations, the machine-learned model 300 can include a generative sequence processing model 302 (e.g., a large language model) that is operable to perform the one or more pairwise comparisons between the first set of text (e.g., passage 1) and the second set of text (e.g., passage 2) based on the query by obtaining an initial ranking 304 of the sets of text (e.g., input data 206), such as a local ordering, in the form of a list. For example, the first entry in the list may be the second set of data (e.g., passage 2) which is to the left and the second entry in the list may be the first set of data (e.g., passage 1) which is to the right and is also the final entry in the list in this example because there are two passages input into the generative sequence processing model 202.” and Fig. 4 and ¶ [0057]: “At 408, the computing system generates, by the generative sequence processing model based on the one or more pairwise comparisons, an output comprising generated text identifying the first set of text or the second set of text as a higher ranked set of text in response to the query. In some examples, the computing system generates an output comprising a first score for the first set of text and a second score for the second set of text in response to the query and determines, based on the first score and the second score, that the first set of text or the second set of text is a higher ranked set of text in response to the query, and the first score identifies a probability of the generative sequence processing model generating the first set of text in response to the query and the second score identifies a probability of the generative sequence processing model generating the second set of text in response to the query.”); and
However, Qin et al. does not explicitly teach, but Sharpe et al. does teach:
sending, to a second device, the final LLM output (see ¶ [0174-0177]: “[0174] Embodiment #3: The method of embodiment #1 further comprising: [0175] determining, by the one or more computing devices, a second sequential data token for the first event based on inserting the first data value and the second data value into a natural language template for the first event data; [0176] providing, by the one or more computing devices, the second sequential data token as input to a second language model that outputs generative text; and [0177] based on the second sequential data token, receiving, by the one or more computing devices, second generative text from the second language model.”).
Qin et al. and Sharpe et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Qin et al. to incorporate the teachings of Sharpe et al. of sending, to a second device, the final LLM output which provides the benefit of improving the quality of output provided by the large language model ([0039] of Sharpe et al.).
As to independent claim 16, Qin et al. teaches:
16. A method (see ¶ [0007] as in claim 1, above.) comprising:
receiving, by a first device, a first large language model (LLM) input comprising a list of items listed in an original order (see ¶ [0007] citation as in preamble above and further: “…generating a prompt comprising a query, a first set of text associated with a first candidate result, and a second set of text associated with a second candidate result. The computer-implemented method can further include prompting a generative sequence processing model with the prompt…”
and ¶ [0041]: “FIG. 2 depicts a block diagram of an example machine-learned model 200 according to example embodiments of the present disclosure. In some implementations, a ranking system 214 (e.g., software or a component of a computing device) can receive a set of input data 204 comprising a query and input data 212 comprising a plurality of sets of text (e.g., passage 1 through passage N),…”);
generating a plurality of lists of the items each list comprising the items listed in an order different from the original order, wherein the order is determined randomly (see ¶ [0007, 0041, and 0049] citations as in claim 1 above and further Fig. 3B (300: machine-learned model, 302: LLM, 308: ordered list, 310: final ranking).);
receiving a plurality of LLM outputs based on the plurality of LLM inputs (see Fig. 3B and ¶ [0007, 0041, 0047, and 0049] citations as in claim 1 above and further Fig. 2 (212: plurality of passages (i.e., Passage 1 [Wingdings font/0xE0] Passage N) and plurality of outputs (304 306 and 308, 310)));
generating a final LLM output by aggregating the plurality of LLM outputs (see Figs. 2 and 3A-B and ¶ [0007, 0041, 0047, and 0049] citations as in claim 1 above and further Fig. 4 and ¶ [0057]: “At 408, the computing system generates, by the generative sequence processing model based on the one or more pairwise comparisons, an output comprising generated text identifying the first set of text or the second set of text as a higher ranked set of text in response to the query…”); and
causing a response to the first LLM input using the final LLM output (see Figs. 2, 3A-B, and 4 and ¶ [0007, 0041, 0047, 0049, and 0057] citations as in claim 1 above.).
However, Qin et al. does not explicitly teach, but Sharpe et al. does teach:
sending, to a second device, a plurality of LLM inputs each comprising one of the plurality of lists (see ¶ [0174-0177]: “[0174] Embodiment #3: The method of embodiment #1 further comprising: [0175] determining, by the one or more computing devices, a second sequential data token for the first event based on inserting the first data value and the second data value into a natural language template for the first event data; [0176] providing, by the one or more computing devices, the second sequential data token as input to a second language model that outputs generative text; and [0177] based on the second sequential data token, receiving, by the one or more computing devices, second generative text from the second language model.”)
Qin et al. and Sharpe et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Qin et al. to incorporate the teachings of Sharpe et al. of sending, to a second device, a plurality of LLM inputs each comprising one of the plurality of lists which provides the benefit of improving the quality of output provided by the large language model ([0039] of Sharpe et al.).
Regarding claim 10, Qin et al. further teaches:
10. (Currently Amended) The method of claim 9, wherein the instructions comprise to sort an order of the list of items (see ¶ [0056]: “In some examples, the computing system performs, by the generative sequence processing model, the one or more pairwise comparisons between the first set of text and the second set of text based on the query by initiating a sorting algorithm with the first set of text and the second set of text and receiving an output of the sorting algorithm comprising an ordered list, and the higher ranked set of text in response to the query can include the first set of text when the first set of text is first in the ordered list and include the second set of text when the second set of text is first in the ordered list. The sorting algorithm may be a heapsort algorithm in some examples.”).
Regarding claim 13, Qin et al. in combination with Sharpe et al. teach the limitations as in claim 9, above.
Qin et al. further teaches:
13. (Currently Amended) The method of claim 9, wherein, for each list of the plurality of the lists of the items, an order of the items is determined randomly (see ¶ [0007, 0041, and 0049] citations as in claim 1 above and further Fig. 3B (300: machine-learned model, 302: LLM, 308: ordered list, 310: final ranking)).
Regarding claim 14, Qin et al. in combination with Sharpe et al. teach the limitations as in claim 9, above.
Qin et al. further teaches:
14. (Currently Amended) The method of claim 9, wherein a quantity of the plurality of inputs is based on a quantity number of items in the list of items (Figs. 2 and 3A-B and ¶ [0007, 0041, 0047, and 0049] citations as in claim 1 above. More specifically: “[0049]… Thus, in some implementations, a ranking system 314 (e.g., software or a component of a computing device) can receive a query and a plurality of sets of text (e.g., passage 1 through passage N), such as documents, and the machine-learned model 300 can include a generative sequence processing model 302 (e.g., a large language model) that is operable to obtain an ordered list 308 of the plurality of sets of text and compare the entries by starting at the bottom of the ordered list 308 (e.g., the passage on the right side) and comparing and swapping the entry to the entry above it on the list (e.g., the passage to its left) with a stride of 1, so one pass requires O(N) complexity where N is the number of documents or passages. For instance, the final entry (e.g., passage 1 on the right side) in the ordered list is compared to the entry above the final entry in the list (e.g., passage 1 is compared with passage 5, which is to the left of passage 1). Next, the entry above the final entry (e.g., passage 1 after the swap) can be compared and swapped with the entry above it in the list (e.g., passage 1 is compared with passage 4, which is to the left of passage 1) with a stride of 1. The comparing and swapping can be performed for each entry in the ordered list until the first entry of the list is compared and swapped to generate a final ranking 310.” and further Fig. 4 and ¶ [0057]: “At 408, the computing system generates, by the generative sequence processing model based on the one or more pairwise comparisons, an output comprising generated text identifying the first set of text or the second set of text as a higher ranked set of text in response to the query. In some examples, the computing system generates an output comprising a first score for the first set of text and a second score for the second set of text in response to the query and determines, based on the first score and the second score, that the first set of text or the second set of text is a higher ranked set of text in response to the query, and the first score identifies a probability of the generative sequence processing model generating the first set of text in response to the query and the second score identifies a probability of the generative sequence processing model generating the second set of text in response to the query.”).
Regarding claim 15, Qin et al. in combination with Sharpe et al. teach the limitations as in claim 9, above.
Qin et al. further teaches:
15. (Currently Amended) The method of claim 9, wherein the first device is a server and the second device is a mobile device or a server (see ¶ [0136]: “For example, model host 31 can operate on a server system that provides a machine-learning service to client device(s) that operate client(s) 32 (e.g., over a local or wide-area network). Client device(s) can be end-user devices used by individuals. Client device(s) can be server systems that operate client(s) 32 to provide various functionality as a service to downstream end-user devices.” and ¶ [0164]: “…Machine-learned model(s) 55 can be received from server computing system(s) 60, model development platform system 70, third party system(s) 80 (e.g., an application distribution platform), or developed locally on computing device 50…”).
Regarding claim 17, Qin et al. in combination with Sharpe et al. teach the limitations as in claim 16, above.
Qin et al. further teaches:
17. The method of claim 16, further comprising sending the final LLM output to a third device (see ¶ [0174-0177]: “[0174] Embodiment #3: The method of embodiment #1 further comprising: [0175] determining, by the one or more computing devices, a second sequential data token for the first event based on inserting the first data value and the second data value into a natural language template for the first event data; [0176] providing, by the one or more computing devices, the second sequential data token as input to a second language model that outputs generative text; and [0177] based on the second sequential data token, receiving, by the one or more computing devices, second generative text from the second language model.”).
Qin et al. and Sharpe et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in large language models. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Qin et al. to incorporate the teachings of Sharpe et al. of sending the final LLM output to a second device which provides the benefit of improving the quality of output provided by the large language model ([0039] of Sharpe et al.).
Regarding claim 20, Qin et al. in combination with Sharpe et al. teach the limitations as in claim 16, above.
Qin et al. further teaches:
20. The method of claim 16, wherein the first device comprises a wireless device and the second device comprises a server (see ¶ [0136]: “For example, model host 31 can operate on a server system that provides a machine-learning service to client device(s) that operate client(s) 32 (e.g., over a local or wide-area network). Client device(s) can be end-user devices used by individuals. Client device(s) can be server systems that operate client(s) 32 to provide various functionality as a service to downstream end-user devices.” and ¶ [0164]: “…Machine-learned model(s) 55 can be received from server computing system(s) 60, model development platform system 70, third party system(s) 80 (e.g., an application distribution platform), or developed locally on computing device 50…”).
Claims 11-12 and 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable Qin et al. (US 20250124067 A1) as applied to claim 1 above, and further in view of Nitsure et al. ("Risk aware benchmarking of large language models." arXiv preprint arXiv:2310.07132 (2023)).
Regarding claim 11, Qin et al. in combination with Sharpe et al. teach the limitations as in claim 9, above.
However, Qin et al. in combination with Sharpe et al. do not explicitly teach, but Nitsure et al. does teach:
11. The method of claim 9, wherein aggregating the plurality of LLM outputs comprises determining a distance between each of the plurality of LLM outputs (see ¶ 5 Intro, ¶ A. Ablation Studies Metrics Aggregation Versus Portfolio, and ¶ F.3 Rank Aggregation citations as in claim 4-5 above. “kendall-tau distance/similarity”).
Qin et al. and Nitsure et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in large language model. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Qin et al. to incorporate the teachings of Nitsure et al. of wherein aggregating the plurality of LLM outputs comprises determining a distance between each of the plurality of LLM outputs which provides the benefit of yielding a single interpretable number assessing each output of a LLM (¶ 5 of 1. Introduction of Nitsure et al.).
Regarding claim 12, Qin et al. in combination with Sharpe et al. teach the limitations as in claim 9, above.
However, Qin et al. in combination with Sharpe et al. do not explicitly teach, but Nitsure et al. does teach:
12. (Currently Amended) The method of claim 9, wherein the generating the final LLM output of the plurality of LLM outputs comprises determining a similarity between each of the plurality of LLM outputs (see ¶ 5 Intro, ¶ A. Ablation Studies Metrics Aggregation Versus Portfolio, and ¶ F.3 Rank Aggregation citations as in claim 4-5 above. “kendall-tau distance/similarity”).
Qin et al. and Nitsure et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in large language model. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Qin et al. to incorporate the teachings of Nitsure et al. of wherein determining the final LLM output of the plurality of LLM outputs comprises determining a similarity between each of the plurality of LLM outputs which provides the benefit of yielding a single interpretable number assessing each output of a LLM (¶ 5 of 1. Introduction of Nitsure et al.).
Regarding claim 18, Qin et al. in combination with Sharpe et al. teach the limitations as in claim 16, above.
However, Qin et al. in combination with Sharpe et al. do not explicitly teach, but Nitsure et al. does teach:
18. The method of claim 16, further comprising determining a similarity between each of the plurality of LLM outputs (see ¶ 5 Intro, ¶ A. Ablation Studies Metrics Aggregation Versus Portfolio, and ¶ F.3 Rank Aggregation citations as in claim 4-5 above. “kendall-tau distance/similarity”); and wherein the aggregation of the plurality of LLM outputs is based on the similarity between the LLM outputs (see ¶ 5 Intro, ¶ A. Ablation Studies Metrics Aggregation Versus Portfolio, and ¶ F.3 Rank Aggregation citations as in claim 4-5 above. “kendall-tau distance/similarity”).
Qin et al. and Nitsure et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in large language model. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Qin et al. to incorporate the teachings of Nitsure et al. of further comprising determining a similarity between each of the plurality of LLM outputs; and wherein the aggregation of the plurality of LLM outputs is based on the similarity between the LLM outputs which provides the benefit of yielding a single interpretable number assessing each output of a LLM (¶ 5 of 1. Introduction of Nitsure et al.).
Regarding claim 19, Qin et al. in combination with Sharpe et al. teach the limitations as in claim 16, above.
However, Qin et al. in combination with Sharpe et al. do not explicitly teach, but Nitsure et al. does teach:
19. (Currently Amended) The method of claim 16, wherein generating the final LLM output comprises determining a distance between each of the plurality of LLM outputs (see ¶ 5 Intro, ¶ A. Ablation Studies Metrics Aggregation Versus Portfolio, and ¶ F.3 Rank Aggregation citations as in claim 4-5 above. “kendall-tau distance/similarity”), wherein the distances are determined based on the Kendall tau distance (see ¶ 5 Intro, ¶ A. Ablation Studies Metrics Aggregation Versus Portfolio, and ¶ F.3 Rank Aggregation citations as in claim 4-5 above. “kendall-tau distance/similarity”); and wherein the aggregation of the plurality of LLM outputs is based on the distances (see ¶ 5 Intro, ¶ A. Ablation Studies Metrics Aggregation Versus Portfolio, and ¶ F.3 Rank Aggregation citations as in claim 4-5 above. “kendall-tau distance/similarity”).
Qin et al. and Nitsure et al. are considered to be analogous to the claimed invention because they are in the same field of endeavor in large language model. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Qin et al. to incorporate the teachings of Nitsure et al. of wherein generating the final LLM output comprises determining a distance between each of the plurality of LLM outputs, wherein the distance is determined based on the Kendall tau distance; and wherein the aggregation of the plurality of LLM outputs is based on the distances which provides the benefit of yielding a single interpretable number assessing each output of a LLM (¶ 5 of 1. Introduction of Nitsure et al.).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Keisha Y Castillo-Torres whose telephone number is (571)272-3975. The examiner can normally be reached Monday - Friday, 9:00 am - 4:00 pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pierre-Louis Desir can be reached at (571)272-7799. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Keisha Y. Castillo-Torres
Examiner
Art Unit 2659
/Keisha Y. Castillo-Torres/Examiner, Art Unit 2659
/PIERRE LOUIS DESIR/Supervisory Patent Examiner, Art Unit 2659
Prosecution Insights