DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on May 12, 2026 has been entered.
Response to Arguments
Applicant’s arguments with respect to 35 USC 103 rejection have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant's arguments towards 35 USC 101 rejection have been fully considered but they are not persuasive.
The Applicant states that “The Claims Do Not Meet the Categories of Abstract Ideas.” The Examiner respectfully disagrees.
The limitations of the claims recite abstract concepts that may be performed in the mind and/or with aid of paper, as detailed in the below rejection.
The Applicant further states that “The Claims Integrate The Alleged Abstract Idea Into A Practical Application.” The Examiner respectfully disagrees.
The Claims also do not integrate the abstract idea into a practical application because the steps of the claims, as well as its purported improvement, restate the abstract ideas discussed in the rejection. Furthermore, while the Applicant brings up Enfish and McRo in support of his arguments, the instant claims do not follow the fact patterns of either case.
As such, the 35 USC 101 rejection is maintained.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Independent Claim 1 recites the following features:
obtaining, by an offline evaluation system, log data from a recommendation system including a new ranker, the log data indicating queries and ranked sets of documents generated at least in part by a current ranker of the recommendation system - Retrieving data is considered extra solution activity as per MPEP 2106.05.
training, by the offline evaluation system, an imitation ranker using the log data - Training data is an abstract concept. Specifically, the machine learning described merely recites “apply” or “perform” the abstract idea by merely invoking a computer/machine learning as a tool in its ordinary capacity as described in MPEP 2106.05(f), as well as linking the abstract idea to the field of use of machine learning. Also, based at least on RECENTIVE ANALYTICS, INC. v. FOX CORP, which states, “Today, we hold only that patents that do no more than claim the application of generic machine learning to new data environments, without disclosing improvements to the machine learning models to be applied, are patent ineligible under § 101.”
where the imitation ranker is trained to simulate an output of the current ranker by at least generating scores that approximate rankings produced by the current ranker - Approximating rankings is something that may be performed in the mind or with help of pen and paper.
cause the imitation ranker to generate a first result including a set of scores associated with document and rank pairs based on a query, the set of scores indicating a probability that the current ranker would determine a particular document is associated with a particular rank - Generating scores is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, generating scores may be interpreted as calculating scores, which is an abstract concept invoking mathematical concepts.
obtaining, from a new ranker, a second result including a ranked set of documents in response to the query – Retrieving data is considered extra solution activity as per MPEP 2106.05.
determining, by the offline evaluation system, a rank distribution indicating propensities associated with the document and rank pairs for a set of impressions, where an impression of the set of impression includes a document and rank pair that is included in the first result and the second result, where determining the rank distribution is performed based on document-rank pairs that appear in both the imitation ranker output and the new ranker output - Calculating probabilities is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating probabilities may be interpreted as mathematical concepts, which is an abstract concept.
determining, by the offline evaluation system, a value associated with the new ranker – Determining a value is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating a value may be interpreted as mathematical concepts, which is an abstract concept.
Independent Claim 8 recites the following features:
obtaining a ranked set of documents generated by an imitation ranker based on a query - Retrieving data is considered extra solution activity as per MPEP 2106.05.
where the imitation ranker is trained to simulate an output of the current ranker by at least generating scores that approximate rankings produced by the current ranker - Approximating rankings is something that may be performed in the mind or with help of pen and paper.
determining a hyperparameter associated with the imitation ranker to modify the ranked set of documents – Determining hyperparameters is a generic function of machine learning functionality (it is also a well-known, routine and conventional functionality. For example see at least Song et al (2022/0180241).
computing a rank distribution for documents included the ranked set of documents for a set of impressions generated by a new ranker of a recommendation system based on the query, where determining the rank distribution is performed based on document rank pairs that appear in both the imitation ranker output and the new ranker output - Calculating rank is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating ranks may be interpreted as mathematical concepts, which is an abstract concept.
computing a set of document and rank propensities based on the rank distribution - Calculating probabilities is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating probabilities may be interpreted as mathematical concepts, which is an abstract concept.
determining a value indicating a performance of the new ranker of the recommendation system based on the set of document and rank propensities - Determining a value is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating a value may be interpreted as mathematical concepts, which is an abstract concept.
Independent Claim 16 recites the following features:
generating, by an imitating ranker, a set of document and rank pairs based on a query – Generating seems to be analogous to retrieving in this claim limitation; Retrieving data is considered extra solution activity as per MPEP 2106.05.
where the imitation ranker is trained to simulate an output of the current ranker by at least generating scores that approximate rankings produced by the current ranker - Approximating rankings is something that may be performed in the mind or with help of pen and paper.
determining, by an offline evaluation system, a set of document and rank propensities based on a rank distribution computed for a set of impressions generated by a ranker based on the query, the set of impression including documents included in the set of document and rank pairs, where determining the rank distribution is performed based on the set of document and rank pairs that appear in both the imitation ranker output and the ranker output - Calculating probabilities is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating probabilities may be interpreted as mathematical concepts, which is an abstract concept.
determining, by the offline evaluation system, a metric indicating a performance of the ranker based on the set of document and rank propensities - Determining a value is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating a value may be interpreted as mathematical concepts, which is an abstract concept.
This judicial exception is not integrated into a practical application. Other than the abstract idea, the claims recite hardware elements such as a memory, a processor, etc. However, the hardware elements are recited at a high level of generality, i.e. as generic computer components performing generic computer functions of information.
As to dependent Claims 2-7, 9-15 and 17-20, these claims fail to recite significantly more than the abstract idea. Rather, the Claims recite more details of
mentally processing graph data, which is additional recitations of the abstract idea identified above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7 are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al (US Patent Application Publication 2010/0082582) in view of Wang et al (US Patent Application Publication 2017/0249312) further in view of Stouffer et al (US Patent Application Publication 2015/0379141) and further in view of Kundu et al (US Patent Application Publication 2024/0071047).
Claim 1: Gao discloses a computer-implemented method comprising:
obtaining, log data from a recommendation system including a new ranker, the log data indicating queries and ranked sets of documents generated at least in part by a current ranker of the recommendation system [0036]. [See at least retrieving queries, documents (i.e. ID) and a score for ranking.]
Gao alone does not explicitly disclose the rest of the limitations.
However, Gao, Wang, Stouffer and Kundu disclose:
training, by the offline evaluation system, an imitation ranker using the log data [Wang [0037, 0061-0062] discloses offline training for a ranker using at least logs.]
cause the imitation ranker to generate a first result including a set of scores associated with document and rank pairs based on a query, the set of scores indicating a probability that the current ranker would determine a particular document is associated with a particular rank [Gao [0035-0036] describes at least relevance scores based on probability for query-id pairs.]
obtaining, from a new ranker, a second result including a ranked set of documents in response to the query [Gao [0036] describes at least retrieving queries, documents (i.e. ID) and a score for ranking. [Gao [0060] describes at least calculating scores for a second time.]
determining, by the offline evaluation system, a value associated with the new ranker [Gao [0035-0036] describes at least relevance scores based on probability for query-id pairs and Wang [0037, 0061-0062] discloses offline evaluation.]
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Wang. One would have been motivated to do so in order to decrease processing power of an online system.
Gao as modified also does not explicitly disclose where the imitation ranker is trained to simulate an output of the current ranker by at least generating scores that approximate ranking produced by the current ranker.
However, Stouffer [0064, 0074-0076] discloses simulating changes in the ranking of documents.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Stouffer. One would have been motivated to do so in order to “to identify the optimal changes” for searching and retrieving results.
Gao as modified also does not explicitly disclose where determining the rank distribution is performed based on the set of document and rank pairs that appear in both the imitation ranker output and the ranker output.
However, Kundu [0081] discloses training a model that includes key pair ranking offline and that model is then used during runtime (i.e. rank pairs that appear in both the imitation ranker output and the ranker output).
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Kundu. One would have been motivated to do so in order to save processing capacity in a runtime environment.
Claim 2: Gao as modified discloses the method of Claim 1 above, and Gao further discloses wherein the recommendation system includes a search engine [0020].
Claim 3: Gao as modified discloses the method of Claim 1 above, and Gao further discloses wherein the new ranker includes one or more modifications to the current ranker [0037-0038]. [See at least modifying a weighting factor.]
Claim 4: Gao as modified discloses the method of Claim 3 above, and Gao further discloses wherein the value includes a metric indicating a performance of the one or more modifications [0037-0038].
Claim 5: Gao as modified discloses the method of Claim 3 above, and Gao further discloses wherein the metric includes at least one of a relevance metric, an impression-level relevance metric, number of clicks, mean reciprocal rank, Kendall tau, expected reciprocal rank, mean average precision, precision at k, and normalize discounted cumulative gain [0041].
Claim 7: Gao as modified discloses the method of Claim 1 above, and Gao further discloses wherein the computer- implemented method further comprises determining a second value associated with a second ranker based at least in part on the imitation ranker without re-training the imitation ranker [0037-0038].
Claims 6 and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al (US Patent Application Publication 2010/0082582) in view of Wang et al (US Patent Application Publication 2017/0249312) further in view of Stouffer et al (US Patent Application Publication 2015/0379141) further in view of Kundu et al (US Patent Application Publication 2024/0071047) and further in view of Song et al (US Patent Application Publication 2022/0180241).
Claim 6: Gao as modified discloses the method of Claim 1 above, but Gao alone does not explicitly disclose wherein the computer- implemented method further comprises tuning the imitation ranker using one or more hyperparameters.
However, Song [0021] discloses tuning hyperparameters for a machine learning model.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Song. One would have been motivated to do so in order to “increase or decrease the rate at which the machine learning model learns from training data, which in turn affects the model's efficiency in generating accurate predictions.”
Claim 16: Gao discloses a computer system comprising: a processor; and a computer storage medium storing computer-useable instructions that, when used by the processor, causes the computer system to perform operations comprising:
generating, by an imitating ranker, a set of document and rank pairs based on a query [0036]. [See at least retrieving documents (i.e. ID) and a score for ranking.]
Gao alone does not explicitly disclose the rest of the limitations.
However, Gao and Wang disclose:
determining, by an offline evaluation system, a set of document and rank propensities based on a rank distribution computed for a set of impressions generated by a ranker based on the query, the set of impression including documents included in the set of document and rank pairs [Gao [0035-0036] describes at least relevance scores based on probability and Wang [0061-0062] discloses offline evaluation.]
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Wang. One would have been motivated to do so in order to decrease processing power of an online system.
Song further discloses: determining, by the offline evaluation system, a metric indicating a performance of the ranker based on the set of document and rank propensities [0019-0020]. [See at least identifying a performance value.]
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Song. One would have been motivated to do so in order to identify a performance level at which a system functions.
Gao as modified also does not explicitly disclose where the imitation ranker is trained to simulate an output of the current ranker by at least generating scores that approximate rankings produced generated by the current ranker.
However, Stouffer [0064, 0074-0076] discloses simulating changes in the ranking of documents.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Stouffer. One would have been motivated to do so in order to “to identify the optimal changes” for searching and retrieving results.
Gao as modified also does not explicitly disclose where determining the rank distribution is performed based on the set of document and rank pairs that appear in both the imitation ranker output and the ranker output.
However, Kundu [0081] discloses training a model that includes key pair ranking offline and that model is then used during runtime (i.e. rank pairs that appear in both the imitation ranker output and the ranker output).
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Kundu. One would have been motivated to do so in order to save processing capacity in a runtime environment.
Claim 17: Gao as modified discloses the system of Claim 16 above, but Gao alone does not explicitly disclose wherein generating the set of documents further comprise tuning the imitating ranker using a hyperparameter.
However, Song [0021] discloses using hyperparameters for a machine learning model.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Song. One would have been motivated to do so in order to “increase or decrease the rate at which the machine learning model learns from training data, which in turn affects the model's efficiency in generating accurate predictions.”
Claim 18: Gao as modified discloses the system of Claim 17 above and Gao in view of Song disclose wherein a value of the hyperparameter is determined based at least in part on the set of document and rank pairs. Gao [0036] describes at least retrieving documents (i.e. ID) and a score for ranking; And Song, for the same reason as above discloses using an input, such as in Gao to tune hyperparameters.
Claim 19: Gao as modified discloses the system of Claim 16 above, and Gao further discloses wherein the set of document and rank pairs includes a score for a document at a rank in a ranked set of documents [0036].
Claim 20: Gao as modified discloses the system of Claim 16 above, and Gao further discloses wherein the metric includes at least one of. a relevance metric, an impression-level relevance metric, number of click, mean reciprocal rank, Kendall tau, expected reciprocal rank, mean average precision, precision at k, and normalize discounted cumulative gain [0041].
Claims 8-14 are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al (US Patent Application Publication 2010/0082582) in view of Song et al (US Patent Application Publication 2022/0180241) further in view of Stouffer et al (US Patent Application Publication 2015/0379141) and further in view of Kundu et al (US Patent Application Publication 2024/0071047).
Claim 8: Gao discloses one or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations, the operations comprising:
obtaining a ranked set of documents generated by an imitation ranker based on a query [0036-0037]. [See at least retrieving ranked documents.]
Gao alone does not explicitly disclose determining a hyperparameter associated with the imitation ranker to modify the ranked set of documents.
However, Song [0021] discloses using hyperparameters for a machine learning model.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Song. One would have been motivated to do so in order to “increase or decrease the rate at which the machine learning model learns from training data, which in turn affects the model's efficiency in generating accurate predictions.”
Gao as modified also does not explicitly disclose where the imitation ranker is trained to simulate an output of the current ranker by at least generating scores that approximate ranking produced by the current ranker.
However, Stouffer [0064, 0074-0076] discloses simulating changes in the ranking of documents.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Stouffer. One would have been motivated to do so in order to “to identify the optimal changes” for searching and retrieving results.
Gao as modified further discloses:
computing a rank distribution for documents included the ranked set of documents for a set of impressions generated by a new ranker of a recommendation system based on the query [0031, 0035-0036]. [See at least relevance scores based on probability for query-id pairs. Also see term frequency.]
computing a set of document and rank propensities based on the rank distribution [0031, 0035-0036]. [See at least relevance scores based on probability.]
determining a value indicating a performance of the new ranker of the recommendation system based on the set of document and rank propensities [0035-0036]. [See at least relevance scores based on probability for query-id pairs. Also see Song [0019-0020] for identifying a performance value.]
Gao as modified also does not explicitly disclose where determining the rank distribution is performed based on the set of document and rank pairs that appear in both the imitation ranker output and the ranker output.
However, Kundu [0081] discloses training a model that includes key pair ranking offline and that model is then used during runtime (i.e. rank pairs that appear in both the imitation ranker output and the ranker output).
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Kundu. One would have been motivated to do so in order to save processing capacity in a runtime environment.
Claim 9: Gao as modified discloses the media of Claim 8 above, and Gao further discloses wherein the recommendation system includes a current ranker that generates log data used to train the imitation ranker [0035-0036].
Claim 10: Gao as modified discloses the media of Claim 9 above, and Gao further discloses wherein the new ranker includes a set of changes to the current ranker and the new ranker generates the set of impressions [0037-0038]. [See at least modifying a weighting factor.]
Claim 11: Gao as modified discloses the media of Claim 10 above, and Song [0021], for the same reasons as above, further discloses wherein the set of changes includes at least one of: a new ranking feature, a modification to a ranking model, a modification to a parameter of the current ranker, and a modification to a hyperparameter of the current ranker. [See at least tuning hyperparameters for a machine learning model.]
Claim 12: Gao as modified discloses the media of Claim 8 above, and Gao further discloses wherein the operations further comprise training the imitation ranker using log data obtained from a second recommendation system [0036]. [See at least retrieving queries, documents (i.e. ID) and a score for ranking. Furthermore, Gao discloses retrieving data from a recommendation system. However, using another recommendation system would not change the overall functionality of Gao’s process. As such, it would have bene obvious for one of ordinary skill in the art before the effective filing date to modify Gao to process data from different systems in order to process data from any system that is specified.]
Claim 13: Gao as modified discloses the media of Claim 12 above, and Gao further discloses wherein the log data includes an indication of at least a ranked set of documents and a user interaction with a document of the ranked set of document generated in response to the query [0033, 0036, 0040].
Claim 14: Gao as modified discloses the media of Claim 8 above, and Gao further discloses wherein determining the value further comprises determining a Number of Clicks metric based on the set of document and rank propensities [0033].
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Gao et al (US Patent Application Publication 2010/0082582) in view of Song et al (US Patent Application Publication 2022/0180241) further in view of Stouffer et al (US Patent Application Publication 2015/0379141) further in view of Kundu et al (US Patent Application Publication 2024/0071047) and further in view of Xiao et al (US Patent Application Publication 2015/0347414).
Claim 15: Gao as modified discloses the media of Claim 8 above, but Gao alone does not explicitly disclose wherein determining the value further comprises determining a Mean Reciprocal Rank metric based on the set of document and rank propensities.
However, Xiao [0027-0031] discloses using Mean Reciprocal Rank (MRR) for retrieved documents.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Xiao. One would have been motivated to do so in order to rank results based at least on “probability of correctness.”
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Stegman (2023/0106319) describes at least identifying rankings during an offline phase.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEX GOFMAN whose telephone number is (571)270-1072. The examiner can normally be reached Monday-Friday 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tony Mahmoudi can be reached at 571-272-4078. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEX GOFMAN/Primary Examiner, Art Unit 2163
Prosecution Insights