DETAILED ACTION
Amendment submitted January 20, 2026 has been considered by examiner. Claims 1-20 are pending.
Response to Arguments
Applicant’s arguments with respect to 35 USC 103 rejection have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant's arguments towards 35 USC 101 rejection have been fully considered but they are not persuasive.
The Applicant states that the claims “cannot be reasonably be characterized as mental steps.” The Examiner respectfully disagrees.
The claim limitations recite steps that may be performed in the mind or with help of pen and paper as discussed in a previous rejection. An additional step is currently added, “where the imitation ranker is trained to simulate an output of the current ranker by at least reproducing the ranked sets of documents generated by the current ranker.” However, copying information in order to identify potential ways of ranking those documents is something that may be performed in the mind or with help of pen and paper.
The Claims also do not integrated the abstract idea into a practical application because the steps of the claims, as well as its purported improvement, restate the abstract ideas discussed in the rejection.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1, 8 and 16 are amended to recite “where the imitation ranker is trained to simulate an output of the current ranker by at least reproducing the ranked sets of documents generated by the current ranker.”
Applicant Remarks (Page 7) states that support for the above amended feature “can be found at least at paragraphs [0024] and [0029]…” However, neither the cited paragraphs, nor the rest of the instant specification, discuss “reproducing the ranked sets of documents generated by the current ranker.” As such, it does not seem that there is support for the above amendment in the instant specification and the amendment introduces new matter.
Dependent Claims 2-7, 9-15 and 17-20 do not remedy the above issue.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Independent Claim 1 recites the following features:
obtaining, by an offline evaluation system, log data from a recommendation system including a new ranker, the log data indicating queries and ranked sets of documents generated at least in part by a current ranker of the recommendation system – Retrieving data is considered extra solution activity as per MPEP 2106.05.
training, by the offline evaluation system, an imitation ranker using the log data – Training data is an abstract concept. Specifically, the machine learning described merely recites “apply” or “perform” the abstract idea by merely invoking a computer/machine learning as a tool in its ordinary capacity as described in MPEP 2106.05(f), as well as linking the abstract idea to the field of use of machine learning. Also, based at least on RECENTIVE ANALYTICS, INC. v. FOX CORP, which states, “Today, we hold only that patents that do no more than claim the application of generic machine learning to new data environments, without disclosing improvements to the machine learning models to be applied, are patent ineligible under § 101.”
where the imitation ranker is trained to simulate an output of the current ranker by at least reproducing the ranked sets of documents generated by the current ranker - Copying information in order to identify potential ways of ranking documents is something that may be performed in the mind or with help of pen and paper.
cause the imitation ranker to generate a first result including a set of scores associated with document and rank pairs based on a query, the set of scores indicating a probability of a particular document being associated with a particular rank – Generating scores is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, generating scores may be interpreted as calculating scores, which is an abstract concept invoking mathematical concepts.
obtaining, from a new ranker, a second result including a ranked set of documents in response to the query – Retrieving data is considered extra solution activity as per MPEP 2106.05.
determining, by the offline evaluation system, a rank distribution indicating propensities associated with the document and rank pairs for a set of impressions, where an impression of the set of impression includes a document and rank pair that is included in the first result and the second result – Calculating probabilities is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating probabilities may be interpreted as mathematical concepts, which is an abstract concept.
determining, by the offline evaluation system, a value associated with the new ranker – Determining a value is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating a value may be interpreted as mathematical concepts, which is an abstract concept.
Independent Claim 8 recites the following features:
obtaining a ranked set of documents generated by an imitation ranker based on a query - Retrieving data is considered extra solution activity as per MPEP 2106.05.
where the imitation ranker is trained to simulate an output of the current ranker by at least reproducing the ranked sets of documents generated by the current ranker - Copying information in order to identify potential ways of ranking documents is something that may be performed in the mind or with help of pen and paper.
determining a hyperparameter associated with the imitation ranker to modify the ranked set of documents – Determining hyperparameters is a generic function of machine learning functionality (it is also a well-known, routine and conventional functionality. For example see at least Song et al (2022/0180241).
computing a rank distribution for documents included the ranked set of documents for a set of impressions generated by a new ranker of a recommendation system based on the query - Calculating rank is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating ranks may be interpreted as mathematical concepts, which is an abstract concept.
computing a set of document and rank propensities based on the rank distribution - Calculating probabilities is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating probabilities may be interpreted as mathematical concepts, which is an abstract concept.
determining a value indicating a performance of the new ranker of the recommendation system based on the set of document and rank propensities - Determining a value is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating a value may be interpreted as mathematical concepts, which is an abstract concept.
Independent Claim 16 recites the following features:
generating, by an imitating ranker, a set of document and rank pairs based on a query – Generating seems to be analogous to retrieving in this claim limitation; Retrieving data is considered extra solution activity as per MPEP 2106.05.
where the imitation ranker is trained to simulate an output of the current ranker by at least reproducing the ranked sets of documents generated by the current ranker - Copying information in order to identify potential ways of ranking documents is something that may be performed in the mind or with help of pen and paper.
determining, by an offline evaluation system, a set of document and rank propensities based on a rank distribution computed for a set of impressions generated by a ranker based on the query, the set of impression including documents included in the set of document and rank pairs - Calculating probabilities is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating probabilities may be interpreted as mathematical concepts, which is an abstract concept.
determining, by the offline evaluation system, a metric indicating a performance of the ranker based on the set of document and rank propensities - Determining a value is a mental process that may be performed in a person’s mind or by a person using a pen and paper. Also, calculating a value may be interpreted as mathematical concepts, which is an abstract concept.
This judicial exception is not integrated into a practical application. Other than the abstract idea, the claims recite hardware elements such as a memory, a processor, etc. However, the hardware elements are recited at a high level of generality, i.e. as generic computer components performing generic computer functions of information.
As to dependent Claims 2-7, 9-15 and 17-20, these claims fail to recite significantly more than the abstract idea. Rather, the Claims recite more details of
mentally processing graph data, which is additional recitations of the abstract idea identified above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 7 are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al (US Patent Application Publication 2010/0082582) in view of Wang et al (US Patent Application Publication 2017/0249312) and further in view of Stouffer et al (US Patent Application Publication 2015/0379141).
Claim 1: Gao discloses a computer-implemented method comprising:
obtaining, , log data from a recommendation system including a new ranker, the log data indicating queries and ranked sets of documents generated at least in part by a current ranker of the recommendation system [0036]. [See at least retrieving queries, documents (i.e. ID) and a score for ranking.]
Gao alone does not explicitly disclose the rest of the limitations.
However, Gao, Wang, and Stouffer disclose:
training, by the offline evaluation system, an imitation ranker using the log data [Wang [0037, 0061-0062] discloses offline training for a ranker using at least logs.]
cause the imitation ranker to generate a first result including a set of scores associated with document and rank pairs based on a query, the set of scores indicating a probability that the current ranker would determine a particular document is associated with a particular rank [Gao [0035-0036] describes at least relevance scores based on probability for query-id pairs.]
obtaining, from a new ranker, a second result including a ranked set of documents in response to the query [Gao [0036] describes at least retrieving queries, documents (i.e. ID) and a score for ranking. [Gao [0060] describes at least calculating scores for a second time.]
determining, by the offline evaluation system, a rank distribution indicating propensities associated with the document and rank pairs for a set of impressions, where an impression of the set of impression includes a document and rank pair that is included in the first result and the second result [Gao [0035-0036] describes at least relevance scores based on probability for query-id pairs and Wang [0061-0062] discloses offline evaluation.]
determining, by the offline evaluation system, a value associated with the new ranker [Gao [0035-0036] describes at least relevance scores based on probability for query-id pairs and Wang [0037, 0061-0062] discloses offline evaluation.]
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Wang. One would have been motivated to do so in order to decrease processing power of an online system.
Gao as modified also does not explicitly disclose where the imitation ranker is trained to simulate an output of the current ranker by at least reproducing the ranked sets of documents generated by the current ranker.
However, Stouffer [0064, 0074-0076] discloses simulating changes in the ranking of document. The simulated environment may be interpreted as a reproduced environment for the purpose of simulation.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Stouffer. One would have been motivated to do so in order to “to identify the optimal changes” for searching and retrieving results.
Claim 2: Gao as modified discloses the method of Claim 1 above, and Gao further discloses wherein the recommendation system includes a search engine [0020].
Claim 3: Gao as modified discloses the method of Claim 1 above, and Gao further discloses wherein the new ranker includes one or more modifications to the current ranker [0037-0038]. [See at least modifying a weighting factor.]
Claim 4: Gao as modified discloses the method of Claim 3 above, and Gao further discloses wherein the value includes a metric indicating a performance of the one or more modifications [0037-0038].
Claim 5: Gao as modified discloses the method of Claim 3 above, and Gao further discloses wherein the metric includes at least one of a relevance metric, an impression-level relevance metric, number of click, mean reciprocal rank, Kendall tau, expected reciprocal rank, mean average precision, precision at k, and normalize discounted cumulative gain [0041].
Claim 7: Gao as modified discloses the method of Claim 1 above, and Gao further discloses wherein the computer- implemented method further comprises determining a second value associated with a second ranker based at least in part on the imitation ranker without re-training the imitation ranker [0037-0038].
Claims 6 and 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al (US Patent Application Publication 2010/0082582) in view of Wang et al (US Patent Application Publication 2017/0249312) further in view of Stouffer et al (US Patent Application Publication 2015/0379141) and further in view of Song et al (US Patent Application Publication 2022/0180241).
Claim 6: Gao as modified discloses the method of Claim 1 above, but Gao alone does not explicitly disclose wherein the computer- implemented method further comprises tuning the imitation ranker using one or more hyperparameters.
However, Song [0021] discloses tuning hyperparameters for a machine learning model.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Song. One would have been motivated to do so in order to “increase or decrease the rate at which the machine learning model learns from training data, which in turn affects the model's efficiency in generating accurate predictions.”
Claim 16: Gao discloses a computer system comprising: a processor; and a computer storage medium storing computer-useable instructions that, when used by the processor, causes the computer system to perform operations comprising:
generating, by an imitating ranker, a set of document and rank pairs based on a query [0036]. [See at least retrieving documents (i.e. ID) and a score for ranking.]
Gao alone does not explicitly disclose the rest of the limitations.
However, Gao and Wang disclose:
determining, by an offline evaluation system, a set of document and rank propensities based on a rank distribution computed for a set of impressions generated by a ranker based on the query, the set of impression including documents included in the set of document and rank pairs [Gao [0035-0036] describes at least relevance scores based on probability and Wang [0061-0062] discloses offline evaluation.]
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Wang. One would have been motivated to do so in order to decrease processing power of an online system.
Song further discloses: determining, by the offline evaluation system, a metric indicating a performance of the ranker based on the set of document and rank propensities [0019-0020]. [See at least identifying a performance value.]
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Song. One would have been motivated to do so in order to identify a performance level at which a system functions.
Gao as modified also does not explicitly disclose where the imitation ranker is trained to simulate an output of the current ranker by at least reproducing the ranked sets of documents generated by the current ranker.
However, Stouffer [0064, 0074-0076] discloses simulating changes in the ranking of document. The simulated environment may be interpreted as a reproduced environment for the purpose of simulation.
Claim 17: Gao as modified discloses the system of Claim 16 above, but Gao alone does not explicitly disclose wherein generating the set of documents further comprise tuning the imitating ranker using a hyperparameter.
However, Song [0021] discloses using hyperparameters for a machine learning model.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Song. One would have been motivated to do so in order to “increase or decrease the rate at which the machine learning model learns from training data, which in turn affects the model's efficiency in generating accurate predictions.”
Claim 18: Gao as modified discloses the system of Claim 17 above and Gao in view of Song disclose wherein a value of the hyperparameter is determined based at least in part on the set of document and rank pairs. Gao [0036] describes at least retrieving documents (i.e. ID) and a score for ranking; And Song, for the same reason as above discloses using an input, such as in Gao to tune hyperparameters.
Claim 19: Gao as modified discloses the system of Claim 16 above, and Gao further discloses wherein the set of document and rank pairs includes a score for a document at a rank in a ranked set of documents [0036].
Claim 20: Gao as modified discloses the system of Claim 16 above, and Gao further discloses wherein the metric includes at least one of. a relevance metric, an impression-level relevance metric, number of click, mean reciprocal rank, Kendall tau, expected reciprocal rank, mean average precision, precision at k, and normalize discounted cumulative gain [0041].
Claims 8-14 are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al (US Patent Application Publication 2010/0082582) in view of Song et al (US Patent Application Publication 2022/0180241) and further in view of Stouffer et al (US Patent Application Publication 2015/0379141).
Claim 8: Gao discloses one or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations, the operations comprising:
obtaining a ranked set of documents generated by an imitation ranker based on a query [0036-0037]. [See at least retrieving ranked documents.]
Gao alone does not explicitly disclose determining a hyperparameter associated with the imitation ranker to modify the ranked set of documents.
However, Song [0021] discloses using hyperparameters for a machine learning model.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Song. One would have been motivated to do so in order to “increase or decrease the rate at which the machine learning model learns from training data, which in turn affects the model's efficiency in generating accurate predictions.”
Gao as modified also does not explicitly disclose where the imitation ranker is trained to simulate an output of the current ranker by at least reproducing the ranked sets of documents generated by the current ranker.
However, Stouffer [0064, 0074-0076] discloses simulating changes in the ranking of document. The simulated environment may be interpreted as a reproduced environment for the purpose of simulation.
Gao as modified further discloses:
computing a rank distribution for documents included the ranked set of documents for a set of impressions generated by a new ranker of a recommendation system based on the query [0031, 0035-0036]. [See at least relevance scores based on probability for query-id pairs. Also see term frequency.]
computing a set of document and rank propensities based on the rank distribution [0031, 0035-0036]. [See at least relevance scores based on probability.]
determining a value indicating a performance of the new ranker of the recommendation system based on the set of document and rank propensities [0035-0036]. [See at least relevance scores based on probability for query-id pairs. Also see Song [0019-0020] for identifying a performance value.]
Claim 9: Gao as modified discloses the media of Claim 8 above, and Gao further discloses wherein the recommendation system includes a current ranker that generates log data used to train the imitation ranker [0035-0036].
Claim 10: Gao as modified discloses the media of Claim 9 above, and Gao further discloses wherein the new ranker includes a set of changes to the current ranker and the new ranker generates the set of impressions [0037-0038]. [See at least modifying a weighting factor.]
Claim 11: Gao as modified discloses the media of Claim 10 above, and Song [0021], for the same reasons as above, further discloses wherein the set of changes includes at least one of: a new ranking feature, a modification to a ranking model, a modification to a parameter of the current ranker, and a modification to a hyperparameter of the current ranker. [See at least tuning hyperparameters for a machine learning model.]
Claim 12: Gao as modified discloses the media of Claim 8 above, and Gao further discloses wherein the operations further comprise training the imitation ranker using log data obtained from a second recommendation system [0036]. [See at least retrieving queries, documents (i.e. ID) and a score for ranking. Furthermore, Gao discloses retrieving data from a recommendation system. However, using another recommendation system would not change the overall functionality of Gao’s process. As such, it would have bene obvious for one of ordinary skill in the art before the effective filing date to modify Gao to process data from different systems in order to process data from any system that is specified.]
Claim 13: Gao as modified discloses the media of Claim 12 above, and Gao further discloses wherein the log data includes an indication of at least a ranked set of documents and a user interaction with a document of the ranked set of document generated in response to the query [0033, 0036, 0040].
Claim 14: Gao as modified discloses the media of Claim 8 above, and Gao further discloses wherein determining the value further comprises determining a Number of Clicks metric based on the set of document and rank propensities [0033].
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Gao et al (US Patent Application Publication 2010/0082582) in view of Song et al (US Patent Application Publication 2022/0180241) further in view of Stouffer et al (US Patent Application Publication 2015/0379141) and further in view of Xiao et al (US Patent Application Publication 2015/0347414).
Claim 15: Gao as modified discloses the media of Claim 8 above, but Gao alone does not explicitly disclose wherein determining the value further comprises determining a Mean Reciprocal Rank metric based on the set of document and rank propensities.
However, Xiao [0027-0031] discloses using Mean Reciprocal Rank (MRR) for retrieved documents.
As such, it would have been obvious for one of ordinary skill in the art before the effective filing date to modify Gao with Xiao. One would have been motivated to do so in order to rank results based at least on “probability of correctness.”
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEX GOFMAN whose telephone number is (571)270-1072. The examiner can normally be reached Monday-Friday 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tony Mahmoudi can be reached at 571-272-4078. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALEX GOFMAN/Primary Examiner, Art Unit 2163