DETAILED ACTION
1. This office action is in response to applicant’s communication filed on 08/12/2026 in response to PTO Office Action mailed 05/12/2026. The Applicant’s remarks and amendments to the claims and/or the specification were considered with the results as follows.
2. In response to the last Office Action, no claims are amended, added or canceled. As a result, claims 21-40 are pending in this office action.
Response to Arguments
3. Applicant's arguments with respect to 35 USC 103 have been fully considered but are not persuasive and the details are as follow:
Applicant’s argument stated as “Ho does not teach processing the list of documents of discard documents based on the indicated responsive document by the machine learning model…Ho does not discard any of the additional text collections from the training data set based on the user interactions. Instead, in Ho, the machine learning model can be trained with only the set training document which is generated to include the set of text collections received since the machine learning model was last trained…”.
In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). As acknowledged in the prior Office Action, Shishkin discloses processing the list of documents using a machine-learning model and output respective class-association values for the documents (See Shishkin, para. [0085], para. [0161] and para. [0202]). Shishkin further discloses associating query-document pairs with assessed classes and user-feedback data, including click-through information indicating user interaction with a respective document, and using those query-document pairs as training objects (See Shishkin, para. [0085], para. [0097] and para. [0103] -para. [0107]). Additionally, Shishkin teaches that class-association values may be determined based on user’s selection of a respective document (See Shishkin, para. [0207]). Ho is relied upon for the feature missing from Shishkin. Specifically, Ho teaches that a subsequent machine-learning model may be trained using an updated set of training documents formed by adding or removing one or more documents from an initial set of documents (See Ho, para. [0028]). Ho further discloses updating a corpus of training documents using additional text collections generated from user interactions and retraining the machine-learning model using the updated training documents (See Ho, para. [0066] and para. [0067]). Thus, in the proposed combination, Shishkin’s user-feedback and class-association information identifies the responsive document and distinguishes that document from other documents in the list, while Ho teaches modifying the training-document set by removing documents. It would have been obvious to use Shishkin’s indication of which document is response when determining which documents should be retained or removed from the training set, such that documents not corresponding to the indicated responsive documents are discarded. Therefore, the combination of Shishkin and Ho teaches or suggests processing the list of documents to discard document based on the indicated responsive document.
Double Patenting
5. The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory obviousness-type double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the conflicting application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement.
Effective January 1, 1994, a registered attorney or agent of record may sign a terminal disclaimer. A terminal disclaimer signed by the assignee must fully comply with 37 CFR 3.73(b).
6. Claims 21-40 are rejected on the ground of nonstatutory obvious double patenting over claims 1-17 of Patent No.: US 12,292,895 B1. The subject matter claimed in the instant application is disclosed in the Patent No.: US 12,292,895 B1. For example:
Patent No.: US 12,292,895 B2
Instant Application: 19/197,329
1. A method for ranking documents in
search results, the method comprising:
retrieving a plurality of search result sets, each search result set being associated with a user query and comprising an ordered plurality of documents;
defining a training data set based on the plurality of search result sets by, for each search result set:
determining an observation window including a pre-defined number of documents ordered after a responsive document from the search result set, the responsive document representative of a document selected by a user from the search result set; and
discarding documents from the search result set that are ordered below the pre-defined number of documents after the responsive document; training a machine learning model via the training data set; receiving a further user query; presenting a list of responsive documents ranked by the trained machine learning model; receiving indication of a responsive document from the presented list of responsive documents; processing the list of responsive documents to discard documents from the list of responsive documents that are outside of the observation window from the indicated responsive document; and adding the processed list of responsive documents to the training data set, wherein, by not including the discarded documents, the training data set reduces a bias representative of a user selection of the responsive document over the discarded documents at the machine learning model.
7. A system comprising: a non-transitory, computer-readable medium storing instructions; and a processor configured to execute the instructions to:
retrieve a plurality of search result sets, each search result set being associated with a user query and comprising an ordered plurality of documents; define a training data set based on the plurality of search result sets by, for each search result set:
determine an observation window including a pre-defined number of documents ordered after a responsive document from the search result set, the responsive document representative of a document selected by a user from the search result set; and discard documents from the search result set that are ordered below the pre-defined number of documents after the responsive document; train a machine learning model via the training data set; receive a further user query; present a list of responsive documents ranked by the trained machine learning model; receiving indication of a responsive document from the presented list of responsive documents; processing the list of responsive documents to discard documents from the list of responsive documents that are outside of the observation window from the indicated responsive document; and adding the processed list of responsive documents to the training data set, wherein, by not including the discarded documents, the training data set reduces a bias representative of a user selection of the responsive document over the discarded documents at the machine learning model.
13. A method for presenting search results, the method comprising: retrieving a plurality of search result sets, each search result set being associated with a user query and comprising an ordered plurality of documents; defining a first training data set based on the plurality of search result sets by, for each search result set:
determining an observation window including a pre-defined number of documents ordered after a responsive document from the search result set, the responsive document representative of a document selected by a user from the search result set; and discarding documents from the search result set that are ordered below the pre-defined number of documents after the responsive document; training a first machine learning model via the first training data set; defining a second training data set based on the responsive document from each of the plurality of search result sets; training a second machine learning model via the second training data set; receiving a further user query; presenting a list of responsive documents ranked according to the first and second trained machine learning models; receiving indication of a responsive document from the presented list of responsive documents; processing the list of responsive documents to discard documents from the list of responsive documents that are outside of the observation window from the indicated responsive document; and adding the processed list of responsive documents to the training data set, wherein, by not including the discarded documents, the first training data set reduces a bias representative of a user selection of the responsive document over the discarded documents at the first machine learning model.
21. A method for ranking documents in search results, the method comprising: receiving a search query from a user; providing the search query to a machine learning model, the machine learning model trained via a training data set including user click behavior of a plurality of users;
presenting a search result set comprising a list of documents ranked by the trained machine learning model based on the user click behavior;
receiving an indication of a responsive document representative of a document selected by the user from the presented list of documents;
processing the list of documents to discard documents based on the indicated responsive document by the machine learning model; and adding the processed list of responsive documents to the training data set; wherein, by not including the discarded documents, the training data set reduces a bias representative of a user selection of the responsive document over the discarded documents at the machine learning model.
22. The method of claim 21, further comprising: determining an observation window including a pre-defined number of documents ordered after the responsive document from the search result set by the machine learning model; discarding documents from the search result set that are ordered below the pre-defined number of documents after the indicated responsive document by the machine learning model; and training the machine learning model via the training data set; wherein the training data set comprises an initial training data set and the processed list of responsive documents.
30. A system comprising: a non-transitory, computer-readable medium storing instructions; and a processor configured to execute the instructions to:
receive a search query from a user; provide the search query to a machine learning model, the machine learning model trained via a training data set including user click behavior of a plurality of users; present a search result set comprising a list of documents ranked by the trained machine learning model based on the user click behavior; receive an indication of a responsive document representative of a document selected by the user from the presented list of documents;
determine an observation window including a pre-defined number of documents ordered after the responsive document from the search result set by the machine learning model; process the list of documents to discard documents from the search result set that are ordered below the pre-defined number of documents after the indicated responsive document by the machine learning model; and add the processed list of responsive documents to the training data set; wherein, by not including the discarded documents, the training data set reduces a bias representative of a user selection of the responsive document over the discarded documents at the machine learning model.
data set; wherein the training data set comprises an initial training data set and the processed list of responsive documents.
36. A computer-implemented method comprising: receiving a search query from a user; providing the search query to a machine learning model, the machine learning model trained via a training data set including user click behavior of a plurality of users; presenting a search result set comprising a list of documents ranked by the trained machine learning model based on the user click behavior; receiving an indication of a responsive document representative of a document selected by the user from the presented list of documents; determining an observation window including a pre-defined number of documents ordered after the responsive document from the search result set by the machine learning model; processing the list of documents to
discard documents from the search result set that are ordered below the pre-defined number of documents after the indicated responsive document by the machine learning model; adding the processed list of responsive documents to a training data set; and training the machine learning model via the training data set; wherein the training data set comprises an initial training data set and the processed list of responsive documents; wherein, by not including the discarded documents, the training data set reduces a bias representative of a user selection of the responsive document over the discarded documents at the machine learning model.
initial training data set and the processed list of responsive documents.
machine learning model according to the one or more current search result sets.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Shishkin (US 2018/0293242 A1) and in view of Ho (CA 3088560 A1).
Referring to claim 21, Shishkin discloses a method for ranking documents in search results (See para. [0025], ranking a plurality of documents on a search results page), the method comprising: receiving a search query from a user (See para. [0012], ranking a plurality of documents on a search engine page in response to a query associated with a user);
providing the search query to a machine learning model (See para. [0006] and para. [0075], the system maintains a pool of documents, a ranking model is used to rank documents in response to the query), the machine learning model trained via a training data set including user click behavior of a plurality of users (See para. [0104]-para. [0108]; providing the search query to a training server, the ranking model is trained via a training process in order to compute a given ranking score for a given document based on a given query-document pair and a user feedback data point associated with the document of the given query-document pair, note in para. [0097], the user feedback data point is indicative of user interactions with the training documents, the click-through data is gathered from users);
presenting a search result set comprising a list of documents ranked by the trained
machine learning model based on the user click behavior (See para. [0114] and para. [0115] and Figure 5, the ranking server is configured to compute the given ranking score for each ranking input set that was inputted, for example, a first ranking score for a first ranking input set that comprises the query 504, the document 511 and previous user feedback data point 521);
receiving an indication of a responsive document representative of a document selected
by the user from the presented list of documents (See para. [0097] and [0207], receiving user feedback or user selection of a respective document, the user feedback comprises information related to a number of times the document has been clicked or selected, an amount of time that users spent viewing the document, a number of clicks executed by users while viewing the training document);
processing the list of documents based on the indicated responsive document by the machine learning model (See para. [0085], para. [0161] and para. [0202], the ranking server outputs a list of documents with associated class values [e.g. 1s t- 7th]); and adding the processed list of responsive documents to the training data set (See para. [0085], the ranking server associates the click-through data with a training document set); wherein, by not including the documents from [other classes], the training data set reduces a bias representative of a user selection of the responsive document over the [other class] documents at the machine learning model (See para. [0085] and para. [0097], para. [0103] and para. [0106] and para. [0107], the training database stores a large number of query-document pairs related to a respective assessed class associated with each query-document pair, the training database also stores user feedback data points associated with each respective query-document pair, note in para. [0105] a respective user feedback data point related to each query-document pair in a set of training objects can be omitted).
Shishkin does not explicitly disclose processing a list of documents to discard documents such that documents of other classes are not included in the training data set.
Ho discloses processing a list of documents to discard documents by a machine learning model (See para. [0028], para. [0064]-para. [0067] and Figure 2, updating a training-document set by adding or removing one or more documents from an initial set of documents, a subsequent machine-learning model may be trained using an updated set of training documents formed by adding or removing one or more documents from an initial set of documents, updating a corpus of training documents to include additional text collections generated from user interactions and retraining the machine-learning model using the updated training documents).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shishkin’s training process to update the training-document set by removing documents, as taught by Ho, in order to improve the relevance and the quality of the training data supplied by the machine-learning model by retaining documents associated with responsive user feedback while excluding comparatively non-responsive documents. Such a modification would have amounted to apply Ho’s known technique of training-set curation to Shishkin’s machine-learning training system to obtain the predictable result of a more relevant training set.
Claims 22, 23, 26-31 and 33-37 are rejected under 35 U.S.C. 103 as being unpatentable over Shishkin (US 2018/0293242 A1) and in view of Ho (CA 3088560 A1) and further in view of Collins et al. (US 2015/0039589 A1).
As to claim 22, Shishkin in view Ho discloses discarding document from the search results that after the indicated responsive document by the machine model; and training the machine learning model via the training data set, wherein the training data set comprises an initial training data set and the processed list of responsive documents (See Ho, Figure 2, para. [0028], para. [0064]- para. [0067] updating training documents to include additional text collections which are associated with presented links via a client device and removing one or more documents from the initial set of documents).
Shishkin in view Ho does not explicitly disclose determining an observation window including a pre-defined number of documents ordered after the responsive document from the result set and discarding documents from the search result set that are ordered bellow the pre-defined number documents.
Collins discloses determining an observation window including a pre-defined number of documents ordered after the responsive document from the result set and discarding documents from the search result set that are ordered bellow the pre-defined number documents (See para. [0053]- para. [0056], discarding or filtering query results or documents in response to a received query, a top N (e.g., five) entities can be determined or defined for each top result based on the total number of occurrences of each determined entity or document in the top results, the system filters an entity/document in the query does not appear at all in the top results or appears less than a first threshold number of times (e.g., two) in the top results).
Therefore, it would have been obvious to a person of ordinary skill in the computer art to modify the Shishkin’s ranking system to discard documents from search result sets that are ordered below a pre-defined number of documents, as taught by Collins. Skilled artisan would have been motivated to reduce the number of irrelevant results and improve user searching experience (See Collins, para. [0016]). In addition, all of the references (Shishkin, Ho and Collins) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as improving search results in search engines. This close relation between all of the references highly suggests an expectation of success.
As to claim 23, Shishkin discloses retrieving a plurality of search result sets, each search result set being associated with a respective user query of a plurality of user queries and comprising an ordered plurality of documents; defining the initial training data set based on the plurality of search result sets; and training the machine learning model via the initial training data set (See para. [0085] and para. [0097], para. [0103] and para. [0106] and para. [0107], the training database stores a large number of query-document pairs related to a respective assessed class associated each query-document pair, the training database also stores user feedback data points associated with each respective query-document pair).
As to claim 26, Shishkin discloses conducting reinforcement learning on the machine learning model to maximize a prediction accuracy of the machine learning model (See para. [0005] and para. [0075], the ranking application includes reinforcement learning to rank relevant documents).
As to claim 27, Shishkin discloses a plurality of previous search result sets; and
one or more current search result sets (See para. [0201] and Figure 9, a comparison between original ranks of each of the plurality of documents and amended ranks of each one of the plurality of documents).
As to claim 28, Shishkin discloses wherein training the machine learning model via the
initial training data set comprises batch training the machine learning model according to the previous search result sets (See para. [0085], para. [0096], the training server comprises class training with respect to previous query-document pairs); and conducting reinforcement learning on the machine learning model according to the one or more current search result sets (See para. [0005] and para. [0075], the ranking application comprises reinforcement learning to rank on-going query-document pairs).
As to claim 29, Shishkin does not explicitly disclose wherein the observation window includes between one and three documents after the responsive document.
Collins discloses wherein the observation window includes between one and three documents after the responsive document (See para. [0053]-para. [0056], a top N (e.g., three, four or five) entities can be determined or defined for each top result based on the total number of occurrences of each determined entity or document in the top results, the system filters an entity/document in the query does not appear at all in the top results or appears less than a first threshold number of times (e.g., three) in the top results).
Therefore, it would have been obvious to a person of ordinary skill in the computer art to modify the Shishkin’s ranking system to include an observation window, as taught by Collins. Skilled artisan would have been motivated to reduce the number of irrelevant results and improve user searching experience (See Collins, para. [0016]). In addition, all of the references (Shishkin, Ho and Collins) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as improving search results in search engines. This close relation between all of the references highly suggests an expectation of success.
Referring to claim 30, Shishkin discloses a system comprising:
a non-transitory, computer-readable medium storing instructions (See para. [0043] and para. [0088] and Figure 3); and a processor (See para. [0044] and para. [0043]) configured to execute the instructions to:
receive a search query from a user (See para. [0012], ranking a plurality of documents on a search engine page in response to a query associated with a user of a search engine);
provide the search query to a machine learning model, the machine learning
model trained via a training data set including user click behavior of a plurality of users (See para. [0104]-para. [0108]; providing the search query to a training server, the ranking model is trained via a training process in order to compute a given ranking score for a given document based on a given query-document pair and a user feedback data point associated with the document of the given query-document pair, note in para. [0097], the user feedback data point is indicative of user interactions with the training documents, the click-through data is gathered from users);
present a search result set comprising a list of documents ranked by the trained
machine learning model based on the user click behavior (See para. [0114] and para. [0115] and Figure 5, the ranking server is configured to compute the given ranking score for each ranking input set that was inputted, for example, a first ranking score for a first ranking input set that comprises the query 504, the document 511 and previous user feedback data point 521);
receive an indication of a responsive document representative of a document
selected by the user from the presented list of documents (See para. [0097] and [0207], receiving user feedback or user selection of a respective document, the user feedback comprises information related to a number of times the document has been clicked or selected, an amount of time that users spent viewing the document, a number of clicks executed by users while viewing the training document);
process the list of documents […] based on the indicated
responsive document by the machine learning model (See para. [0085], para. [0161] and para. [0202], the ranking server outputs a list of documents with associated class values [e.g. 1s t- 7th]; and add the processed list of responsive documents to the training data set (See para. [0085], the ranking server associates the click-through data with a training document set); wherein, by not including the documents from [other classes], the training data set reduces a bias representative of a user selection of the responsive document over the
[other classes] at the machine learning model See para. [0085] and para. [0097], para. [0103] and para. [0106] and para. [0107], the training database stores a large number of query-document pairs related to a respective assessed class associated each query-document pair, the training database also stores user feedback data points associated with each respective query-document pair, note in para. [0105] a respective user feedback data point related to each query-document pair in a set of training objects can be omitted).
Shishkin does not explicitly disclose processing a list of documents to discard documents such that documents of other classes are not included in the training data set.
Ho discloses processing a list of documents to discard documents by a machine learning model (See para. [0028], para. [0064]-para. [0067] and Figure 2, updating a training-document set by adding or removing one or more documents from an initial set of documents, a subsequent machine-learning model may be trained using an updated set of training documents formed by adding or removing one or more documents from an initial set of documents, updating a corpus of training documents to include additional text collections generated from user interactions and retraining the machine-learning model using the updated training documents).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shishkin’s training process to update the training-document set by removing documents, as taught by Ho, in order to improve the relevance and the quality of the training data supplied by the machine-learning model by retaining documents associated with responsive user feedback while excluding comparatively non-responsive documents. Such a modification would have amounted to apply Ho’s known technique of training-set curation to Shishkin’s machine-learning training system to obtain the predictable result of a more relevant training set.
Shishkin in view Ho does not explicitly disclose determining an observation window including a pre-defined number of documents ordered after the responsive document from the result set.
Collins discloses determining an observation window including a pre-defined number of documents ordered after the responsive document from the result set and discarding documents from the search result set that are ordered below the pre-defined number documents (See para. [0053]- para. [0056], discarding or filtering query results or documents in response to a received query, a top N (e.g., five) entities can be determined or defined for each top result based on the total number of occurrences of each determined entity or document in the top results, the system filters an entity/document in the query does not appear at all in the top results or appears less than a first threshold number of times (e.g., two) in the top results).
Therefore, it would have been obvious to a person of ordinary skill in the computer art to modify the Shishkin/Ho’s training system to retain a predefined number of ranked documents and discard documents falling outside the predefined number, as taught by Collins, in order to reduce irrelevant search results and improve the usefulness of the resulting search data (See Collins, para. [0016]). In addition, all of the references (Shishkin, Ho and Collins) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as improving search results in search engines. This close relation between all of the references highly suggests an expectation of success.
As to claim 31, Shishkin discloses retrieving a plurality of search result sets, each search result set being associated with a respective user query of a plurality of user queries and comprising an ordered plurality of documents; defining an initial training dataset based on the plurality of search result sets; and training the machine learning model via the training data set;
wherein the training data set comprises the initial training data set and the processed list
of responsive documents (See para. [0085] and para. [0097], para. [0103] and para. [0106] and para. [0107], the training database stores a large number of query-document pairs related to a respective assessed class associated each query-document pair, the training database also stores user feedback data points associated with each respective query-document pair).
As to claim 33, Shishkin discloses wherein the plurality of search result sets comprises:
a plurality of previous search result sets; and one or more current search result sets (See para. [0201] and Figure 9, a comparison between original ranks of each of the plurality of documents and amended ranks of each one of the plurality of documents).
As to claim 34, Shishkin discloses wherein training the machine learning model via the
initial training data set comprises: batch training the machine learning model according to the previous search result sets (See para. [0085], para. [0096], the training server comprises class training with respect to previous query-document pairs); and conducting reinforcement learning on the machine learning model according to the one or more current search result sets (See para. [0005] and para. [0075], the ranking application comprises reinforcement learning to rank on-going query-document pairs).
As to claim 35, Shishkin does not explicitly disclose wherein the observation window includes between one and three documents after the responsive document.
Collins discloses wherein the observation window includes between one and three documents after the responsive document (See para. [0053]-para. [0056], a top N (e.g., three, four or five) entities can be determined or defined for each top result based on the total number of occurrences of each determined entity or document in the top results, the system filters an entity/document in the query does not appear at all in the top results or appears less than a first threshold number of times (e.g., three) in the top results).
Therefore, it would have been obvious to a person of ordinary skill in the computer art to modify the Shishkin’s ranking system to include an observation window, as taught by Collins. Skilled artisan would have been motivated to reduce the number of irrelevant results and improve user searching experience (See Collins, para. [0016]). In addition, all of the references (Shishkin, Ho and Collins) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as improving search results in search engines. This close relation between all of the references highly suggests an expectation of success
Referring to claim 36, Shishkin discloses a computer-implemented method comprising:
receiving a search query from a user (See para. [0012], ranking a plurality of documents on a search engine page in response to a query associated with a user of a search engine);
providing the search query to a machine learning model, the machine learning model
trained via a training data set including user click behavior of a plurality of users (See para. [0104]-para. [0108]; providing the search query to a training server, the ranking model is trained via a training process in order to compute a given ranking score for a given document based on a given query-document pair and a user feedback data point associated with the document of the given query-document pair, note in para. [0097], the user feedback data point is indicative of user interactions with the training documents, the click-through data is gathered from users);
presenting a search result set comprising a list of documents ranked by the trained
machine learning model based on the user click behavior (See para. [0114] and para. [0115] and Figure 5, the ranking server is configured to compute the given ranking score for each ranking input set that was inputted, for example, a first ranking score for a first ranking input set that comprises the query 504, the document 511 and previous user feedback data point 521);
receiving an indication of a responsive document representative of a document selected
by the user from the presented list of documents (See para. [0097] and [0207], receiving user feedback or user selection of a respective document, the user feedback comprises information related to a number of times the document has been clicked or selected, an amount of time that users spent viewing the document, a number of clicks executed by users while viewing the training document); […]
processing the list of documents […] based on the indicated responsive
document by the machine learning model (See para. [0085], para. [0161] and para. [0202], the ranking server outputs a list of documents with associated class values [e.g. 1s t- 7th]);
adding the processed list of responsive documents to a training data set (See para. [0085], para. [0161] and para. [0202], the ranking server outputs a list of documents with associated class values [e.g. 1s t- 7th]);
training the machine learning model via the training data set; wherein the training data set comprises an initial training data set and the processed list of responsive documents (See para. [0085] and para. [0097], para. [0103] and para. [0106] and para. [0107], the training database stores a large number of query-document pairs related to a respective assessed class associated each query-document pair, the training database also stores user feedback data points associated with each respective query-document pair) and wherein by not including from [other classes], the training data set reduces a bias representative of a user selection of the responsive document over [other classes] documents at the machine learning model (See para. [0085] and para. [0097], para. [0103] and para. [0106] and para. [0107], the training database stores a large number of query-document pairs related to a respective assessed class associated with each query-document pair, the training database also stores user feedback data points associated with each respective query-document pair, note in para. [0105] a respective user feedback data point related to each query-document pair in a set of training objects can be omitted).
Shishkin does not explicitly disclose processing a list of documents to discard documents such that documents of other classes are not included in the training data set.
Ho discloses processing a list of documents to discard documents by a machine learning model (See para. [0028], para. [0064]-para. [0067] and Figure 2, updating a training-document set by adding or removing one or more documents from an initial set of documents, a subsequent machine-learning model may be trained using an updated set of training documents formed by adding or removing one or more documents from an initial set of documents, updating a corpus of training documents to include additional text collections generated from user interactions and retraining the machine-learning model using the updated training documents).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shishkin’s training process to update the training-document set by removing documents, as taught by Ho, in order to improve the relevance and the quality of the training data supplied by the machine-learning model by retaining documents associated with responsive user feedback while excluding comparatively non-responsive documents. Such a modification would have amounted to apply Ho’s known technique of training-set curation to Shishkin’s machine-learning training system to obtain the predictable result of a more relevant training set.
Shishkin in view Ho does not explicitly disclose determining an observation window including a pre-defined number of documents ordered after the responsive document from the result set.
Collins discloses determining an observation window including a pre-defined number of documents ordered after the responsive document from the result set and discarding documents from the search result set that are ordered below the pre-defined number documents (See para. [0053]- para. [0056], discarding or filtering query results or documents in response to a received query, a top N (e.g., five) entities can be determined or defined for each top result based on the total number of occurrences of each determined entity or document in the top results, the system filters an entity/document in the query does not appear at all in the top results or appears less than a first threshold number of times (e.g., two) in the top results).
Therefore, it would have been obvious to a person of ordinary skill in the computer art to modify the Shishkin/Ho’s training system to retain a predefined number of ranked documents and discard documents falling outside the predefined number, as taught by Collins, in order to reduce irrelevant search results and improve the usefulness of the resulting search data (See Collins, para. [0016]). In addition, all of the references (Shishkin, Ho and Collins) teach features that are directed to analogous art and they are directed to the same field of endeavor, such as improving search results in search engines. This close relation between all of the references highly suggests an expectation of success.
As to claim 37, Shishkin discloses retrieving a plurality of search result sets, each search result set being associated with a respective user query of a plurality of user queries and comprising an ordered plurality of documents; defining the initial training data set based on the plurality of search result sets; and training the machine learning model via the initial training data set and wherein the training data set comprises the initial training data set and the processed list of responsive documents (See para. [0085] and para. [0097], para. [0103] and para. [0106] and para. [0107], the training database stores a large number of query-document pairs related to a respective assessed class associated with each query-document pair, the training database also stores user feedback data points associated with each respective query-document pair).
Allowable Subject Matter
Claims 24, 25, 32, 38-40 are objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YUK TING CHOI whose telephone number is (571)270-1637. The examiner can normally be reached Monday-Friday 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, AMY NG can be reached at 5712701698. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YUK TING CHOI/Primary Examiner, Art Unit 2164
Prosecution Insights