DETAILED ACTION
This Office Action is in response to the remarks entered on 09/03/2025. Claims 1, 8, 15 are amended. Claims 1-20 are pending.
Notice of Pre-AIA or AIA Status
The present application is being examined under the pre-AIA first to invent provisions.
Claim Objections
Claims 1, 8 and 15 are objected to because of the following informalities: claims 1 and 8 recite the word “category”. This word should be replaced by “category”. Claim 15 recites “A method document category review”, and it should recite “A method for document category review”; in addition, it further recites the word “catagories”, which should recite “category”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Independent Claims 1, 8 and 15 recite “training machine learning for the coding category utilizing an adaptive identification cycle trained with the initial set of review documents” and then “updating the machine learning by utilizing an adaptive identification cycle with the additional review documents until a confidence threshold validation is met, the confidence threshold validation calculated using the correction from the human reviewer of the subject document and the identifying of the subject document by the machine learning”. Therefore, it is unclear whether these two recitations of the “adaptive identification cycle” are meant to be different types of identification cycles or if they are the same, and this renders the claims unclear and indefinite. For purposes of examination, Examiner will interpret them to be the same and the subsequent limitation as follows “updating the machine learning by utilizing the adaptive identification cycle with the additional review documents until a confidence threshold validation is met, the confidence threshold validation calculated using the correction from the human reviewer of the subject document and the identifying of the subject document by the machine learning”. Clarification is required.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 16 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. Claim 15, as amended, now includes this limitation with a slight change that does not change the scope of the limitation; therefore dependent claim 16 does not further limit the subject matter of the claim. Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-19 of U.S. Patent No. 11,023,828. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant application claims are a broader version of the claims that appear in the Patent and they are both directed to document review by incorporating user input to identify a subject or category and coding documents accordingly, therefore, the inventive concept is the same and they are not patentably distinct from each other.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 6-18 of U.S. Patent No. 8,489,538. Although the claims at issue are not identical, they are not patentably distinct from each other because both are directed to document review by incorporating user input to identify a subject or category and coding documents accordingly, therefore, the inventive concept is the same and they are not patentably distinct from each other.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 2-4, 6, 8-18 of U.S. Patent No. 8,554,716. Although the claims at issue are not identical, they are not patentably distinct from each other because both are directed to document review by incorporating user input to identify a subject or category and coding documents accordingly, therefore, the inventive concept is the same and they are not patentably distinct from each other.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-6 of U.S. Patent No. 7,933,859. Although the claims at issue are not identical, they are not patentably distinct from each other because both are directed to document review by incorporating user input to identify a subject or category and coding documents accordingly, therefore, the inventive concept is the same and they are not patentably distinct from each other.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 11,282,000. Although the claims at issue are not identical, they are not patentably distinct from each other because they are both directed to document review by incorporating user input to identify a subject or category and coding documents accordingly, therefore, the inventive concept is the same and they are not patentably distinct from each other.
Claim 1-20 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-9, 11-21 of copending Application No. 17/504,374. Although the claims at issue are not identical, they are not patentably distinct from each other because they are both directed to document review by incorporating user input to identify a subject or category and coding documents accordingly, therefore, the inventive concept is the same and they are not patentably distinct from each other. This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Claim 1-20 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of copending Application No. 17/216,362. Although the claims at issue are not identical, they are not patentably distinct from each other because they are both directed to document review by incorporating user input to identify a subject or category and coding documents accordingly, therefore, the inventive concept is the same and they are not patentably distinct from each other. This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Claim 1-20 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-8, 10-18, 20-22 of copending Application No. 17/220,445. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant application claims and the claims that appear in the co-pending application are both directed to document review by incorporating user input to identify a subject or category and coding documents accordingly, therefore, the inventive concept is the same and they are not patentably distinct from each other. This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Claim 1-20 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-21 of copending Application No. 17/222,283. Although the claims at issue are not identical, they are not patentably distinct from each other because they are both directed to document review by incorporating user input to identify a subject or category and coding documents accordingly, therefore, the inventive concept is the same and they are not patentably distinct from each other. This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under pre-AIA 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA 35 U.S.C. 103(c) and potential pre-AIA 35 U.S.C. 102(e), (f) or (g) prior art under pre-AIA 35 U.S.C. 103(a).
Claims 1-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Davis et al (US 7,089,238, as submitted in IDS dated 9/9/2022- hereinafter Davis) in view of Roitblat et al (NPL: “Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review”- January 2010, hereinafter Roitblat), in view of Johnson et al (US 2005/0027664, as submitted in IDS dated 9/9/2022- hereinafter Johnson) and further in view of Hennig et al (US 20120101965, as submitted in IDS dated 9/9/2022- hereinafter Hennig).
Referring to Claim 1, Davis teaches a method of predictive coding for document category review comprising:
receiving training documents that comprise an initial set of review documents (see Davis at Column 2: lines 23-30; Davis teaches an initial set of documents, which are used for creating a training set);
generating a coded control set of data based on the training documents using coding determinations for the coded control set (see Davis at Column 2: lines 34-53; Davis teaches addition of documents by receiving documents from multiple feeds and selecting a portion of them to add to the training set used in production for automatic classification of incoming documents. A categorization engine 211 is used to identify nearest neighbors and calculate similarity and category scores. The category score is higher or lower, corresponding to a degree of confidence in assignment of a particular document to a particular category. Therefore, since the initial training data is used for training the categorization engine, this corresponds to ‘generating a coded control set based on the training data), the coding determinations performed using a predictive coding of the review documents (see Davis at Column 2: lines 34-53; Davis teaches addition of documents by receiving documents from multiple feeds and selecting a portion of them to add to the training set used in production for automatic classification of incoming documents. A categorization engine 211 is used to identify nearest neighbors and calculate similarity and category scores. The category score is higher or lower, corresponding to a degree of confidence in assignment of a particular document to a particular category. Therefore, since the initial training data is used for training the categorization engine for automatic classification, this corresponds to ‘automated coding of the relevant documents by a predictive coding system);
coding additional review documents using the coded control set (see Davis at Column 2: lines 34-53; Davis teaches a categorization engine 211 is used to identify nearest neighbors and calculate similarity and category scores. The category score is higher or lower, corresponding to a degree of confidence in assignment of a particular document to a particular category. Therefore, since the initial training data is used for training the categorization engine for automatic classification, this corresponds to coding of additional documents);
presenting a subject document from the additional review documents to a human reviewer (see Davis at Column 2: line 51- Column 3 line 3; Davis teaches the process of editorial review for quality control of either random documents or documents that didn’t meet a confidence threshold. Further, Davis teaches that this editorial review is done by a human user);
receiving a correction from the human reviewer to correct at least a portion of the predictive coding by performing a hard coding correction on the subject document, the hard coding correction comprising updating the predictive coding (see Davis at Column 2: line 51- Column 3 line 3; Davis teaches the process of editorial review for quality control of either random documents or documents that didn’t meet a confidence threshold. Further, Davis teaches that this editorial review is done by a human user, wherein the corrected classified document is flagged as being reviewed by the human);
receiving the hard coding correction and updating the coded control set with the subject document having the hard coding correction (see Davis at Column 2: lines 54-56; Davis teaches that documents verified by editorial review are collected in a verified documents set 214 and used for incremental updating of the training set 223);
applying the updated coded control set to another set of additional documents to predictively code the additional documents (see Davis at Abstract; Davis teaches methods for incrementally updating the accuracy provided by documents in training set of used for automatic categorization. Furthermore, at Column 1: lines 49-51, Davis teaches that once the training set has been retuned, it can be used for categorization of documents. Therefore, the updated training set is used for automatic categorization/coding of additional documents).
However, Davis fails to teach:
receiving a coding catagory that includes one of responsiveness, issues, and privileges;
training machine learning for the coding category utilizing an adaptive identification cycle trained with the initial set of review documents;
identifying similar documents to the initial set of review documents utilizing the machine learning to detect concepts within the initial set of review documents;
updating the machine learning by utilizing an adaptive identification cycle with the additional review documents until a confidence threshold validation is met, the confidence threshold validation calculated using the correction from the human reviewer of the subject document and the identifying of the subject document by the machine learning; and
supplementing the initial set of review documents with the similar documents.
Roitblat teaches, in an analogous system,
receiving a coding catagory that includes one of responsiveness, issues, and privileges (see Roitblat at p. 70: Introduction: “In litigation, particularly civil litigation in the US Federal Courts, the parties are required, when requested, to produce documents that are potentially relevant to the issues and facts of the matter. This is a part of the process called “discovery.” When it involves electronic documents, or more formally, “electronically stored information (ESI),” it is called eDiscovery. The potentially relevant documents are said to be responsive”. Further at p. 72: Research Design: Methods: “One solution to the problem of the exploding cost of eDiscovery is to use technology to reduce the effort required to identify responsive and privileged documents. Like the TREC legal track, the goal of the present research is to evaluate the ability of information retrieval technology to meet the needs of the legal community for tools to identify the responsive documents in a collection”).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify the teachings of Davis with the above teachings of Roitblat by receiving a hard coding correction to at least a portion of an automated coding of documents, as taught by Davis, wherein the documents are legal documents classified as relevant, responsive and privileged, as taught by Roitblat. The modification would have been obvious because one of ordinary skill in the art would be motivated to evaluate the ability of information retrieval technology to meet the needs of the legal community for tools to identify the responsive documents in a collection (as suggested by Roitblat at p. 72).
Johnson teaches, in an analogous system,
training machine learning for the coding category utilizing an adaptive identification cycle trained with the initial set of review documents (see Johnson at Abstract: “Through iterative interactive training sessions with a user the system trains annotators, and these are in turn used to discover more annotations in the text data. Once all of the text data or a sufficient amount of the text data is annotated, at the user's discretion, the system learns a final annotator or annotators, which are exported and available to annotate new textual data”. Further, see Claim 1: “iteratively learning annotators for the at least one named entity or class using a machine learning algorithm; applying the learned annotators to text data resulting in the annotation of at least one named entity or class annotation instance; and selectively presenting for review and correction, if determined, representations of the at least one named entity or class annotation instance identified by the applying of the learned annotators”. This iterative corresponds to the claimed adaptive identification life cycle);
updating the machine learning by utilizing an adaptive identification cycle with the additional review documents until a confidence threshold validation is met, the confidence threshold validation calculated using the correction from the human reviewer of the subject document and the identifying of the subject document by the machine learning (see Johnson at Claim 1: “iteratively learning annotators for the at least one named entity or class using a machine learning algorithm; applying the learned annotators to text data resulting in the annotation of at least one named entity or class annotation instance; and selectively presenting for review and correction, if determined, representations of the at least one named entity or class annotation instance identified by the applying of the learned annotators”. This iterative learning corresponds to the claimed updating of the machine learning using an adaptive identification life cycle. Also, see Claim 2: “annotations instances are selectively presented for review and correction, if determined, based on a predetermined threshold value of a confidence level”, therefore, this corresponds to the claimed “confidence threshold”. Further, see [0016]: “At the end of each iteration, any annotation, generated from the learned annotators, having a confidence level within a confidence level range is corrected based on feedback”).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify the combination of Davis and Roitblat with the above teachings of Johnson by receiving a hard coding correction to at least a portion of an automated coding of documents wherein the documents are legal documents classified as relevant, responsive and privileged, as taught by Davis and Roitblat, and training a machine learning comprising an adaptive identification lifecycle to perform the automatic classification of the documents, as taught by Johnson. The modification would have been obvious because one of ordinary skill in the art would be motivated to incrementally improve its ability to assign annotations correctly and also allows for mechanisms for selectively presenting results and guiding the user in the evaluation and correction process (as suggested by Johnson at 0114).
Hennig teaches, in an analogous system,
identifying similar documents to the initial set of review documents utilizing machine learning to detect concepts within the initial set of review documents (see Hennig at [0041]; Hennig teaches “the distribution of probabilities of the corpus/topic distribution prediction may be compared with a distribution of probabilities of a document/topic distribution of an input document to determine whether the input document concerns similar topics as the document corpus. In another example, one or more documents within the document corpus having semantically similar topics may be grouped as determined by the topic model’. Moreover, at [0049]; Hennig teaches “in one example, the training component 306 may be configured to group one or more documents within the document corpus 302 having semantically similar topics as determined by the topic model 308”. Therefore, the comparison of a document with the corpus using probabilities of the corpus/topic distribution to determine if they are semantically similar corresponds to the claimed identification of similar documents); and
supplementing the initial set of review documents with the similar documents (see Hennig at [0036] and Fig. 3; Hennig teaches “an addition of a new document to the document corpus may be detected. A new document representation of the new document and new features of the new document may be processed using the topic model. In this way, the topic model may be updated based upon the processing of the new document (e.g., the topic model may be trained by updating current parameters with parameters specified during the processing of the new document representation and/or the new features”. Moreover, it can be seen at Fig. 3 the updating of the Topic model by the training component. Therefore, this updating of the topic model with new documents based on similarity corresponds to the claimed supplementing the initial set of documents).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify the combination of Davis, Roitblat and Johnson with the above teachings of Hennig by receiving a hard coding correction to at least a portion of an automated coding of documents wherein the documents are legal documents classified as relevant, responsive and privileged, and training a machine learning comprising an adaptive identification lifecycle to perform the automatic classification of the documents, as taught by Davis, Roitblat and Johnson, and identifying contextually similar documents to an initial set of relevant documents, as taught by Hennig. The modification would have been obvious because one of ordinary skill in the art would be motivated to update one or more current feature/topic parameters with specified feature/topic parameters in the training of the topic model during sequential processing of document representations and/or features (as suggested by Hennig at [0032]).
Referring to Claim 2, the combination of Davis, Roitblat, Johnson and Hennig teaches the method according to claim 1, further comprising applying the coding determinations of the coded control set to contextually similar data in a corpus of documents (see Hennig at [0036] and Fig. 3; Hennig teaches “an addition of a new document to the document corpus may be detected. A new document representation of the new document and new features of the new document may be processed using the topic model. In this way, the topic model may be updated based upon the processing of the new document (e.g., the topic model may be trained by updating current parameters with parameters specified during the processing of the new document representation and/or the new features”. Moreover, it can be seen at Fig. 3 the updating of the Topic model by the training component. Therefore, this updating of the topic model with new documents based on similarity corresponds to the claimed applying coding determination to contextually similar data).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify the combination of Davis, Roitblat and Johnson with the above teachings of Hennig by receiving a hard coding correction to at least a portion of an automated coding of documents wherein the documents are legal documents classified as relevant, responsive and privileged, and training a machine learning comprising an adaptive identification lifecycle to perform the automatic classification of the documents, as taught by Davis, Roitblat and Johnson, and identifying contextually similar documents to an initial set of relevant documents, as taught by Hennig. The modification would have been obvious because one of ordinary skill in the art would be motivated to update one or more current feature/topic parameters with specified feature/topic parameters in the training of the topic model during sequential processing of document representations and/or features (as suggested by Hennig at [0032]).
Referring to Claim 3, the combination of Davis, Roitblat, Johnson and Hennig teaches the method according to claim 1, further comprising receiving a review of each document in the updated coded control set to ensure that proper coding has been implemented (see Davis at Column 4: lines 42-49; Davis teaches the status column provides information regarding confidence in coding of a document. "Okay" may be used to indicate that a document has been correctly categorized; "missing" may be used to indicate that a document with a high score has not been assigned to a topic; and "suspicious" may indicate that a document with a low score has been assigned to the topic. This is interpreted as ensuring proper coding).
Referring to Claim 4, the combination of Davis, Roitblat, Johnson and Hennig teaches the method according to claim 1, wherein generating the coded control set of data comprises generating the coded control set of data based on a plurality of uncoded documents from a corpus, the plurality of uncoded documents comprising documents that are relevant or selected randomly (see Davis at Column 2: lines 28-30; Davis teaches “[u|ncoded documents 101 are loaded and registered 102 into a workfile. A user codes the documents to create a training set’. Moreover, at Column 2: lines 41-47: “[t]he documents 201 may be coded or uncoded. An input queue 202 may be used to organize addition of documents 201 to the training set, for instance, when a news dissemination service is receiving documents from multiple feeds and selecting a portion of them to add to the training set used in production for automatic classification of incoming documents”. Therefore, the coded control set is based on uncoded documents from a corpus, and since the documents are selected as a portion from multiple feeds, this corresponds to the documents being relevantly or randomly selected).
Referring to Claim 5, the combination of Davis, Roitblat, Johnson and Hennig teaches the method according to claim 4, wherein the portion of the documents are randomly sampled from an un-reviewed document population (see Davis at Column 2: lines 56-62; Davis teaches editorial review, for quality control or other purposes, may also include a random sample 212 of documents that were above a confidence threshold during coding. Selection of a random sample 212 for editorial review balances addition to the training set of difficult cases, with low confidence scores, and easier cases, with higher confidence scores. Therefore, this corresponds to ‘portion of the documents are randomly sampled”).
Referring to Claim 6, the combination of Davis, Roitblat, Johnson and Hennig teaches the method according to claim 1, further comprising receiving a determination if the document coded using the coded control set was miscoded, prior to the step of receiving the hard coding correction to the document (see Davis at Column 4: lines 42-49; Davis teaches the status column provides information regarding confidence in coding of a document. "Okay" may be used to indicate that a document has been correctly categorized; "missing" may be used to indicate that a document with a high score has not been assigned to a topic; and "suspicious" may indicate that a document with a low score has been assigned to the topic. "Missing" and "suspicious" documents may be referred to a human for editorial review; therefore, the miscoded determination is prior to have a hard coding correction by the editorial reviewer).
Referring to Claim 7, the combination of Davis, Roitblat, Johnson and Hennig teaches the method according to claim 1, further comprising updating the training data with each new coded document created using the coded control set (see Davis at Column 2: lines 54-56; Davis teaches that documents verified by editorial review are collected in a verified documents set 214 and used for incremental updating of the training set 223).
Referring to independent Claim 8, it is rejected on the same basis as independent claim 1 since they are analogous claims.
Referring to dependent Claim 9, it is rejected on the same basis as dependent claim 2 since they are analogous claims.
Referring to dependent Claim 10, it is rejected on the same basis as dependent claim 3 since they are analogous claims.
Referring to dependent Claim 11, it is rejected on the same basis as dependent claim 4 since they are analogous claims.
Referring to dependent Claim 12, it is rejected on the same basis as dependent claim 5 since they are analogous claims.
Referring to dependent Claim 13, it is rejected on the same basis as dependent claim 6 since they are analogous claims.
Referring to dependent Claim 14, it is rejected on the same basis as dependent claim 7 since they are analogous claims.
Referring to Claim 15, Davis teaches a method document category review comprising:
generating a coded control set of data based on received training documents using coding determinations for the coded control set (see Davis at Column 2: lines 34-53; Davis teaches addition of documents by receiving documents from multiple feeds and selecting a portion of them to add to the training set used in production for automatic classification of incoming documents. A categorization engine 211 is used to identify nearest neighbors and calculate similarity and category scores. The category score is higher or lower, corresponding to a degree of confidence in assignment of a particular document to a particular category. Therefore, since the initial training data is used for training the categorization engine, this corresponds to ‘generating a coded control set based on the training data), the coding determinations performed using a predictive coding of review documents (see Davis at Column 2: lines 34-53; Davis teaches addition of documents by receiving documents from multiple feeds and selecting a portion of them to add to the training set used in production for automatic classification of incoming documents. A categorization engine 211 is used to identify nearest neighbors and calculate similarity and category scores. The category score is higher or lower, corresponding to a degree of confidence in assignment of a particular document to a particular category. Therefore, since the initial training data is used for training the categorization engine for automatic classification, this corresponds to ‘automated coding of the relevant documents by a predictive coding system);
coding additional review documents using the coded control set (see Davis at Column 2: lines 34-53; Davis teaches a categorization engine 211 is used to identify nearest neighbors and calculate similarity and category scores. The category score is higher or lower, corresponding to a degree of confidence in assignment of a particular document to a particular category. Therefore, since the initial training data is used for training the categorization engine for automatic classification, this corresponds to coding of additional documents);
presenting a subject document from additional review documents to a human reviewer, the additional review documents being generated from the coded control set (see Davis at Column 2: line 51- Column 3 line 3; Davis teaches the process of editorial review for quality control of either random documents or documents that didn’t meet a confidence threshold. Further, Davis teaches that this editorial review is done by a human user);
receiving a correction from the human reviewer to correct at least a portion of the predictive coding, the correction comprising a hard coding correction on the subject document (see Davis at Column 2: line 51- Column 3 line 3; Davis teaches the process of editorial review for quality control of either random documents or documents that didn’t meet a confidence threshold. Further, Davis teaches that this editorial review is done by a human user, wherein the corrected classified document is flagged as being reviewed by the human), the hard coding correction being used to update the predictive coding by updating the coded control set with the subject document having the hard coding correction (see Davis at Column 2: lines 54-56; Davis teaches that documents verified by editorial review are collected in a verified documents set 214 and used for incremental updating of the training set 223); and
applying the updated coded control set to another set of additional documents to predictively code the additional documents (see Davis at Abstract; Davis teaches methods for incrementally updating the accuracy provided by documents in training set of used for automatic categorization. Furthermore, at Column 1: lines 49-51, Davis teaches that once the training set has been retuned, it can be used for categorization of documents. Therefore, the updated training set is used for automatic categorization/coding of additional documents).
However, Davis fails to teach:
receiving a coding catagories that includes one of responsiveness, issues, and privileges;
training machine learning for the coding category utilizing an adaptive identification cycle trained with the initial set of review documents;
identifying similar documents to the initial set of review documents utilizing the machine learning to detect concepts within the initial set of review documents; and
updating the machine learning by utilizing an adaptive identification cycle with the additional review documents until a confidence threshold validation is met, the confidence threshold validation calculated using the correction from the human reviewer of the subject document and the identifying of the subject document by the machine learning.
Roitblat teaches, in an analogous system,
receiving a coding catagories that includes one of responsiveness, issues, and privileges (see Roitblat at p. 70: Introduction: “In litigation, particularly civil litigation in the US Federal Courts, the parties are required, when requested, to produce documents that are potentially relevant to the issues and facts of the matter. This is a part of the process called “discovery.” When it involves electronic documents, or more formally, “electronically stored information (ESI),” it is called eDiscovery. The potentially relevant documents are said to be responsive”. Further at p. 72: Research Design: Methods: “One solution to the problem of the exploding cost of eDiscovery is to use technology to reduce the effort required to identify responsive and privileged documents. Like the TREC legal track, the goal of the present research is to evaluate the ability of information retrieval technology to meet the needs of the legal community for tools to identify the responsive documents in a collection”).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify the teachings of Davis with the above teachings of Roitblat by receiving a hard coding correction to at least a portion of an automated coding of documents, as taught by Davis, wherein the documents are legal documents classified as relevant, responsive and privileged, as taught by Roitblat. The modification would have been obvious because one of ordinary skill in the art would be motivated to evaluate the ability of information retrieval technology to meet the needs of the legal community for tools to identify the responsive documents in a collection (as suggested by Roitblat at p. 72).
Johnson teaches, in an analogous system,
training machine learning for the coding category utilizing an adaptive identification cycle trained with the initial set of review documents (see Johnson at Abstract: “Through iterative interactive training sessions with a user the system trains annotators, and these are in turn used to discover more annotations in the text data. Once all of the text data or a sufficient amount of the text data is annotated, at the user's discretion, the system learns a final annotator or annotators, which are exported and available to annotate new textual data”. Further, see Claim 1: “iteratively learning annotators for the at least one named entity or class using a machine learning algorithm; applying the learned annotators to text data resulting in the annotation of at least one named entity or class annotation instance; and selectively presenting for review and correction, if determined, representations of the at least one named entity or class annotation instance identified by the applying of the learned annotators”. This iterative corresponds to the claimed adaptive identification life cycle);
updating the machine learning by utilizing an adaptive identification cycle with the additional review documents until a confidence threshold validation is met, the confidence threshold validation calculated using the correction from the human reviewer of the subject document and the identifying of the subject document by the machine learning (see Johnson at Claim 1: “iteratively learning annotators for the at least one named entity or class using a machine learning algorithm; applying the learned annotators to text data resulting in the annotation of at least one named entity or class annotation instance; and selectively presenting for review and correction, if determined, representations of the at least one named entity or class annotation instance identified by the applying of the learned annotators”. This iterative learning corresponds to the claimed updating of the machine learning using an adaptive identification life cycle. Also, see Claim 2: “annotations instances are selectively presented for review and correction, if determined, based on a predetermined threshold value of a confidence level”, therefore, this corresponds to the claimed “confidence threshold”. Further, see [0016]: “At the end of each iteration, any annotation, generated from the learned annotators, having a confidence level within a confidence level range is corrected based on feedback”).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify the combination of Davis and Roitblat with the above teachings of Johnson by receiving a hard coding correction to at least a portion of an automated coding of documents wherein the documents are legal documents classified as relevant, responsive and privileged, as taught by Davis and Roitblat, and training a machine learning comprising an adaptive identification lifecycle to perform the automatic classification of the documents, as taught by Johnson. The modification would have been obvious because one of ordinary skill in the art would be motivated to incrementally improve its ability to assign annotations correctly and also allows for mechanisms for selectively presenting results and guiding the user in the evaluation and correction process (as suggested by Johnson at 0114).
Hennig teaches, in an analogous system,
identifying similar documents to the initial set of review documents utilizing the machine learning to detect concepts within the initial set of review documents (see Hennig at [0041]; Hennig teaches “the distribution of probabilities of the corpus/topic distribution prediction may be compared with a distribution of probabilities of a document/topic distribution of an input document to determine whether the input document concerns similar topics as the document corpus. In another example, one or more documents within the document corpus having semantically similar topics may be grouped as determined by the topic model’. Moreover, at [0049]; Hennig teaches “in one example, the training component 306 may be configured to group one or more documents within the document corpus 302 having semantically similar topics as determined by the topic model 308”. Therefore, the comparison of a document with the corpus using probabilities of the corpus/topic distribution to determine if they are semantically similar corresponds to the claimed identification of similar documents).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify the combination of Davis, Roitblat and Johnson with the above teachings of Hennig by receiving a hard coding correction to at least a portion of an automated coding of documents wherein the documents are legal documents classified as relevant, responsive and privileged, and training a machine learning comprising an adaptive identification lifecycle to perform the automatic classification of the documents, as taught by Davis, Roitblat and Johnson, and identifying contextually similar documents to an initial set of relevant documents, as taught by Hennig. The modification would have been obvious because one of ordinary skill in the art would be motivated to update one or more current feature/topic parameters with specified feature/topic parameters in the training of the topic model during sequential processing of document representations and/or features (as suggested by Hennig at [0032]).
Referring to Claim 16, the combination of Davis, Roitblat, Johnson and Hennig teaches the method according to claim 15, further comprising identifying similar documents to the review documents utilizing machine learning to detect concepts within the review documents (see Hennig at [0041]; Hennig teaches “the distribution of probabilities of the corpus/topic distribution prediction may be compared with a distribution of probabilities of a document/topic distribution of an input document to determine whether the input document concerns similar topics as the document corpus. In another example, one or more documents within the document corpus having semantically similar topics may be grouped as determined by the topic model’. Moreover, at [0049]; Hennig teaches “in one example, the training component 306 may be configured to group one or more documents within the document corpus 302 having semantically similar topics as determined by the topic model 308”. Therefore, the comparison of a document with the corpus using probabilities of the corpus/topic distribution to determine if they are semantically similar corresponds to the claimed identification of similar documents).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify the combination of Davis, Roitblat and Johnson with the above teachings of Hennig by receiving a hard coding correction to at least a portion of an automated coding of documents wherein the documents are legal documents classified as relevant, responsive and privileged, and training a machine learning comprising an adaptive identification lifecycle to perform the automatic classification of the documents, as taught by Davis, Roitblat and Johnson, and identifying contextually similar documents to an initial set of relevant documents, as taught by Hennig. The modification would have been obvious because one of ordinary skill in the art would be motivated to update one or more current feature/topic parameters with specified feature/topic parameters in the training of the topic model during sequential processing of document representations and/or features (as suggested by Hennig at [0032]).
Referring to Claim 17, the combination of Davis, Roitblat, Johnson and Hennig teaches the method according to claim 16, further comprising supplementing the review documents with the similar documents (see Hennig at [0036] and Fig. 3; Hennig teaches “an addition of a new d