DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on April 12, 2024, was considered by the examiner. The submission is in compliance with the provisions of 37 CFR 1.97.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefore, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea (mental process) without significantly more.
Claim 1:
Regarding claim 1, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “A computer-implemented method, comprising operations for: generating a first overall quality score for annotated documents; determining that the first overall quality score is below a quality threshold; generating a ranked list of annotated documents for review; determining that one or more of the annotated documents in the ranked list of annotated documents have been updated; generating a second overall quality score for the annotated documents; determining that the second overall quality score is above the quality threshold; training an extraction machine learning model with the annotated documents; and using the extraction machine learning model to extract data items from the annotated documents,” and a method is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a mental process but for recitation of generic computer components:
A computer-implemented method, comprising operations for: generating a first overall quality score for annotated documents; (mental process, a person can mentally evaluate and generate a score for annotated documents, see MPEP 2106.04(a)(2)(III)),
determining that the first overall quality score is below a quality threshold; (mental process, a person can mentally evaluate and determine that the first score is below a threshold value, see MPEP 2106.04(a)(2)(III)),
generating a ranked list of annotated documents for review; (mental process, a person can mentally evaluate and generate a ranked or ordered list of annotated documents to review, see MPEP 2106.04(a)(2)(III)),
determining that one or more of the annotated documents in the ranked list of annotated documents have been updated; (mental process, a person can mentally evaluate, observe, and determine if a change occurred in a ranked list of annotated documents, see MPEP 2106.04(a)(2)(III)),
generating a second overall quality score for the annotated documents; (mental process, a person can mentally evaluate and generate a second quality score for annotated documents, see MPEP 2106.04(a)(2)(III)),
determining that the second overall quality score is above the quality threshold; (mental process, a person can mentally evaluate and determine if a second quality score is above a threshold value, see MPEP 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, cover performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
training an extraction machine learning model with the annotated documents; (Mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
and using the extraction machine learning model to extract data items from the annotated documents, (In step 2A, prong 2, extract data items recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional elements vii recites mere instructions to apply the judicial exception using generic computer components, which is not indicative of significantly more. The additional element viii recites mere data gathering, and is considered insignificant extra-solution activity. In step 2B, this insignificant extra-solution activity is well understood routine and conventional activity, which includes court case v. Electronically scanning or extracting data from a physical document, Content Extraction and Transmission, LLC v. Wells Fargo Bank, 776 F.3d 1343, 1348, 113 USPQ2d 1354, 1358 (Fed. Cir. 2014) (optical character recognition);
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim 2:
Regarding claim 2, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 2 recites the following additional element:
The computer-implemented method of claim 1, further comprising operations for: performing a technique selected from a group of techniques comprising a base score technique, a pattern technique, and a semantic analysis technique. (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)).
If claim limitations, under their broadest reasonable interpretation, cover performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 3:
Regarding claim 3, it is dependent upon claim 2, and thereby incorporates the limitations of, and corresponding analysis applied to claim 2. Further, claim 3 recites the following abstract idea:
… generates the ranked list of annotated documents based on confidence scores of positions of fields in the annotated documents, (This recites a mental process, a person can mentally evaluate and generate a ranked or ordered list of annotated documents based on confidence score values, see MPEP 2106.04(a)(2)(III)),
Further, claim 3 recites the following additional element:
The computer-implemented method of claim 2, wherein the base score technique … (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)).
If claim limitations, under their broadest reasonable interpretation, cover performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 4:
Regarding claim 4, it is dependent upon claim 2, and thereby incorporates the limitations of, and corresponding analysis applied to claim 2. Claim 4 recites the following abstract idea:
… generates the ranked list of annotated documents based on confidence scores of pattern of fields in the annotated documents, (This recites a mental process, a person can mentally evaluate and generate a ranked or ordered list of annotated documents based on confidence score values, see MPEP 2106.04(a)(2)(III)),
Further, claim 4 recites the following additional element:
The computer-implemented method of claim 2, wherein the pattern technique … (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)).
If claim limitations, under their broadest reasonable interpretation, cover performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 5:
Regarding claim 5, it is dependent upon claim 2, and thereby incorporates the limitations of, and corresponding analysis applied to claim 2. Claim 5 recites the following abstract idea:
… generates the ranked list of annotated documents based on confidence scores of semantic analysis of fields in the annotated documents, (This recites a mental process, a person can mentally evaluate and generate a ranked or ordered list of annotated documents based on confidence score values, see MPEP 2106.04(a)(2)(III)),
Further, claim 5 recites the following additional element:
The computer-implemented method of claim 2, wherein the semantic analysis technique… (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)).
If claim limitations, under their broadest reasonable interpretation, cover performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 6:
Regarding claim 6, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 6 recites the following additional elements:
The computer-implemented method of claim 1, further comprising operations for: receiving a search request that refers to a model quality measure; (In step 2A, prong2, receiving a search request recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g),). In step 2B, this insignificant extra-solution activity is well understood routine and conventional activity which includes receiving or transmitting data over a network from court case Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016), – see MPEP 2106.05(d) (II)(i)).
and returning one or more of the annotated documents that match the model quality measure. (In step 2A, prong2, returning or transmitting documents recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g),). In step 2B, this insignificant extra-solution activity is well understood routine and conventional activity which includes receiving or transmitting data over a network from court case Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016), – see MPEP 2106.05(d) (II)(i)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 7:
Regarding claim 7, it is dependent upon claim 1, and thereby incorporates the limitations of, and corresponding analysis applied to claim 1. Further, claim 7 recites the following additional element:
The computer-implemented method of claim 1, further comprising operations for: receiving updated, annotated documents; (In step 2A, prong2, receiving updated, annotated documents recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g),). In step 2B, this insignificant extra-solution activity is well understood routine and conventional activity which includes receiving or transmitting data over a network from court case Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016), – see MPEP 2106.05(d) (II)(i)).
and fine tuning the extraction machine learning model with the updated, annotated documents based on a new overall quality score exceeding the quality threshold. (In step 2A, prong 2, this is considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)). (In step 2B, this is also considered mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)).
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 8:
Regarding claim 8, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform operations for: generating a first overall quality score for annotated documents; determining that the first overall quality score is below a quality threshold; generating a ranked list of annotated documents for review; determining that one or more of the annotated documents in the ranked list of annotated documents have been updated; generating a second overall quality score for the annotated documents; determining that the second overall quality score is above the quality threshold; training an extraction machine learning model with the annotated documents; and using the extraction machine learning model to extract data items from the annotated documents,” and a computer program product or machine is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a mental process but for recitation of generic computer components:
… generating a first overall quality score for annotated documents; (mental process, a person can mentally evaluate and generate a score for annotated documents, see MPEP 2106.04(a)(2)(III)),
determining that the first overall quality score is below a quality threshold; (mental process, a person can mentally evaluate and determine that the first score is below a threshold value, see MPEP 2106.04(a)(2)(III)),
generating a ranked list of annotated documents for review; (mental process, a person can mentally evaluate and generate a ranked or ordered list of annotated documents to review, see MPEP 2106.04(a)(2)(III)),
determining that one or more of the annotated documents in the ranked list of annotated documents have been updated; (mental process, a person can mentally evaluate, observe, and determine if a change occurred in a ranked list of annotated documents, see MPEP 2106.04(a)(2)(III)),
generating a second overall quality score for the annotated documents; (mental process, a person can mentally evaluate and generate a second quality score for annotated documents, see MPEP 2106.04(a)(2)(III)),
determining that the second overall quality score is above the quality threshold; (mental process, a person can mentally evaluate and determine if a second quality score is above a threshold value, see MPEP 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, cover performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform operations for: … (In step 2A, prong 2, this is considered a generic computer component being used as a tool. – see MPEP 2106.05(f)),
training an extraction machine learning model with the annotated documents; (Mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
and using the extraction machine learning model to extract data items from the annotated documents, (In step 2A, prong 2, extract data items recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional element vii recites a generic computer component being used as a tool, and additional elements viii recites mere instructions to apply the judicial exception using generic computer components, which are not indicative of significantly more. The additional element ix recites mere data gathering, and is considered an insignificant extra-solution activity. In step 2B, this insignificant extra-solution activity is well understood routine and conventional activity, which includes court case v. Electronically scanning or extracting data from a physical document, Content Extraction and Transmission, LLC v. Wells Fargo Bank, 776 F.3d 1343, 1348, 113 USPQ2d 1354, 1358 (Fed. Cir. 2014) (optical character recognition);
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 9-13:
Regarding claims 9-13, claims 9-13 recite similar limitations as corresponding claims 2-6 listed above, and are rejected for similar reasons under 35 U.S.C. 101.
Claim 14:
Regarding claim 14, it comprises of similar additional limitations as claim 7, and is rejected for similar reasons under 35 U.S.C. 101.
Claim 15:
Regarding claim 15, in step 1 of the 101-analysis set forth in MPEP 2106, the claim recites “A computer system, comprising: one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices; and program instructions, stored on at least one of the one or more computer-readable, tangible storage devices for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, to perform operations comprising: generating a first overall quality score for annotated documents; determining that the first overall quality score is below a quality threshold; generating a ranked list of annotated documents for review; determining that one or more of the annotated documents in the ranked list of annotated documents have been updated; generating a second overall quality score for the annotated documents; determining that the second overall quality score is above the quality threshold; training an extraction machine learning model with the annotated documents; and using the extraction machine learning model to extract data items from the annotated documents,” and a system or machine is one of the four statutory categories of invention.
In step 2A prong 1 of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, covers a mental process but for recitation of generic computer components:
… generating a first overall quality score for annotated documents; (mental process, a person can mentally evaluate and generate a score for annotated documents, see MPEP 2106.04(a)(2)(III)),
determining that the first overall quality score is below a quality threshold; (mental process, a person can mentally evaluate and determine that the first score is below a threshold value, see MPEP 2106.04(a)(2)(III)),
generating a ranked list of annotated documents for review; (mental process, a person can mentally evaluate and generate a ranked or ordered list of annotated documents to review, see MPEP 2106.04(a)(2)(III)),
determining that one or more of the annotated documents in the ranked list of annotated documents have been updated; (mental process, a person can mentally evaluate, observe, and determine if a change occurred in a ranked list of annotated documents, see MPEP 2106.04(a)(2)(III)),
generating a second overall quality score for the annotated documents; (mental process, a person can mentally evaluate and generate a second quality score for annotated documents, see MPEP 2106.04(a)(2)(III)),
determining that the second overall quality score is above the quality threshold; (mental process, a person can mentally evaluate and determine if a second quality score is above a threshold value, see MPEP 2106.04(a)(2)(III)),
If claim limitations, under their broadest reasonable interpretation, cover performance of the limitations as a mental process but for the recitation of generic computer components, then it falls within the mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
In step 2A prong 2 of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following additional elements do not integrate this judicial exception into a practical application:
A computer system, comprising: one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices; and program instructions, stored on at least one of the one or more computer-readable, tangible storage devices for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, to perform operations comprising… (In step 2A, prong 2, this is considered a generic computer components being used as a tool. – see MPEP 2106.05(f)),
training an extraction machine learning model with the annotated documents; (Mere instructions to apply an exception using generic computer – see MPEP 2106.05(f)),
and using the extraction machine learning model to extract data items from the annotated documents, (In step 2A, prong 2, extract data items recites mere data gathering, which is considered insignificant extra-solution activity – see MPEP 2106.05(g)),
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
In step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As discussed above, additional element vii recites a generic computer components being used as a tool, and additional element viii recites mere instructions to apply the judicial exception using generic computer components, which are not indicative of significantly more.
The additional element ix recites mere data gathering, and is considered an insignificant extra-solution activity. In step 2B, this insignificant extra-solution activity is well understood routine and conventional activity, which includes court case v. Electronically scanning or extracting data from a physical document, Content Extraction and Transmission, LLC v. Wells Fargo Bank, 776 F.3d 1343, 1348, 113 USPQ2d 1354, 1358 (Fed. Cir. 2014) (optical character recognition);
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 16-19:
Regarding claims 16-19, claims 16-19 recite similar limitations as corresponding claims 2-5 listed above, and are rejected for similar reasons under 35 U.S.C. 101.
Claim 20:
Regarding claim 20, it comprises of similar additional limitations as claim 6, and is rejected for similar reasons under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 7, 8, 14, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zhdanov, F. in U.S. Patent publication US11048979B1, published on June 29, 2021, (hereafter, Zhdanov), in view of Chang, Y. et al., in US PG Pub. No. US20210334315A1, published on October 28, 2021, (hereafter, Chang).
Claim 1:
Regarding claim 1, Zhdanov teaches “ A computer-implemented method, comprising operations for: generating a first overall quality score for annotated documents;”
See Zhdanov in col. 6, lines 8-29, describe “the annotation consolidation service 122 can maintain a label score and a worker score when performing annotation consolidation. During consolidation, annotation consolidation service 122 can fetch current label scores for each piece of data in the dataset (e.g., image, video frame, audio utterance, etc.) ... The label score can be compared to a specified threshold. If the label score is higher than the threshold then no additional annotations are required. If the label score is lower than the threshold then the data may be passed to additional annotators to be further annotated by annotating service 120. At numeral 10, once the label score is higher than the threshold, then the core engine 110 can be updated to indicate that the subset of the input dataset has been labeled. The active learning loop may continue to execute with the core engine invoking the active learning service 112 to label a new subset of the input dataset”. Here, Zhdanov shows generating scores for annotated data.
Further, see Zhdanov in col. 9, lines 17-22 mention “At numeral 4, the active learning service 112 can pass the subset identified by the active learning service 112 to be manually annotated to WIS 118. As shown, WIS 118 may include a dataset list 300, which includes each object of the dataset (e.g., image file, text file, video file, video frame, audio utterance, etc.).” Here, Zhdanov shows using a text file as part of data, and a text file can also be viewed as a form of document.
Further, see Zhdanov in col. 9, lines 63-67, through col. 10, lines 1-2 describe "As shown in FIG. 4, an input dataset 400 can be provided to an active learning service 402 (such as data labeling service 108). Active learning service 402 can use one or more active learning algorithms and one or more machine learning models 406 to determine a portion of the dataset to be auto-annotated and a portion of the dataset to be manually annotated." Here, Zhdanov shows that the data, in any format from text files to image file documents, can be labeled or annotated.
Also, see Zhdanov in col. 9, lines 36-61 mention “The annotation consolidation service 122 can determine a consolidated annotation (e.g., label) and quality score for each annotated object in the annotated datasets. If the annotation threshold has been reached, the annotation consolidation service 122 can store the resulting labels to output data store 206 at numeral 8. If the threshold has not been reached, then at numeral 9, the annotation consolidation service can send a request to the annotation manager 302 to extend the annotation workflow execution. This may include requesting a configurable number of additional annotators annotate the dataset. The number of additional annotators may be determined based on the difference between the desired quality threshold and the current threshold. This may continue to loop (e.g., operations depicted as numerals 5-9) for each object of the dataset until all objects of the dataset have been annotated and determined to have a quality score higher than the threshold value.” Here, Zhdanov mentions an active learning loop in which annotated data is given a quality score, and the method mentions this keeps repeating until all objects of the dataset have a quality score higher than a threshold value. In this step, Zhdanov mentions a method using an iterative process, that generates a first, second, and subsequent quality scores for annotated documents.
See Zhdanov in col. 10, lines 53-56 note "As shown in FIG. 4, the active learning loop can be performed iteratively, incrementally training a more accurate machine learning model and incrementally auto-annotating a larger portion of the input dataset 400." Here, Zhdanov emphasizes this method is iterative. See Zhdanov in col. 7, lines 49- 52 describe for more details.
Further, Zhdanov teaches “determining that the first overall quality score is below a quality threshold;”
See Zhdanov in col. 9, lines 36-61 mention “The annotation consolidation service 122 can determine a consolidated annotation (e.g., label) and quality score for each annotated object in the annotated datasets. If the annotation threshold has been reached, the annotation consolidation service 122 can store the resulting labels to output data store 206 at numeral 8. If the threshold has not been reached, then at numeral 9, the annotation consolidation service can send a request to the annotation manager 302 to extend the annotation workflow execution. This may include requesting a configurable number of additional annotators annotate the dataset. The number of additional annotators may be determined based on the difference between the desired quality threshold and the current threshold. This may continue to loop (e.g., operations depicted as numerals 5-9) for each object of the dataset until all objects of the dataset have been annotated and determined to have a quality score higher than the threshold value.” Here, Zhdanov mentions an active learning loop in which annotated data is given a quality score, and the method mentions this keeps repeating until all objects of the dataset have a quality score higher than a threshold value. Specifically, Zhdanov mentions ‘If the threshold has not been reached, then at numeral 9, the annotation consolidation service can send a request to the annotation manager 302 to extend the annotation workflow execution.’ Here, Zhdanov shows if a threshold value is not met, then annotation is continuously performed for the data until all items in the dataset have a quality score above threshold value. In this step, Zhdanov mentions a method using an iterative process, that generates a first, second, and subsequent quality scores for annotated documents, and ensures that the scores are above threshold value.
Further, Zhdanov teaches “generating … of annotated documents for review;”
See Zhdanov in col. 6, lines 8-29 describe “…During consolidation, annotation consolidation service 122 can fetch current label scores for each piece of data in the dataset (e.g., image, video frame, audio utterance, etc.) ... The label score can be compared to a specified threshold. If the label score is higher than the threshold then no additional annotations are required. If the label score is lower than the threshold then the data may be passed to additional annotators to be further annotated by annotating service 120. At numeral 10, once the label score is higher than the threshold, then the core engine 110 can be updated to indicate that the subset of the input dataset has been labeled.” Here, Zhdanov shows generating labeled or annotated data (which include text file documents mentioned from col. 9, lines 63-67) for review by comparing the annotated data with a threshold value.
Further, see Zhdanov in col. 9, lines 63-67, through col. 10, lines 1-2 describe "As shown in FIG. 4, an input dataset 400 can be provided to an active learning service 402 (such as data labeling service 108). Active learning service 402 can use one or more active learning algorithms and one or more machine learning models 406 to determine a portion of the dataset to be auto-annotated and a portion of the dataset to be manually annotated." Here, Zhdanov shows that the data, in any format from text files to image file documents, can be labeled or annotated.
Further, Zhdanov teaches “determining that one or more of the annotated documents … of annotated documents have been updated”
See Zhdanov in col. 6, lines 8-29, describe “the annotation consolidation service 122 can maintain a label score and a worker score when performing annotation consolidation. During consolidation, annotation consolidation service 122 can fetch current label scores for each piece of data in the dataset (e.g., image, video frame, audio utterance, etc.) ... The label score can be compared to a specified threshold. If the label score is higher than the threshold then no additional annotations are required. If the label score is lower than the threshold then the data may be passed to additional annotators to be further annotated by annotating service 120. At numeral 10, once the label score is higher than the threshold, then the core engine 110 can be updated to indicate that the subset of the input dataset has been labeled. The active learning loop may continue to execute with the core engine invoking the active learning service 112 to label a new subset of the input dataset”. Here, Zhdanov shows generating scores for annotated data. Zhdanov also shows that once a score is higher than threshold, then the engine can be updated to show that the subset of the input dataset has been labeled (i.e. determining that one or more of the annotated documents have been updated). The system is set up, as described by Zhdanov, such that the system is modified if any one of the labelled data (which includes text files) is changed.
Further, Zhdanov teaches “generating a second overall quality score for the annotated documents;”
See Zhdanov in col. 9, lines 36-61 mention “The annotation consolidation service 122 can determine a consolidated annotation (e.g., label) and quality score for each annotated object in the annotated datasets. If the annotation threshold has been reached, the annotation consolidation service 122 can store the resulting labels to output data store 206 at numeral 8. If the threshold has not been reached, then at numeral 9, the annotation consolidation service can send a request to the annotation manager 302 to extend the annotation workflow execution. This may include requesting a configurable number of additional annotators annotate the dataset. The number of additional annotators may be determined based on the difference between the desired quality threshold and the current threshold. This may continue to loop (e.g., operations depicted as numerals 5-9) for each object of the dataset until all objects of the dataset have been annotated and determined to have a quality score higher than the threshold value.” Here, Zhdanov mentions an active learning loop in which annotated data is given a quality score, and the method mentions this keeps repeating until all objects of the dataset have a quality score higher than a threshold value. Since Zhdanov mentions this is an iterative process, this method includes generating a first, second, and subsequent quality scores for annotated documents.
Also, see Zhdanov in col. 10, lines 5-12 note “Depending on the application, the machine learning model may perform classification, segmentation, object detection, etc. The machine learning model can output a confidence score for the feature it is trained to identify in each item of data in the dataset (e.g., each image in an image dataset, each word in a text dataset, etc.). A threshold can be set above which auto-annotation can be performed with an acceptable level of accuracy.” Here, Zhdanov shows that annotated data can include annotated text documents in a text dataset.
Further, see Zhdanov in col. 4, lines 13-18 describe “Labels are the outputs of annotations after the annotations have been consolidated and have achieved a quality score above a given threshold. As such, as used herein, a label refers to the true underlying object property, while annotations refer to the tags or other outputs by a labeling task (e.g., by a human labeler or machine annotation).” Here, Zhdanov shows creating a quality score, and since this method is iterative, this also shows generating a second quality score for the annotated data (which include text documents).
Further, Zhdanov teaches “determining that the second overall quality score is above the quality threshold”
See Zhdanov in col. 9, lines 36-61 mention “The annotation consolidation service 122 can determine a consolidated annotation (e.g., label) and quality score for each annotated object in the annotated datasets. If the annotation threshold has been reached, the annotation consolidation service 122 can store the resulting labels to output data store 206 at numeral 8. If the threshold has not been reached, then at numeral 9, the annotation consolidation service can send a request to the annotation manager 302 to extend the annotation workflow execution. This may include requesting a configurable number of additional annotators annotate the dataset. The number of additional annotators may be determined based on the difference between the desired quality threshold and the current threshold. This may continue to loop (e.g., operations depicted as numerals 5-9) for each object of the dataset until all objects of the dataset have been annotated and determined to have a quality score higher than the threshold value.” Here, Zhdanov mentions an active learning loop in which annotated data is given a quality score, and the method mentions this keeps repeating until all objects of the dataset have a quality score higher than a threshold value. Since Zhdanov mentions this is an iterative process, this method includes generating a second, and subsequent quality scores for annotated documents, and determines if the quality score is higher than a threshold value.
Further, Zhdanov teaches “training an extraction machine learning model with the annotated documents”
See Zhdanov in col. 10, lines 53-57 describe "As shown in FIG. 4, the active learning loop can be performed iteratively, incrementally training a more accurate machine learning model and incrementally auto-annotating a larger portion of the input dataset 400. This provides an at least partially trained model 406". Further, see Zhdanov in col. 10, lines 6-67 - col. 11, lines 1-4 "In some embodiments, inference can be performed in batches while iteratively training the machine learning model. This incremental training and inference allows each iteration to build on the last, increasing the speed at which inference can be performed and improving the accuracy of the model with each iteration." Here, Zhdanov shows the model is trained iteratively with annotated data, which includes text file documents as specified by col. 9, lines 17-22.
Further, Zhdanov teaches “and using the extraction machine learning model to extract data items from the annotated documents.”
See Zhdanov in col. 8, lines 44-56 mention “In some embodiments, the dataset may include a manifest file which describes dataset properties and records. A record may include named attributes, including metadata such as image size, or labels such as “dog” or “cat”. Other attributes may include raw data which needs labeling, such as image or sentences in natural language processing (NLP). In some embodiments, a manifest file for a dataset may be generated automatically by extracting metadata from files in the input data store 204 and generating the manifest file based on the metadata.” Here, Zhdanov shows this is a form of extraction from input data.
Further, see Zhdanov in col. 10, lines 2-12, describe “For example, the dataset can be passed through a machine learning model 406 to identify features of the dataset. Depending on the application, the machine learning model may perform classification, segmentation, object detection, etc. The machine learning model can output a confidence score for the feature it is trained to identify in each item of data in the dataset (e.g., each image in an image dataset, each word in a text dataset, etc.).” Identifying features, described by Zhdanov, shows a similar concept to extracting data items from data. Examiner construes data items to be features or any information that is part of the data. Since Zhdanov shows that data can include words from text data or documents, this model is also used for identifying features from text data.
Also, see Zhdanov in col. 6, lines 37-40 mention “The machine annotation service 114 may include a training service that can generate a new model, or update the previously used model, using the labeled subset of the input dataset.” Here, Zhdanov mentions that a model can be updated using a labeled or annotated input data, where updated here is construed to be synonymous with fine-tuning a machine learning model.
However, Zhdanov did not teach “generating a ranked list of annotated documents …;”
or “determining that one or more of the annotated documents in the ranked list of annotated documents …,”
In an analogous art, Chang teaches “generating a ranked list of annotated documents …;”
See Chang for ranked list of annotated documents in [0003] describe “a method for ranking and displaying candidate documents for human annotation task includes retrieving a document set, and displaying a list of documents from the document set for human annotation. The method then performs a real-time ranking candidate documents for human annotation loop that includes receiving a human annotation of a first unannotated document in the list of documents from the document set for human annotation, updating an annotated entities and corresponding entity types set based on the human annotation of the document from the document set, …, calculating a score for each document in the remaining set of documents in the document set based on the auto-mapping of annotated entities to corresponding entity types on the remaining set of documents in the document set, and updating an order of the remaining set of documents being displayed for human annotation based on the calculated score for each document in the remaining set of documents in the document set.” Chang here shows a ranked list of annotated documents.
Also, see Chang in [0026] note “If there are predefined dictionaries, the method 200, at step 206, pre-annotates documents in the document set using the predefined dictionaries, and calculates, at step 208, a pre-score for each document in the document set based on the pre-annotations. Pre-annotation means annotations performed on the documents prior to the human annotations (e.g., based on the predefined dictionaries). A pre-score is a score generated based on the pre-annotation of the documents.” The documents were already annotated before human annotation review. Chang mentions receiving pre-scores ( indicating quality scores) of the list of annotated documents
Further, see Chang in [0027] note “At step 210, the method 200 displays the list of documents from the document set for human annotation. Human annotation means that a human annotates texts within a document with suitable metadata that help machines to understand the speeches with sentence-level accuracy of the document. Human annotation can be performed for all types of texts available in various languages. If the documents were pre-annotated at step 206 and pre-scored at step 208, the method 200, at step 210, displays the list of documents from the document set for human annotation based on the pre-score for each document in the document set.” Examiner construes annotated to mean any form of annotation or labeling, including pre-annotation or labeling with OCR, named entity recognition tagging, or parts of speech tagging, or other methods of labeling the information.
Further, Chang teaches “determining that one or more of the annotated documents in the ranked list of annotated documents …”
See Chang for ranked list of annotated documents in [0003] describe “in an embodiment, a method for ranking and displaying candidate documents for human annotation task includes retrieving a document set, and displaying a list of documents from the document set for human annotation. The method then performs a real-time ranking candidate documents for human annotation loop that includes receiving a human annotation of a first unannotated document in the list of documents from the document set for human annotation, updating an annotated entities and corresponding entity types set based on the human annotation of the document from the document set, …, calculating a score for each document in the remaining set of documents in the document set based on the auto-mapping of annotated entities to corresponding entity types on the remaining set of documents in the document set, and updating an order of the remaining set of documents being displayed for human annotation based on the calculated score for each document in the remaining set of documents in the document set.” Chang here shows a ranked list of annotated documents. Chang also shows ‘updating an order of the remaining set of documents being displayed for human annotation based on the calculated score for each document’ to indicate updating the ranked list of annotated documents, where ranking is by the calculated score per document.
Further, Chang shows in abstract “… and update an order of the remaining set of documents being displayed for human annotation based on the calculated score for each document in the remaining set of documents in the document set.” Here, Chang shows the updating step on the annotated documents in the ranked list.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhdanov and incorporate into the teachings of Chang because both references teach a method of using annotated documents and rank them by quality score before training a model.
One of ordinary skill in the art would be motivated to do so because this would achieve a “ result is that a user can finish the annotation task using a relatively smaller document set by annotating documents in an order that helps balance entity distribution, as well as being able to select a better candidate document set through all of the available documents if necessary. The disclosed embodiments can improve work effectivity, annotation quality, and the performance of the machine learning model”, (see Chang in [0019]).
Claim 7:
Regarding claim 7, Zhdanov in view of Chang, teach the limitations of claim 1.
Further, Chang teaches “7. The computer-implemented method of claim 1, further comprising operations for: receiving updated, annotated documents;”
See Chang in [0038-0039] mention “At step 228, the method 200 determines whether the F-score of the model is higher than a threshold or whether all the documents in the documents set have been annotated. The threshold can be defined by a user. For example, a user can specify that they want the model to at least have an F-score of 0.9. If the method 200 determines that the F-score of the model is higher than the threshold, the method 200 terminates because the model has been sufficiently trained. The model can then be used to automatically annotate any received document in performing NLP processing. [0039] If, at step 228, the F-score of the model does not satisfy the threshold and there are additional documents to be annotated in the document set, the method 200 repeats the real-time ranking of candidate documents for human annotation loop (steps 212-222) with the next/top unannotated document in the remaining set of documents in the document set. In an embodiment, if all the documents in the documents set have been annotated before the F-score of the model is higher than the threshold, the method 200 terminates and additional document sets can be used to further train the model in accordance with the method 200.”
Here, Chang shows receiving additional annotated documents to further train the model.
Further, Zhdanov teaches “ and fine tuning the extraction machine learning model with the updated, annotated documents based on a new overall quality score exceeding the quality threshold”
See Zhdanov in col. 6, lines 8-29, describe “the annotation consolidation service 122 can maintain a label score and a worker score when performing annotation consolidation. During consolidation, annotation consolidation service 122 can fetch current label scores for each piece of data in the dataset (e.g., image, video frame, audio utterance, etc.) ... The label score can be compared to a specified threshold. If the label score is higher than the threshold then no additional annotations are required. If the label score is lower than the threshold then the data may be passed to additional annotators to be further annotated by annotating service 120. At numeral 10, once the label score is higher than the threshold, then the core engine 110 can be updated to indicate that the subset of the input dataset has been labeled. The active learning loop may continue to execute with the core engine invoking the active learning service 112 to label a new subset of the input dataset”. Here, Zhdanov shows generating scores for annotated data. Zhdanov also shows that once a score is higher than threshold, then the engine can be updated to show that the subset of the input dataset has been labeled (i.e. determining that one or more of the annotated documents have been updated). The system is set up, as described by Zhdanov, such that the system is modified if any one of the labelled data (which includes text files) is changed. Zhdanov shows that the method is iterative, so that once a new score is higher than the threshold value for the labeled data, then the system is updated to start the active learning service unit 112.
Also, see Zhdanov in col. 6, lines 37-40 mention “The machine annotation service 114 may include a training service that can generate a new model, or update the previously used model, using the labeled subset of the input dataset.” Here, Zhdanov mentions that a model can be updated using a labeled or annotated input data, where updated here is construed to be synonymous with fine-tuning a machine learning model. Once the system is updated, this also updates the machine learning model using the updated labeled subset (i.e. annotated documents).
PNG
media_image1.png
930
720
media_image1.png
Greyscale
Claim 8:
Regarding claim 8, the claim recites similar limitations as corresponding independent claim 1 and is rejected for similar reasons as claim 1 using similar teachings and rationale.
Further, Chang teaches “A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform operations…”
See Chang in [0049] mention “The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.” Here, Chang shows the computer program product has instructions run on the processor to perform tasks and other operations.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhdanov and incorporate into the teachings of Chang because both references teach a method of using annotated documents and rank them by quality score before training a model.
One of ordinary skill in the art would be motivated to do so because this would achieve a “ result is that a user can finish the annotation task using a relatively smaller document set by annotating documents in an order that helps balance entity distribution, as well as being able to select a better candidate document set through all of the available documents if necessary. The disclosed embodiments can improve work effectivity, annotation quality, and the performance of the machine learning model”, (see Chang in [0019]).
Claim 14:
Regarding claim 14, it comprises of similar additional limitations as corresponding claim 7, and is rejected under the same rationale and for similar reasons under 35 U.S.C. 103.
Claim 15:
Regarding claim 15, the claim recites similar limitations as corresponding independent claim 1 and is rejected for similar reasons as claim 1 using similar teachings and rationale.
Further, Chang teaches “15. A computer system, comprising: one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices; and program instructions, stored on at least one of the one or more computer-readable, tangible storage devices for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, to perform operations”
See Chang in paragraphs [0049-0050] describe “The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention. The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.” Here, Chang shows a computer system with computer medium and devices that perform operations.
Further, see Chang in paragraph [0004] mention “The system includes memory for storing instructions, and a processor configured to execute the instructions to: receive a human annotation of a first unannotated document in a list of documents.” Here, Chang mentions a memory and processor that are part of the computer system.
It would have been obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to combine the base reference of Zhdanov and incorporate into the teachings of Chang because both references teach a method of using annotated documents and rank them by quality score before training a model.
One of ordinary skill in the art would be motivated to do so because this would achieve a “ result is that a user can finish the annotation task using a relatively smaller document set by annotating documents in an order that helps balance entity distribution, as well as being able to select a better candidate document set through all of the available documents if necessary. The disclosed embodiments can improve work effectivity, annotation quality, and the performance of the machine learning model, ” (see Chang in [0019]).
Claims 2, 3, 4, 5, 9, 10, 11, 12, 16, 17, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Zhdanov in view of Chang, and further in view of Baviskar D. et al., “Multi-Layout Unstructured Invoice Documents Dataset: A Dataset for Template-Free Invoice Processing and Its Evaluation Using AI Approaches,” published on July 12, 2021, available at https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9481217&tag=1 , (hereafter, Baviskar_July21), and further in view of Hajiali, M., in “OCR post-processing using large language models,” published on August 15, 2023, available at https://oasis.library.unlv.edu/thesesdissertations/4811/, (hereafter, Hajiali).
Claim 2:
Regarding claim 2, Zhdanov in view of Chang, teach the limitations of claim 1. However, Zhdanov in view of Chang, did not teach “2. The computer-implemented method of claim 1, further comprising operations for: performing a technique selected from a group of techniques comprising a base score technique, a pattern technique, and a semantic analysis technique.”
In an analogous art, Baviskar_July21 teaches “2. The computer-implemented method of claim 1, further comprising operations for: performing a technique selected from a group of techniques comprising a base score technique, …, and a semantic analysis technique.
See Baviskar_July21 in page 101500, part C. Named Entity Recognition (NER) mention “The study [19] proposed a Convolutional Universal Text Information Extractor (CUTIE) approach that uses CNN to extract key information from the ICDAR-2019 receipts dataset and other self-built datasets. It uses the semantic information and positional information of entities to train CNN and the word embedding layer…Another study [41] combined CNN and RNN for Chinese medical invoice recognition tasks. CNN is used for extracting image features from medical invoices. RNN is used for identifying semantic information from the extracted features.” Here, Baviskar_July21 shows using semantic analysis techniques (like RNN or CNN) to identify semantic information from annotated documents such as receipts or medical invoices.
Also, see Baviskar_July21 in page 101502, section III. Proposed process flow, A. Proposed framework, part 1. Data collection mention “All the images are then converted into their text file using Google Vision OCR. X-Y position coordinates (spatial distribution) of each extracted word and confidence score are also obtained as OCR output along with the extracted text.”. Here, Baviskar_July21 mentions using a X-Y position coordinates of each extracted word per OCR-annotated document, and relates to using a base score method. Each word relates to each field label that is annotated by the OCR method. The base score technique is specified by the specification in [0105-0106] “The base quality score focuses on position and area of each annotated field label. Suppose the ontology has m field labels and each field label is annotated in each of the q documents, the field label may be identified using field type, character length, and location for every document class. In certain embodiments, the field type, character length, and location are provided when creating the field label. In certain embodiments, the base quality score focuses on identifying annotations that are anomalies based on the position of fields”. Using both semantic meaning and position (such as X-Y location of fields of text), Baviskar_July21 shows using both semantic analysis and base score techniques.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Zhdanov and Chang, and incorporate with the teachings of Baviskar_July21 by using the teachings of Zhdanov and Chang, for a method of using annotated documents and rank them by quality score before training a model, with Baviskar_July21’s teaching of a base score technique and a semantic analysis technique.
One of ordinary skill in the art would be motivated to do so because by integrating Baviskar_July21’s framework into the methods of Zhdanov and Chang, one with ordinary skill in the art would achieve the goal of providing a method “after fine-tuning, BERT can efficiently perform many popular NLP tasks such as NER. BERT is significant, since it can process the data bidirectionally with a contextual understanding of data,” (see page 101500 , second to last paragraph, in Baviskar_July21).
However, Zhdanov in view of Chang, and further in view of Baviskar_July21 did not teach “The computer-implemented method of claim 1, further comprising operations for: performing a technique selected from a group of techniques comprising …, a pattern technique, …”
In an analogous art, Hajiali teaches “The computer-implemented method of claim 1, further comprising operations for: performing a technique selected from a group of techniques comprising …, a pattern technique, …”
See Hajiali in page 49, in section 4.1 Detecting the OCR Errors, Finding the wrong word, mention “After calculating the scores, we would have a list of numbers representing scores associated with words. The next step would be finding outliers, which are the words with the highest scores that are significantly larger than the others. To achieve this, we utilized the Isolation Forest algorithm [49] for outlier detection in a list of numbers. The technique employed by this approach involves segregating data points by randomly choosing a dividing threshold within the range of values in the dataset.”. Here, Hajiali shows that after calculating scores, such as confidence scores, from the annotated documents, and generating a list of those documents that is ranked as stated from page 44, section 3.6.4, Hajiali shows here using an outlier detection method, which is part of the pattern technique, as stated in the specification in [0059] “In certain embodiments, the pattern technique 224 (i.e., a second technique) may be described as a template based technique (anomaly pattern detection) or a technique for identifying accidental user errors.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Zhdanov, Chang, and Baviskar_July21, and incorporate with the teachings of Hajiali by using the teachings of Zhdanov, Chang, and Baviskar_July21 for a method of using annotated documents and rank them by quality score before training a model, with Hajiali’s teaching of a pattern technique.
One of ordinary skill in the art would be motivated to do so because by integrating Hajiali’s framework into the methods of Zhdanov, Chang, and Baviskar_July21, one with ordinary skill in the art would achieve the goal where “these methods are generally more efficient in training and inference than neural networks. This efficiency is due to the lower complexity of the models, which results in faster training times and lower memory requirements,” (see Hajiali in page 5, section 2.3.1 Traditional Machine Learning Methods).
Claim 3:
Regarding claim 3, Zhdanov in view of Chang, further in view of Baviskar_July21, and further in view of Hajiali, teach the limitations of claim 2.
Further, Baviskar_July21 teaches “3. The computer-implemented method of claim 2, wherein the base score technique generates the … list of annotated documents based on confidence scores of positions of fields in the annotated documents.”
See Baviskar_July21 in page 101502, section III. Proposed process flow, A. Proposed framework, part 1. Data collection mention “All the images are then converted into their text file using Google Vision OCR. X-Y position coordinates (spatial distribution) of each extracted word and confidence score are also obtained as OCR output along with the extracted text.”. Here, Baviskar_July21 mentions using a X-Y position coordinates of each extracted word per OCR-annotated document, and relates to using a base score method. Baviskar_July21 also mentions a list of annotated documents based on confidence scores per each OCR output of extracted text. The confidence scores were obtained along with the X-Y position coordinates of each extracted word, where the X-Y position coordinates correspond to position of field labels of words. With each field text, there is an X-Y coordinate, indicating the location of where each word is placed within a document that is labeled or annotated. Each word relates to each field label that is annotated by the OCR method. The base score technique is specified by the specification in [0105-0106] “The base quality score focuses on position and area of each annotated field label. Suppose the ontology has m field labels and each field label is annotated in each of the q documents, the field label may be identified using field type, character length, and location for every document class. In certain embodiments, the field type, character length, and location are provided when creating the field label. In certain embodiments, the base quality score focuses on identifying annotations that are anomalies based on the position of fields”. The examiner construes OCR post-processing as a form of annotation of a document, such as letters, invoices, receipts, etc.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Zhdanov and Chang, and incorporate with the teachings of Baviskar_July21 by using the teachings of Zhdanov and Chang, for a method of using annotated documents and rank them by quality score before training a model, with Baviskar_July21’s teaching of a base score technique and a semantic analysis technique.
One of ordinary skill in the art would be motivated to do so because by integrating Baviskar_July21’s framework into the methods of Zhdanov and Chang, one with ordinary skill in the art would achieve the goal of providing a method “after fine-tuning, BERT can efficiently perform many popular NLP tasks such as NER. BERT is significant, since it can process the data bidirectionally with a contextual understanding of data,” (see page 101500 , second to last paragraph, in Baviskar_July21).
Further, Hajiali teaches “3. The computer-implemented method of claim 2, wherein the base score technique generates the ranked list of annotated documents based on confidence scores …”
See Hajiali in page 44, section 3.6.4. describe “3.6.4 Machine learning based methods OCR post-processing using machine learning-based techniques involves training models to learn from different features to enhance candidate selection. These models generate potential candidates, extract their features, and rank them to improve the accuracy and robustness of the OCR process. In a study by Mei et al. [45] conducted in 2016, they presented a method capable of rectifying 61.5% of OCR errors. Their approach involves utilizing the Damerau-Levenshtein distance measure to generate a list of candidates. The method creates several features, such as string similarity, language popularity, and lexicon existence, to evaluate the candidates and rank them based on their likelihood of being the correct correction for the OCR error. Moreover, they frame the problem as a regression task. By utilizing the scores of potential features, they make estimations on the likelihood that each candidate serves as a correction for the erroneous word. This confidence score is then used to rank candidates for each error, with the method selecting the candidate with the highest confidence score as the correction for the error.” Here, Hajiali mentions the candidates are the annotated documents by OCR, which are then ranked by confidence score for evaluating the accuracy of the annotations.
Also, see Hajiali in page 42, from section 3.5.2 Text detection, mention “OCR software typically uses a combination of pattern matching and feature extraction algorithms to achieve the best possible text recognition results.” Here, Hajiali mentions OCR also uses pattern matching and feature extraction methods to annotate text to extract information from documents.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Zhdanov, Chang, and Baviskar_July21, and incorporate with the teachings of Hajiali by using the teachings of Zhdanov, Chang, and Baviskar_July21 for a method of using annotated documents and rank them by quality score before training a model, with Hajiali’s teaching of a ranked list of annotated documents based on confidence scores.
One of ordinary skill in the art would be motivated to do so because by integrating Hajiali’s framework into the methods of Zhdanov, Chang, and Baviskar_July21, one with ordinary skill in the art would achieve the goal where “these methods are generally more efficient in training and inference than neural networks. This efficiency is due to the lower complexity of the models, which results in faster training times and lower memory requirements,” (see Hajiali in page 5, section 2.3.1 Traditional Machine Learning Methods).
Claim 4:
Regarding claim 4, Zhdanov in view of Chang, further in view of Baviskar_July21, and further in view of Hajiali, teach the limitations of claim 2.
Further, Hajiali teaches “4. The computer-implemented method of claim 2, wherein the pattern technique generates the ranked list of annotated documents based on confidence scores of pattern of fields in the annotated documents.”
See Hajiali in page 49, in section 4.1 Detecting the OCR Errors, Finding the wrong word, mention “After calculating the scores, we would have a list of numbers representing scores associated with words. The next step would be finding outliers, which are the words with the highest scores that are significantly larger than the others. To achieve this, we utilized the Isolation Forest algorithm [49] for outlier detection in a list of numbers. The technique employed by this approach involves segregating data points by randomly choosing a dividing threshold within the range of values in the dataset.” Here, Hajiali shows that after calculating scores, such as confidence scores, from the annotated documents, and generating a list of those documents that is ranked as stated from page 44, section 3.6.4, Hajiali shows here using an outlier detection method, which is part of the pattern technique, as stated in the specification in [0059] “In certain embodiments, the pattern technique 224 (i.e., a second technique) may be described as a template based technique (anomaly pattern detection) or a technique for identifying accidental user errors.”
See Hajiali in page 44, section 3.6.4. describe “3.6.4 Machine learning based methods OCR post-processing using machine learning-based techniques involves training models to learn from different features to enhance candidate selection. These models generate potential candidates, extract their features, and rank them to improve the accuracy and robustness of the OCR process. In a study by Mei et al. [45] conducted in 2016, they presented a method capable of rectifying 61.5% of OCR errors. Their approach involves utilizing the Damerau-Levenshtein distance measure to generate a list of candidates. The method creates several features, such as string similarity, language popularity, and lexicon existence, to evaluate the candidates and rank them based on their likelihood of being the correct correction for the OCR error. Moreover, they frame the problem as a regression task. By utilizing the scores of potential features, they make estimations on the likelihood that each candidate serves as a correction for the erroneous word. This confidence score is then used to rank candidates for each error, with the method selecting the candidate with the highest confidence score as the correction for the error.” Here, Hajiali mentions the candidates are the annotated documents by OCR, which are then ranked by confidence score for evaluating the accuracy of the annotations.
Also, see Hajiali in page 42, from section 3.5.2 Text detection, mention “OCR software typically uses a combination of pattern matching and feature extraction algorithms to achieve the best possible text recognition results.” Here, Hajiali mentions OCR also uses pattern matching and feature extraction methods to annotate text to extract information from documents. The examiner construes OCR post-processing as a form of annotation of a document, such as letters, invoices, receipts, etc.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Zhdanov, Chang, and Baviskar_July21, and incorporate with the teachings of Hajiali by using the teachings of Zhdanov, Chang, and Baviskar_July21 for a method of using annotated documents and rank them by quality score before training a model, with Hajiali’s teaching of a ranked list of annotated documents based on confidence scores.
One of ordinary skill in the art would be motivated to do so because by integrating Hajiali’s framework into the methods of Zhdanov, Chang, and Baviskar_July21, one with ordinary skill in the art would achieve the goal where “these methods are generally more efficient in training and inference than neural networks. This efficiency is due to the lower complexity of the models, which results in faster training times and lower memory requirements,” (see Hajiali in page 5, section 2.3.1 Traditional Machine Learning Methods).
Claim 5:
Regarding claim 5, Zhdanov in view of Chang, further in view of Baviskar_July21, and further in view of Hajiali, teach the limitations of claim 2.
Further, Hajiali teaches “5. The computer-implemented method of claim 2, wherein the semantic analysis technique generates the ranked list of annotated documents based on confidence scores of semantic analysis of fields in the annotated documents.”
See Hajiali in page 44, section 3.6.4. describe “3.6.4 Machine learning based methods OCR post-processing using machine learning-based techniques involves training models to learn from different features to enhance candidate selection. These models generate potential candidates, extract their features, and rank them to improve the accuracy and robustness of the OCR process. In a study by Mei et al. [45] conducted in 2016, they presented a method capable of rectifying 61.5% of OCR errors. Their approach involves utilizing the Damerau-Levenshtein distance measure to generate a list of candidates. The method creates several features, such as string similarity, language popularity, and lexicon existence, to evaluate the candidates and rank them based on their likelihood of being the correct correction for the OCR error. Moreover, they frame the problem as a regression task. By utilizing the scores of potential features, they make estimations on the likelihood that each candidate serves as a correction for the erroneous word. This confidence score is then used to rank candidates for each error, with the method selecting the candidate with the highest confidence score as the correction for the error.” Here, Hajiali mentions the candidates are the annotated documents by OCR, which are then ranked by confidence score for evaluating the accuracy of the annotations. The examiner construes OCR post-processing as a form of annotation of a document, such as letters, invoices, receipts, etc. The confidence score used is based on semantic analysis of methods such as lexicon existence, string similarity, and if each feature is scored as a measure of how confident the feature is in being correct.
Further, see Hajiali in page 13, section 2.5 Word embedding, describe “These models utilize neural networks to learn the semantic associations between words and represent them as continuous vector spaces, generating word embeddings that capture the semantic similarity between words. As a result, word embeddings have become a widely adopted technique in various NLP tasks, thanks to their ability to accurately capture the meaning of textual data.” Here, Hajiali mentions using method of semantic associations between words, which is part of using a semantic analysis technique.
For more details, see Hajiali in page 42, from section 3.5.2 Text detection, mention “OCR software typically uses a combination of pattern matching and feature extraction algorithms to achieve the best possible text recognition results.” Here, Hajiali mentions OCR also uses pattern matching and feature extraction methods to annotate text to extract information from documents. The examiner construes OCR post-processing as a form of annotation of a document, such as letters, invoices, receipts, etc.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Zhdanov, Chang, and Baviskar_July21, and incorporate with the teachings of Hajiali by using the teachings of Zhdanov, Chang, and Baviskar_July21 for a method of using annotated documents and rank them by quality score before training a model, with Hajiali’s teaching of a ranked list of annotated documents based on confidence scores.
One of ordinary skill in the art would be motivated to do so because by integrating Hajiali’s framework into the methods of Zhdanov, Chang, and Baviskar_July21, one with ordinary skill in the art would achieve the goal where “these methods are generally more efficient in training and inference than neural networks. This efficiency is due to the lower complexity of the models, which results in faster training times and lower memory requirements,” (see Hajiali in page 5, section 2.3.1 Traditional Machine Learning Methods).
Claims 9 - 12:
Regarding claims 9-12, they comprise of similar additional limitations as corresponding claims 2-5, respectively, and are rejected under the same rationale, and for similar reasons under 35 U.S.C. 103.
Claims 16 - 19:
Regarding claims 16-19, they comprise of similar additional limitations as corresponding claims 2-5, respectively, and are rejected under the same rationale for similar reasons under 35 U.S.C. 103.
Claims 6, 13, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zhdanov in view of Chang, and further in view of Sy, M. et al., in “User centered and ontology based information retrieval system for life sciences”, published on January 25th, 2012, available at https://link.springer.com/article/10.1186/1471-2105-13-S1-S4 , (hereafter, Sy).
Claim 6:
Regarding claim 6, Zhdanov in view of Chang, teach the limitations of claim 1.
However, Zhdanov in view of Chang, did not teach “6. The computer-implemented method of claim 1, further comprising operations for: receiving a search request that refers to a model quality measure; and returning one or more of the annotated documents that match the model quality measure.”
In an analogous method, Sy teaches “6. The computer-implemented method of claim 1, further comprising operations for: receiving a search request that refers to a model quality measure;”
See Sy on page 3, section Information retrieval systems overview, describe "The search is the core process of an IRS. It contains the system strategy for retrieving documents that match a query. An IRS selects and ranks relevant documents according to a score strategy that is highly dependent on their indexation." Here, Sy explicitly mentions a user’s search query is the core step for the information retrieval system (i.e. model), and the information retrieval system chooses relevant documents according to a score.
Further, see Sy on page 2, Background section, second paragraph on page, describe "Aggregation operators we use are preference models that capture end user expectations. The retrieved resources are ordered according to their overall scores, so that the most relevant resources (indexed with the exact query concepts) are ranked higher than the least relevant ones (indexed with hypernyms or hyponyms of query concepts). More interestingly, defining an overall adequacy based on partial similarities enables a precise score to be assigned to each resource w.r.t. every concept of the query. We summarize this detailed information in a small explanatory pictogram and use an interactive semantic map to display top ranked resources. Thanks to this approach, the end user can easily tune the aggregation process, identify, at a simple glance, the most relevant resources, recognize entities adequacy w.r.t. each query concept," Sy describes that with each user query input, the retrieved results are ordered according to overall adequacy score, which relate to a model quality measure since the score measures how well the model performs in retrieving relevant results.
Also, see Sy in page 1, abstract describe “ an information retrieval system that relies on domain ontology to widen the set of relevant documents that is retrieved and that uses a graphical rendering of query results to favor user interactions. Semantic proximities between ontology concepts and aggregating models are used to assess documents adequacy with respect to a query. The selection of documents is displayed in a semantic map to provide graphical indications that make explicit to what extent they match the user's query; this man/machine interface favors a more interactive and iterative exploration of data corpus, by facilitating query concepts weighting and visual explanation.” Here, Sy shows that models are evaluated by adequacy or how well they reflect the original user query’s requirements.
Further, see Sy in page 2, Background, mention “It details particularly operators that are used to aggregate different query concepts, query expansion and the different approaches of similarity measurement used in this context. Then, the methods section describes a new resource-query matching model based on multi-level aggregation of relevance scores. The results section starts by comparing OBIRS engine with some other methods on a benchmark.” Here, Sy mentions using models to perform search requests.
See Sy in page 9, figure 3 mention an interface where the user set a score threshold to retrieve relevant documents.
PNG
media_image2.png
812
1036
media_image2.png
Greyscale
See Sy in page 4, Query expansion section for details.
Also, see Sy in page 4, Query expansion mention “This query expansion technique uses the documents that are judged to be relevant by the user after an initial query to produce a new one using reformulation, re-weighting and expansion”. Here, Sy teaches a query that looks for documents sorted by relevance by adequacy score from page 2.
Further, Sy teaches “and returning one or more of the annotated documents that match the model quality measure.”
See Sy in page 6, in section Proximity measurement between a document and a query, mention “After determining similarities between each concept of the query and (the index of) a document, the next step consists in combining them in a single score that reflects the global relevance of the document w.r.t. the query. User's preferences have to be taken into account during this process in order to determine the overall relevance of a document w.r.t. a query, i.e. its RSV… computing documents’ RSV enables them to be ranked according to their relevance. Furthermore, having the score details of a document for each query concept allows us to justify and compare the source of the match of each document with the query.” Here, Sy describes using a user input query (i.e. search request), to gather documents based on the relevance to user query with an RSV score (i.e. a model quality measure).
Also, see Sy in page 1, abstract describe “ … Semantic proximities between ontology concepts and aggregating models are used to assess documents adequacy with respect to a query. The selection of documents is displayed in a semantic map to provide graphical indications that make explicit to what extent they match the user's query.” Here, Sy mentions the models evaluate documents based on a quality score, and then display documents in a semantic map graphically that match the user’s query (i.e. return documents that match the model quality measure).
Further, see Sy in page 3, from Information Retrieval Systems Overview, mention the model “assigns a score to each document (called RSV- Retrieval Status Value) depending on how well it matches the query.” Here, Sy teaches that the model returns the document that match the user query.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the references of Zhdanov and Chang, and incorporate with the teachings of Sy by using the teachings of Zhdanov and Chang, for a method of using annotated documents and rank them by quality score before training a model, with Sy’s teaching of receiving a search request then returning one or more of the annotated documents that match the model quality measure from that search request.
One of ordinary skill in the art would be motivated to do so because by integrating Sy’s framework into the methods of Zhdanov and Chang, one with ordinary skill in the art would achieve the goal of providing a method where a “ 3-stage relevance model (which allows RSVs to be computed) integrates both the semantic expressiveness of the ontology based data structure and the end-user's preferences. The more user friendly the man-machine interface, the more efficient the interaction between the IRS and the end-user,” (See Sy in page 7, Results section).
Claim 13:
Regarding claim 13, it comprises of similar additional limitations as corresponding claim 6, and is rejected under the same rationale and for similar reasons under 35 U.S.C. 103.
Claim 20:
Regarding claim 20, it comprises of similar additional limitations as corresponding claim 6, and is rejected under the same rationale for similar reasons under 35 U.S.C. 103.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WENWEI ZENG whose telephone number is (571)272-7111. The examiner can normally be reached Monday-Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WenWei Zeng/Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights