CTNF 18/279,590 CTNF 100537 DETAILED ACTION This office action is in response to submission of application on 08/30/2023. Claims 1-8 are presented for examination. Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Information Disclosure Statement The information disclosure statement (IDS) submitted on 08/30/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner. Specification 06-11 AIA The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-8 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Claim 1 : Step 1: The claim is directed to a device, which falls within the statutory category of a machine/manufacture. Step 2A Prong 1: The claim is directed to an abstract idea. Specifically, the claim recites: generate evaluation results of the training data creators on a basis of comparison between correct labels corresponding to elements included in the training data and inference labels of the elements ( Abstract idea – mental process. Generating evaluation results of training data creators by comparing training data annotations and inferences can practically be performed in the human mind or with the aid of pen and paper, for example, by viewing the annotations and inferences on a display and mentally determining a degree of similarity or accuracy. The courts have recognized that claims can recite a mental process even if they are claimed as being performed on a computer. See MPEP 2106.04(a)(2)(III).) Step 2A Prong 2: The additional elements recited in the claim do not integrate the abstract idea into a practical application, individually or in combination. Specifically, the claim recites the additional elements: the support device comprising processing circuitry configured to: (This limitation is interpreted as implementation of the disclosed functionality in a generic computing environment, and thus amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).) infer inference labels that are labels corresponding to elements included in the training data using a model that is learned using the training data and infers labels corresponding to the elements; (Generating inference labels using a generic trained machine learning model is standard in the field of machine learning, and thus amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).) Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Specifically, the claim recites the additional elements: the support device comprising processing circuitry configured to: (This limitation is interpreted as implementation of the disclosed functionality in a generic computing environment, and thus amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).) infer inference labels that are labels corresponding to elements included in the training data using a model that is learned using the training data and infers labels corresponding to the elements; (Generating inference labels using a generic trained machine learning model is standard in the field of machine learning, and thus amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).) Claims 2-8 : Claim 2 recites The support device according to claim 1, wherein the training data includes a plurality of element groups each including a plurality of elements in series, and the processing circuitry generates evaluation results for the respective element groups based on comparison between the correct labels corresponding to elements included in corresponding element groups and the inference labels such that the evaluation results can be confirmed for each of the training data creators. This claim merely specifies that the training data upon which the training data creator evaluation (i.e. the mental process ) is performed includes groups of series of elements. Therefore, the claim merges with the abstract idea recited in claim 1, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea. Claim 3 recites The support device according to claim 1, wherein the training data includes a plurality of element groups each including a plurality of elements in series, and the processing circuitry generates training data confirmation screens for the respective element groups including elements included in corresponding element groups, correct labels corresponding to the elements, and inference labels of the elements and are switchable between the element groups for each of the training data creators. Generating training data confirmation screens including the training elements, annotations, and inference labels amounts to adding insignificant extra-solution activity (data outputting) to the judicial exception (see MPEP2106.05(g)), which is well-understood, routine, and conventional (see MPEP 2106.05(d)). Therefore, the claim merges with the abstract idea recited in claim 1, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea. Claim 4 recites The support device according to claim 1, wherein the processing circuitry generates evaluation results for respective elements included in the training data based on comparison between the correct labels and the inference labels for the respective training data creators. This claim merely specifies that the training data creator evaluation (i.e. the mental process ) is performed on individual elements of training data. Therefore, the claim merges with the abstract idea recited in claim 1, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea. Claim 5 recites The support device according to claim 4, wherein the processing circuitry generates training data confirmation screens for the respective elements including the elements, correct labels corresponding to the elements, and inference labels of the elements such that the training data confirmation screens can be confirmed for each of the training data creators. Generating training data confirmation screens including the training elements, annotations, and inference labels amounts to adding insignificant extra-solution activity (data outputting) to the judicial exception (see MPEP2106.05(g)), which is well-understood, routine, and conventional (see MPEP 2106.05(d)). Therefore, the claim merges with the abstract idea recited in claim 4, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea. Claim 6 recites The support device according to claim 4, wherein the processing circuitry includes, in the evaluation results, a difference pattern that is a pattern in which one of the correct labels and one of the inference labels are different and confusion is likely to occur. Including a difference pattern as part of the evaluation can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process ), for example, by viewing the training data, annotations, and inference labels on a display and mentally identifying elements where the annotation and inference label are different. Therefore, the claim merges with the abstract idea recited in claim 4, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea. Claim 7 is a method claim containing substantially the same elements as system claim 1, and is rejected on the same grounds under 35 U.S.C. 101 as claim 1, mutatis mutandis. Claim 8 is a product claim containing substantially the same elements as system claim 1, and is rejected on the same grounds under 35 U.S.C. 101 as claim 1, mutatis mutandis. The additional components of A non-transitory computer readable recording medium recording a program for causing a computer to function as the support device are interpreted as a general-purpose computer and mere instructions to apply the judicial exception on the computer. Therefore, the claims do not recite additional elements that are sufficient to amount to significantly more than the abstract idea. Claim Rejections - 35 USC § 102 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-12-aia AIA (a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention. 07-15 AIA Claim s 1-2, 4, and 6-8 are rejected under 35 U.S.C. 102( a)(1 ) as being anticipated by Kwant et al. (hereinafter Kwant), U.S. Patent Application Publication US-20190102656-A1 (published 04/04/2019). Regarding Claim 1, Kwant teaches A support device for supporting evaluation of training data creators who create training data including sets of elements and correct labels corresponding to the elements, the support device comprising processing circuitry (0004: “According to another embodiment, an apparatus for providing quality assurance for training a prediction model to label one or more features comprises at least one processor, and at least one memory including computer program code for one or more computer programs…”) Kwant teaches configured to: infer inference labels that are labels corresponding to elements included in the training data using a model that is learned using the training data and infers labels corresponding to the elements; and (0004: “train, by a processor, the feature prediction model to label the one or more features (e.g., features detected in an image to support autonomous driving) by using a training data set comprising a plurality of data items (e.g., a plurality of images) with manually marked feature labels. The apparatus is also caused to process the training data set using the trained feature prediction model to generate automatically marked feature labels for the plurality of data items of the training data set.” Automatically marked feature labels (i.e. inference labels) are inferred for items of the training data set using a model which was trained on the training data set.) generate evaluation results of the training data creators on a basis of comparison between correct labels corresponding to elements included in the training data and inference labels of the elements. (0004: “The apparatus is further caused to compute precision data indicating a respective precision between the manually marked feature labels and the automatically marked feature labels for each of the plurality of images in the training data set.” Precision data (i.e. evaluation results) is computed based on precision (i.e. comparison) between the manually marked feature labels (i.e. correct labels) and the automatically marked feature labels (i.e. inference labels) for the items of the training data set.) Regarding Claim 2, Kwant teaches The support device according to claim 1, as shown above. Kwant also teaches wherein the training data includes a plurality of element groups each including a plurality of elements in series, and (0035: “For example, when the feature detector 103 is used in combination with a computer vision system 105 to detect features or objects depicted in input images (e.g., to support visual odometry for autonomous or semi-autonomous navigation), the training data set can be a set of images that have been manually labeled with features of interest (e.g., lane markings, road signs, buildings, and/or used for visual odometry).” 0057: “In one embodiment, each data item or image in the training data set can identify a labeler (e.g., human) that created the manually marked feature labels for the item. For example, each data item or image can include a data field with a labeler ID identifying the labeler.” The training data includes a set of images (i.e. a plurality of elements in series) each associated with a labeler (i.e. an element group).) the processing circuitry generates evaluation results for the respective element groups based on comparison between the correct labels corresponding to elements included in corresponding element groups and the inference labels such that the evaluation results can be confirmed for each of the training data creators. (0057: “In one embodiment, the feature detector 103 then computes a labeling performance value for one or more labelers identified in the training data set based on the precision data.” Labeling performance values (i.e. evaluation results) are generated for each labeler (i.e. for each element group, such that the evaluation results can be confirmed for each training data creator) based on the precision data (i.e. based on the comparison between correct labels and inference labels).) Regarding Claim 4, Kwant teaches The support device according to claim 1, as shown above. Kwant also teaches wherein the processing circuitry generates evaluation results for respective elements included in the training data based on comparison between the correct labels and the inference labels for the respective training data creators. (0049: “Returning to FIG. 2, in step 205, the feature detector 103 computes precision data indicating a respective precision between the manually marked feature labels and the automatically marked feature labels for one or more of the plurality of data items (e.g., images) in the training data set.” Precision data (i.e. evaluation results) is computed based on precision (i.e. comparison) between the manually marked feature labels (i.e. correct labels) and the automatically marked feature labels (i.e. inference labels) for the items (i.e. elements) of the training data set labeled by the labelers (i.e. training data creators).) Regarding Claim 6, Kwant teaches The support device according to claim 4, as shown above. Kwant also teaches wherein the processing circuitry includes, in the evaluation results, a difference pattern that is a pattern in which one of the correct labels and one of the inference labels are different and confusion is likely to occur. (0056: “In one embodiment, the feature detector 103 can determine the characteristics (e.g., overexposed areas as shown in FIG. 3C), that commonly result in instances of precision between manually marked and automatically marked feature labels that do not meet the QA criterion.” The system determines characteristics that commonly result in low precision between manual and automatic labels (i.e. a pattern in which correct labels and inference labels are different and confusion is likely).) Claim 7 is a method claim containing substantially the same elements as system claim 1. Kwant teaches the elements of claim 1, as shown above. Regarding Claim 8, Kwant teaches the support device according to claim 1 , as shown above. Kwant also teaches A non-transitory computer readable recording medium recording a program for causing a computer to function as the support device according to claim 1. (0005: “[A] non-transitory computer-readable storage medium for providing quality assurance for training a prediction model to label one or more features carries one or more sequences of one or more instructions which, when executed by one or more processors, cause, at least in part, an apparatus to [perform the same steps as described above in regard to claim 1].”) Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim s 3 and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Kwant in view of Elisha et al. (hereinafter Elisha), U.S. Patent Application Publication US-20210256420-A1 (filed 02/19/2020). Regarding Claim 3, Kwant teaches The support device according to claim 1, as shown above. Kwant also teaches wherein the training data includes a plurality of element groups each including a plurality of elements in series, and (See the portions of 0035 and 0057 cited above in regard to claim 2. The training data includes a set of images (i.e. a plurality of elements in series) each associated with a labeler (i.e. an element group).) the processing circuitry generates training data confirmation screens for the respective element groups [including elements included in corresponding element groups, correct labels corresponding to the elements, and inference labels of the elements] and are switchable between the element groups for each of the training data creators. (0060: “FIG. 5C is an example of a UI 541 for presenting QA reports based on the label precision data generated according to the various embodiments described herein. As shown, the UI 541 can present a QA report section 543 displaying the performance of Labelers A and B who generated some of the manually marked feature labels in the training data set. In this example, Labeler A is calculated to have 90% of his/her images or data items marked with acceptable precision (e.g., precision above the QA criteria), while Labeler B is calculated to have 60% of his/her images or data items marked with acceptable precision.” A user interface (UI) (i.e. confirmation screen) is generated to present the precision performance of each labeler (i.e. training data creator) on their respective data items (i.e. element group). As can be seen in figure 5C, the precision data can be viewed individually for each labeler group (i.e. is switchable between element groups).) Kwant does not appear to explicitly disclose confirmation screens including elements included in corresponding element groups, correct labels corresponding to the elements, and inference labels of the elements However, Elisha teaches confirmation screens including elements included in corresponding element groups, correct labels corresponding to the elements, and inference labels of the elements (0008-0009: “FIG. 3 shows a block diagram of a confusion matrix in accordance with an embodiment. FIG. 4 shows a block diagram of a confusion matrix in which each of its row elements are populated with exemplary values in accordance with an embodiment.” As can be seen in figures 3 and 4, the confusion matrix includes rows and columns corresponding to actual labels (i.e. correct labels) and predicted labels (i.e. inference labels), and values indicating the number of data items (i.e. elements) associated with each combination of actual and predicted label.) It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Kwant and Elisha. Kwant teaches quality assurance for machine learning model training by comparing training data labels to model inferences on the training data and evaluating the precision of labels assigned by different labelers. Elisha teaches improving machine learning models by detecting and removing inaccurate training samples, including generating a training data confusion matrix which presents a comparison between actual and predicted labels. One of ordinary skill would have motivation to combine Kwant and Elisha because a confusion matrix is a visualization of the training data which allows for “determin[ing] whether two labels are problematic (e.g., two labels similar (or have a conflict), whether a label is weak, or whether a label disturbs another label)” (Elisha, 0071). After determining problematic labels and samples, “ML model accuracy may be improved by training on a more accurate revised training set” (Elisha, 0025). Regarding Claim 5, Kwant teaches The support device according to claim 4, as shown above. Kwant also teaches wherein the processing circuitry evaluation unit generates training data confirmation screens for the respective elements [including the elements, correct labels corresponding to the elements, and inference labels of the elements] such that the training data confirmation screens can be confirmed for each of the training data creators. (0060: “FIG. 5C is an example of a UI 541 for presenting QA reports based on the label precision data generated according to the various embodiments described herein. As shown, the UI 541 can present a QA report section 543 displaying the performance of Labelers A and B who generated some of the manually marked feature labels in the training data set. In this example, Labeler A is calculated to have 90% of his/her images or data items marked with acceptable precision (e.g., precision above the QA criteria), while Labeler B is calculated to have 60% of his/her images or data items marked with acceptable precision.” A user interface (UI) (i.e. confirmation screen) is generated to present the precision performance for the data items (i.e. elements), aggregated by labeler (i.e. such that the confirmation screens can be confirmed for each training data creator).) Kwant does not appear to explicitly disclose confirmation screens including the elements, correct labels corresponding to the elements, and inference labels of the elements However, Elisha teaches confirmation screens including the elements, correct labels corresponding to the elements, and inference labels of the elements (0008-0009: “FIG. 3 shows a block diagram of a confusion matrix in accordance with an embodiment. FIG. 4 shows a block diagram of a confusion matrix in which each of its row elements are populated with exemplary values in accordance with an embodiment.” As can be seen in figures 3 and 4, the confusion matrix includes rows and columns corresponding to actual labels (i.e. correct labels) and predicted labels (i.e. inference labels), and values indicating the number of data items (i.e. elements) associated with each combination of actual and predicted label.) Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN M ROHD whose telephone number is (571)272-6445. The examiner can normally be reached Mon-Thurs 8:00-6:00 EST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /B.M.R./Examiner, Art Unit 2147 /ERIC NILSSON/Primary Examiner, Art Unit 2151 Application/Control Number: 18/279,590 Page 2 Art Unit: 2147 Application/Control Number: 18/279,590 Page 3 Art Unit: 2147 Application/Control Number: 18/279,590 Page 4 Art Unit: 2147 Application/Control Number: 18/279,590 Page 5 Art Unit: 2147 Application/Control Number: 18/279,590 Page 6 Art Unit: 2147 Application/Control Number: 18/279,590 Page 7 Art Unit: 2147 Application/Control Number: 18/279,590 Page 8 Art Unit: 2147 Application/Control Number: 18/279,590 Page 9 Art Unit: 2147 Application/Control Number: 18/279,590 Page 10 Art Unit: 2147 Application/Control Number: 18/279,590 Page 11 Art Unit: 2147 Application/Control Number: 18/279,590 Page 12 Art Unit: 2147 Application/Control Number: 18/279,590 Page 13 Art Unit: 2147 Application/Control Number: 18/279,590 Page 14 Art Unit: 2147 Application/Control Number: 18/279,590 Page 15 Art Unit: 2147