DETAILED ACTION
This action is responsive to the application filed on 08/21/2024. Claims 1-9 are pending in the case. Claims 1, 8, and 9 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application
filed in Japan on 09/11/2023. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/21/2024 is being considered by the examiner.
The information disclosure statement (IDS) submitted on 03/06/2025 is being considered by the examiner.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 5-7 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 5 recites “the same data” in lines 2-3. There is insufficient antecedent basis for this limitation in the claim. It is unclear if applicant is attempting to recite a new claim limitation or if applicant is attempting to refer to a previously recited claim element. For examination purposes, this limitation has been interpreted to mean “a same data”, reciting a new claim element.
Claim 6 recites “the same data” in line 2. There is insufficient antecedent basis for this limitation in the claim. It is unclear if applicant is attempting to recite a new claim limitation or if applicant is attempting to refer to a previously recited claim element. For examination purposes, this limitation has been interpreted to refer to the “same data” recited in claim 5.
Claim 7 is rejected as being dependent upon a rejected base claim without curing any of the deficiencies.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-9 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1:
Step 1 Statutory Category: Claim 1 is directed to a machine, which falls within one of the four statutory categories.
Step 2A Prong 1 Judicial exception: Claim 1 recites in part, “evaluate each of the plurality of trained models based on an evaluation indicator defined by a user”. This limitation covers the recitation of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case an evaluation. See MPEP § 2106.04(a)(2)(III).
Step 2A Prong 2 Integration into a practical application: This judicial exception is not integrated into a practical application. In particular the claim recites: “An information processing apparatus, comprising circuitry”. This is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “acquire a plurality of trained models that are trained respectively using different datasets” and “output screen data for displaying the evaluation indicator relating to each of the plurality of trained models” – “for displaying” in this limitation is an intended use limitation and bears no patentable weight. These limitations are additional elements that amount to adding insignificant extra-solution activity to the judicial exception. See MPEP §2106.05(g).
Step 2B Significantly more: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element: “An information processing apparatus, comprising circuitry” amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. Further, the additional elements: “acquire a plurality of trained models that are trained respectively using different datasets” and “output screen data for displaying the evaluation indicator relating to each of the plurality of trained models” amount to adding insignificant extra-solution activity to the judicial exception, and further, is directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). The claim is not patent eligible.
Regarding claim 2, the rejection of claim 1 is incorporated, and further, the claim recites: “wherein the evaluation indicator is an indicator relating to a usage of the plurality of trained models”. This limitation is a continuation of the “evaluate each of the plurality of trained models based on an evaluation indicator defined by a user” limitation identified as an abstract idea in the rejection of the parent claim. Thus, the claim recites an abstract idea.
The claim does not include any additional elements that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 3, the rejection of claim 2 is incorporated, and further, the claim recites: “wherein the usage is a usage relating to a task or work of the user”. This limitation is a continuation of the “wherein the evaluation indicator is an indicator relating to a usage of the plurality of trained models” limitation identified as an abstract idea in the rejection of the parent claim. Thus, the claim recites an abstract idea.
The claim does not include any additional elements that amount to an integration of the judicial exception into a practical application, nor to significantly more than the judicial exception. The claim is not patent eligible.
Regarding claim 4, the rejection of claim 1 is incorporated, and further, the claim recites: “wherein the screen data displays a general-purpose evaluation indicator indicating a prediction accuracy of each of the plurality of trained models and the evaluation indicator in a manner that the general-purpose evaluation indicator and the evaluation indicator can be compared with each other”. This limitation simply provides more detail of the screen data, thus, this limitation is a continuation of “output screen data for displaying the evaluation indicator relating to each of the plurality of trained models” which is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception, See MPEP §2106.05(g); and further, is directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). The claim is not patent eligible.
Regarding claim 5, the rejection of claim 1 is incorporated, and further, the claim recites: “wherein the circuitry outputs second screen data for displaying prediction results of the plurality of trained models for the same data” - “for displaying” in this limitation is an intended use limitation and bears no patentable weight. This limitation is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception, See MPEP §2106.05(g); and further, is directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). The claim is not patent eligible.
Regarding claim 6, the rejection of claim 5 is incorporated, and further, the claim recites: “wherein the second screen data displays one or more of the prediction results relating only to the same data for which the prediction results differ between the plurality of trained models”. This limitation simply provides more detail of the screen data, thus, this limitation is a continuation of “wherein the circuitry outputs second screen data for displaying prediction results of the plurality of trained models for the same data” which is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception, See MPEP §2106.05(g); and further, is directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). The claim is not patent eligible.
Regarding claim 7, the rejection of claim 5 is incorporated, and further, the claim recites: “wherein the second screen data displays one or more of the prediction results of one or more of the plurality of trained models selected by the user”. This limitation simply provides more detail of the screen data, thus, this limitation is a continuation of “wherein the circuitry outputs second screen data for displaying prediction results of the plurality of trained models for the same data” which is an additional element that amounts to adding insignificant extra-solution activity to the judicial exception, See MPEP §2106.05(g); and further, is directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). The claim is not patent eligible.
Regarding claim 8:
Step 1 Statutory Category: Claim 8 is directed to a method, which falls within one of the four statutory categories.
Step 2A Prong 1 Judicial exception: Claim 8 recites in part, “evaluating each of the plurality of trained models based on an evaluation indicator defined by a user”. This limitation covers the recitation of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case an evaluation. See MPEP § 2106.04(a)(2)(III).
Step 2A Prong 2 Integration into a practical application: This judicial exception is not integrated into a practical application. In particular the claim recites: “a computer”. This is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “acquiring a plurality of trained models that are trained respectively using different datasets” and “outputting screen data for displaying the evaluation indicator relating to each of the plurality of trained models” – “for displaying” in this limitation is an intended use limitation and bears no patentable weight. These limitations are additional elements that amount to adding insignificant extra-solution activity to the judicial exception. See MPEP §2106.05(g).
Step 2B Significantly more: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element: “a computer” amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. Further, the additional elements: “acquiring a plurality of trained models that are trained respectively using different datasets” and “outputting screen data for displaying the evaluation indicator relating to each of the plurality of trained models” amount to adding insignificant extra-solution activity to the judicial exception, and further, is directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). The claim is not patent eligible.
Regarding claim 9:
Step 1 Statutory Category: Claim 9 is directed to an article of manufacture, which falls within one of the four statutory categories.
Step 2A Prong 1 Judicial exception: Claim 9 recites in part, “evaluating each of the plurality of trained models based on an evaluation indicator defined by a user”. This limitation covers the recitation of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), in this case an evaluation. See MPEP § 2106.04(a)(2)(III).
Step 2A Prong 2 Integration into a practical application: This judicial exception is not integrated into a practical application. In particular the claim recites: “A non-transitory recording medium storing a plurality of instructions which, when executed by one or more processors, causes the one or more processors to perform a method”. This is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. See MPEP §2106.05(f). Further, the claim recites: “acquiring a plurality of trained models that are trained respectively using different datasets” and “outputting screen data for displaying the evaluation indicator relating to each of the plurality of trained models” – “for displaying” in this limitation is an intended use limitation and bears no patentable weight. These limitations are additional elements that amount to adding insignificant extra-solution activity to the judicial exception. See MPEP §2106.05(g).
Step 2B Significantly more: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element: “A non-transitory recording medium storing a plurality of instructions which, when executed by one or more processors, causes the one or more processors to perform a method” amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process. Elements that merely amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer in its ordinary capacity as a tool to perform an existing process cannot provide an inventive concept. Further, the additional elements: “acquiring a plurality of trained models that are trained respectively using different datasets” and “outputting screen data for displaying the evaluation indicator relating to each of the plurality of trained models” amount to adding insignificant extra-solution activity to the judicial exception, and further, is directed to receiving or transmitting data over a network which courts have recognized as well-understood, routine, and conventional when they are claimed in a generic manner, see MPEP §2106.05(d)(II). The claim is not patent eligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-5 and 7-9 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Dasgupta et al., U.S. Patent No. 10719301, hereinafter referred to as “Dasgupta”.
Regarding claim 1, Dasgupta teaches An information processing apparatus, comprising circuitry (Dasgupta, Col 60, Lines 52-55, “FIG. 30 is a block diagram illustrating an example computer system that can be used to one or more portions of an MDE that allows users to develop models through iterative model experiments, according to some embodiments”; A person of ordinary skill in the art would recognize that a “computer system” provides evidence for “circuitry”) configured to:
acquire a plurality of trained models that are trained respectively using different datasets (Dasgupta, Col 27, Lines 49-51, “At operation 1140, a model experiment is performed according to the user input received at the model experiment interface”; Dasgupta, Col 27, Lines 60-65, “As shown, operation 1140 includes two sub-operations 1142 and 1144. At operation 1142, a training run of the model is executed using the training data set. Operation 1142 may be performed, for example, by the model trainer 160 of FIG. 1 and using one or more components of the model update layer 240 of FIG. 2”; “training” the models is considered to be acquiring the models; Dasgupta, Col 27, Lines 15-25, “At operation 1120, a training data set of media data for a machine learning media model is annotated according to the user input. In some embodiments, the media data management interface may contain user control elements that allow a user to import, export, and label image sets managed by the MDE … In some embodiments, the media data management interface may allow a user to create, a training data set and a test data set from the media data to perform model experiments on ML media models; See also Dasgupta, Figure 11A; The method is performed iteratively, thus “a plurality of trained models” are acquired, and input data is annotated each iteration, thus each model is trained with a different dataset);
evaluate each of the plurality of trained models based on an evaluation indicator defined by a user (Dasgupta, Col 27, Line 65 – Col 28, Line 3, “At operation 1144, a test run of the model is executed using the test data set, and performance metrics for the ML media model for the test run are generated. In some embodiments, operation 1144 may be performed by the model tester 170 of FIG. 1 and using the process shown in FIG. 4; see also Dasgupta, Figure 9; the “precision” metric is considered to be the “evaluation indicator”; Dasgupta, Col 5, Lines 4-13, “Customizable metrics: In some embodiments, the MDE may implement functionality to compute an array of different accuracy metrics to be used to track model performance. Such performance statistics may include metrics such as precision, recall, and F1 scores, AUC (Area Under the Curve), ROC (Receiver Operating Characteristic), MAP (Mean Average Precision) scores. Functionality to compute these metrics may be pre-implemented in the MDE. In some embodiments, the system is extensible to allow a user to introduce additional customized metrics logic”); and
output screen data for displaying the evaluation indicator relating to each of the plurality of trained models (Dasgupta, Col 28, Lines 19-27, “At operation 1150, a model diagnosis interface is generated to display the performance metrics. In some embodiments, the model diagnosis interface may be the model diagnosis interface 148, as discussed in connection with FIG. 1. In some embodiments, the model experiment interface may be a graphical user interface and include one or multiple windows. In some embodiments, the media data management interface may be a web interface that includes one or more webpages generated by a web server”; Dasgupta, Col 28, Lines 31-34, “ In some embodiments, the model diagnosis interface may present the performance data, along with particular media samples, to allow a user to gain insight into the model's performance”; Dasgupta, Col 25, Lines 36-40, “As shown, the user interface 900 may allow users to select a group of experiments to visually compare. For example, the user interface 900 includes a button 912, which allows a user to add or remove experiments from the group of compared experiments; Each experiment is a different model, thus data relating to each of the “plurality of trained models” is output).
Regarding claim 2, the rejection of claim 1 is incorporated, and further, Dasgupta teaches wherein the evaluation indicator is an indicator relating to a usage of the plurality of trained models (Dasgupta, Col 27, Line 65 – Col 28, Line 3, “At operation 1144, a test run of the model is executed using the test data set, and performance metrics for the ML media model for the test run are generated. In some embodiments, operation 1144 may be performed by the model tester 170 of FIG. 1 and using the process shown in FIG. 4; see also Dasgupta, Figure 9; the “precision” metric is considered to be the “evaluation indicator”; A person of ordinary skill in the art would recognize that “precision” of a machine learning model is directed related to how the model is used, thus related to “a usage” of the model).
Regarding claim 3, the rejection of claim 2 is incorporated, and further, Dasgupta teaches wherein the usage is a usage relating to a task of the user (Dasgupta, Col 27, Line 65 – Col 28, Line 3, “At operation 1144, a test run of the model is executed using the test data set, and performance metrics for the ML media model for the test run are generated. In some embodiments, operation 1144 may be performed by the model tester 170 of FIG. 1 and using the process shown in FIG. 4; see also Dasgupta, Figure 9; the “precision” metric is considered to be the “evaluation indicator”; see also, Dasgupta, Figure 7A, These experiment settings include the datasets used and the goal of the experiment, which can be seen to be “classify”, thus using the model relates to the user-specified “task” of classification).
Regarding claim 4, the rejection of claim 1 is incorporated, and further, Dasgupta teaches wherein the screen data displays a general-purpose evaluation indicator indicating a prediction accuracy of each of the plurality of trained models and the evaluation indicator in a manner that the general-purpose evaluation indicator and the evaluation indicator can be compared with each other (Dasgupta, Col 28, Lines 19-27, “At operation 1150, a model diagnosis interface is generated to display the performance metrics. In some embodiments, the model diagnosis interface may be the model diagnosis interface 148, as discussed in connection with FIG. 1. In some embodiments, the model experiment interface may be a graphical user interface and include one or multiple windows. In some embodiments, the media data management interface may be a web interface that includes one or more webpages generated by a web server”; Dasgupta, Col 28, Lines 31-34, “ In some embodiments, the model diagnosis interface may present the performance data, along with particular media samples, to allow a user to gain insight into the model's performance”; Dasgupta, Col 25, Lines 36-40, “As shown, the user interface 900 may allow users to select a group of experiments to visually compare. For example, the user interface 900 includes a button 912, which allows a user to add or remove experiments from the group of compared experiments; Each experiment is a different model, thus data relating to each of the “plurality of trained models” is output; The “precision” is considered to be the “evaluation indicator” and the “accuracy” is considered to be the “general-purpose evaluation indicator”; Dasgupta, Figure 9, shows the evaluation indicators are displayed in a manner such that they can be compared).
Regarding claim 5, the rejection of claim 1 is incorporated, and further, Dasgupta teaches wherein the circuitry outputs second screen data for displaying prediction results of the plurality of trained models for the same data (Dasgupta, Col 28, Lines 31-37, “In some embodiments, the model diagnosis interface may present the performance data, along with particular media samples, to allow a user to gain insight into the model's performance and diagnosis certain problems with model. In some embodiments, for example, the model diagnosis interface may display misclassified samples from the testing set to the user”; Dasgupta, Col 25, Lines 41-47, “In some embodiments, the results for the three experiments (or any number of comparison experiments) may be obtained from a “golden” test set. The golden test set may be constructed ahead of time, and be used as a standard test set for different versions of models, thus allowing the models to be compared under the same test data”).
Regarding claim 7, the rejection of claim 5 is incorporated, and further, Dasgupta teaches wherein the second screen data displays one or more of the prediction results of one or more of the plurality of trained models selected by the user (Dasgupta, Col 28, Lines 31-37, “In some embodiments, the model diagnosis interface may present the performance data, along with particular media samples, to allow a user to gain insight into the model's performance and diagnosis certain problems with model. In some embodiments, for example, the model diagnosis interface may display misclassified samples from the testing set to the user”; This step is part of an iterative process, thus each of the data of the plurality of trained models selected by the user will be output; Dasgupta, Col 25, Lines 41-47, “In some embodiments, the results for the three experiments (or any number of comparison experiments) may be obtained from a “golden” test set. The golden test set may be constructed ahead of time, and be used as a standard test set for different versions of models, thus allowing the models to be compared under the same test data”).
Regarding claim 8, Dasgupta teaches An information processing method performed by a computer (Dasgupta, Col 60, Lines 52-55, “FIG. 30 is a block diagram illustrating an example computer system that can be used to one or more portions of an MDE that allows users to develop models through iterative model experiments, according to some embodiments”), the method comprising:
acquiring a plurality of trained models that are trained respectively using different datasets (Dasgupta, Col 27, Lines 49-51, “At operation 1140, a model experiment is performed according to the user input received at the model experiment interface”; Dasgupta, Col 27, Lines 60-65, “As shown, operation 1140 includes two sub-operations 1142 and 1144. At operation 1142, a training run of the model is executed using the training data set. Operation 1142 may be performed, for example, by the model trainer 160 of FIG. 1 and using one or more components of the model update layer 240 of FIG. 2”; “training” the models is considered to be acquiring the models; Dasgupta, Col 27, Lines 15-25, “At operation 1120, a training data set of media data for a machine learning media model is annotated according to the user input. In some embodiments, the media data management interface may contain user control elements that allow a user to import, export, and label image sets managed by the MDE … In some embodiments, the media data management interface may allow a user to create, a training data set and a test data set from the media data to perform model experiments on ML media models; See also Dasgupta, Figure 11A; The method is performed iteratively, thus “a plurality of trained models” are acquired, and input data is annotated each iteration, thus each model is trained with a different dataset);
evaluating each of the plurality of trained models based on an evaluation indicator defined by a user (Dasgupta, Col 27, Line 65 – Col 28, Line 3, “At operation 1144, a test run of the model is executed using the test data set, and performance metrics for the ML media model for the test run are generated. In some embodiments, operation 1144 may be performed by the model tester 170 of FIG. 1 and using the process shown in FIG. 4; see also Dasgupta, Figure 9; the “precision” metric is considered to be the “evaluation indicator”; Dasgupta, Col 5, Lines 4-13, “Customizable metrics: In some embodiments, the MDE may implement functionality to compute an array of different accuracy metrics to be used to track model performance. Such performance statistics may include metrics such as precision, recall, and F1 scores, AUC (Area Under the Curve), ROC (Receiver Operating Characteristic), MAP (Mean Average Precision) scores. Functionality to compute these metrics may be pre-implemented in the MDE. In some embodiments, the system is extensible to allow a user to introduce additional customized metrics logic”); and
outputting screen data for displaying the evaluation indicator relating to each of the plurality of trained models (Dasgupta, Col 28, Lines 19-27, “At operation 1150, a model diagnosis interface is generated to display the performance metrics. In some embodiments, the model diagnosis interface may be the model diagnosis interface 148, as discussed in connection with FIG. 1. In some embodiments, the model experiment interface may be a graphical user interface and include one or multiple windows. In some embodiments, the media data management interface may be a web interface that includes one or more webpages generated by a web server”; Dasgupta, Col 28, Lines 31-34, “ In some embodiments, the model diagnosis interface may present the performance data, along with particular media samples, to allow a user to gain insight into the model's performance”; Dasgupta, Col 25, Lines 36-40, “As shown, the user interface 900 may allow users to select a group of experiments to visually compare. For example, the user interface 900 includes a button 912, which allows a user to add or remove experiments from the group of compared experiments; Each experiment is a different model, thus data relating to each of the “plurality of trained models” is output).
Regarding claim 9, Dasgupta teaches A non-transitory recording medium storing a plurality of instructions which, when executed by one or more processors, causes the one or more processors to perform a method (Dasgupta, Col 65, Lines 38-41, “One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors implement a model development environment (MDE)”), the method comprising:
acquiring a plurality of trained models that are trained respectively using different datasets (Dasgupta, Col 27, Lines 49-51, “At operation 1140, a model experiment is performed according to the user input received at the model experiment interface”; Dasgupta, Col 27, Lines 60-65, “As shown, operation 1140 includes two sub-operations 1142 and 1144. At operation 1142, a training run of the model is executed using the training data set. Operation 1142 may be performed, for example, by the model trainer 160 of FIG. 1 and using one or more components of the model update layer 240 of FIG. 2”; “training” the models is considered to be acquiring the models; Dasgupta, Col 27, Lines 15-25, “At operation 1120, a training data set of media data for a machine learning media model is annotated according to the user input. In some embodiments, the media data management interface may contain user control elements that allow a user to import, export, and label image sets managed by the MDE … In some embodiments, the media data management interface may allow a user to create, a training data set and a test data set from the media data to perform model experiments on ML media models; See also Dasgupta, Figure 11A; The method is performed iteratively, thus “a plurality of trained models” are acquired, and input data is annotated each iteration, thus each model is trained with a different dataset);
evaluating each of the plurality of trained models based on an evaluation indicator defined by a user (Dasgupta, Col 27, Line 65 – Col 28, Line 3, “At operation 1144, a test run of the model is executed using the test data set, and performance metrics for the ML media model for the test run are generated. In some embodiments, operation 1144 may be performed by the model tester 170 of FIG. 1 and using the process shown in FIG. 4; see also Dasgupta, Figure 9; the “precision” metric is considered to be the “evaluation indicator”; Dasgupta, Col 5, Lines 4-13, “Customizable metrics: In some embodiments, the MDE may implement functionality to compute an array of different accuracy metrics to be used to track model performance. Such performance statistics may include metrics such as precision, recall, and F1 scores, AUC (Area Under the Curve), ROC (Receiver Operating Characteristic), MAP (Mean Average Precision) scores. Functionality to compute these metrics may be pre-implemented in the MDE. In some embodiments, the system is extensible to allow a user to introduce additional customized metrics logic”); and
outputting screen data for displaying the evaluation indicator relating to each of the plurality of trained models (Dasgupta, Col 28, Lines 19-27, “At operation 1150, a model diagnosis interface is generated to display the performance metrics. In some embodiments, the model diagnosis interface may be the model diagnosis interface 148, as discussed in connection with FIG. 1. In some embodiments, the model experiment interface may be a graphical user interface and include one or multiple windows. In some embodiments, the media data management interface may be a web interface that includes one or more webpages generated by a web server”; Dasgupta, Col 28, Lines 31-34, “ In some embodiments, the model diagnosis interface may present the performance data, along with particular media samples, to allow a user to gain insight into the model's performance”; Dasgupta, Col 25, Lines 36-40, “As shown, the user interface 900 may allow users to select a group of experiments to visually compare. For example, the user interface 900 includes a button 912, which allows a user to add or remove experiments from the group of compared experiments; Each experiment is a different model, thus data relating to each of the “plurality of trained models” is output).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Dasgupta in view of Wang et al., HetVis: A Visual Analysis Approach for Identifying Data Heterogeneity in Horizontal Federated Learning, 12/2/2022, IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 29, NO. 1, https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9912364, hereinafter referred to as “Wang”.
Regarding claim 6, the rejection of claim 5 is incorporated, and further, Dasgupta does not explicitly teach wherein the second screen data displays one or more of the prediction results relating only to the same data for which the prediction results differ between the plurality of trained models.
Wang teaches wherein the second screen data displays one or more of the prediction results relating only to the same data for which the prediction results differ between the plurality of trained models (Wang, Page 313, Section 4.2, Lines 1-3, “In the second stage, users need to indirectly compare global data and the local data by comparing prediction behaviors of the two models trained respectively; Wang, Page 313, Section 4.2, Lines 6-11, “To collect various outputs, we provide three sets of data records as input (R2.2, Figure 2(d)), which are local data and two automatically generated datasets (please refer to Section 5.1 for details). The records with inconsistent outputs (records classified as different labels by the two models), or we say in the following, inconsistent records, are then identified (Figure 2(e))”; see also Wang, Page 313, Figure 2e).
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the invention, to have modified the screen data of the information processing apparatus taught by Dasgupta, to include one or more of the prediction results relating only to the same data for which the prediction results differ between the plurality of trained models as taught by Wang. The motivation to do so would have been that identifying inconsistencies between the models’ outputs would allow the user to investigate the heterogeneity in the datasets used to train the different models (Wang, Page 313, Section 4.3, Lines 1-6, “In the third stage, users examine an inconsistency cluster to study the corresponding heterogeneity issue and understand the impacts of the heterogeneity issue. As shown in Figure 2(g), users can check data space characteristics of the records in the selected inconsistency cluster. Related findings inspire users to come up with hypotheses on suspicious heterogeneity”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Murugesan et al., DeepCompare: Visual and Interactive Comparison of Deep Learning, 08/21/2019, IEEE Computer Graphics and Applications, vol. 39, no. 5, pp. 47-59, doi: 10.1109/MCG.2019.2919033 teaches a visual analytic approach to systematically compare the results of deep learning models to provide insight into the models and assess tradeoffs between them; the tool allows users to evaluate model results, identify and compare activation patterns for misclassification, and more.
Dudley et al., A Review of User Interface Design for Interactive Machine Learning, June 2018, ACM Trans. Interact. Intell. Syst. 8, 2, Article 8 (June 2018), 37 pages, https://doi.org/10.1145/3185517 teaches solution principles for building effective interfaces for interactive machine learning, which is an interactive paradigm in which a user iteratively builds and refines a model through cycles of input and review.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOLLY CLARKE SIPPEL whose telephone number is (571)272-3270. The examiner can normally be reached Monday - Friday, 7:30 a.m. - 4:30 p.m. ET..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571)272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.C.S./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights