Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are presented in the case.
Priority
Acknowledgment is made of applicant's claim for foreign priority based on application 2021-121122 filed in Japan on 07/23/2021. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statements submitted on 01/16/2024 and 11/19/2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested: “Systems and Methods for Evaluating Machine-Learning Training Data Using Physics-Based Simulation”.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: a “determination unit that determines a characteristic”, “a presentation unit that presents an evaluation result”, “an input unit that inputs a user operation”, “a model update unit that updates the machine learning model” in claims 9, 19-20. “unit” is the placeholder.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. (See USPGPUB ¶117-119)
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50 (“2019 PEG”)
Claims 1 and 9 have the following abstract idea analysis.
Step 1: The claims are directed to “a method and system”. The claims are directed to the statutory categories accordingly.
Step 2A Prong 1: claims recite the abstract idea limitation of "a determination step of determining a characteristic of each piece of the training data on a basis of an inference result of the machine learning model for the training data;". The limitation include mathematical concept see MPEP § 2106.04(a)(2)) where it cites textual recitation can still be mathematical "determining a ration of A to B". The specification also provides example math using simulations and calculations based on mass and force (See USPGPUB ¶70-71). See USPTO 2024 example 48 where STFT conversion and determining vectors by formula were treated as mathematical operations. Thus, the limitation is an abstract idea in the “mathematical concept”. Other sections of the claims such as "training data", "train a machine learning model" and "a presentation step of presenting an evaluation result" are advanced processes, too generic or high level to be listed as a judicial exception given the available descriptions and MPEP comparisons.
Step 2A Prong 2: The judicial exceptions recited in these claims are not integrated into a practical application. Merely invoking "apparatus", "processing systems", "a determination unit ", "a presentation unit " or "learning model" does not yield eligibility. Claims are still in line with mathematical concepts such as claim 1 and 9 are not specific to a practical application. The additional elements as such are processors and instructions which do not include specialized hardware. See MPEP § 2106.05(a). The math is just being used to produce a result. Claim 1 and 9 do not include a more specific field but even doing so may not be sufficient to overcome the abstract idea rejection. Merely applying an math to a field without an advancement in the new field or new hardware is ineligible. See MPEP § 2106.05(h).
Step 2B: The claims do not contain significantly more than their judicial exceptions. Processors, memory and other hardware are in their standard forms in the field. Note generic processors are recited not new quantum processors. These additional elements are well-understood, routine, and conventional activity, see MPEP 2106.05(d)(II). Claims lacks any particular "how" or algorithm for a solution in a field in a novel way. Claims require more specificity on processes that would be incapable of simple mathematics, mental processes or use more substantial structure than conventional devices such as non-textbook implementations.
Regarding claims 2-8, and 10-20 they merely narrow the previously recited abstract idea limitations with more abstract concepts and/or routine fundamental processes. For the reasons described above with respect to claim 1 this judicial exception is not meaningfully integrated into a practical application, or significantly more than the abstract idea. Abstract idea steps 1, 2A prong 1 and 2 remain the same as independent analysis above. See specification for more practical application concepts as none are seen in claims 2-8, and 10-20.
With respect to step 2B These claims disclose similar limitations described for the dependent claims above and do not provide anything significantly more than organizing human activity concepts. Claims 2-8, and 10-20 recite the additional elements of "the determination unit determines a physical characteristic of an object corresponding to each piece of the training data on a basis of the inference result of the machine learning model, and performs physical simulation calculation between objects each having the determined physical characteristic, and the presentation unit presents the object corresponding each piece of the training data on a basis of a result of the physical simulation calculation. the determination unit determines the physical characteristic of the object corresponding to the training data on a basis of an expected value for each label output by the machine learning model for the training data. the determination unit determines mass, buoyancy, or a size of the object corresponding to the training data on a basis of the expected value for a correct answer label. the determination unit determines at least one of attractive force or repulsive force that acts between the objects corresponding to each piece of the training data on a basis of the expected value for a correct answer label. the determination unit calculates motion information of each object by the physical simulation calculation on a basis of the physical characteristic determined for the object corresponding each piece of the training data, and the presentation unit moves and displays each object on a screen of a display device on a basis of the motion information calculated by the determination unit. an input unit that inputs a user operation on the object displayed on the screen of the display device. in response to a predetermined operation performed on the object displayed on the screen through the input unit, the presentation unit further presents detailed information related to the training data corresponding to the object. the determination unit determines the characteristic of each piece of the training data and the presentation unit presents the evaluation result of the training data each time the machine learning model is updated. a first device that includes the determination unit; and a second device that includes the presentation unit. the second device includes a display device that displays, on a screen, the evaluation result of the training data based on the determined characteristic, and an input unit that inputs a user operation on the screen. a third device that includes a model update unit that updates the machine learning model by training using the training data.". These elements are more abstract concepts, generic applications to a field of use or well-understood, routine, conventional activity (see MPEP § 2106.05(d) and can't be simply appended to qualify as significantly more or being a practical application. What type of application, or structure of components beyond generic machine learning is still unknown for these claims. Therefore claims 2-8, and 10-20 also recites abstract ideas that do not integrate into a practical application or amount to significantly more than the judicial exception, and are rejected under U.S.C. 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 8-9 and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over NUSHI et al. (US 20200349395 A1) hereinafter Nushi in view of Givental et al. (US 20210264025 A1) hereinafter Givental.
As to independent claim 1, Nushi teaches an information processing method that performs processing related to training data to be used to train a machine learning model, the information processing method comprising: [trains a model with a test/training dataset ¶18 "a collection of training instances used in training a machine learning system"…"training dataset"]
a determination step of determining a characteristic of [each piece] of the training data on a basis of an inference result of the machine learning model for the training data; and [model generates outputs and then compares outputs to ground truth ¶12 "learning system trained to generate one or more outputs based on one or more inputs"], [determines error labels, confidence, status values (characteristics) derived from output by evaluating model ¶14, ¶33 "the model evaluation system 106 generates error labels for the instances where the outputs from the machine learning system 108 are incorrect or otherwise inaccurate with respect to the ground truth data." ]
a presentation step of presenting an evaluation result of the training data based on the determined characteristic. [presents performance views with evaluation results Fig. 5A 502-506 ¶79 " model evaluation system 106 may provide the performance views 502-506 to the client device 116 for presentation of one or more performance views"]
Nushi does not specifically teach determining a characteristic of [each piece] of the training data on a basis of an inference result of the machine learning model.
However, Givental teaches determining a characteristic of [each piece] of the training data on a basis of an inference result of the machine learning model. [each data instance in training data gets ranking, and confidence values (characteristics) ¶27-28 "For each data instance (e.g., log) in the level two training dataset, a loss function-based ranking of each of the ML models' predictions is formed based on performance factors of the corresponding ML model. It should be appreciated that the predictions that the ML models generate are probability values, also referred to as confidence values"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the model evaluation disclosed by Nushi by incorporating the determining a characteristic of [each piece] of the training data on a basis of an inference result of the machine learning model disclosed by Givental because both techniques address the same field of machine learning and by incorporating Givental into Nushi improves investigation of results with batter identification of anomalous classifications [Givental ¶24]
As to dependent claim 8, the rejection of claim 1 is incorporated, Nushi and Givental further teach wherein the characteristic of each piece of the training data is determined in the determination step and the evaluation result of the training data is presented in the presentation step each time the machine learning model is updated. [Nushi updates with additional training data and fine tuning then presents ¶59-60 "generate performance views including updated performance statistics."]
As to independent claim 9, Nushi teaches an information processing system [training system and environment ¶26] that performs processing related to training data to be used to train a machine learning model, the information processing system comprising: [trains a model with a test/training dataset ¶18 "a collection of training instances used in training a machine learning system"…"training dataset"]
a determination unit [processor ¶85] that determines a characteristic of [each piece] of the training data on a basis of an inference result of the machine learning model for the training data; and [model generates outputs and then compares outputs to ground truth ¶12 "learning system trained to generate one or more outputs based on one or more inputs"], [determines error labels, confidence, status values (characteristics) derived from output by evaluating model ¶14, ¶33 "the model evaluation system 106 generates error labels for the instances where the outputs from the machine learning system 108 are incorrect or otherwise inaccurate with respect to the ground truth data." ]
a presentation unit [client device with display ¶28-¶30] that presents an evaluation result of the training data based on the determined characteristic. [presents performance views with evaluation results Fig. 5A 502-506 ¶79 " model evaluation system 106 may provide the performance views 502-506 to the client device 116 for presentation of one or more performance views"]
Nushi does not specifically teach determining a characteristic of [each piece] of the training data on a basis of an inference result of the machine learning model.
However, Givental teaches determining a characteristic of [each piece] of the training data on a basis of an inference result of the machine learning model. [each data instance in training data gets ranking, and confidence values (characteristics) ¶27-28 "For each data instance (e.g., log) in the level two training dataset, a loss function-based ranking of each of the ML models' predictions is formed based on performance factors of the corresponding ML model. It should be appreciated that the predictions that the ML models generate are probability values, also referred to as confidence values"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the model evaluation disclosed by Nushi by incorporating the determining a characteristic of [each piece] of the training data on a basis of an inference result of the machine learning model disclosed by Givental because both techniques address the same field of machine learning and by incorporating Givental into Nushi improves investigation of results with batter identification of anomalous classifications [Givental ¶24]
As to dependent claim 17, the rejection of claim 9 is incorporated, Nushi and Givental further teach the determination unit determines the characteristic of each piece of the training data and the presentation unit presents the evaluation result of the training data each time the machine learning model is updated. [Nushi updates with additional training data and fine tuning then presents ¶59-60 "generate performance views including updated performance statistics."]
As to dependent claim 18, the rejection of claim 9 is incorporated, Nushi and Givental further teach a first device that includes the determination unit; and [Nushi server that evaluates and determines Fig. 1 102, 106 ¶29-30]
a second device that includes the presentation unit. [Nushi client device with software and display Fig. 1 116, 118 ¶30]
As to dependent claim 19, the rejection of claim 18 is incorporated, Nushi and Givental further teach the second device includes a display device that displays, on a screen, the evaluation result of the training data based on the determined characteristic, and an input unit that inputs a user operation on the screen. [Nushi displays performance views ¶30 , Fig. 5A 508 ¶79], [Nushi keyboard, mouse ¶100]
As to dependent claim 20, the rejection of claim 18 is incorporated, Nushi and Givental further teach a third device that includes a model update unit that updates the machine learning model by training using the training data. [Nushi training system Fig. 1 110 and updates (fine-tunes) ¶42]
Claims 2-7 and 10-16 are rejected under 35 U.S.C. 103 as being unpatentable over Nushi and Givental, as applied in the rejection of claims 1 above, and further in view of Shin et al. (US 20220067273 A1) hereinafter Shin.
As to dependent claim 2, the rejection of claim 1 is incorporated.
Nushi and Givental do not specifically teach in the determination step, a physical characteristic of an object corresponding to each piece of the training data is determined on a basis of the inference result of the machine learning model, and physical simulation calculation between objects each having the determined physical characteristic is performed, and in the presentation step, the object corresponding each piece of the training data is presented on a basis of a result of the physical simulation calculation.
However, Shin teaches in the determination step, a physical characteristic of an object corresponding to each piece of the training data is determined on a basis of the inference result of the machine learning model, and physical simulation calculation between objects each having the determined physical characteristic is performed, and [forces (physical characteristics) applied to nodes (objects) representing training data (¶91) and results of physical simulation ¶160-165 " a variety of forces are applied to the nodes in a physics based simulation subject to the edge length constraints referred to above. In this example, the nodes are subjected to four different forces which are applied in a 2-dimensional coordinate system to determine the node spatial locations."]
in the presentation step, the object corresponding each piece of the training data is presented on a basis of a result of the physical simulation calculation. [displays graph with the nodes with location based on the simulation ¶172, ¶155 " the node spatial locations are determined by a force-directed model which operates to simulate the application of different physical forces simultaneously to the nodes to achieve an overall positioning of the node spatial locations"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the predictions disclosed by Nushi and Givental by incorporating the in the determination step, a physical characteristic of an object corresponding to each piece of the training data is determined on a basis of the inference result of the machine learning model, and physical simulation calculation between objects each having the determined physical characteristic is performed, and in the presentation step, the object corresponding each piece of the training data is presented on a basis of a result of the physical simulation calculation disclosed by Shin because all techniques address the same field of machine learning systems and by incorporating Shin into Nushi and Givental enhance visualization for more quantifiable and understandable differences [Shin ¶5]
As to dependent claim 3, the rejection of claim 2 is incorporated, Nushi, Givental and Shin further teach in the determination step, the physical characteristic of the object corresponding to the training data is determined on a basis of an expected value for each label output by the machine learning model for the training data. [Givental scores (values) for each label (classes) ¶26-¶28 " ML model may calculate a value of 0.74 for ESCALATE and 0.26 value for CLOSE"]
As to dependent claim 4, the rejection of claim 3 is incorporated, Nushi, Givental and Shin further teach in the determination step, mass, buoyancy, or a size of the object corresponding to the training data is determined on a basis of the expected value for a correct answer label. [Shin mass ¶165 and size based on parameter ¶161]
As to dependent claim 5, the rejection of claim 3 is incorporated, Nushi, Givental and Shin further teach in the determination step, at least one of attractive force or repulsive force that acts between the objects corresponding to each piece of the training data is determined on a basis of the expected value for a correct answer label. [Shin attractive force ¶163 and repulsive force ¶162 based on values from simulation ¶160]
As to dependent claim 6, the rejection of claim 2 is incorporated, Nushi, Givental and Shin further teach in the determination step, motion information of each object is calculated by the physical simulation calculation on a basis of the physical characteristic determined for the object corresponding each piece of the training data, and [Shin velocity (motion) ¶167]
in the presentation step, each object is moved and displayed on a screen of a display device on a basis of the motion information calculated in the determination step. [Shin displayed accordingly on suitable display ¶172, ¶179 "nodes 560, 580 move in opposite directions"
As to dependent claim 7, the rejection of claim 6 is incorporated, Nushi, Givental and Shin further teach an input step of inputting a user operation on the object displayed on the screen of the display device. [Nushi user interactions ¶79 "a user of the client device 116 may interact with the performance views to navigate through different clusters, nodes, and/or test instances "]
As to dependent claim 10, the rejection of claim 9 is incorporated.
Nushi and Givental do not specifically teach the determination unit determines a physical characteristic of an object corresponding to each piece of the training data on a basis of the inference result of the machine learning model, and performs physical simulation calculation between objects each having the determined physical characteristic, and the presentation unit presents the object corresponding each piece of the training data on a basis of a result of the physical simulation calculation.
However, Shin teaches the determination unit determines a physical characteristic of an object corresponding to each piece of the training data on a basis of the inference result of the machine learning model, and performs physical simulation calculation between objects each having the determined physical characteristic, and [forces (physical characteristics) applied to nodes (objects) representing training data (¶91) and results of physical simulation ¶160-165 " a variety of forces are applied to the nodes in a physics based simulation subject to the edge length constraints referred to above. In this example, the nodes are subjected to four different forces which are applied in a 2-dimensional coordinate system to determine the node spatial locations."]
the presentation unit presents the object corresponding each piece of the training data on a basis of a result of the physical simulation calculation. [displays graph with the nodes with location based on the simulation ¶172, ¶155 " the node spatial locations are determined by a force-directed model which operates to simulate the application of different physical forces simultaneously to the nodes to achieve an overall positioning of the node spatial locations"]
Accordingly, it would have been obvious to a person of ordinary skill in the art before the effective filling date of the claimed invention to modify the predictions disclosed by Nushi and Givental by incorporating the the determination unit determines a physical characteristic of an object corresponding to each piece of the training data on a basis of the inference result of the machine learning model, and performs physical simulation calculation between objects each having the determined physical characteristic, and the presentation unit presents the object corresponding each piece of the training data on a basis of a result of the physical simulation calculation disclosed by Shin because all techniques address the same field of machine learning systems and by incorporating Shin into Nushi and Givental enhance visualization for more quantifiable and understandable differences [Shin ¶5]
As to dependent claim 11, the rejection of claim 10 is incorporated, Nushi, Givental and Shin further teach the determination unit determines the physical characteristic of the object corresponding to the training data on a basis of an expected value for each label output by the machine learning model for the training data. [Givental scores (values) for each label (classes) ¶26-¶28 " ML model may calculate a value of 0.74 for ESCALATE and 0.26 value for CLOSE"]
As to dependent claim 12, the rejection of claim 11 is incorporated, Nushi, Givental and Shin further teach the determination unit determines at least one of attractive force or repulsive force that acts between the objects corresponding to each piece of the training data on a basis of the expected value for a correct answer label. [Shin mass ¶165 and size based on parameter ¶161]
As to dependent claim 13, the rejection of claim 11 is incorporated, Nushi, Givental and Shin further teach the determination unit determines at least one of attractive force or repulsive force that acts between the objects corresponding to each piece of the training data on a basis of the expected value for a correct answer label. [Shin attractive force ¶163 and repulsive force ¶162 based on values from simulation ¶160]
As to dependent claim 14, the rejection of claim 10 is incorporated, Nushi, Givental and Shin further teach the determination unit calculates motion information of each object by the physical simulation calculation on a basis of the physical characteristic determined for the object corresponding each piece of the training data, and [Shin velocity (motion) ¶167]
the presentation unit moves and displays each object on a screen of a display device on a basis of the motion information calculated by the determination unit. [Shin displayed accordingly on suitable display ¶172, ¶179 "nodes 560, 580 move in opposite directions"
As to dependent claim 15, the rejection of claim 14 is incorporated, Nushi, Givental and Shin further teach an input unit [Nushi keyboard, mouse ¶100] that inputs a user operation on the object displayed on the screen of the display device. [Nushi user interactions ¶79 "a user of the client device 116 may interact with the performance views to navigate through different clusters, nodes, and/or test instances "]
As to dependent claim 16, the rejection of claim 15 is incorporated, Nushi, Givental and Shin further teach in response to a predetermined operation performed on the object displayed on the screen through the input unit, the presentation unit further presents detailed information related to the training data corresponding to the object. [Nushi navigation ¶79 and drill down operation with data ¶82]
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
PERRY et al. (US 20220319096 A1) teaches machine learning of objects in a virtual environment including physics simulation (see ¶24).
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Beau Spratt whose telephone number is 571 272 9919. The examiner can normally be reached 8:30am to 5:00pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached at 571 272 7212. The fax phone number for the organization where this application or proceeding is assigned is 571 483 7388.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866 217 9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800 786 9199 (IN USA OR CANADA) or 571 272 1000.
/BEAU D SPRATT/Primary Examiner, Art Unit 2143
Prosecution Insights