DETAILED ACTION
This action is responsive to the claims filed on 03/23/2026. Claims 1-3 and 5-9 are pending for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. DE 10 2021 214 329.6, filed on December 14, 2021.
Response to Amendments
Applicant’s arguments with respect to the rejection under 35 U.S.C. §101 have been considered but are not persuasive. Applicant amended claim 1 to further recite that “the coverage is an indication that the data set includes the trigger events at a certain percentage,” “generating synthetic data depending on the coverage,” and “retraining the machine learning system based on the extended data set by the synthetic data.” These amendments do not cause the claim to be patent eligible. The amended “certain percentage” limitation merely further characterizes the previously identified abstract idea of determining coverage by expressing the evaluated coverage as a numerical amount or percentage, which remains an evaluation/judgment that can practically be performed mentally or with the aid of pen and paper. The “generating synthetic data depending on the coverage” limitation is recited at a high level of generality and merely uses the result of the abstract coverage evaluation to determine what additional data should be generated, without reciting a particular technological improvement to the operation of the computer or machine learning system itself. The “retraining” limitation likewise does not integrate the judicial exception into a practical application because, as supported by the specification, retraining the machine learning system is performed by applying data to the machine learning system so that model parameters/weights are adjusted, which constitutes mathematical calculation/optimization performed using a generic machine learning system. Thus, the amended limitations amount to no more than using generic computer/ML components as a tool to apply the abstract idea and do not add significantly more than the judicial exception. Accordingly, the rejection of claim 1 under 35 U.S.C. §101 is maintained.
Applicant’s arguments with respect to claims 1-3 and 5-9 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-3 and 5-9 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 8 is rejected under 35 U.S.C. § 101 as being directed to nonstatutory subject matter.
Step 1: Statutory Categories:
Claims 1-7 are directed to a method.
Claim 9 is directed to a Computer-Readable Medium.
Claim 8 recites a “device configured to” perform the claimed evaluation functions, but does not positively recite sufficient tangible hardware structure, such as a processor, memory, circuitry, or other particular machine components, that limits the claim to a statutory machine or manufacture. Under the broadest reasonable interpretation in light of the specification, which states that the disclosed methods may be implemented as a computer program executed by a processor, the recited “device” encompasses software or a computer program per se. Software per se, without a computer-readable medium or other structural embodiment, does not fall within one of the statutory categories of process, machine, manufacture, or composition of matter under 35 U.S.C. § 101. See MPEP § 2106.03. Accordingly, claim 8 is rejected as being directed to non-statutory subject matter.
Independent Claims – 1, 8, and 9
Step 2A Prong 1: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Independent claims 1, 8, and 9 recites the following limitations that are abstract ideas in the form of mental processes:
Claim 1 recites:
validating the machine learning system on at least a part of the data set, wherein for recurring incorrect outputs of the machine learning system with the same objects or the same environmental conditions, the objects or environmental conditions are identified as trigger events; (“validating” in the context of the claim encompasses a user manually evaluate the results of the machine learning system based on supplied images of different types and identifying specific types of images as trigger events, and therefore constitutes a mental process of evaluation which can reasonably be performed in the human mind or with aid of pen and paper, see MPEP 2106.04(a)(2)(III) for more information on mental processes)
and determining a coverage of the trigger events by the data set depending on the semantic domain model. (“determining a coverage of the trigger events” encompasses a user manually associating specific types of images that result in incorrect outputs according to a domain model and, therefore, constitutes a mental process of evaluation which can reasonably be performed in the human mind or with aid of pen and paper);
wherein the coverage is an indication that the data set includes the trigger events at a certain percentage; (the coverage further being an indication that the data set includes trigger events to a certain percentage does not elevate the limitation past a mental process of evaluation which can reasonably be performed in the human mind or with aid of pen and paper)
and generating synthetic data depending on the coverage; (generating synthetic data depending on a predetermined coverage at a high level of generality is being considered as a mental process of evaluation which can reasonably be performed in the human mind or with aid of pen and paper)
and retraining the machine learning system based on the extended data set by the synthetic data. (this limitation is merely directed to mathematical concepts in the form of mathematical formulas, calculation, or algorithms, see specification pages 8-9 for the related mathematical disclosure)
Claim 1 also recites the following additional elements for the purposes of Step 2A Prong Two analysis:
A method of evaluating a data set with respect to its coverage of trigger events, which can produce erroneous outputs when processed by a machine learning system, the method comprising the following steps providing a semantic domain model (SDM) and the data set; (providing data is merely data gathering and is considered insignificant extra-solution activity, the type of data does not cause the judicial exception to be practically integrated, see MPEP 2106.05(g)).
The additional limitations fail step 2A Prong 2 of the 101 analysis because they do not transform the claim into a practical application. These limitations are too abstract or lack technical improvement that would make the concept practically useful. Without clear utility or integration into a specific field, the claim does not relate to any particular application. It does not meet the requirements of Step 2A Prong 2, as it fails to make the concept meaningfully applicable in practice.
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea.
This claim recites the following additional elements for the purposes of Step 2B analysis:
A method of evaluating a data set with respect to its coverage of trigger events, which can produce erroneous outputs when processed by a machine learning system, the method comprising the following steps providing a semantic domain model (SDM) and the data set; (providing data is merely data gathering and is considered insignificant extra-solution activity, the type of data does not cause the judicial exception to be practically integrated, see MPEP 2106.05(g), for the purpose of step 2b, it should be noted that the courts have recognized receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information) as well-understood, routine, and conventional data).
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 8 and 9 recite limitations substantially similar to claim 1 and as such a similar analysis applies.
Claim 8 also recites the following additional limitation for consideration:
A device configured to evaluate a data set with respect to its coverage of trigger events, which can produce erroneous outputs when processed by a machine learning system, the device configured to (Under step 2A prong 2 and Step 2b computer components (a machine learning system) used at a high level of generality is being considered mere instructions to apply the exception by using computers or machinery merely as a tool, see MPEP 2106.06(f))
Claim 9 also recites the following additional limitation for consideration:
A non-transitory machine-readable storage medium on which is stored a computer program for evaluating a data set with respect to its coverage of trigger events, which can produce erroneous outputs when processed by a machine learning system, the computer program, when executed by a computer, causing the computer to perform the following steps: (Under step 2a prong 2 and step 2b, computer components used at a high level of generality is being considered mere instructions to apply the exception by using computers or machinery merely as a tool, see MPEP 2106.06(f))
Dependent Claims
The remaining dependent claims do not recite additional elements, whether considered individually or in combination, that are sufficient to integrate the judicial exception into a practical application or amount to significantly more than the judicial exception.
Claim 2 recites the further limitation of:
The method according to claim 1, wherein the coverage is determined based on metrics, wherein the metrics characterize a coverage of the trigger events by the data set and/or a coverage of the trigger events with respect to elements of the SDM and/or coverage of the data with respect to the elements of the SDM. (a high level of generality when applying the metrics is being considered a mental process of evaluation which can reasonably be performed in the human mind or with aid of pen and paper);
If claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process but for the recitation of generic computer components, then it falls under the mental process grouping of abstract ideas. Accordingly, the claim "recites" an abstract idea. Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 3 recites the further limitation of:
This claim recites the following non-abstract additional elements for the purposes of Step 2A Prong Two and Step 2B analysis.
The method according to claim 1, wherein the semantic domain model characterizes a description of an input space including an environment of the machine learning system. (a semantic domain model to characterize an input space used at a high level of generality is being considered mere instructions to apply the exception by using computers or machinery merely as a tool, see MPEP 2106.06(f))
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 5 recites the further limitation of:
The method according to claim 1, further comprising: depending on the coverage, outputting whether the data set can be used for training for safety-critical applications or whether the trained machine learning system can be released with the data set for safety-critical applications. (outputting data is merely data gathering and is considered insignificant extra-solution activity, the type of data does not cause the judicial exception to be practically integrated, see MPEP 2106.05(g), for the purpose of step 2b, it should be noted that the courts have recognized receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information) as well-understood, routine, and conventional data);
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 6 recites the further limitation of:
The method according to claim 5, further comprising: based on the data set being used for a safety-critical application, controlling a technical system depending on determined outputs of the machine learning system. (Under Step 2A Prong II and step 2B: controlling a technical system based on the outputs without any further instruction or limitation on the controlling is being considered as mere instruction to apply an exception.);
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 7 recites the further limitation of:
The method according to claim 1, wherein the input variables are images and the machine learning system is an image classifier. (Under step 2a prong 2 and step 2b the general use of a classifier without any further modification to the model is being considered as mere instruction to apply an exception, see MPEP 2106.05(f));
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3 and 7-9 are rejected under 35 U.S.C. 103 as being unpatentable by Brill et al., (US 11409992 B2), hereafter referred to as Brill, in view of Surazhsky et al. (US 20180268255 A1), hereafter referred to as Surazhsky.
Claim 1: Brill teaches the following:
A method of evaluating a data set with respect to its coverage of trigger events, which can produce erroneous outputs when processed by a machine learning system, the method comprising the following steps:
providing a semantic domain model (SDM) and the data set; (Brill, claim 1, “obtaining a functional model representing system requirements of a system utilizing a machine learning prediction model… wherein the functional model comprises a set of attributes, each of which having a respective domain of values,”, Brills’s “functional model” is a semantic description (attributes + value domains) of the input/environment space.
Brill, claim 1, “determining a set of data slices based on the functional model, wherein each data slice of the set of data slices is associated with a different valuation of one or more attributes of the functional model;”, data slices from the set are used to evaluate machine learning performance, as disclosed further below.)
validating the machine learning system on at least a part of the data set, (Brill, claim 1, “computing, for each data slice, a performance measurement of the machine learning prediction model over the data slice,”, applies the model to slice-mapped test data and computes per-slice metrics—i.e., validation on a dataset portion.)
wherein for recurring incorrect outputs of the machine learning system with the same objects or the same environmental conditions, the objects or environmental conditions are identified as trigger events; (Brill, col. 9, line 27, “On Step 160, the performance measurement of each data slice may be checked to determine whether the machine learning prediction model adheres to a target performance requirement. In some exemplary embodiments, data slices whose performance is outside the expected range considering the overall machine learning performance may be detected… On Step 220, a determination whether the data instance is mapped to an underperforming data slice may be performed. In some exemplary embodiments, the data instance may be mapped to one or more data slices, in accordance with the functional model attribute values combinations associated therewith. Under-performing data slices may be indicative that the system performance can be neither determined nor trusted for io coverage gaps.”, A data slice performing outside the expected range is detected (identified as a trigger event); repeatedly poor results in the same slice = recurring errors (the incorrect output).
Brill, col. 14, line 22, “Functional Model 320 may comprise a set of Attributes 322, each of which having a respective domain of Values 324. Attributes 322 may comprise at least one metadata-based attribute that is not comprised Feature Vector 315. As an example, Attributes 322 may comprise gender, type of the medical imaging device, imaging modality, or the like.”, In Brill, each attribute is a semantic descriptor of the input domain (e.g. gender being an object). Because the data slices of Brill are defined by specific valuations of these attributes, Brills “under-performing slices” are the recurring incorrect outputs under the same object or environmental conditions (the trigger events))
and determining a coverage of the trigger events by the data set depending on the semantic domain model. (Brill, claim 5, “wherein said computing comprises determining, for each data slice, a number of testing data instances that are mapped to the data slice”, Coverage/support is counted per SDM-defined slice.
Brill, col. 9, line 38, “On Step 170, under-performing data slice may be reported. In some exemplary embodiments, if the number of test data instances in a data slice is below a predetermined threshold, the data slice may be reported as a coverage gap.”, low-support (under threshold) slices are flagged as “coverage gaps” against a defined coverage goal.
Brill, col. 14, line 22, “In some exemplary embodiments, Functional Model 320 may be configured to represent system requirements of System 380. Functional Model 320 may comprise a set of Attributes 322, each of which having a respective domain of Values 324”, under a broadest reasonable interpretation, semantic domain model is interpreted as “a conceptual framework representing the meanings and relationships of terms and concepts within a particular domain”. Because Brill’s functional model defines the domain semantics as named attributes with allowed values and encodes relationships/constraints to them, it is interpreted that Brill teaches upon a determining a coverage of the trigger events by the data set depending on a semantic domain model.)
Surazhsky, in the same field of neural networks, teaches the following which Brill fails to teach:
wherein the coverage is an indication that the data set includes the trigger events at a certain percentage; (Surazhsky, paragraph 49, “The training engine 110 may determine, based at least on a result of processing the one or more additional synthetic images, whether a performance of the machine learning model meets a threshold value (413). For example, the training engine 110 (e.g., the performance auditor 214) may determine whether the convolutional neural network is able to correctly classify a threshold quantity (e.g., number and/or percentage) of images in a training set and/or a validation set, which may include synthetic images having changed perspectives. Alternately and/or additionally, the training engine 110 (e.g., the performance auditor 214) may determine whether a quantity (e.g., number and/or percentage) of misclassified images is below a threshold value. In some example embodiments, the training engine 110 (e.g., the performance auditor 214) may further gauge the performance of the convolutional neural network based on the error function or cost function associated with the convolutional neural network.”, Surazhsky teaches evaluating model performance using threshold quantities that may be expressed as a number or percentage. Surazhsky specifically teaches determining whether a convolutional neural network correctly classifies a threshold quantity of images in a training set or validation set, and also determining whether a quantity of misclassified images is below a threshold value. Therefore, Surazhsky teaches that the relevant amount of data associated with a model-failure condition may be indicated as a percentage of the evaluated data set.)
and generating synthetic data depending on the coverage; (Surazhsky, paragraph 3, “The operations may include: training a machine learning model by at least processing a training set with the machine learning model, the training set including at least one synthetic image that is generated by applying one or more modifications to a non-synthetic image; determining, based at least on a result of the processing of the mixed training set, that the machine learning model is unable to classify images having a specific modification; and training the machine learning model with additional training data that includes one or more additional synthetic images having the specific modification.”, Surazhsky teaches determining that a machine learning model misclassifies synthetic images having a specific modification, such as altered perspective. Surazhsky then teaches generating additional synthetic images having that specific modification. The generation of the synthetic images depends on the determined classification deficiency for images having the particular modification. Therefore, Surazhsky teaches generating synthetic data depending on the evaluated condition associated with the data, because the additional synthetic images are generated for the condition that the model failed to classify successfully.)
and retraining the machine learning system based on the extended data set by the synthetic data. (Surazhsky, paragraph 6, “In some variations, the training of the machine learning model with the additional training data may be repeated until a performance of the machine learning model meets a threshold value. The performance of the convolutional network may meet the threshold value, when the machine learning model is able to correctly classify a threshold quantity of synthetic images having the specific modification. The performance of the machine learning model may meet the threshold value, when the machine learning model achieves convergence, and wherein the machine learning model achieves convergence when an error function associated with the machine learning model is at minima.”, Surazhsky teaches training the machine learning model with additional training data that includes additional synthetic images having the specific modification. Surazhsky further teaches that this training may be repeated until the performance of the machine learning model meets a threshold value. Therefore, Surazhsky teaches retraining the machine learning system based on a data set extended by the synthetic data, because the additional synthetic images are included in the additional training data used to further train the model.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Brill’s machine-learning data-slice evaluation method to include Surazhsky’s percentage-based synthetic-data retraining technique. Brill teaches identifying under-performing or low-support data slices and identifying the need for additional data with desired feature values. Surazhsky teaches generating additional synthetic images for a specific modification that causes misclassification and training the model using that additional synthetic data. The combination would have predictably improved Brill’s machine-learning system by providing targeted additional training data for the semantic condition corresponding to the under-performing or insufficiently covered data slice.
Claim 2: Brill and Surazhsky teaches the limitations of claim 1, Brill further teaches:
The method according to claim 1, wherein the coverage is determined based on metrics, wherein the metrics characterize a coverage of the trigger events by the data set and/or a coverage of the trigger events with respect to elements of the SDM and/or coverage of the data with respect to the elements of the SDM. (Brill, col. 9, line 3, “On Step 150, a performance measurement of the machine learning prediction model over each data slice may be computed. The computation of the performance measurement may be performed based on an application of the machine learning prediction model on each testing data instance that is mapped to the data slice.
In some exemplary embodiments, the performance measurement may be configured to measure the performance of the model, based on the classification accuracy. Additionally or alternatively, the performance measurement may be an evaluation of the accuracy of the machine learning prediction model along with other metrics such as logarithmic loss, confusion matrix, area under curve, f1-score, mean absolute error, mean squared error, or the like.”, discloses per-slice performance measurements (accuracy, AUC, F1, etc.) which is used to determine coverage as disclosed above for claim 1)
Claim 3: Brill and Surazhsky teaches the limitations of claim 1, Brill further teaches:
The method according to claim 1, wherein the semantic domain model characterizes a description of an input space including an environment of the machine learning system. (Brill, claim 1, “obtaining a functional model representing system requirements of a system utilizing a machine learning prediction model,… wherein the functional model comprises a set of attributes, each of which having a respective domain of values, wherein the set of attributes comprises at least one metadata-based attribute”, Attributes (including metadata about conditions) semantically define the environment/input space)
Claim 7: Brill and Surazhsky teaches the limitations of claim 1, Brill further teaches:
The method according to claim 1, wherein the input variables are images and the machine learning system is an image classifier. (Brill, col. 10, line 52, “Different machine learning prediction models may be configured to provide estimated predictions based on valuations of different feature vectors. In some cases different machine learning prediction models may provide different estimated predictions for the same data instance, may have different performance, different confidence intervals of the machine learning prediction model, or the like. As an example, in a case of image annotation, different classifiers may be utilized for day and night images. The classifier that is configured to provide a high confidence classification in images captured in a day light may not perform well in night images or images captured in the dark.”, images are used as input to determine performance over various machine learning, such as image classifiers.)
Claims 8 and 9 have limitations substantially similar to claim 1, as such a similar analysis applies.
Claim 8 also recites the following additional limitation which Brill further teaches:
A device configured to evaluate a data set with respect to its coverage of trigger events, which can produce erroneous outputs when processed by a machine learning system, the device configured to… (Brill, Fig. 3, “block diagram of an apparatus”; Brill, Fig. 3, “Apparatus 300 may comprise Memory 307”; Brill, Fig. 3, “program code operative to cause Processor 302 to perform acts”; Brill, abstract, “identification and improvement of machine learning (ML) under-performance”; Brill teaches an apparatus including processor(s), memory, I/O, and program code configured to perform the disclosed machine-learning performance testing and data-slicing operations. Brill further teaches evaluating testing data associated with functional-model-defined data slices, detecting under-performing data slices, and performing coverage-gap analysis for such slices. Therefore, Brill teaches a device configured to evaluate a data set with respect to its coverage of trigger-event-like conditions that can produce erroneous or unreliable outputs when processed by a machine learning system.)
Claim 9 also recites the following additional limitation which Brill further teaches:
A non-transitory machine-readable storage medium on which is stored a computer program for evaluating a data set with respect to its coverage of trigger events, which can produce erroneous outputs when processed by a machine learning system, the computer program, when executed by a computer, causing the computer to perform the following steps: (Brill, claim 9, “A computer program product comprising a non-transitory computer readable storage medium retaining program instructions, which program instructions when read by a processor, cause the processor to perform a method comprising:”)
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Brill in view of Surazhsky and Kang et al. (US 10679100 B2), hereafter referred to as Kang.
Claim 5: Brill and Surazhsky teaches the limitations of claim 1, Brill further teaches:
safety-critical applications. (Brill, col. 14, line 10, “As an example, Machine Learning Model 320 may be configured to detect breast cancer in mammographic images and provide a prediction of malignant or benign for the tumor. Testing Data 350 may X-Ray mammographic images along with labels thereof.”, medical diagnosis (breast-cancer detection on mammograms) is a safety-critical application where erroneous model outputs can affect the care of a patient.).)
Kang, in the same field of machine learning performance evaluation, teaches the following limitations which Brill and Surazhsky fails to teach:
The method according to claim 1, further comprising: depending on the coverage, outputting whether the data set can be used for training for safety-critical applications or whether the trained machine learning system can be released with the data set for (Kang, col. 16, line 60, “In this way, a quality of the machine learning training data may be known in advance of (time and computing resource consuming) training of a machine learning model and therefore, a point of performance improvement for a given machine learning model may be clarified and/or determined in advance of training based on whether the coverage metric for a given corpus of machine learning training data satisfies a quality threshold”, Kang expressly gates training/deployment on a coverage metric threshold.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings disclosed by Brill and Surazhsky with the teachings disclosed by Kang (i.e. training a machine learning system based on a performance coverage). Although Brill and Surazhsky does not disclose the use of outputting a trained machine learning model when a performance coverage meets a certain threshold, when put in combination, Kang teaches that exact method which Brills lacks. A motivation for the combination is to have a method to gate training/release decisions using measured data coverage to avoid wasted compute and ensure dataset adequacy. ( Kang, claim 1, “identifies whether to train at least one machine learning classifier of the machine learning-based dialogue system using the corpora of raw machine learning training data based on whether the calculated coverage metric value satisfies a predetermined coverage metric threshold.”).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Brill in view of Surazhsky and Kang as applied to claim 5 above, and further in view of Iandola et al., (US 10678244 B2), hereafter referred to as Iandola.
Claim 6: Brill, Surazhsky and Kang teach the limitations of claim 5, Iandola, in the same field of machine learning performance evaluation, teaches the following limitations which Brill, Surazhsky and Kang fails to teach:
The method according to claim 5, further comprising: based on the data set being used for a safety-critical application, controlling a technical system depending on determined outputs of the machine learning system. (Iandola, col. 4, line 34, “The autonomous control system 110 performs various detection and control algorithms based on sensor data of the physical environment to guide the vehicles in a safe and efficient manner. For example, the autonomous control system 110 may detect various objects (e.g., lamp post, cars) that are proximate to a vehicle in the captured sensor data of the environment, and guide the vehicle away from the objects to prevent collision of the vehicle with the objects. As another example, the autonomous control system 110 may detect boundaries of lanes on the road such that the vehicle can be guided within the appropriate lane with the flow of traffic.”, Iandola discloses using ML-driven detections to actuate vehicle control—i.e., controlling a technical system based on ML outputs using a data set used for safety-critical applications (driver safety).)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have incorporated the teachings disclosed by Iandola with the teachings disclosed by Brill, Surazhsky, or Kang. Although Brill, Surazhsky, and Kang does not disclose the use of generating synthetic data, when put in combination, Iandola teaches a method of generating synthetic data which Brill can use to retrain their machine learning models. A motivation for the combination is to have a method to increase model robustness by augmenting training data and covering missing scenarios. (Iandola, abstract, “In general, this allows autonomous control systems to augment training data to improve performance of computer models, simulate scenarios that are not included in existing training data”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chung, Y., Kraska, T., Polyzotis, N., Tae, K. H., & Whang, S. E. (2019, April). Slice finder: Automated data slicing for model validation. In 2019 IEEE 35th International Conference on Data Engineering (ICDE) (pp. 1550-1553). IEEE.
US 11,215,999 B2 - Data pipeline and deep learning system for autonomous driving
Asudeh, A., Jin, Z., & Jagadish, H. V. (2019, April). Assessing and remedying coverage for a given dataset. In 2019 IEEE 35th International Conference on Data Engineering (ICDE) (pp. 554-565). IEEE.
US 2019/0354613 A1
US 11,714,877 B1
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HYUNGJUN B YI whose telephone number is (703)756-4799. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Jung can be reached at (571) 270-3779. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.B.Y./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights