DETAILED ACTION
This office action is in response to amendments filed on 03/18/2026.
Claims 20, 22, and 32 have been amended. Claim 34 has been canceled. Claims 20-33 and 35-38 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/06/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Arguments
Claim Objections:
In light of applicant’s amendments to the claims (pg. 2-9), the objections to the claims have been withdrawn.
Rejections Under 35 USC § 112(b):
In light of applicant’s amendments to the claims (pg. 2-9), the rejections under 35 USC § 112(b) have been withdrawn.
Rejections Under 35 USC § 101:
Applicant's arguments regarding the rejection under 35 USC § 101 (pg. 15-17) have been fully considered but they are not persuasive.
Applicant argues that the amended claim 20 does not recite an abstract idea because it includes collecting real-world measurements using sensors and using the model to control a technical system such as a vehicle. Examiner notes that, as can be seen in the rejection below, the claim recites mental steps such as determining and comparing data coverage quality ratings, and the steps of data collection and vehicle control are analyzed as additional elements which are not sufficient to integrate the abstract idea into a practical application.
Applicant additionally argues that the amended claim 20 contains an improvement to technology in the safety-critical field of autonomous vehicles, as corrective action can be taken to remedy identified areas of input space which lack a sufficient basis of training examples. Examiner notes that while the claim describes evaluating the coverage of the collected training data, it does not recite taking corrective action to achieve the alleged improvement to safety.
The rejections under 35 USC § 101 have been updated to include the amended limitations and to clarify the reasoning given for the limitations that were not amended.
Prior Art Rejections:
Applicant's arguments regarding the rejection under 35 USC § 103 have been fully considered but they are not persuasive.
Applicant argues (pg. 17-18) that the method of Grichnik operates on a single dataset, while the claims require a first and second example set. Examiner notes that, as can be seen in the rejection below, Grichnik is not relied upon to teach the first and second example sets. Grichnik teaches performing operations on a single dataset to measure data quality, and Dredze motivates extension of these operations to a first and second dataset to identify domain shift.
Similarly, applicant argues (pg. 18-20) that Grichnik’s measurement of data coverage is different from the claimed determination of quality ratings because Grichnik’s measurements are computed within a single dataset as an internal consistency check, while the claimed quality ratings are computed for the purpose of comparison between the first and second example sets. Examiner notes that both Grichnik’s data coverage measurements and the claimed data quality ratings are computed within a single dataset (the claimed first quality rating is computed within the first example set, and the claimed second quality rating is computed within the second example set). Thus, Grichnik’s data coverage measurements fall within the broadest reasonable interpretation of the claimed quality ratings. While Grichnik may not explicitly suggest using data coverage measurements to compare two datasets, one of ordinary skill in the art would have been motivated to use them for this purpose by Dredze, which teaches comparing the distribution of two datasets in order to detect domain shift.
Applicant argues (pg. 20-22) that the first
n
samples and most recent
n
samples taught by Dredze come from the same continuous data stream, while the claimed first and second example sets are independently acquired in separate acquisition steps. Applicant points to the instant application’s specification, pg. 31, lines 12-16, which describes collecting the second example set by taking into account knowledge regarding the first example set. Applicant is reminded it is improper to import claim limitations from the specification See MPEP 2111.01. “Though understanding the claim language may be aided by explanations contained in the written description, it is important not to import into a claim limitations that are not part of the claim. For example, a particular embodiment appearing in the written description may not be read into a claim when the claim language is broader than the embodiment.” Superguide Corp. v. DirecTV Enterprises, Inc., 358 F.3d 870, 875, 69 USPQ2d 1865, 1868 (Fed. Cir. 2004). Examiner notes that nothing in the claim precludes the first and second example sets from being acquired from the same continuous data stream, and thus the first
n
samples and most recent
n
samples taught by Dredze fall within the broadest reasonable interpretation of the claimed first and second example sets.
Applicant argues (pg. 22-23) that the distributions
P
and
P
’
are probability distributions, while the claimed first and second quality ratings are based on an absolute count of examples assigned to representatives. Examiner notes that, as can be seen in the rejection below, the specific count-based nature of the quality ratings is taught by Grichnik’s measurement of data distribution. Dredze simply motivates obtaining this representation for two different data distributions so that they can be compared to identify a domain shift.
Applicant argues (pg. 23-24) that Dredze’s method of comparing data distributions requires collapse of each example’s features to a single value, which is incompatible with the input space analysis of amended claim 20. Examiner disagrees with this conclusion of incompatibility. Nothing in the claim precludes the input space from being one dimensional, in which case the first and second example sets would consist of single values and their distributions could be straightforwardly compared using the A-distance method taught by Dredze. Further, examiner notes that the test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one or all of the references. Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981). In this case, secondary reference Dredze would have motivated one of ordinary skill in the art to compare the distributions of two datasets in order to identify a domain shift.
Applicant argues (pg. 24) that the amended claim 20 requires the representatives to be distributed at the same positions in the input space for both the first and second example sets. Applicant points to the instant application’s specification, pg. 10, lines 23-29 as evidence of this requirement, and asserts that this feature has no counterpart in the cited references. Applicant is reminded it is improper to import claim limitations from the specification See MPEP 2111.01. “Though understanding the claim language may be aided by explanations contained in the written description, it is important not to import into a claim limitations that are not part of the claim. For example, a particular embodiment appearing in the written description may not be read into a claim when the claim language is broader than the embodiment.” Superguide Corp. v. DirecTV Enterprises, Inc., 358 F.3d 870, 875, 69 USPQ2d 1865, 1868 (Fed. Cir. 2004). Examiner notes that this requirement on representative positioning is not recited in the claim.
Applicant argues (pg. 25-26) that cited references Grichnik and Dredze have different technical purposes, and one of ordinary skill in the art would not have had motivation to combine them. It has been held that a prior art reference must either be in the field of the inventor’s endeavor or, if not, then be reasonably pertinent to the particular problem with which the inventor was concerned, in order to be relied upon as a basis for rejection of the claimed invention. See In re Oetiker, 977 F.2d 1443, 24 USPQ2d 1443 (Fed. Cir. 1992). In this case, the claimed invention is directed to quality assurance of training data by analyzing and comparing the coverage of input space by two datasets to detect data drift. The cited reference Grichnik is concerned with quality assurance of training data by analyzing the coverage of input space. The cited reference Dredze is concerned with quality assurance of training data by comparing the distributions of two datasets to detect data drift. The teachings of both Grichnik and Dredze are reasonably pertinent to the problem with which the inventor is concerned, and one of ordinary skill in the art would have been motivated to combine them because Grichnik provides a method to measure the distribution of input data, and Dredze teaches comparing two distributions of input data to detect a domain shift. As can also be seen in the rejection below, theses references, in combination with Khodabandeh, disclose or suggest each feature of the amended claim 20.
The prior art rejections have been updated to include the amended limitations and to clarify the reasoning given for the limitations that were not amended.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 20-33 and 35-38 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1:
Step 1: The claim is directed to a method, which falls within the statutory category of a
process.
Step 2A Prong 1: The claim is directed to an abstract idea. Specifically, the claim recites:
collecting a first example set including a plurality of examples and a second example set including a plurality of examples; (Abstract idea – mental process. Collecting a first and second example set can practically be performed in the human mind or with the aid of pen and paper, for example, by viewing a plurality of example data points written out on a sheet of paper and mentally dividing the examples into two sets. See MPEP 2106.04(a)(2)(III).)
determining a first quality rating representing coverage of the input space by the examples of the first example set based on a distribution of input values in the input space, distributing representatives within the input space, assigning a number of examples to the representatives, and determining the quality rating based on a number of examples assigned to a respective representative; (Abstract idea – mental process. Distributing representatives within the input space, assigning examples to the representatives, and determining a quality rating representing input space coverage of the example set can practically be performed in the human mind or with the aid of pen and paper, for example, by viewing a two-dimensional graphical representation of the examples plotted in input space on a sheet of paper, dividing it into a grid, mentally counting the number of examples which fall within each cell of the grid, and mentally evaluating the data coverage of the grid. See MPEP 2106.04(a)(2)(III).)
determining a second quality rating representing coverage of the input space by the examples of the second example set based on the distribution of input values in the input space, distributing representatives within the input space, assigning a number of examples to the representatives, and determining the quality rating based on a number of examples assigned to a respective representative; and (Abstract idea – mental process. Distributing representatives within the input space, assigning examples to the representatives, and determining a quality rating representing input space coverage of the example set can practically be performed in the human mind or with the aid of pen and paper, for example, by viewing a two-dimensional graphical representation of the examples plotted in input space on a sheet of paper, dividing it into a grid, mentally counting the number of examples which fall within each cell of the grid, and mentally evaluating the data coverage of the grid. See MPEP 2106.04(a)(2)(III).)
comparing the first quality rating and the second quality rating with one another, determining a comparative variable corresponding to an amount of a difference between the number of examples of the first example set assigned to the representative and the number of examples of the second example set assigned to the representative, and (Abstract idea – mental process. Comparing two quality ratings and determining a difference between the number of examples assigned to representatives is an evaluation/judgement which can practically be performed in the human mind or with the aid of pen and paper. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2: The additional elements recited in the claim do not integrate the abstract idea into a practical application, individually or in combination. Specifically, the claim recites the additional elements:
creating and training the example-based system based on collected examples forming an example set, using sensors of an object recognition system to collect the examples; (Creating and training a generic neural network (i.e. example-based system, see instant application specification, 0002) based on an example set obtained using sensors is standard in the field of machine learning, and thus amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).)
a respective example of the example set including an input value lying in an input space; (Generic training examples including input values in input space are standard in the field of machine learning, and thus this limitation amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).)
using the object recognition system in an automated operation of at least one of a vehicle, a track-bound vehicle, a motor vehicle, an aircraft, a watercraft or a spacecraft. (Using the object recognition model for automated operation of a vehicle amounts to generally linking the use of a judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Specifically, the claim recites the additional elements:
creating and training the example-based system based on collected examples forming an example set, using sensors of an object recognition system to collect the examples; (Creating and training a generic neural network (i.e. example-based system, see instant application specification, 0002) based on an example set obtained using sensors is standard in the field of machine learning, and thus amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).)
a respective example of the example set including an input value lying in an input space; (Generic training examples including input values in input space are standard in the field of machine learning, and thus this limitation amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).)
using the object recognition system in an automated operation of at least one of a vehicle, a track-bound vehicle, a motor vehicle, an aircraft, a watercraft or a spacecraft. (Using the object recognition model for automated operation of a vehicle amounts to generally linking the use of a judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h).)
Claims 21-33 and 35-38:
Claim 21 recites The method according to claim 20, which further comprises: forming a third example set from the first and second example sets; and determining a third quality rating representing a coverage of the input space by examples of the third example set based on the distribution of the input values in the input space; and comparing the first quality rating, the second quality rating and the third quality rating. Combining two example sets into a third set, determining a quality rating representing input space coverage of the third set, and comparing the quality ratings can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process), for example, by plotting the examples from the first and second example sets by hand in a two-dimensional graphical representation of input space and mentally comparing the coverage of the combined third example set with the coverage of the individual first and second example sets. See MPEP 2106.04(a)(2)(III). Therefore, the claim merges with the abstract idea recited in claim 20, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 22 recites The method according to claim 21, which further comprises: carrying out the determination of any one of the first quality rating, the second quality rating or the third quality rating by: distributing representatives in the input space, and assigning a plurality of examples of the example set to a respective representative; locating the examples assigned to the representative in a surrounding area of the input space surrounding the representative; and at least one of: determining, as the first quality rating, a local quality rating for the surrounding area based on the examples of the first example set assigned to the representative, or determining, as the second quality rating, a local quality rating for the surrounding area based on the examples of the second example set assigned to the representative. Distributing representatives in input space, assigning examples to their surrounding areas, and determining a local quality rating for a surrounding area representing input space coverage of the assigned examples can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process), for example, by viewing a two-dimensional graphical representation of the input space on a sheet of paper, dividing it into a grid, and mentally counting the number of examples which fall within each cell of the grid. See MPEP 2106.04(a)(2)(III). Therefore, the claim merges with the abstract idea recited in claim 21, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 23 recites The method according to claim 22, which further comprises providing the third quality rating as a local quality rating for the surrounding area and determining the third quality rating based on the examples of the third example set assigned to the representative. Providing a local quality rating for a surrounding area representing input space coverage of the assigned examples from the combined example set can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process), for example, by viewing a two-dimensional graphical representation of the grid-partitioned input space on a sheet of paper and mentally counting the number of examples of the combined example set which fall within each cell of the grid. See MPEP 2106.04(a)(2)(III). Therefore, the claim merges with the abstract idea recited in claim 22, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 24 recites The method according to claim 22, which further comprises providing the quality rating with at least one of statistical measures determined based on the set of examples or examples assigned to a respective representative. Determining the quality rating as a statistical measure based on the examples assigned to a representative can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process), for example, by mentally counting the number of examples assigned to the representative. See MPEP 2106.04(a)(2)(III). Therefore, the claim merges with the abstract idea recited in claim 22, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 25 recites The method according to claim 24, which further comprises determining as a statistical mean at least one of a statistical measure, a mean, a median, a minimum or quantiles of the plurality of examples assigned to a representative. Determining a statistical mean, median, minimum, or quantiles is a mathematical calculation (i.e. mathematical concept). See MPEP 2106.04(a)(2)(I). Therefore, the claim merges with the abstract idea recited in claim 24, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 26 recites The method according to claim 22, which further comprises determining adjacent surrounding areas in the input space having a respective representative assigned a plurality of examples fulfilling a predetermined quality criterion of the quality rating. Determining adjacent surrounding areas with representatives assigned examples fulfilling a quality criterion can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process), for example, by viewing a two-dimensional graphical representation of the grid-partitioned input space on a sheet of paper and mentally identifying adjacent regions of the grid which contain less than a threshold number of examples. See MPEP 2106.04(a)(2)(III). Therefore, the claim merges with the abstract idea recited in claim 22, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 27 recites The method according to claim 26, which further comprises determining a connection area formed of adjacent surrounding areas within the input space and assigning each representative of the surrounding areas a number of examples fulfilling a predetermined quality criterion of the quality rating. Determining a connection area formed of adjacent surrounding areas in input space and assigning each associated representative examples fulfilling a quality criterion can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process), for example, by viewing a two-dimensional graphical representation of the grid-partitioned input space on a sheet of paper, mentally identifying adjacent regions of the grid which contain less than a threshold number of examples, mentally designating the identified regions as a connection area, and plotting additional examples in the adjacent surrounding areas by hand until the threshold number of examples per region is met. See MPEP 2106.04(a)(2)(III). Therefore, the claim merges with the abstract idea recited in claim 26, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 28 recites The method according to claim 22, which further comprises: providing the respective example with an output value located in an output space; determining for the respective surrounding area a local complexity rating representing a complexity of a task of the example-based system, the complexity being defined by the examples of the surrounding area; and determining the local complexity rating by a relative position of the examples of the surrounding area with respect to one another in the input space and the output space. Generic training examples including output values in output space are standard in the field of machine learning, and thus amount to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f). Determining a local complexity rating for a surrounding area based on the relative positions of its assigned examples in input and output space can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process), for example, by viewing a two-dimensional graphical representation of the examples in the surrounding area plotted in input space and output space on a sheet of paper and mentally comparing the distances between pairs of points in input and output space. See MPEP 2106.04(a)(2)(III). Therefore, the claim merges with the abstract idea recited in claim 22, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 29 recites The method according to claim 28, which further comprises: determining a first local complexity rating for the examples of the first example set, determining a second local complexity rating for the examples of the second example set and determining a third local complexity rating for the examples of the third example set; and comparing the third local complexity rating with at least one of the first or second local complexity rating. Determining local complexity ratings for each of the three datasets and comparing them can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process), for example, by mentally determining a local complexity rating based on examples from each of the datasets and mentally evaluating the differences between the determined ratings. See MPEP 2106.04(a)(2)(III). Therefore, the claim merges with the abstract idea recited in claim 28, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 30 recites The method according to claim 28, which further comprises a complexity distribution is determined by using a histogram representation of the complexity rating. Determining a complexity distribution using a histogram representation of the complexity rating can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process), for example, by mentally determining a complexity rating for each surrounding area in the input space, mentally determining bins encompassing ranges of complexity rating values, and drawing a histogram by hand representing the number of surrounding areas with complexity ratings falling within each bin. See MPEP 2106.04(a)(2)(III). Therefore, the claim merges with the abstract idea recited in claim 28, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 31 recites The method according to claim 30, which further comprises determining a complexity distribution over k-nearest neighbors of an example in the input space. Determining a complexity distribution over the k-nearest neighbors of an example can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process), for example, by mentally determining complexity ratings for each example based on a comparison between its distances in input and output space to each of its k nearest neighbors, which can be identified visually by viewing a two-dimensional graphical representation of the examples, and then drawing a histogram by hand as above to represent the distribution of complexity ratings. See MPEP 2106.04(a)(2)(III). Therefore, the claim merges with the abstract idea recited in claim 30, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 32 recites The method according to claim 28, which further comprises: providing the complexity rating as an integrated quality indicator
Q
I
2
; determining the integrated quality indicator based on a definition as follows:
Q
I
2
P
=
1
|
P
2
|
∑
x
i
∈
P
2
(
d
N
R
E
x
i
-
d
N
R
A
x
i
)
2
wherein:
d
N
R
E
x
=
d
R
E
x
∑
y
∈
P
2
d
R
E
y
|
P
2
|
is a normalized distance of the represented inputs and
d
N
R
A
x
=
d
R
A
x
∑
y
∈
P
2
d
R
A
y
|
P
2
|
is a normalized distance of the represented outputs, wherein
x
is a pair
(
x
1
,
x
2
,
)
formed two examples
x
1
and
x
2
, wherein
x
1
and
x
2
are examples from an example set
P
, wherein
P
=
{
p
1
,
p
1
,
.
.
.
,
p
|
P
|
}
is a set of elements of a multiset
B
A
G
P
, and wherein
|
P
2
|
is a number of pairs of elements of the multiset
B
A
G
P
; wherein
d
R
E
x
is a distance between two points in the input space, and wherein
d
R
A
x
is a distance between two points in the output space. This claim is directed to a mathematical formula/equation (i.e. mathematical concept). See MPEP 2106.04(a)(2)(I). Therefore, the claim merges with the abstract idea recited in claim 28, and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 33 recites The method according to claim 20, which further comprises providing the example-based system for use in a safety-related function and providing the safety-related function with object recognition based on an image recognition, in which the object is recognized by using the example-based system. Using the example-based system for object recognition in a safety-related function amounts to generally linking the use of a judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h). Therefore, the claim does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 35 recites The method according to claim 20, which further comprises providing the example-based system for use in a safety-related function and using the safety-related function to represent a classification based on sensor data of organisms or a safe control of industrial plants, including a classification of chemical substances, a classification of signatures of vehicles or a control in a field of industrial automation. Using the example-based system in a safety-related function for classification based on sensor data or safe control of industrial plants amounts to generally linking the use of a judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h). Therefore, the claim does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 36 recites The method according to claim 20, which further comprises providing the example-based system with: a system with supervised learning, or an artificial neural network with one or more layers of neurons not being input neurons or output neurons and being trained with back-propagation, or a convolutional neural network, or a single-shot multibox detector network. Systems with supervised learning, neural networks with hidden layers trained with back-propagation, convolutional neural networks, and single-shot multibox detector networks are generic models which are standard in the field of machine learning, and thus this limitation amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f). Therefore, the claim does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 37 recites A computer program stored on a non-transitory computer-readable storage medium, the computer program comprising commands which, when the program is executed by a computing unit, cause the computing unit to perform the method according to claim 20. This limitation is interpreted as implementation of the claimed method in a generic computing environment, and thus amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f). The courts have recognized that claims can recite a mental process even if they are claimed as being performed on a computer. Therefore, the claim does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 38 recites A non-transitory computer-readable storage medium, comprising commands which, when executed by a computing unit, cause the computing unit to perform the method according to claim 20. This limitation is interpreted as implementation of the claimed method in a generic computing environment, and thus amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f). The courts have recognized that claims can recite a mental process even if they are claimed as being performed on a computer. Therefore, the claim does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 20-25, 33, and 35-38 are rejected under 35 U.S.C. 103 as being unpatentable over
Grichnik et al. (hereinafter Grichnik), U.S. Patent Application Publication US-20090300052-A1 (published 12-03-2009) in view of
Dredze et al. (hereinafter Dredze), “We’re Not in Kansas Anymore: Detecting Domain Changes in Streams” (published 10-09-2010) and
Khodabandeh et al. (hereinafter Khodabandeh), “A Robust Learning Approach to Domain Adaptive Object Detection” (published 11-18-2019).
Regarding Claim 20,
Grichnik teaches A method for quality assurance of an example-based system, the method comprising:
creating and training the example-based system based on collected examples forming an example set, (0009: “The method may also include generating a computational model indicative of interrelationships between the plurality of input parameters and the one or more output parameters based on the data records.” 0039: “The neural network computational model may be trained by using the data records.” A neural network computational model (i.e. example-based system, see instant application specification, 0002) is generated (i.e. created) and trained using data records (i.e. collected examples).)
a respective example of the example set including an input value lying in an input space; ([0022]: “For each data record, there may be a set of output parameter values that corresponds to a particular set of input variable values.” Each data record (i.e. example of the example set) includes a set of input variable values (i.e. an input value in input space).)
collecting a first example set including a plurality of examples and [a second example set including a plurality of examples]; (0009: “The method may include obtaining data records relating to a plurality of input variables and one or more output parameters and selecting a plurality of input parameters from the plurality of input variables.” Data records (i.e. a plurality of examples) are obtained (i.e. collected).)
determining a first quality rating representing coverage of the input space by the examples of the first example set based on a distribution of input values in the input space, distributing representatives within the input space, assigning a number of examples to the representatives, and determining the quality rating based on a number of examples assigned to a respective representative; (0030-0031: “After input parameters are selected in step 202, processor 102 may evaluate the coverage of the data records in the modeling space… at step 203, processor 102 may determine data densities for the different regions in the modeling space and determine a distribution of the data density.” 0044: “FIG. 3 illustrates a flowchart of an exemplary hyper-quadrant density inspection process 300 for evaluating data coverage, consistent with certain disclosed embodiments. At step 301, the modeling space of interest may be divided into a plurality of hyper-quadrants… The span of each dimension may correspond the range of each input parameter.” 0047: “At step 302, processor 102 may calculate data density in each hyper-quadrant. In one embodiment, processor 102 may count the data records or vectors in each hyper-quadrant.” The coverage (i.e. distribution) of the data records (i.e. examples/input values of the first example set) in the modeling space (i.e. input space) is evaluated by the process illustrated in figure 3 (see below), which includes step 304 ‘determine statistical difference in data density distribution’ (i.e. determine a first quality rating). A hyper-quadrant in modeling space corresponds to a representative distributed in input space. Data records (i.e. examples) located within a hyper-quadrant are assigned to that hyper-quadrant (i.e. representative). The data records are evaluated for coverage by counting the number of data records in each hyper-quadrant (i.e. number of examples assigned to each representative).)
PNG
media_image1.png
660
586
media_image1.png
Greyscale
Grichnik does not appear to explicitly disclose
collecting […] a second example set including a plurality of examples;
determining a second quality rating representing coverage of the input space by the examples of the second example set based on the distribution of input values in the input space, distributing representatives within the input space, assigning a number of examples to the representatives, and determining the quality rating based on a number of examples assigned to a respective representative; and
comparing the first quality rating and the second quality rating with one another, determining a comparative variable corresponding to an amount of a difference between the number of examples of the first example set assigned to the representative and the number of examples of the second example set assigned to the representative, and
However, Dredze teaches collecting a first example set including a plurality of examples and a second example set including a plurality of examples; (Pg. 588, section 4: “Since the
A
-distance processes a stream of real numbers, we need to represent an example using a real number, such as the classification margin for that example. The first
n
of these numbers in the stream are a sample from
P
, and the most recent
n
are a sample from
P
’
.” The first
n
numbers in the data stream, defined by distribution
P
, are a first example set, and the most recent
n
numbers in the data stream, defined by distribution
P
’
, are a second example set.)
determining a second quality rating representing coverage of the input space by the examples of the second example set based on the distribution of input values in the input space, distributing representatives within the input space, assigning a number of examples to the representatives, and determining the quality rating based on a number of examples assigned to a respective representative; and (Examiner notes that this limitation is identical to the previous limitation taught by Grichnik, except that it determines a second quality rating based on the examples of the second example set. Pg. 588, section 4: “Let
A
be a set of real intervals and let
A
∈
A
be one such interval. For that interval,
P
(
A
)
is the probability that a value drawn from some unknown distribution falls in
A
… To compute
P
and
P
’
, one needs to specify
A
and
n
…” Distribution
P
is determined, representing the coverage of the input space (i.e. a first quality rating) by the first
n
numbers in the data stream (i.e. first example set), and distribution
P
'
is determined, representing coverage of the input space (i.e. a second quality rating) by the most recent
n
numbers in the data stream (i.e. second example set).)
comparing the first quality rating and the second quality rating with one another, determining a comparative variable corresponding to an amount of a difference between the number of examples of the first example set assigned to the representative and the number of examples of the second example set assigned to the representative, and (Pg. 588, section 4: “The
A
-distance between
P
and
P
’
, i.e. the difference between two distributions over the intervals, is defined as follows:
d
A
P
,
P
’
=
2
s
u
p
A
∈
A
|
P
A
–
P
’
(
A
)
|
. Two distributions are said to be different when, for a user-specified threshold
ϵ
,
d
A
(
P
,
P
'
)
>
ϵ
.” A difference is found between
P
and
P
’
(i.e. the first and second quality ratings are compared). In combination with Grichnik’s measure of data coverage, which is based on the number of examples assigned to each hyper-quadrant representative, this difference amounts to a comparative variable corresponding to the difference between the number of examples in each example set assigned to each representative.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Grichnik and Dredze. Grichnik teaches evaluating and improving the quality of data for a machine learning model by measuring data coverage and density in input space. Dredze teaches detecting domain shifts in data for a machine learning model by comparing the distributions of two windows of data. One of ordinary skill would have motivation to combine Grichnik and Dredze because Grichnik provides a method to measure the distribution of input data, and Dredze compares two distributions of input data to detect a domain shift, which is necessary for maintaining an accurate model: “[t]his problem of domain shift is a pervasive problem… that negatively impact[s] classification accuracy” (Dredze, pg. 585, section 1). Dredze presents “methods for automatically detecting such domain shifts from a stream of (unlabeled) examples that require limited computation and memory by virtue of operating on fixed-size windows of data… and are shown to be sensitive to shifts while maintaining a low rate of false positives” (Dredze, pg. 594, section 10).
Grichnik and Dredze do not appear to explicitly disclose
using sensors of an object recognition system to collect the examples;
using the object recognition system in an automated operation of at least one of a vehicle, a track-bound vehicle, a motor vehicle, an aircraft, a watercraft or a spacecraft.
However, Khodabandeh teaches using sensors of an object recognition system to collect the examples; (Pg. 1-2, section 1: “Object detection lies at the core of computer vision and finds application in surveillance, medical imaging, self-driving cars, face analysis, and industrial manufacturing…” Pg. 6, section 4: “Following [2] we evaluate performance on multi- and single-label object detection tasks using three different datasets… Cityscapes [4] is a dataset of real urban scenes containing 3,475 images captured by a dash-cam, 2,975 images are used for training and the remaining 500 for validation.” The object detection (i.e. recognition) system is trained using data collected by dash-cams (i.e. sensors).)
using the object recognition system in an automated operation of at least one of a vehicle, a track-bound vehicle, a motor vehicle, an aircraft, a watercraft or a spacecraft. (See the portions of sections 1 and 4 cited above. The object detection (i.e. recognition) system is used in self-driving cars (i.e. automated operation of a vehicle).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Grichnik, Dredze, and Khodabandeh. Grichnik teaches evaluating and improving the quality of data for a machine learning model by measuring data coverage and density in input space. Dredze teaches detecting domain shifts in data for a machine learning model by comparing the distributions of two windows of data. Khodabandeh teaches adapting to domain shifts in data for an object detection model for self-driving cars. One of ordinary skill would have motivation to combine Grichnik, Dredze, and Khodabandeh because “Domain shift is unavoidable in real-world applications of object detection. For example, in self-driving cars, the target domain consists of unconstrained road environments which cannot all possibly be observed in training data… In this paper, we address the domain adaptation problem… We evaluate the accuracy of our approach in various source/target domain pairs and demonstrate that the model significantly improves the state-of-the-art on multiple domain adaptation scenarios on the SIM10K, Cityscapes and KITTI datasets” (Khodabandeh, pg. 1, abstract).
Regarding Claim 21, Grichnik, Dredze, and Khodabandeh teach The method according to claim 20, as shown above.
Grichnik also teaches which further comprises:
forming a third example set from the first and second example sets; and (Examiner notes that according to specification paragraph 0022 of the instant application, the third example set can be comprised of the first example set and additional examples collected to fill the gaps in the first example set. 0032: “If a data coverage condition is detected (step 204: Yes), data coverage may be recommended before a model is constructed based on the data records. At step 205, modeling system 100 may receive a user choice on how to modify the data coverage… include[ing] having the user provide additional data records for sparse regions in the modeling space… [or] authorizing processor 102 to create additional data records for sparse regions in the modeling space…” The data is modified (i.e. a third example set is formed) by combining the current data records (i.e. the first example set) and newly provided/created data records (i.e. the second example set).)
determining a third quality rating representing a coverage of the input space by examples of the third example set based on the distribution of the input values in the input space; and (0037: “Once step 206, 207, 208 or any combination of them are performed for modifying the data coverage of the modeling space, process 200 may return to step 203 and re-evaluate the data coverage.” 0044: “FIG. 3 illustrates a flowchart of an exemplary hyper-quadrant density inspection process 300 for evaluating data coverage, consistent with certain disclosed embodiments.” The modified data (i.e. the third example set) is re-evaluated for coverage (i.e. a third quality rating is determined). In figure 3, step 304 ‘determine statistical difference in data density distribution’ corresponds to determining the quality rating.)
comparing the first quality rating, the second quality rating and the third quality rating. (0050: “At step 305, the statistical difference may be compared to a threshold… If the statistical difference exceeds the threshold (step 305: Yes), processor 102 may identify a data coverage condition (step 306).” The statistical difference corresponding to the modified data (i.e. the third quality rating) is compared to the threshold that was exceeded by the statistical difference corresponding to the original data (i.e. the first quality rating or the second quality rating) to determine if coverage has improved.)
Regarding Claim 22, Grichnik, Dredze, and Khodabandeh teach The method according to claim 21, as shown above.
Grichnik also teaches which further comprises:
carrying out the determination of any one of the first quality rating, the second quality rating or the third quality rating by: (0044: “FIG. 3 illustrates a flowchart of an exemplary hyper-quadrant density inspection process 300 for evaluating data coverage, consistent with certain disclosed embodiments.” Evaluating data coverage corresponds to determining the quality rating.)
distributing representatives in the input space, and (0044: “At step 301, the modeling space of interest may be divided into a plurality of hyper-quadrants… The span of each dimension may correspond the range of each input parameter.” A hyper-quadrant in modeling space corresponds to a representative in input space.)
assigning a plurality of examples of the example set to a respective representative; (0047: “At step 302, processor 102 may calculate data density in each hyper-quadrant. In one embodiment, processor 102 may count the data records or vectors in each hyper-quadrant.” Data records (i.e. examples) located within a hyper-quadrant are assigned to that hyper-quadrant (i.e. representative).)
locating the examples assigned to the representative in a surrounding area of the input space surrounding the representative; and (0047: “At step 302, processor 102 may calculate data density in each hyper-quadrant. In one embodiment, processor 102 may count the data records or vectors in each hyper-quadrant.” Data records (i.e. examples) located within a hyper-quadrant (i.e. located in a representative’s surrounding area in input space) are assigned to that hyper-quadrant (i.e. representative).)
at least one of: determining, as the first quality rating, a local quality rating for the surrounding area based on the examples of the first example set assigned to the representative, or determining, as the second quality rating, a local quality rating for the surrounding area based on the examples of the second example set assigned to the representative. (0047: “At step 302, processor 102 may calculate data density in each hyper-quadrant. In one embodiment, processor 102 may count the data records or vectors in each hyper-quadrant.” The data records (i.e. examples of the first example set) are evaluated for coverage (i.e. the first quality rating is determined) by calculating a data density (i.e. local quality rating) for each hyper-quadrant (i.e. representative) based on the data records (i.e. examples of the first example set) located within the hyper-quadrant (i.e. in the representative’s surrounding area and thus assigned to the representative).)
Regarding Claim 23, Grichnik, Dredze, and Khodabandeh teach The method according to claim 22, as shown above.
Grichnik also teaches which further comprises providing the third quality rating as a local quality rating for the surrounding area and determining the third quality rating based on the examples of the third example set assigned to the representative. (0037: “Once step 206, 207, 208 or any combination of them are performed for modifying the data coverage of the modeling space, process 200 may return to step 203 and re-evaluate the data coverage.” The modified data records (i.e. examples of the third example set) are re-evaluated for coverage by the same method as is described above in regard to claim 22 (i.e. the third quality rating is determined) by calculating a data density (i.e. local quality rating) for each hyper-quadrant (i.e. representative) based on the modified data records (i.e. examples of the third example set) located within the hyper-quadrant (i.e. in the representative’s surrounding area and thus assigned to the representative).)
Regarding Claim 24, Grichnik, Dredze, and Khodabandeh teach The method according to claim 22, as shown above.
Grichnik also teaches which further comprises providing the quality rating with at least one of statistical measures determined based on the set of examples or examples assigned to a respective representative. (0044: “FIG. 3 illustrates a flowchart of an exemplary hyper-quadrant density inspection process 300 for evaluating data coverage, consistent with certain disclosed embodiments.” As can be seen in figure 3, step 304, data coverage is evaluated (i.e. the quality rating is provided) by determining statistical difference (i.e. statistical measure) in data distribution (i.e. based on the set of examples).)
Regarding Claim 25, Grichnik, Dredze, and Khodabandeh teach The method according to claim 24, as shown above.
Grichnik also teaches which further comprises determining as a statistical mean at least one of a statistical measure, a mean, a median, a minimum or quantiles of the plurality of examples assigned to a representative. (0049: “Alternatively, processor 102 may determine the statistical difference as the standard deviation of the data density distribution, and normalize it with the mean of the distribution.” Standard deviation and mean of the data density distribution is a statistical measure of the data records (i.e. examples) assigned to each hyper-quadrant (i.e. representative).)
Regarding Claim 33, Grichnik, Dredze, and Khodabandeh teach The method according to claim 20, as shown above.
Khodabandeh also teaches which further comprises providing the example-based system for use in a safety-related function and providing the safety-related function with object recognition based on an image recognition, in which the object is recognized by using the example-based system. (Pg. 1-2, section 1: “Object detection lies at the core of computer vision and finds application in surveillance, medical imaging, self-driving cars, face analysis, and industrial manufacturing… We provide the first (to the best of our knowledge) formulation of domain adaptation in object detection as robust learning.” Pg. 6, section 4: “Following [2] we evaluate performance on multi- and single-label object detection tasks using three different datasets… SIM 10K [28] is a simulated dataset containing 10,000 images synthesized by the Grand Theft Auto game engine. In this dataset, which simulates car driving scenes captured by a dash-cam, there are 58,701 annotated car instances with bounding boxes.” The framework provides an object detection (i.e. image recognition) model for self-driving cars (i.e. a safety-related function).)
Regarding Claim 35, Grichnik, Dredze, and Khodabandeh teach The method according to claim 20, as shown above.
Khodabandeh also teaches which further comprises providing the example-based system for use in a safety-related function and using the safety-related function to represent a classification based on sensor data of organisms or a safe control of industrial plants, including a classification of chemical substances, a classification of signatures of vehicles or a control in a field of industrial automation. (Pg. 1-2, section 1: “Object detection lies at the core of computer vision and finds application in surveillance, medical imaging, self-driving cars, face analysis, and industrial manufacturing…” Pg. 6, section 4: “Following [2] we evaluate performance on multi- and single-label object detection tasks using three different datasets… Cityscapes [4] is a dataset of real urban scenes containing 3,475 images captured by a dash-cam… There are 8 different object categories in this dataset including person, rider, car, truck, bus, train, motorcycle and bicycle.” The framework provides an object detection model for self-driving cars (i.e. a safety-related function) which performs classification of objects including cars, trucks, and buses (i.e. classification of signatures of vehicles) based on images captured by dash-cam (i.e. sensor data).)
Regarding Claim 36, Grichnik, Dredze, and Khodabandeh teach The method according to claim 20, as shown above.
Grichnik also teaches which further comprises providing the example-based system with: a system with supervised learning, or an artificial neural network with one or more layers of neurons not being input neurons or output neurons and being trained with back-propagation, or a convolutional neural network, or a single-shot multibox detector network. (0009: “The method may also include generating a computational model indicative of interrelationships between the plurality of input parameters and the one or more output parameters based on the data records.” 0038-0039: “Any appropriate type of neural network may be used to build the computational model. The type of neural network models used may include back propagation, feed forward models, cascaded neural networks, and/or hybrid neural networks, etc… The neural network computational model may be trained by using the data records.” A neural network trained with backpropagation using data records including input and output parameters is a system with supervised learning.)
Regarding Claim 37, Grichnik, Dredze, and Khodabandeh teach the method according to claim 20, as shown above.
Grichnik also teaches A computer program stored on a non-transitory computer-readable storage medium, the computer program comprising commands which, when the program is executed by a computing unit, cause the computing unit to perform the method (0018-0019: “Modeling system 100 may include a processor 102, a memory module 104, a database 106, an I/O interface 108, and a network interface 110… Processor 102 may execute sequences of computer program instructions to perform various processes that will be explained later.” Claim 18 also explicitly refers to “A computer readable medium having stored thereon instructions for modifying data coverage in a modeling system…”)
Regarding Claim 38, Grichnik, Dredze, and Khodabandeh teach the method according to claim 20, as shown above.
Grichnik also teaches A non-transitory computer-readable storage medium, comprising commands which, when executed by a computing unit, cause the computing unit to perform the method (0018-0019: “Modeling system 100 may include a processor 102, a memory module 104, a database 106, an I/O interface 108, and a network interface 110… Processor 102 may execute sequences of computer program instructions to perform various processes that will be explained later.” Claim 18 also explicitly refers to “A computer readable medium having stored thereon instructions for modifying data coverage in a modeling system…”)
Claims 26-27 are rejected under 35 U.S.C. 103 as being unpatentable over Grichnik in view of Dredze and Khodabandeh, and further in view of
Boedihardjo et al. (hereinafter Boedihardjo), U.S. Patent Application Publication US-20200380307-A1 (filed 05-29-2020).
Regarding Claim 26, Grichnik, Dredze, and Khodabandeh teach The method according to claim 22, as shown above.
Grichnik also teaches which further comprises determining [adjacent] surrounding areas in the input space having a respective representative assigned a plurality of examples fulfilling a predetermined quality criterion of the quality rating. (0053: “…processor 102 may mark sparse hyper-quadrants with data densities below a density threshold (step 309).” Hyper-quadrants (i.e. surrounding areas in the input space) with data densities below a threshold (i.e. with representative-assigned examples fulfilling a predetermined quality criterion) are marked (i.e. determined).)
Grichnik, Dredze, and Khodabandeh do not appear to explicitly disclose determining adjacent surrounding areas in the input space
However, Boedihardjo teaches determining adjacent surrounding areas in the input space (0049: “In some instances, the intractable input space may be limited to a continuous region defined by the data set on which the prediction model is trained…” The sparse/low-density areas identified by Grichnik represent intractable input space, and Boedihardjo teaches that an intractable input space is continuous (i.e. the areas are adjacent). Figure 3 additionally shows the continuous intractable input space
α
being comprised of adjacent surrounding areas.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Grichnik, Dredze, Khodabandeh, and Boedihardjo. Grichnik teaches evaluating and improving the quality of data for a machine learning model by measuring data coverage and density in input space. Dredze teaches detecting domain shifts in data for a machine learning model by comparing the distributions of two windows of data. Khodabandeh teaches adapting to domain shifts in data for an object detection model for self-driving cars. Boedihardjo teaches improving the performance of a machine learning model on an intractable input space. One of ordinary skill would have motivation to combine Grichnik, Dredze, Khodabandeh, and Boedihardjo because Boedihardjo provides “technical solutions to technical problems associated with the difficulties in locating where innate ‘biases’ in ML models and/or AI systems might lie, e.g., because of poorly trained models, intentionally manipulated training data sets, and/or the like” (Boedihardjo, 0066).
Regarding Claim 27, Grichnik, Dredze, Khodabandeh, and Boedihardjo teach The method according to claim 26, as shown above.
Boedihardjo also teaches which further comprises determining a connection area formed of adjacent surrounding areas within the input space and (0049: “In some instances, the intractable input space may be limited to a continuous region defined by the data set on which the prediction model is trained…” The intractable input space is a continuous region (i.e. a connection area formed of adjacent surrounding areas). Figure 3 additionally shows the continuous intractable input space
α
(i.e. connection area) being comprised of adjacent surrounding areas.)
Grichnik teaches assigning each representative of the surrounding areas a number of examples fulfilling a predetermined quality criterion of the quality rating. (0060: “For example, additional data records may be created in the sparse regions of the modeling space, using a symmetric random scatter process. This additional feature may further enhance the accuracy of the modeling process.” Additional data records (i.e. examples) are created in the sparse regions (i.e. assigned to the representatives of the surrounding areas) in order to enhance accuracy (i.e. fulfill a predetermined quality criterion).)
Claims 28-31 are rejected under 35 U.S.C. 103 as being unpatentable over Grichnik in view of Dredze and Khodabandeh, and further in view of
Waschulzik, (hereinafter Waschulzik-1999) “Quality-assured efficient development of feedforward artificial neural networks with supervised learning (QUEEN)” (published 05-07-1999).
Regarding Claim 28, Grichnik, Dredze, and Khodabandeh teach The method according to claim 22, as shown above.
Grichnik also teaches which further comprises:
providing the respective example with an output value located in an output space; (0022: “At step 201, modeling system 100 may obtain data records relating to input variables and output parameters… For each data record, there may be a set of output parameter values that corresponds to a particular set of input variable values.” Each data record (i.e. example) has a set of output parameter values (i.e. an output value located in output space).)
Grichnik, Dredze, and Khodabandeh do not appear to explicitly disclose
determining for the respective surrounding area a local complexity rating representing a complexity of a task of the example-based system, the complexity being defined by the examples of the surrounding area; and
determining the local complexity rating by a relative position of the examples of the surrounding area with respect to one another in the input space and the output space.
However, Waschulzik-1999 teaches determining for the respective surrounding area a local complexity rating representing a complexity of a task of the example-based system, the complexity being defined by the examples of the surrounding area; and (Pg. 32, section 4.2: “For the evaluation and quality control of the results of the development process before the creation of the neural network, based on the consideration of the ‘Similarity of the distances of the examples in the input space to the distances in the output space’, quality indicators are defined in the following sections… Local quality indicators are used to determine particularly simple or complex areas of the task.” Pg. 38, section 4.6: “If only one indicator for the quality of coding is to be used in the development of the representation or coding, the integrated quality indicator
Q
I
²
(QUEEN's integrated quality indicator) can be used.” Pg. 43, section 4.7: “For estimating the complexity of a mapping and for building a model… local
Q
I
²
are computed over subsets of
P
[the example set].” For a subset of the example set
P
(i.e. a surrounding area), a local quality indicator
Q
I
²
(i.e. local complexity rating) is computed (i.e. determined) representing the complexity of a mapping for model building (i.e. a task of the example-based system), which is defined by the similarity of distances of examples in the subset (i.e. surrounding area).)
determining the local complexity rating by a relative position of the examples of the surrounding area with respect to one another in the input space and the output space. (Pg. 32, section 4.2: “For the evaluation and quality control of the results of the development process before the creation of the neural network, based on the consideration of the ‘Similarity of the distances of the examples in the input space to the distances in the output space’, quality indicators are defined in the following sections… Local quality indicators are used to determine particularly simple or complex areas of the task.” Pg. 38, section 4.6: “If only one indicator for the quality of coding is to be used in the development of the representation or coding, the integrated quality indicator
Q
I
²
(QUEEN's integrated quality indicator) can be used.” Pg. 43, section 4.7: “For estimating the complexity of a mapping and for building a model… local
Q
I
²
are computed over subsets of
P
[the example set].” The local quality indicator
Q
I
²
(i.e. local complexity rating) is determined by the similarity of the distances (i.e. relative position) of the local examples (i.e. examples of the surrounding area) in the input space to the distances in the output space.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Grichnik, Dredze, Khodabandeh, and Waschulzik-1999. Grichnik teaches evaluating and improving the quality of data for a machine learning model by measuring data coverage and density in input space. Dredze teaches detecting domain shifts in data for a machine learning model by comparing the distributions of two windows of data. Khodabandeh teaches adapting to domain shifts in data for an object detection model for self-driving cars. Waschulzik-1999 teaches evaluating the complexity of data for a machine learning model using quality indicators which measure relative distances between examples in input and output space. One of ordinary skill would have motivation to combine Grichnik, Dredze, Khodabandeh, and Waschulzik-1999 because Waschulzik’s quality indicators “can be used in the treatment of problematic situations to detect errors in the collection, recording, pre-processing or coding of the examples, but on the other hand, they can also be used to break down the task into subtasks that are easier to process” (Waschulzik-1999, pg. 32, section 4.2).
Regarding Claim 29, Grichnik, Dredze, Khodabandeh, and Waschulzik-1999 teach The method according to claim 28, as shown above.
Grichnik and Dredze also teach which further comprises: determining a first local [complexity] rating for the examples of the first example set, determining a second local [complexity] rating for the examples of the second example set and determining a third local [complexity] rating for the examples of the third example set; and comparing the third local [complexity] rating with at least one of the first or second local [complexity] rating. (Grichnik and Dredze teach determining a first, second, and third local quality rating and comparing them, as shown above in regard to claims 20-22.)
Grichnik and Dredze do not appear to explicitly disclose that the rating is a complexity rating.
However, Waschulzik-1999 teaches a local complexity rating (Examiner notes that determining a local complexity rating for examples of an example set, as recited in claim 29, is the same as determining a local complexity rating for examples of the surrounding areas, as recited in claim 28. Waschulzik-1999 teaches determining a local complexity rating, as shown above in regard to claim 28.)
Regarding Claim 30, Grichnik, Dredze, Khodabandeh, and Waschulzik-1999 teach The method according to claim 28, as shown above.
Waschulzik-1999 also teaches which further comprises a complexity distribution is determined by using a histogram representation of the complexity rating. (Pg. 43-44, section 4.7 & 4.7.1: “The evaluation of the local
Q
I
²
is based on the matrix of the local
Q
I
²
, which is denoted by
M
L
Q
I
²
(
P
)
… In addition to looking at the individual values of
M
L
Q
I
²
, the histogram
H
L
Q
I
²
(
P
)
calculated from
M
L
Q
I
²
(
P
)
and
B
L
Q
I
²
(
P
)
also provides further information about the mapping given by examples P…” A complexity distribution is determined using a histogram representation of the matrix of local quality indicator
Q
I
²
(i.e. complexity rating).)
Regarding Claim 31, Grichnik, Dredze, Khodabandeh, and Waschulzik-1999 teach The method according to claim 30, as shown above.
Waschulzik-1999 also teaches which further comprises determining a complexity distribution over k-nearest neighbors of an example in the input space. (Pg. 42, section 4.6.2: “In order to automatically adjust the size of the local environments to the density of examples in a subspace, the size of the local environment determined by the number of
k
nearest examples.” The histogram represents a distribution of local quality indicators (i.e. complexity ratings), and locality is defined by the
k
nearest examples (i.e. k-nearest neighbors of an example in input space).)
Claim 32 is rejected under 35 U.S.C. 103 as being unpatentable over Grichnik in view of Dredze, Khodabandeh, and Waschulzik-1999, and further in view of
Waschulzik et al. (hereinafter Waschulzik-2000), “Quality Assured Efficient Engineering of Feedforward Neural Networks with Supervised Learning (QUEEN) Evaluated with the ‘Pima Indians Diabetes Database’” (published 07-27-2000).
Regarding Claim 32, Grichnik, Dredze, Khodabandeh, and Waschulzik-1999 teach The method according to claim 28, as shown above.
Waschulzik-1999 also teaches which further comprises:
providing the complexity rating as an integrated quality indicator
Q
I
²
; (Pg. 38, section 4.6: “If only one indicator for the quality of coding is to be used in the development of the representation or coding, the integrated quality indicator
Q
I
²
(QUEEN's integrated quality indicator) can be used.”)
Grichnik, Dredze, Khodabandeh, and Waschulzik-1999 do not appear to clearly disclose the remaining limitations of claim 32.
However, Waschulzik-2000 teaches
determining the integrated quality indicator based on a definition as follows:
Q
I
2
P
=
1
|
P
2
|
∑
x
i
∈
P
2
(
d
N
R
E
x
i
-
d
N
R
A
x
i
)
2
wherein:
d
N
R
E
x
=
d
R
E
x
∑
y
∈
P
2
d
R
E
y
|
P
2
|
is a normalized distance of the represented inputs and
d
N
R
A
x
=
d
R
A
x
∑
y
∈
P
2
d
R
A
y
|
P
2
|
is a normalized distance of the represented outputs, (Pg. 98-99, section 2.2: “The result of the modelling is the QUEEN integrated quality indicator (
Q
I
Q
I
or
Q
I
2
) given in (1).
Q
I
2
E
1
=
1
|
E
1
2
|
∑
x
i
∈
E
1
2
(
d
N
C
I
x
i
-
d
N
C
O
x
i
)
2
(1)
with:
d
N
C
I
x
i
=
d
C
I
x
i
∑
y
∈
E
1
2
d
C
I
y
|
E
1
2
|
(2)
d
N
C
O
x
i
=
d
C
O
x
i
∑
y
∈
E
1
2
d
C
O
y
|
E
1
2
|
(3)
…”)
wherein
x
is a pair
(
x
1
,
x
2
,
)
formed two examples
x
1
and
x
2
, (Pg. 99, section 2.2: “
x
i
,
k
: first (
k
=
1
) or second (
k
=
2
) example in
x
i
”)
wherein
x
1
and
x
2
are examples from an example set
P
, (Pg. 99, section 2.2: “
E
1
: set of examples”)
wherein
P
=
{
p
1
,
p
1
,
.
.
.
,
p
|
P
|
}
is a set of elements of a multiset
B
A
G
P
, (Pg. 99, section 2.2: “
E
1
: set of examples”)
wherein
|
P
2
|
is a number of pairs of elements of the multiset
B
A
G
P
; (Pg. 99, section 2.2: “
E
1
2
: set of all possible pairs
x
i
of the examples from
E
1
”)
wherein
d
R
E
x
is a distance between two points in the input space, and (Pg. 99, section 2.2: “
d
C
I
x
i
: abbreviation for
d
c
o
d
(
c
o
d
i
n
p
u
t
x
i
,
1
,
c
o
d
(
i
n
p
u
t
x
i
,
2
)
)
…
d
c
o
d
: distance between two encodings (e.g. the euclidean distance).”)
wherein
d
R
A
x
is a distance between two points in the output space. (Pg. 99, section 2.2: “
d
C
O
x
i
: abbreviation for
d
c
o
d
(
c
o
d
o
u
t
p
u
t
x
i
,
1
,
c
o
d
(
o
u
t
p
u
t
x
i
,
2
)
)
…
d
c
o
d
: distance between two encodings (e.g. the euclidean distance).”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Grichnik, Dredze, Khodabandeh, Waschulzik-1999, and Waschulzik-2000. Grichnik teaches evaluating and improving the quality of data for a machine learning model by measuring data coverage and density in input space. Dredze teaches detecting domain shifts in data for a machine learning model by comparing the distributions of two windows of data. Khodabandeh teaches adapting to domain shifts in data for an object detection model for self-driving cars. Waschulzik-1999 teaches the QUEEN procedure for evaluating the complexity of data for a machine learning model using quality indicators which measure relative distances between examples in input and output space. Waschulzik-2000 teaches applying the QUEEN procedure to set up a neural network. One of ordinary skill would have motivation to combine Grichnik, Dredze, Khodabandeh, Waschulzik-1999, and Waschulzik-2000 because Waschulzik-2000 is an extension of Waschulzik-1999 which deals with the same field of endeavor and attempts to solve the same problem in the same manner.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN M ROHD whose telephone number is (571)272-6445. The examiner can normally be reached Mon-Thurs 8:00-6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.M.R./Examiner, Art Unit 2147
/ERIC NILSSON/Primary Examiner, Art Unit 2151
Prosecution Insights