DETAILED ACTION
This correspondence is responsive to the application filed on January 2, 2024. Claims 1-14 are pending in the case with claims 1 and 13-14 in independent form.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Summary of Detailed Action
Claims 1-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
Claim 2 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
Claims 1, 3-5, 7-8, and 10-14 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1-2, 9, and 11-14 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Tian et al.
Claims 3-5 are rejected under 35 U.S.C. 103 as being unpatentable over Tian et al., and further in view of Joshi et al.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Tian, and further in view of Pezzotti et al. EP (Kloninklikjke Phillips N. V., Pub. No. EP 3869401, cited on IDS filed March 19, 2024).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Tian and further in view of Alvarez et al. (Alvarez et al. cited on IDS filed March 19, 2024).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Tian and further in view of Pezzotti et al. WO (Kloninklikjke Phillips N.V., WO 2022/078922 A1 hereinafter Pezzotti WO, cited on IDS filed March 19, 2024.)
Claim 6 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and if the rejections as being indefinite are overcome.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Independent claims 1, 13 and 14 recite determining, from each training example of the training examples, “a training style” that characterizes the domain and/or distribution to which the training example belongs; determining, from the given input record of measurement data, a test style that characterizes the domain and/or distribution to which the given record of measurement data belongs; evaluating, based on “the training styles” and the test style, to which extent the test style is a member of the distribution of “the training styles.” There is insufficient antecedent basis for “the training styles” limitation in the claim. It is further unclear whether each training example has a different training style determined or if all of training examples have the same determined training styles or if some training examples have the same determined training styles while other training examples have different training styles. Applicant may cancel claims 1, 13 and 14 or amend claims 1, 13 and 14 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claims 2-12 depend from claim 1 and are rejected for the same reasons discussed above with respect to claim 1.
Claim 2 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 2 depends form claim 1 and recites the limitation for processing, by a trained feature extractor network, … and processing, by “the trained feature network.” There is insufficient antecedent basis for this limitation in the claim. The examiner notes that it appears that the claim should recite by the trained feature extractor network.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3-5, 7-8, and 10-14 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) recite(s) subject matter at a general, high-level of a method for detecting whether a given input record of measurement data is in a domain and/or distribution of training examples with which the machine learning model was trained, the method comprising the following steps: determining, from each training example of the training examples, a training style that characterizes the domain and/or distribution to which the training example belongs; determining, from the given input record of measurement data, a test style that characterizes the domain and/or distribution to which the given record of measurement data belongs; evaluating, based on the training styles and the test style, to which extent the test style is a member of the distribution of the training styles; and based at least in part on an outcome of the evaluation, determining whether the given record of measurement data is in the domain and/or distribution of the training examples, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the above recitations are also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I). This judicial exception is not integrated into a practical application and the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Claims 1-14 recite one of the four statutory categories of patent able subject matter and belong to the statutory class(es) of a process (method claims 1-12), a machine (system/apparatus claim 14), and an article of manufacture (non-transitory computer readable media claim 13).
Claim 1 recites a method, thus a process and one of the four statutory categories of patentable subject matter. However, claim 1 further recites for detecting whether a given input record of measurement data is in a domain and/or distribution of training examples with which the machine learning model was trained, the method comprising the following steps: determining, from each training example of the training examples, a training style that characterizes the domain and/or distribution to which the training example belongs; determining, from the given input record of measurement data, a test style that characterizes the domain and/or distribution to which the given record of measurement data belongs; evaluating, based on the training styles and the test style, to which extent the test style is a member of the distribution of the training styles; and based at least in part on an outcome of the evaluation, determining whether the given record of measurement data is in the domain and/or distribution of the training examples, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the above recitations are also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
that is inputted to a trained machine learning model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more, and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim 3, dependent on claim 1, recites only additional abstract ideas for wherein the evaluating includes: aggregating the training styles to form an aggregate; and determining to which extent the test style is a member of the distribution of the training styles of the training examples based on a distance between the test style and the aggregate, and/or based on a value of a rating function that is dependent on the distance, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the above recitations are also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
Claim 4, dependent on claim 3, recites only additional abstract ideas for wherein the aggregate includes a parametrized statistical distribution that is fitted to the training styles, and/or a centroid of a cluster of the training styles, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
Claim 5, dependent on claim 3, recites only additional abstract ideas for wherein a Mahalanobis distance is chosen as a measure for the distance, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
Claim 7, dependent on claim 1, recites additional abstract ideas for further comprising, in response to determining that the given record of measurement data is not in the domain and/or distribution of the training examples: repeating the method with the same given input record of measurement data, but with a second dataset of training examples; and in response to determining that the given input record of measurement data is in the domain and/or distribution of training examples in the second dataset, determining that the second machine learning model is more appropriate for processing the given input record of measurement data than the previous machine learning model, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the above recitations are also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
a second machine learning model that has been trained on (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 8, dependent on claim 1, recites additional abstract ideas for further comprising, in response to determining that the given record of measurement data is not in the domain and/or distribution of the training examples, repeating the method but with the new record of measurement data; and in response to determining that the new record of measurement data is in the domain and/or distribution of the training examples, determining that the new record of measurement data is more credible than the given record of measurement data, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the above recitations are also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
obtaining a new record of measurement data from a sensor that is different from a sensor with which the given input record of measurement data was acquired; (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
with the same machine learning model, (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 10, dependent on claim 1, recites additional abstract ideas for further comprising, in response to determining that the given record of measurement data is not in the domain and/or distribution of the training examples: applying, to the given record of measurement data, a candidate remedial procedure for a problem and/or deficiency that may affect the given record of measurement data, thereby creating a modified record of measurement data; repeating the method but with the modified record of measurement data; and in response to determining that the modified record of measurement data is in the domain and/or distribution of the training examples, determining that the given input record of measurement data is affected with the problem and/or deficiency remedied by the candidate remedial procedure, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the above recitations are also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
with the same machine learning model, (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 11, dependent on claim 1, recites additional abstract ideas for wherein input records of measurement data are chosen as the given input record, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
that have been captured by at least one sensor carried on board a vehicle or robot (This additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 12, dependent on claim 11, does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
wherein: the training styles, and/or an aggregate of the training styles, are determined on an external computing system outside the vehicle or robot (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)) and transmitted to the vehicle or robot; (An additional element of extra-solution activity that courts have identified is well understood, routine and conventional activity for receiving or transmitting data over a network, e.g., using the internet to gather data. See also, MPEP 2106.05(d)(II), MPEP 2106.05(g), 2019 Guidance, 84 FR 50 at 55, 2019 Guidance, 84 FR 50, footnote 31.).
and the remaining steps of the method are performed on board the vehicle or robot. (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 13 recites a non-transitory machine-readable data carrier, thus an article of manufacture and one of the four statutory categories of patentable subject matter. However, claim 13 further recites for detecting whether a given input record of measurement data is in a domain and/or distribution of training examples with which the machine learning model was trained, determining, from each training example of the training examples, a training style that characterizes the domain and/or distribution to which the training example belongs; determining, from the given input record of measurement data, a test style that characterizes the domain and/or distribution to which the given record of measurement data belongs; evaluating, based on the training styles and the test style, to which extent the test style is a member of the distribution of the training styles; and based at least in part on an outcome of the evaluation, determining whether the given record of measurement data is in the domain and/or distribution of the training examples, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the above recitations are also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
A non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
that is inputted to a trained machine learning model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
the instructions, when executed by one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more, and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim 14 recites one or more computers, thus a machine and one of the four statutory categories of patentable subject matter. However, claim 14 further recites for detecting whether a given input record of measurement data is in a domain and/or distribution of training examples with which the machine learning model was trained, determining, from each training example of the training examples, a training style that characterizes the domain and/or distribution to which the training example belongs; determining, from the given input record of measurement data, a test style that characterizes the domain and/or distribution to which the given record of measurement data belongs; evaluating, based on the training styles and the test style, to which extent the test style is a member of the distribution of the training styles; and based at least in part on an outcome of the evaluation, determining whether the given record of measurement data is in the domain and/or distribution of the training examples, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper. MPEP 210604(a)(2)(III). Additionally, the above recitations are also mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations. MPEP 210604(a)(2)(I).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
One or more computers with a non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions for (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
that is inputted to a trained machine learning model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
the instructions, when executed by the one or more computers, causing the one or more computers to perform the following steps: (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more, and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-2, 9, and 11-14 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Tian et al. (Pub. No. US 2023/0107006 A1, filed September 12, 2022) hereinafter Tian.
Regarding claim 1, Tian teaches:
A method for detecting whether a given input record of measurement data that is inputted to a trained machine learning model is in a domain and/or distribution of training examples with which the machine learning model was trained, the method comprising the following steps: (i.e., This disclosure relates to disentangled out-of-distribution (OOD) calibration and data detection. In a first embodiment, a method (A method) includes providing, using at least one processing device of an electronic device, input data to a machine learning model. Tian, para 4-5. In this example, the scoring function 216 can represent a combination of scores generated by a covariate shift scoring function 218 and a concept shift scoring function 220. A covariate shift generally refers to a change in the style of input data 202 being received (compared to the training data used to train the machine learning model). A concept shift generally refers to a change in the semantics of input data 202 being received (compared to the training data used to train the machine learning model). Both types of shifts can be used to characterize the spectrum of OOD data (A method for detecting (scoring function) whether a given input record of measurement data (input data of sensor captured image, scene measurement data, para 39, 46, 48, 69, Figures 1-2, 5, 7) that is inputted to a trained machine learning model is in a domain and/or distribution of training examples with which the machine learning model was trained (whether the input data is in a same or a changed domain and/or distribution of training data examples used to train the trained ml model)). To represent these two distribution shifts, the two scoring functions 218 and 220 can be derived, such as based on Kullback-Leibler (KL) divergence, and applied to the input data 202 (A method for detecting (scoring function) whether a given input record of measurement data (input data of sensor captured image, scene measurement data, para 39, 46, 48, 69, Figures 1-2, 5, 7) that is inputted to a trained machine learning model is in a domain and/or distribution of training examples with which the machine learning model was trained (whether the input data is in a same or changed domain and/or distribution of training data examples used to train the trained ml model)). The scoring function 216 may combine the scores produced by the scoring functions 218-220 in order to generate a final score representing a measure of whether input data 202 is likely in-distribution or out-of-distribution. By breaking out multiple types of shifting that can lead to data being out-of-distribution, the architecture 200 here disentangles the different types of shifting, which can lead to more-accurate OOD detection. Tian, Figs 1-7, para 54, 53-56, 30, 66, 39, 46, 48, 69, 5.)
determining, from each training example of the training examples, a training style that characterizes the domain and/or distribution to which the training example belongs; (i.e., In this example, the scoring function 216 can represent a combination of scores generated by a covariate shift scoring function 218 and a concept shift scoring function 220. A covariate shift generally refers to a change in the style of input data 202 being received (compared to the training data used to train the machine learning model) (determining, from each training example of the training examples, a training style that characterizes the domain and/or distribution to which the training example belongs (covariate shift scoring function for comparing input data test style to a determined training style that characterizes the domain and/or distribution to which each training data example belongs, the determined training style determined from each training data example of the training data examples). A concept shift generally refers to a change in the semantics of input data 202 being received (compared to the training data used to train the machine learning model). Both types of shifts can be used to characterize the spectrum of OOD data (determining, from each training example of the training examples, a training style that characterizes the domain and/or distribution to which the training example belongs (covariate shift scoring function for comparing input data test style to a determined training style that characterizes the domain and/or distribution to which each training data example belongs, the determined training style determined from each training data example of the training data examples). To represent these two distribution shifts, the two scoring functions 218 and 220 can be derived, such as based on Kullback-Leibler (KL) divergence, and applied to the input data 202. The scoring function 216 may combine the scores produced by the scoring functions 218-220 in order to generate a final score representing a measure of whether input data 202 is likely in-distribution or out-of-distribution. By breaking out multiple types of shifting that can lead to data being out-of-distribution, the architecture 200 here disentangles the different types of shifting, which can lead to more-accurate OOD detection. Tian, Figs 1-7, para 54, 53-56, 30, 66, 39, 46, 48, 69)
determining, from the given input record of measurement data, a test style that characterizes the domain and/or distribution to which the given record of measurement data belongs; (i.e., [0054] In this example, the scoring function 216 can represent a combination of scores generated by a covariate shift scoring function 218 and a concept shift scoring function 220. A covariate shift generally refers to a change in the style of input data 202 being received (compared to the training data used to train the machine learning model) (determining, from the given input record of measurement data (input data of sensor captured image, scene measurement data, para 39, 46, 48, 69, Figures 1-2, 5, 7), a test style that characterizes the domain and/or distribution to which the given record of measurement data belongs (covariate shift scoring function for comparing a determined input data test style that characterizes the domain and/or distribution to which the given record of input data of measurement data belongs to the training style of the training data). A concept shift generally refers to a change in the semantics of input data 202 being received (compared to the training data used to train the machine learning model). Both types of shifts can be used to characterize the spectrum of OOD data (determining, from the given input record of measurement data (input data of sensor captured image, scene measurement data, para 39, 46, 48, 69, Figures 1-2, 5, 7), a test style that characterizes the domain and/or distribution to which the given record of measurement data belongs (covariate shift scoring function for comparing a determined input data test style that characterizes the domain and/or distribution to which the given record of input data of measurement data belongs to the training style of the training data). To represent these two distribution shifts, the two scoring functions 218 and 220 can be derived, such as based on Kullback-Leibler (KL) divergence, and applied to the input data 202. The scoring function 216 may combine the scores produced by the scoring functions 218-220 in order to generate a final score representing a measure of whether input data 202 is likely in-distribution or out-of-distribution. By breaking out multiple types of shifting that can lead to data being out-of-distribution, the architecture 200 here disentangles the different types of shifting, which can lead to more-accurate OOD detection. Tian, Figs 1-7, para 54, 53-56, 30, 66, 39, 46, 48, 69)
evaluating, based on the training styles and the test style, to which extent the test style is a member of the distribution of the training styles; and
(i.e., In this example, the scoring function 216 can represent a combination of scores generated by a covariate shift scoring function 218 and a concept shift scoring function 220. A covariate shift generally refers to a change in the style of input data 202 being received (compared to the training data used to train the machine learning model) (evaluating, based on the training styles and the test style, to which extent the test style is a member of the distribution of the training styles (covariate shift scoring function for evaluating (scoring), based on the training data training styles and the input data test style, to which extent (score) the input data test style is a member of the distribution of the training data training styles)). A concept shift generally refers to a change in the semantics of input data 202 being received (compared to the training data used to train the machine learning model). Both types of shifts can be used to characterize the spectrum of OOD data (covariate shift scoring function for evaluating (scoring), based on the training data training styles and the input data test style, to which extent (score) the input data test style is a member of the distribution of the training data training styles)). To represent these two distribution shifts, the two scoring functions 218 and 220 can be derived, such as based on Kullback-Leibler (KL) divergence, and applied to the input data 202. The scoring function 216 may combine the scores produced by the scoring functions 218-220 in order to generate a final score representing a measure of whether input data 202 is likely in-distribution or out-of-distribution. By breaking out multiple types of shifting that can lead to data being out-of-distribution, the architecture 200 here disentangles the different types of shifting, which can lead to more-accurate OOD detection. Tian, Figs 1-7, para 54, 53-56, 30, 66, 39, 46, 48, 69-71)
based at least in part on an outcome of the evaluation, determining whether the given record of measurement data is in the domain and/or distribution of the training example. (i.e., [0082] An OOD score associated with the input data is calculated at step 708. This may include, for example, the processor 120 of the electronic device 101 performing the scoring function 216 (based on the scoring functions 218-220) using the outputs of the geometric transformation 208 in order to identify an OOD score associated with the input data 202. The OOD score is compared to a threshold at step 710, and a determination is made whether the OOD score exceeds the threshold at step 712. This may include, for example, the processor 120 of the electronic device 101 comparing the OOD score to a value above which is indicative of OOD data (based at least in part on an outcome of the evaluation, determining whether the given record of measurement data is in the domain and/or distribution of the training example). If the threshold is exceeded, an OOD notification is generated at step 714. This may include, for example, the processor 120 of the electronic device 101 generating a notification for a user. If the threshold is not exceeded, an OOD condition is not detected (based at least in part on an outcome of the evaluation, determining whether the given record of measurement data is in the domain and/or distribution of the training example), and the one or more predictive probability distributions 212 produced by the machine learning model may be used in any suitable manner (such as when provided to a user or provided to another machine learning model or other logic for further processing). Tian, Figs 1-7, para 82, 54, 53-56, 30, 66, 39, 46, 48, 69-71.).
Regarding claim 2, which depends from claim 1 and recites:
wherein the determining of the training style, and the determining of the test style, include: processing, by a trained feature extractor network, the training example into a feature map for the training example, and processing, by the trained feature network, the input record of measurement data, into a feature map for the input record; and
(i.e., Tian teaches the method of claim 1 from which claim 2 depends, including the determining of the training style, and the determining of the test style. Tian teaches the method of claim 1 from which claim 2 depends, including the training example and the input record of measurement data. Tian teaches that, [0047] The input data 202 is provided to a feature extractor 204, which processes the input data 202 in order to identify a set of features (f) 206 associated with the input data 202. The features 206 generally represents aspects or characteristics of the input data 202 that are extracted from the input data 202 or generated using the input data 202. Typically, the feature extractor 204 is implemented using one or more convolution layers or other layers of a machine learning model, where the machine learning model learns during training which aspects or characteristics of the input data 202 are useful or relevant in performing a specific task (processing, by a trained feature extractor network (processing by trained feature extractor network 204, that learns and is trained during training, see also Figures 3-4, para 62, adjusted, trained feature extractor network 204), the training example into a feature map for the training example, and processing, by the trained feature network the input record of measurement data, into a feature map for the input record (process training example data by trained feature extractor implemented using one or more convolutional layers that produce feature map for training example, Figures 2-3, and process input record of measurement data, para 39, 46-48, 69, Figures 1-2, 5, by trained feature extractor implemented using one or more convolutional layers that produce feature map for input record of measurement data Figures 2-4 ). The feature extractor 204 can use any suitable technique to identify features 206 associated with input data 202. Various approaches for implementing feature extractors have been developed, and additional approaches are sure to be developed in the future. Tian, Figs 1-7, para 47, 48, 62, 65-66, 39, 46-48, 54, 69.).
determining, from the feature map for the training example, features of the training example that characterize a domain and/or distribution of the training example, and determining, from the feature map for the input record, features of the input record that characterize a domain and/or distribution of the input record.
(i.e., Tian teaches that, The outputs of the geometric transformation 208 (the logits 210) are converted into confidences (the probability distributions 212) and are used to represent uncertainties 404 associated with predictions made by the machine learning model. The confidences here are calibrated confidences due to the performance of the post-training calibration process 402. The outputs of the geometric transformation 208 (the logits 210) are also provided to the scoring function 216, which generates one or more scores indicative of whether the input data 202 is in-distribution or out-of-distribution. For example, the scores generated by the scoring function 216 can be produced based on scores produced using the covariate shift scoring function 218 and the concept shift scoring function 220 (which may actually form parts of the same scoring function 216). Each score can be compared to a threshold. If the threshold is exceeded, an OOD detection 406 can be produced indicating that out-of-distribution data has been received (determining, from the feature map for the training example (from the feature extractor convolutional layers feature map, para 47), features of the training example that characterize a domain and/or distribution of the training example (determining covariate shift by determining training data style features that characterize a domain or distribution of the training data examples, para 54), and determining, from the feature map for the input record (from the feature extractor convolutional layers feature map, para 47), features of the input record that characterize a domain and/or distribution of the input record (determining covariate shift by determining training data style features that characterize a domain or distribution of the training data examples and input record style features that characterize a domain or distribution of the input record, para 54)). The OOD detection 406 can be used in any suitable manner, such as to notify a user of the electronic device 101 and ask for input. Tian, Figures 1-7, para 66, 65, 54, 47-48, 62, 65-66, 39, 46-48, 69.).
Regarding claim 9, which depends from claim 1 and recites:
further comprising, in response to determining that the given record of measurement data is not in the domain and/or distribution of the training examples: suppressing transmission of an output that the machine learning model computes from the given record of measurement data to a downstream technical system; and/or actuating a downstream technical system that uses outputs of the machine learning model to move the technical system into an operational state where it can better tolerate noisy or incorrect outputs.
Tian teaches the method of claim 1 from which claim 9 depends, including in response to determining that the given record of measurement data is not in the domain and/or distribution. Tian, Figures 1-7, para 54, 53-56, 30, 66, 39, 46, 48, 69-71, 5, 82. Tian teaches that, Each score can be compared to a threshold. If the threshold is exceeded, an OOD detection 406 can be produced indicating that out-of-distribution data has been received. The OOD detection 406 can be used in any suitable manner, such as to notify a user of the electronic device 101 and ask for input (suppressing transmission of an output that the machine learning model computes from the given record of measurement data to a downstream technical system (suppress transmission of output that the machine learning model computes from the given record of measurement data to a downstream technical system to notify user and ask for input); and/or actuating a downstream technical system that uses outputs of the machine learning model to move the technical system into an operational state where it can better tolerate noisy or incorrect outputs). Tian, Figs 1-7, para 66, 82, 54, 54, 53-56, 30, 66, 39, 46, 48, 69-71, 5.
Regarding claim 11, which depends from claim 1 and recites:
wherein input records of measurement data that have been captured by at least one sensor carried on board a vehicle or robot are chosen as the given input record.
Tian teaches the method of claim 1 from which claim 11 depends, including the input records of measurement data. Tian teaches input records of measurement data that have been captured by at least one sensor carried on board a vehicle or robot are chosen as the given input record in that Tian discloses that, Out-of-distribution data can routinely cause a trained machine learning model to fail, such as when the trained machine learning model fails to produce reliable predictions. This can be problematic in safety-critical applications like self-driving vehicles (such as when models are used to detect people or objects in front of vehicles) and medical imaging analysis (such as when models are used to detect tumors in patients). Tian, Figs 1-2, 4-5, para 29, 39, 42, 71. The electronic device 101 further includes one or more sensors … For example, one or more sensors 180 may include one or more cameras or other imaging sensors, which may be used to capture images of scenes. Tian, Figs 1-2, 4-5, para 39, 38, 42, 44, 69, 71, 29. As shown in FIG. 5, the electronic device 101 here includes or has access to a machine learning model 502. The machine learning model 502 may have the architecture 200 shown in FIG. 2 and may be trained and calibrated based on the processes 300 and 400 shown in FIGS. 3 and 4. In this example, the machine learning model 502 has been trained to perform image classification, which means that the machine learning model 502 can receive input data in the form of images 504 and generate outputs 506 identifying the estimated contents of the images 504. The images 504 may, for example, represent images captured using one or more sensors 180 of the electronic device 101 (wherein input records of measurement data that have been captured by at least one sensor carried on board a vehicle (sensors 180 of electronic device vehicle head units, see Figures 1, 4-5, para 15, 32, 38-39, 42, 44, 46) or robot are chosen (chosen scenes Figure 5) as the given input record). Tian, Figs 1, 4-5, para 69, 70, 71, 39, 71, 29. As a particular example, when used in a self-driving vehicle, the outputs 506 may represent determined classifications of people or objects in front of or around a vehicle, and the outputs 506 may be used by one or more other machine learning models or other logic to control operation of the vehicle. Tian, Figs 1, 5, para 29, 71.
Regarding claim 12, which depends from claim 11 and recites:
wherein: the training styles, and/or an aggregate of the training styles, are determined on an external computing system outside the vehicle or robot and transmitted to the vehicle or robot; and the remaining steps of the method are performed on board the vehicle or robot.
Tian teaches the method of claim 11 from which claim 12 depends, including the input records of measurement data that have been captured by at least one sensor carried on board a vehicle or robot are chosen as the given input record. As similarly discussed above, Tian teaches that, Out-of-distribution data can routinely cause a trained machine learning model to fail, such as when the trained machine learning model fails to produce reliable predictions. This can be problematic in safety-critical applications like self-driving vehicles (such as when models are used to detect people or objects in front of vehicles) and medical imaging analysis (such as when models are used to detect tumors in patients). Tian, Figs 1-2, 4-5, para 29, 38-39, 71. The electronic device 101 further includes one or more sensors … For example, one or more sensors 180 may include one or more cameras or other imaging sensors, which may be used to capture images of scenes. Tian, Figs 1-2, 4-5, para 39, 69, 71, 29. As shown in FIG. 5, the electronic device 101 here includes or has access to a machine learning model 502. The machine learning model 502 may have the architecture 200 shown in FIG. 2 and may be trained and calibrated based on the processes 300 and 400 shown in FIGS. 3 and 4 (the training styles, and/or an aggregate of the training styles, are determined on an external computing system outside the vehicle or robot and transmitted to the vehicle or robot (ml model trained by external computing device, post training OOD (Figure 4 para 64-66) training styles accessed by and transmitted to the vehicle, Figures 1-2, 4-5, para 15, 32, 39, 46, 64-66); and the remaining steps of the method are performed on board the vehicle (remaining steps performed on vehicle, Figure 1-2, 4-5, para 69-71, 15, 32, 39, 46, 64-66) or robot). In this example, the machine learning model 502 has been trained to perform image classification, which means that the machine learning model 502 can receive input data in the form of images 504 and generate outputs 506 identifying the estimated contents of the images 504. The images 504 may, for example, represent images captured using one or more sensors 180 of the electronic device 101. Tian, Figs 1-2, 4-5, para 69, 70, 71, 39, 71, 29, 64-66. As a particular example, when used in a self-driving vehicle, the outputs 506 may represent determined classifications of people or objects in front of or around a vehicle, and the outputs 506 may be used by one or more other machine learning models or other logic to control operation of the vehicle. Tian, Figs 1-2, 4-5, para 29, 71.
Claim 13 recites a non-transitory machine-readable data carrier that parallels the method of claim 1. Therefore, the analysis discussed above with respect to claim 1 also applies to claim 13. Accordingly, claim 13 is rejected based on substantially the same rationale as set forth above with respect to claim 1. More specifically regarding A non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions, the instructions, when executed by one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps (i.e., Trian, Fig 1, para 7, 10, 32-43).
Claim 14 recites one or more computers that parallels the method of claim 1. Therefore, the analysis discussed above with respect to claim 1 also applies to claim 14. Accordingly, claim 14 is rejected based on substantially the same rationale as set forth above with respect to claim 1. More specifically regarding One or more computers with a non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions … the instructions, when executed by the one or more computers, causing the one or more computers to perform the following steps (i.e., Trian, Fig 1, para 6, 10, 32-43).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3-5 are rejected under 35 U.S.C. 103 as being unpatentable over Tian as applied to claim 1 above, and further in view of Joshi et al. (Pub. No. US 2021/0341911 A1, published November 4, 2021) hereinafter Joshi.
Regarding claim 3, which depends from claim 1 and recites:
wherein the evaluating includes: aggregating the training styles to form an aggregate; and determining to which extent the test style is a member of the distribution of the training styles of the training examples based on a distance between the test style and the aggregate, and/or based on a value of a rating function that is dependent on the distance.
Tian teaches the method of claim 1 from which claim 3 depends, including the evaluating, training styles and test style. Tian does not specifically disclose aggregating the training styles to form an aggregate; and determining to which extent the test style is a member of the distribution of the training styles of the training examples based on a distance between the test style and the aggregate, and/or based on a value of a rating function that is dependent on the distance.
However, Joshi teaches in the field related to identifying maintenance issues of vehicle power modules by using vehicle contextual information in an autoencoder. Joshi, para 1, 2-3. Joshi, which is analogous to the claimed invention because Joshi is directed to measurement data and detecting anomalies, teaches that, In a secondary training phase, the decoder is not used. The training datasets are input to the encoder, where the training datasets are associated with contextual features, such as elevation gain, weight-on-wheels, mileage, geographic region, climate, location or mode of travel (e.g., urban, suburban, rural, payload, etc. The contextual features can be represented by normalized values, so that they are input in a range from zero to one. For example, a climate feature on a sunny day may be zero, on a rainy data may be 0.5, on a snowy day may be 0.8 and in a blizzard may be 1.0. [0034] Aspects of the present disclosure utilize the encoded feature space with K-means clustering to form clusters of data defined by contextual features (aggregating (clustering) the training styles (training contexts styles features) to form an aggregate (an aggregate of clusters);). The clusters are indicative of the operating points of the PE devices under external operating conditions, such as, but not limited to average drain-source current in MOSFET devices and duration on (ON time). Prognostics of the PE modules under test can be determined by applying data from the test devices to a trained encoder and determining the minimum squared distance of the encoded features to the training clusters (and determining to which extent the test style is a member of the distribution of the training styles of the training examples based on a distance between the test style and the aggregate, and/or based on a value of a rating function that is dependent on the distance). This distance metric can be calculated in many ways, but a popular method is called the Mahalanobis distance. Joshi, para 33-34, 31-36, 71.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the method for detecting whether a given input record of measurement data that is inputted to a trained machine learning model is in a domain and/or distribution of training examples with which the machine learning model was trained, and evaluating, based on the training styles and the test style, to which extent the test style is a member of the distribution of the training styles of Tian using the features for aggregating the training styles to form an aggregate; and determining to which extent the test style is a member of the distribution of the training styles of the training examples based on a distance between the test style and the aggregate, and/or based on a value of a rating function that is dependent on the distance of Joshi, with a reasonable expectation of success, in order to provide for utilizing sensed data to detect anomalies and thus predicting the state or potential failure of the PE module and to provide further data analysis of the sensed data to determine which PE module may be in danger of imminent failure during operation and to provide methods for performing prognostics of PE devices installed in vehicles by utilizing autoencoders trained with datasets including contextual information. Joshi, para 2-3, 31-36, 71. This would have provided the advantages of providing further analysis of measurement data and contextual information to detect a sudden shift.
Regarding claim 4, which depends from claim 3 and recites:
wherein the aggregate includes a parametrized statistical distribution that is fitted to the training styles, and/or a centroid of a cluster of the training styles.
Tian in view of Joshi teaches the method of claim 3 from which claim 4 depends, including the evaluating, training styles and test style and aggregate. Tian does not specifically disclose wherein the aggregate includes a parametrized statistical distribution that is fitted to the training styles, and/or a centroid of a cluster of the training styles.
However, Joshi teaches in the field related to identifying maintenance issues of vehicle power modules by using vehicle contextual information in an autoencoder. Joshi, para 1, 2-3. Joshi, which is analogous to the claimed invention because Joshi is directed to measurement data and detecting anomalies, teaches that, The training datasets are input to the encoder, where the training datasets are associated with contextual features, such as elevation gain, weight-on-wheels, mileage, geographic region, climate, location or mode of travel (e.g., urban, suburban, rural, payload, etc. The contextual features can be represented by normalized values, so that they are input in a range from zero to one. For example, a climate feature on a sunny day may be zero, on a rainy data may be 0.5, on a snowy day may be 0.8 and in a blizzard may be 1.0. [0034] Aspects of the present disclosure utilize the encoded feature space with K-means clustering to form clusters of data defined by contextual features (aggregating (clustering) the training styles (training contexts styles features) to form an aggregate (an aggregate of clusters);). The clusters are indicative of the operating points of the PE devices under external operating conditions, such as, but not limited to average drain-source current in MOSFET devices and duration on (ON time). Prognostics of the PE modules under test can be determined by applying data from the test devices to a trained encoder and determining the minimum squared distance of the encoded features to the training clusters (and determining to which extent the test style is a member of the distribution of the training styles of the training examples based on a distance between the test style and the aggregate, and/or based on a value of a rating function that is dependent on the distance). This distance metric can be calculated in many ways, but a popular method is called the Mahalanobis distance. Joshi, para 33-34, 31-36, 71. The Mahalanobis distance measures distances between points, even correlated points for multiple variables. The Mahalanobis distance measures distance relative to the centroid, i.e, a base or central point which can be thought of as an overall mean for multivariate data of the cluster (wherein the aggregate includes a parametrized statistical distribution that is fitted to the training styles, and/or a centroid of a cluster of the training styles). The centroid is a point in multivariate space where all means from all variables intersect. The larger the Mahalanobis distance, the further away from the centroid the data point is. Joshi, para 59, 31-36, 71.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the method for detecting whether a given input record of measurement data that is inputted to a trained machine learning model is in a domain and/or distribution of training examples with which the machine learning model was trained, and evaluating, based on the training styles and the test style, to which extent the test style is a member of the distribution of the training styles of Tian using the features for aggregating the training styles to form an aggregate; and determining to which extent the test style is a member of the distribution of the training styles of the training examples based on a distance between the test style and the aggregate, and/or based on a value of a rating function that is dependent on the distance and wherein the aggregate includes a parametrized statistical distribution that is fitted to the training styles, and/or a centroid of a cluster of the training styles of Joshi, with a reasonable expectation of success, in order to provide for utilizing sensed data to detect anomalies and thus predicting the state or potential failure of the PE module and to provide further data analysis of the sensed data to determine which PE module may be in danger of imminent failure during operation and to provide methods for performing prognostics of PE devices installed in vehicles by utilizing autoencoders trained with datasets including contextual information. Joshi, para 2-3, 31-36, 71. This would have provided the advantages of providing further analysis of measurement data and contextual information to detect a sudden shift.
Regarding claim 5, which depends from claim 3 and recites:
wherein a Mahalanobis distance is chosen as a measure for the distance.
Tian in view of Joshi teaches the method of claim 3 from which claim 5 depends, including the evaluating, training styles and test style and aggregate. Tian does not specifically disclose wherein a Mahalanobis distance is chosen as a measure for the distance.
However, Joshi teaches in the field related to identifying maintenance issues of vehicle power modules by using vehicle contextual information in an autoencoder. Joshi, para 1, 2-3. Joshi, which is analogous to the claimed invention because Joshi is directed to measurement data and detecting anomalies, teaches that, The training datasets are input to the encoder, where the training datasets are associated with contextual features, such as elevation gain, weight-on-wheels, mileage, geographic region, climate, location or mode of travel (e.g., urban, suburban, rural, payload, etc. The contextual features can be represented by normalized values, so that they are input in a range from zero to one. For example, a climate feature on a sunny day may be zero, on a rainy data may be 0.5, on a snowy day may be 0.8 and in a blizzard may be 1.0. Joshi, para 33. [0034] Aspects of the present disclosure utilize the encoded feature space with K-means clustering to form clusters of data defined by contextual features (aggregating (clustering) the training styles (training contexts styles features) to form an aggregate (an aggregate of clusters);). The clusters are indicative of the operating points of the PE devices under external operating conditions, such as, but not limited to average drain-source current in MOSFET devices and duration on (ON time). Prognostics of the PE modules under test can be determined by applying data from the test devices to a trained encoder and determining the minimum squared distance of the encoded features to the training clusters (and determining to which extent the test style is a member of the distribution of the training styles of the training examples based on a distance between the test style and the aggregate, and/or based on a value of a rating function that is dependent on the distance). This distance metric can be calculated in many ways, but a popular method is called the Mahalanobis distance (wherein a Mahalanobis distance is chosen as a measure for the distance). Joshi, para 34, 33, 31-36, 71, 58-59, 80.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the method for detecting whether a given input record of measurement data that is inputted to a trained machine learning model is in a domain and/or distribution of training examples with which the machine learning model was trained, and evaluating, based on the training styles and the test style, to which extent the test style is a member of the distribution of the training styles of Tian using the features for aggregating the training styles to form an aggregate; and determining to which extent the test style is a member of the distribution of the training styles of the training examples based on a distance between the test style and the aggregate, and/or based on a value of a rating function that is dependent on the distance wherein a Mahalanobis distance is chosen as a measure for the distance of Joshi, with a reasonable expectation of success, in order to provide for utilizing sensed data to detect anomalies and thus predicting the state or potential failure of the PE module and to provide further data analysis of the sensed data to determine which PE module may be in danger of imminent failure during operation and to provide methods for performing prognostics of PE devices installed in vehicles by utilizing autoencoders trained with datasets including contextual information. Joshi, para 2-3, 31-36, 71. This would have provided the advantages of providing further analysis of measurement data and contextual information to detect a sudden shift.
Claim(s) 7 is rejected under 35 U.S.C. 103 as being unpatentable over Tian as applied to claim 1 above, and further in view of Pezzotti et al. (Pub. No. EP 3869401 A1, filed February 21, 2020) hereinafter Pezzotti The Examiner notes that Pezzotti (Kloninklikjke Phillips N.V.) is cited on Applicant’s Information Disclosure Statement filed on March 19, 2024.
Regarding claim 7, which depends from claim 1 and recites:
further comprising, in response to determining that the given record of measurement data is not in the domain and/or distribution of the training examples: repeating the method with the same given input record of measurement data, but with a second machine learning model that has been trained on a second dataset of training examples; and in response to determining that the given input record of measurement data is in the domain and/or distribution of training examples in the second dataset, determining that the second machine learning model is more appropriate for processing the given input record of measurement data than the previous machine learning model.
Tian teaches the method of claim 1 from which claim 7 depends, including the further comprising, in response to determining that the given record of measurement data is not in the domain and/or distribution of the training examples and in response to determining that the given record of measurement data is not in the domain or distribution of the training examples, as to notify a user of the electronic device 101 and ask for input. Tian, Figs 1-7, para 82, 54, 53-56, 30, 66, 39, 46, 48, 69-71. Tian does not specifically disclose repeating the method with the same given input record of measurement data, but with a second machine learning model that has been trained on a second dataset of training examples; and in response to determining that the given input record of measurement data is in the domain and/or distribution of training examples in the second dataset, determining that the second machine learning model is more appropriate for processing the given input record of measurement data than the previous machine learning model.
However, Pezzotti teaches in the field related to out-of-distribution detection of input instances to a model, and to a corresponding computer-implemented method. Pezzotti, para 1. Pezzotti, which is analogous to the claimed invention because Pezzotti is directed to out-of-distribution detection of input instances to a model, teaches that, Another possible cause for an input instance being OOD is that the subject being measured by the input instance is out-of-distribution. For example, the subject being scanned may have a pathology that is underrepresented in the dataset. This cause may be recognized by determining that multiple input instances representing the same subjects are OOD, but input instances of other subjects are not OOD. When it is determined that the subject is OOD, for example, the subject may be reported to a user, e.g., a clinician, for further analysis. Optionally, upon detecting that the subject is OOD (determining that the given record of measurement data is not in the domain and/or distribution), another output image for the input instance may be determined using a fallback model and the other output image may be used instead of the output image of the main model (repeating (repeating, using instead) the method with the same given input record of measurement data, but with a second machine learning model (fallback second model) that has been trained on a second dataset (fallback second model training dataset) of training examples; and in response to determining that the given input record of measurement data is in the domain and/or distribution of training examples in the second dataset, determining that the second machine learning model (fallback second model) is more appropriate for processing the given input record of measurement data than the previous machine learning model). The fallback model can for instance be a non-trainable model. For example, for MR scans, a SENSE-based reconstruction as known per se may be used as a fallback if the subject is OOD. Pezzotti, para 23.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the method for detecting whether a given input record of measurement data that is inputted to a trained machine learning model is in a domain and/or distribution of training examples with which the machine learning model was trained, and determining that the given record of measurement data is not in the domain and/or distribution of the training examples and in response to determining that the given record of measurement data is not in the domain or distribution of the training examples, as to notify a user and ask for input of Tian using the features for in response to determining that the given record of measurement data is not in the domain and/or distribution repeating the method with the same given input record of measurement data, but with a second machine learning model that has been trained on a second dataset of training examples; and in response to determining that the given input record of measurement data is in the domain and/or distribution of training examples in the second dataset, determining that the second machine learning model is more appropriate for processing the given input record of measurement data than the previous machine learning model of Pezzotti, with a reasonable expectation of success, in order to provide OOD detection techniques to image processing models and to determine if a model that may be expected to produce reliable results. Pezzotti, para 5-6. This would have provided the advantages of providing further analysis of measurement data from a more reliable model.
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over Tian as applied to claim 1 above, and further in view of Alvarez et al. (Pub. No. US 2020/0226790 A1, published July 16, 2020) hereinafter Alvarez. The Examiner notes that Alvarez is cited on Applicant’s Information Disclosure Statement filed on March 19, 2024.
Regarding claim 8, which depends from claim 1 and recites:
further comprising, in response to determining that the given record of measurement data is not in the domain and/or distribution of the training examples: obtaining a new record of measurement data from a sensor that is different from a sensor with which the given input record of measurement data was acquired; repeating the method with the same machine learning model, but with the new record of measurement data; and in response to determining that the new record of measurement data is in the domain and/or distribution of the training examples, determining that the new record of measurement data is more credible than the given record of measurement data.
Tian teaches the method of claim 1 from which claim 8 depends, including the further comprising, in response to determining that the given record of measurement data is not in the domain and/or distribution of the training examples and in response to determining that the given record of measurement data is not in the domain or distribution of the training examples, as to notify a user of the electronic device 101 and ask for input. Tian, Figs 1-7, para 82, 54, 53-56, 30, 66, 39, 46, 48, 69-71. Tian does not specifically disclose, obtaining a new record of measurement data from a sensor that is different from a sensor with which the given input record of measurement data was acquired; repeating the method with the same machine learning model, but with the new record of measurement data; and in response to determining that the new record of measurement data is in the domain and/or distribution, determining that the new record of measurement data is more credible than the given record of measurement data.
However, Alvarez teaches in the field related to sensor calibration for autonomous driving systems and/or the detection of sensor calibration. Alvarez, para 1. Alvarez, which is analogous to the claimed invention because Alvarez is directed to sensor calibration detection and detecting differences between sensor data outside a predetermined range, teaches that, Thus, the one or more processors one can first estimate the mean and variance of the distribution and perform a Wald test or other suitable test to detect out-of-distribution data that is uploaded from vehicles with problematic sensors (determining that the given record of measurement data is not in the domain and/or distribution), Alvarez, para 142, 145. It is noted that data sensors themselves can be noisy. As such, it may be desirable to include a system to evaluate the errors at multiple locations or even using data from a plurality of different sensors such as Lidar, Rader et. al. on the same vehicle, and to calibrate them via an ensemble model (obtaining a new record of measurement data from a sensor that is different from a sensor with which the given input record of measurement data was acquired (Lidar, Radar sensor data different from problematic sensor data); repeating the method with the same machine learning model, but with the new record of measurement data (new Lidar, Radar sensor data using same ensemble model); and in response to determining that the new record of measurement data is in the domain and/or distribution (not problematic sensor data in domain and/or distribution), determining that the new record of measurement data is more credible (new Lidar, Radar sensor data is in distribution and Lidar, Radar sensor data not problematic and more credible) than the given record of measurement data). That is to say, using this optional methodology, the sensor may only be considered problematic if a measure of the y data is out-of-distribution and if transformation parameters are calculated as the average of estimations at all these locations. In this case, the measure of the y data that must be out-of-distribution can be any measure desired for the implementation, including, but not limited to, a majority of the data, a fixed percent of the data, etc. Alvarez, para 145.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the method for detecting whether a given input record of measurement data that is inputted to a trained machine learning model is in a domain and/or distribution of training examples with which the machine learning model was trained, and determining that the given record of measurement data is not in the domain and/or distribution of the training examples and in response to determining that the given record of measurement data is not in the domain or distribution of the training examples, as to notify a user and ask for input of Tian using the features for determining that the given record of measurement data is not in the domain and/or distribution and obtaining a new record of measurement data from a sensor that is different from a sensor with which the given input record of measurement data was acquired, repeating the method with the same machine learning model, but with the new record of measurement data, and in response to determining that the new record of measurement data is in the domain and/or distribution, determining that the new record of measurement data is more credible than the given record of measurement data of Alvarez, with a reasonable expectation of success, in order to provide autonomous vehicles and semi-autonomous vehicles that utilize a variety of sensors with increased safety of vehicle operation. Joshi, para 2-3, 31-36, 71. This would have provided the advantages of providing further analysis of measurement data from different sensors for increasing data reliability and safety.
Claim(s) 10 is rejected under 35 U.S.C. 103 as being unpatentable over Tian as applied to claim 1 above, and further in view of Pezzotti et al. (Pub. No. WO 2022/078922 A1, filed October 11, 2021) hereinafter Pezzotti WO. The Examiner notes that Pezzotti WO (Kloninklikjke Phillips N.V.) is cited on Applicant’s Information Disclosure Statement filed on March 19, 2024.
Regarding claim 10, which depends from claim 1 and recites:
further comprising, in response to determining that the given record of measurement data is not in the domain and/or distribution of the training examples: applying, to the given record of measurement data, a candidate remedial procedure for a problem and/or deficiency that may affect the given record of measurement data, thereby creating a modified record of measurement data; repeating the method with the same machine learning model, but with the modified record of measurement data; and in response to determining that the modified record of measurement data is in the domain and/or distribution of the training examples, determining that the given input record of measurement data is affected with the problem and/or deficiency remedied by the candidate remedial procedure.
Tian teaches the method of claim 1 from which claim 10 depends, including the further comprising, in response to determining that the given record of measurement data is not in the domain and/or distribution of the training examples and in response to determining that the given record of measurement data is not in the domain or distribution of the training examples, as to notify a user of the electronic device 101 and ask for input. Tian, Figs 1-7, para 82, 54, 53-56, 30, 66, 39, 46, 48, 69-71. Tian does not specifically disclose applying, to the given record of measurement data, a candidate remedial procedure for a problem and/or deficiency that may affect the given record of measurement data, thereby creating a modified record of measurement data; repeating the method with the same machine learning model, but with the modified record of measurement data; and in response to determining that the modified record of measurement data is in the domain and/or distribution of the training examples, determining that the given input record of measurement data is affected with the problem and/or deficiency remedied by the candidate remedial procedure.
However, Pezzotti WO teaches in the field related to medical imaging, in particular to medical imaging techniques that use machine learning for image reconstruction or enhancement and sequential out of distribution detection for medical imaging. Pezzotti, p. 1, lines 1-5, Title. Pezzotti WO, which is analogous to the claimed invention because Pezzotti is directed to out-of-distribution detection for medical imaging, teaches that, the OOD score (the out-of-distribution score, in-distribution accuracy score, and anomaly estimation score ) the can be used for example: To reject the output of the main model and revert back to non-Al based algorithms To inform the user of problems in the data and asking for correction, e.g., reacquisition of the data (applying, to the given record of measurement data, a candidate remedial procedure (reacquisition, reconstructed, rescan) for a problem and/or deficiency that may affect the given record of measurement data, thereby creating a modified record of measurement data; repeating the method with the same machine learning model, but with the modified record of measurement data (reacquisition, reconstructed, rescan); and in response to determining that the modified record of measurement data is in the domain and/or distribution of the training examples, determining that the given input record of measurement data is affected with the problem and/or deficiency remedied by the candidate remedial procedure). Possible problems are: Faulty components in the imaging device Artefacts caused, for example, by motion or metal Problems in the processing pipeline. Pezzotti WO, Figures 2, 8-9, 11, page 22, lines 11-20; page 21, line12- page 22, line 20; page 28, lines 4-6. In another example, the described OOD score is calculated repeatedly during an MR acquisition to estimate the final image quality for an image reconstruction network. The acquisition is stopped once the OOD score is below a pre-defined threshold ((applying, to the given record of measurement data, a candidate remedial procedure (reacquisition, reconstructed, rescan) for a problem and/or deficiency that may affect the given record of measurement data, thereby creating a modified record of measurement data; repeating the method with the same machine learning model, but with the modified record of measurement data (reacquisition, reconstructed, rescan); and in response to determining that the modified record of measurement data is in the domain and/or distribution of the training examples, determining that the given input record of measurement data is affected with the problem and/or deficiency remedied by the candidate remedial procedure)). Pezzotti WO, Figures 2, 8-9, 11, page 28, lines 4-6, page 22, lines 11-20; page 21, line12- page 22, line 20.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement the method for detecting whether a given input record of measurement data that is inputted to a trained machine learning model is in a domain and/or distribution of training examples with which the machine learning model was trained, and determining that the given record of measurement data is not in the domain and/or distribution of the training examples and in response to determining that the given record of measurement data is not in the domain or distribution of the training examples, as to notify a user and ask for input of Tian using the features for applying, to the given record of measurement data, a candidate remedial procedure for a problem and/or deficiency that may affect the given record of measurement data, thereby creating a modified record of measurement data, repeating the method with the same machine learning model, but with the modified record of measurement data; and in response to determining that the modified record of measurement data is in the domain and/or distribution of the training examples, determining that the given input record of measurement data is affected with the problem and/or deficiency remedied by the candidate remedial procedure of Pezzotti WO, with a reasonable expectation of success, in order to provide a better means of detecting if the measured medical image data input into a trainable machine learning module will yield correct results or not. Pezzotti WO, page 2, lines 9-14. This would have provided the advantages of providing analysis of measurement data that is more reliable.
Allowable Subject Matter
Claim 6 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and if the rejections as being indefinite are overcome.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. US-10685057-B1, US-20210109528-A1, US-20210182687-A1, US-20220038620-A1, US-20220245422-A1, US-20220391724-A1, US-20230044354-A1, US-20230104127-A1, US-20230107505-A1, US-20240095983-A1, US-20250149177-A1, US-20250363996-A1, US-12488570-B2, US-20250384667-A1, US-20260087786-A1, US-12657351-B2.
Abdal, Rameen et al., StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows, Association for Computing Machinery, https://doi.org/10.1145/3447648, June 2021.
Sheng Han et al., A lightweight and style-robust neural network for autonomous driving in end side devices, published December 28, 2022
Denis Gudovskiy et al., CFLOW-AD: Real-Time Unsupervised Anomaly Detection with Localization via Conditional Normalizing Flows, 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2022, pp. 1819-1828, doi:10.1109/WACV51458.2022.00188
Xu Yao et al., Feature-Style Encoder for Style-Based GAN Inversion, 2022, https://doi.org/10.48550/arXiv.2202.02183.
Nishant Kumar et al., InFlow: Robust outlier detection utilizing Normalizing Flow, 16 Nov 2021, https://doi.org/10.48550/arXiv.2106.12894.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BARBARA LEVEL whose telephone number is (303)297-4748. The examiner can normally be reached Monday through Friday 8:00 AM - 5:00 PM MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BARBARA M LEVEL/ Examiner, Art Unit 2142
Prosecution Insights