DETAILED ACTION
This correspondence is responsive to the Application filed on December 12, 2023. Claims 1-19 are pending in the case.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in EP on December 14, 2022. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Summary of Detailed Action
The drawings are objected to regarding Figure 6A grey scale text that is not legible.
I. Claims 15-17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
II. Claim 16 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
III. Claim 17 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
IV. Claim 11 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
V. Claim 11 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
VI. Claim 12 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
VII. Claim 17 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
VIII. Claim 19 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite.
IX. Claims 13-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite
Claims 1-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claims 1-6, 8-9, 12, 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Kuppers et al. in view of Patel et al.
Claims 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over Kuppers and Patel, and further in view of Krishnan et al.
Drawings
The drawings are objected to because Figure 6A contains grey scale text that is not legible or reproducible for publication purposes. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
I. Claims 15-17 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Independent claim 15 recites a computer system comprising a plurality of computer hardware components “configured to” determine a conversion rule for an object prediction model, the computer hardware components being “configured to.” It is not clear if the computer system hardware components of claim 15 are merely configured to and capable of performing the recited process but do not actually perform, or cause the computer system to perform, any process whatsoever. It is further unclear if the claim 15 computer system hardware components are configured and capable of performing the recited process but in actuality only perform other processes not included in the recited limitations. It is yet further unclear if the claim 15 computer system hardware components are configured and capable of performing the recited process, but only actually perform some but not all of the recited limitations. Thus the boundaries of claim 15 are not clear and the claim is indefinite. Applicant may cancel claim 15 or amend claim 15 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 16 includes the computer system of claim 15 and claim 17 depends from claim 15 and both claims 16 and 17 are rejected for the same reasons discussed above with respect to claim 15.
II. Claim 16 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 16 recites A vehicle that includes the computer system of claim 15 and at least one sensor. It is not entirely clear if claim 16 was actually intended to independently claim a vehicle with the computer system of claim 15 and a sensor or if claim 16 was intended to depend from claim 15 and further recite that the computer system is included in a vehicle that includes a sensor. It is also not clear if the vehicle sensor data of claim 16 provides the sensor data of claim 15 or if the sensor data is from one or more different sensors. Thus the boundaries of claim 16 are not clear and the claim is indefinite. Applicant may cancel claim 16 or amend claim 16 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
III. Claim 17 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 17 depends from claim 15 and recites that the computer hardware components are further configured to apply the conversion rule to an output of an object prediction model. It is not clear if claim 17 only requires that the computer system hardware components be further configured to and capable of performing the recited applying limitation but are not required to actually perform, or cause the computer system to perform, any further process whatsoever. It is further unclear if the claim 17 computer system hardware components might be further configured and capable of performing all of the recited process but in actuality only perform some other processes not included in the recited limitation. It is yet further unclear if the claim 17 computer system hardware components are further configured and capable of performing the recited process, but only actually perform some but not all of the recited limitation. Thus the boundaries of claim 17 are not clear and the claim is indefinite. Applicant may cancel claim 17 or amend claim 17 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
IV. Claim 11 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 11 depends from claim 8 and recites the limitation wherein each score comprises a minimum distance to “the curve.” There is insufficient antecedent basis for this limitation in the claim.
V. Claim 11 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 11 depends from claim 8 and recites the limitation “wherein each score” comprises a minimum distance to the curve. There is insufficient antecedent basis for this limitation in the claim.
VI. Claim 11 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 11 depends from claim 8 and recites the limitation wherein each score comprises a minimum distance to “the curve.” There is insufficient antecedent basis for this limitation in the claim.
VII. Claim 17 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 17 depends from claim 15 and recites to apply the conversion rule to an output of “an object prediction model.” However, claim 15 already recites an object prediction model, so claim 17 is unclear in terms of whether or not the conversion rule is applied to an output of the object prediction model of claim 15 or if the conversion rule is applied to an output of another different object prediction model. Thus, the boundaries of the claim 17 are unclear and the claim is indefinite. Applicant may cancel claim 17 or amend claim 17 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
VIII. Claim 19 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 19 depends from claim 18 and recites to apply the conversion rule to an output of “an object prediction model.” However, claim 18 already recites an object prediction model, so claim 19 is unclear in terms of whether or not the conversion rule is applied to an output of the object prediction model of claim 18 or if the conversion rule is applied to an output of another different object prediction model. Thus, the boundaries of the claim 19 are unclear and the claim is indefinite. Applicant may cancel claim 19 or amend claim 19 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
IX. Claims 13-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 13 recites determining a tracker parameter for a unified tracking module based on the conversion rule such that the unified tracking module is applicable to the object prediction model. It is not clear what a tracker parameter for a unified tracking module is or is not. For example, is a tracking parameter a parameter for what objects or types of objects to detect and track? Or, is a tracker parameter a certain value, threshold or performance metric such as accuracy, precision, recall to track? Or is a tracker parameter a condition or context for object detection tracking? Or is a tracking parameter something else entirely? It is further unclear what a unified tracking module is or is not. For example, is a unified tracking module a module that monitors and tracks a certain unified type of object detector? Or, is a unified tracking module any module that unifies tracked parameters? It is yet further unclear how the tracking parameter for the unified tracking module is based on the conversion rule and applicable to the object prediction. Thus, the boundaries of claim 13 are unclear and the claim is indefinite. Claim 14 depends from claim 13 and is indefinite for at least the same reasons discussed above with respect to claim 13. Applicant may cancel claims 13-14 or amend claims 13-14 to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) recite(s) subject matter at a general, high-level of a method for determining a conversion rule for an object prediction, the method comprising the following steps, determining a plurality of predictions based on sensor data using the object prediction, wherein each prediction comprises a respective prediction value and a respective confidence value of the respective prediction value; determining the conversion rule for the object prediction by carrying out the following steps: determining a plurality of sampling values for assessing a performance of the object prediction; for each sampling value of the plurality of sampling values, determining a corresponding statistical value based on ground-truth data and the plurality of confidence values, wherein the ground-truth data is associated with the sensor data; and determining the conversion rule for the object prediction based on the plurality of sampling values and the plurality of corresponding statistical values, which are mental processes and mathematical concepts. MPEP 210604(a)(2)(III)
This judicial exception is not integrated into a practical application and the claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Claims 1-19 recite one of the four statutory categories of patent able subject matter and belong to the statutory class(es) of a process (method claims 1-14), a machine (system/apparatus claims 15-17), and an article of manufacture (non-transitory computer readable media claims 18-19).
Claim 1 recites a method, thus a process and one of the four statutory categories of patentable subject matter. However, claim 1 further recites for determining a conversion rule for an object prediction (mental process, mathematical concepts), the method comprising the following steps, determining a plurality of predictions based on sensor data using the object prediction (mental process), wherein each prediction comprises a respective prediction value (mental process) and a respective confidence value of the respective prediction value (mental process, mathematical concepts); determining the conversion rule for the object prediction by carrying out the following steps (mental process, mathematical concepts): determining a plurality of sampling values for assessing a performance of the object prediction (mental process); for each sampling value of the plurality of sampling values, determining a corresponding statistical value based on ground-truth data and the plurality of confidence values, wherein the ground-truth data is associated with the sensor data (mental process, mathematical concepts); and determining the conversion rule for the object prediction based on the plurality of sampling values and the plurality of corresponding statistical values (mental process, mathematical concepts), which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)) and mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
computer implemented (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
carried out by computer hardware components (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim 2, dependent on claim 1, recites only additional abstract ideas for wherein the plurality of sampling values corresponds to the plurality of confidence values, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)) and mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
Claim 3, dependent on claim 1, recites only additional abstract ideas for wherein each statistical value of the plurality of statistical values comprises a true-positive rate corresponding to the respective sampling value, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
Claim 4, dependent on claim 1, recites additional abstract ideas for further comprising the following steps, filtering the ground-truth data based on a condition, wherein the condition is at least one of an object class, a scene type, or bounding box properties (mental processes); and for each sampling value of the plurality of sampling values, determining the corresponding statistical value (mathematical concepts) based on the filtered ground-truth data and the plurality of confidence values (mental processes, mathematical concepts), which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)) and mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
carried out by computer hardware components (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
Claim 5, dependent on claim 1, recites only additional abstract ideas for wherein the prediction values comprise data describing the respective prediction associated to an object class of a plurality of object classes, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)).
Claim 6, dependent on claim 1, recites only additional abstract ideas for wherein the prediction values comprise data describing the respective prediction associated to bounding box properties, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)).
Claim 7, dependent on claim 1, recites additional abstract ideas for further comprising the following steps, for each sampling value (mental processes) of the plurality of sampling values: determining a first number as a number of predictions with a respective confidence value greater or equal than the sampling value (mental processes and mathematical concepts); determining a second number as a number of predictions with a respective confidence value greater or equal than the sample value, wherein the predictions with a respective confidence value greater or equal than the sample value correspond to a corresponding object in the ground-truth data (mental processes and mathematical concepts); and determining the statistical value by dividing the second number by the first number (mathematical concepts), which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)) and mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
carried out by computer hardware components (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
Claim 8, dependent on claim 1, recites only additional abstract ideas for wherein determining the conversion rule comprises a fitting of the conversion rule to the plurality of sampling values and the plurality of corresponding statistical values, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
Claim 9, dependent on claim 8, recites only additional abstract ideas for wherein the fitting of the conversion rule to the plurality of sampling values and the plurality of corresponding statistical values comprises using a regression method, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
Claim 10, dependent on claim 8, recites only additional abstract ideas for wherein the fitting of the conversion rule approximates a curve based on a plurality of scores, wherein each score of the plurality of scores represents a statistical value of the plurality of statistical values and the corresponding sampling value of the plurality of sampling values, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
Claim 11, dependent on claim 8, recites only additional abstract ideas for wherein each score comprises a minimum distance to the curve, which are mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
Claim 12, dependent on claim 1, recites additional abstract ideas for further comprising the following step, applying the conversion rule to an object prediction, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)) and mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
carried out by computer hardware components (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
an output of (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 13, dependent on claim 12, recites additional abstract ideas for further comprising the following step, determining a tracker parameter for a unified tracking based on the conversion rule such that the unified tracking is applicable to the object prediction, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)) and mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
carried out by computer hardware components (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
module (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 14, dependent on claim 13, recites additional abstract ideas for wherein the object prediction once applied to the conversion rule is used as input to the unified tracking, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)) and mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
the output of (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
module (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
Claim 15 recites a system, thus a machine and one of the four statutory categories of patentable subject matter. However, claim 15 further recites to determine a conversion rule for an object prediction (mental process, mathematical concepts), determine a plurality of predictions based on sensor data using the object prediction (mental process), wherein each prediction comprises a respective prediction value (mental process) and a respective confidence value of the respective prediction value (mental process, mathematical concepts), determine a plurality of predictions based on sensor data using the object prediction (mental process); determine the conversion rule for the object prediction by carrying out the following steps (mental process, mathematical concepts): determining a plurality of sampling values for assessing a performance of the object prediction model (mental process); for each sampling value of the plurality of sampling values, determining a corresponding statistical value based on ground-truth data and the plurality of confidence values, wherein the ground-truth data is associated with the sensor data (mental process, mathematical concepts); and determining the conversion rule for the object prediction based on the plurality of sampling values and the plurality of corresponding statistical values (mental process, mathematical concepts), which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)) and mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
A computer system comprising a plurality of computer hardware components configured to (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
the computer hardware components being configured to (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim 16, dependent on system of claim 15, does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
a vehicle (This additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
computer system (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
at least one sensor (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
Claim 17, dependent on system of claim 15, recites additional abstract ideas to apply the conversion rule to an object prediction, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)) and mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
computer system (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
computer hardware components are further configured to an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
an output of (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim 18 recites a computer readable medium, thus an article of manufacture and one of the four statutory categories of patentable subject matter. However, claim 18 further recites to determine a conversion rule for an object prediction (mental process, mathematical concepts), by determining a plurality of predictions based on sensor data using the object prediction (mental process), wherein each prediction comprises a respective prediction value (mental process) and a respective confidence value of the respective prediction value (mental process, mathematical concepts); determining the conversion rule for the object prediction by carrying out the following steps (mental process, mathematical concepts): determining a plurality of sampling values for assessing a performance of the object prediction (mental process), for each sampling value of the plurality of sampling values, determining a corresponding statistical value based on ground-truth data and the plurality of confidence values, wherein the ground-truth data is associated with the sensor data (mental process, mathematical concepts); and determining the conversion rule for the object prediction based on the plurality of sampling values and the plurality of corresponding statistical values (mental process, mathematical concepts), which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)) and mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
A non-transitory computer readable medium storing instructions that, when executed by a computer, cause the computer to (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Thus, the claim is directed to the abstract idea.
Further, the additional elements, alone or in combination, do not provide significantly more than the abstract idea itself, because implementation on a computer (MPEP 2106.05(f)) cannot provide significantly more and generally linking the use of the judicial exception to a particular technological field of use does not meaningfully limit the claims (MPEP 2106.04(d)) and the combination of additional elements does not provide an inventive concept. Thus, the claim is ineligible.
Claim 19, dependent on claim 18, recites additional abstract ideas to apply the conversion rule to an object prediction, which are mental processes or concepts that can be performed in the human mind, including observation, evaluation, judgment or opinion, or by a human using pen and paper (MPEP 210604(a)(2)(III)) and mathematical concepts including mathematical relationships, mathematical formulas or equations, and mathematical calculations (MPEP 210604(a)(2)(I)).
The claim does not include any additional elements which integrate the abstract idea into a practical application since the additional elements consist of:
instructions further cause the computer to (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
an output of (an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea. See also, MPEP 2106.05(f), MPEP 2106.04(d), 2019 Guidance, 84 FR 50 at 55, footnote 30.).
model (This additional element amounts to merely the words to “apply it” (or an equivalent) or are mere instructions to implement an abstract idea or other exception on a computer. MPEP 2106.05(f).) Also, this additional element amounts to no more than generally linking the use of the judicial exception to a particular technologic environment or field of use - The application or use of the judicial exception in this manner does not meaningfully limit the claim by going beyond generally linking the use of the judicial exception to a particular technological environment. MPEP 2106.05(h)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 8-9, 12, 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Kuppers et al., Multivariate Confidence Calibration for Object Detection, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPRW), hereinafter Kuppers in view of Patel et al. (US 2024/0070516 A1, filed August 24, 2022) hereinafter Patel.
Regarding claim 1, Kuppers teaches:
A computer implemented method for determining a conversion rule for an object prediction model, the method comprising the following steps carried out
(i.e., 3. Methods for Confidence Calibration In this section we introduce our calibration methodology for object detectors (computer implemented method (computer implemented, Experiments, Results Section 5-5.2, p.1327,1328) method for determining a conversion rule (calibration mapping conversion rules) for an object prediction model (object detector prediction model), the method comprising the following steps carried out). Kuppers, Sections 3,3.1-3.4, p. 1324-1325. 5 Experiments 5.1 Evaluation Protocol We apply our extended calibration methods to several pretrained object detectors (computer implemented method for determining a conversion rule (calibration mapping conversion rules) for an object predictor model) available in [4]. … 5.2 Results. Kuppers, Section 5, p. 1327, 1328. Even though it is possible to evaluate the distribution parameters directly, we use a discriminative approach to obtain a better calibration mapping [16] (determining a conversion rule (calibration mapping conversion rules) for an object prediction model (object detector prediction model, p.1324-1325)). Thus, the calibration process can then be posed as an optimization of a logistic regression problem. Nevertheless, using different models f(s|m) results in an extremely varying expressiveness of the calibration mapping, as we explain in Fig. 3.Kuppers, Sections 3,3.1-3.4, p. 1324-1325.)
determining a plurality of predictions based on sensor data using the object prediction model, wherein each prediction comprises a respective prediction value and a respective confidence value of the respective prediction value;
(i.e., 3.1 Object Detector Modern object detectors typically take an image as the input x and deliver a prediction in form of a class label y, a confidence score p and a bounding box r = (cx, cy, w, h), where (cx, cy) is the position of the box center and the scale is determined by the width and height (w, h). We interpret these quantities as random variables that follow a joint ground-truth distribution π(x, y ,r) = π(y, r|x)π(x). The detector can be interpreted as function h(x)=(ˆy, ˆp, ˆr), that takes the input images and generates predictions for the class, confidence and location (determining, generating a plurality of predictions based on sensor data images using the object detection prediction model, where each object detection prediction includes a respective prediction value and confidence of the respective prediction (3.1 where each prediction includes a bounding box r=(cx, cy, w, h) (prediction value) and confidence score p (confidence value)). Kuppers, Section 3.1 p. 1324, 1325. Section 5 Experiments, 5.1 Evaluation Protocol, We apply our extended calibration methods to several pretrained object detectors available in [4]. … We use the COCO validation dataset 2017 [11] with images licensed for commercial use only to demonstrate the effectiveness of our approach. Table 1 Calibration scores for different networks with predictions including class, confidence, and bounding boxes. Kuppers, Section 5,5.1-5.2, Table 1, p. 1327-1328)
determining the conversion rule for the object prediction model by carrying out the following steps:
(i.e., 3.2. Multivariate Calibration Framework Our goal is to develop calibration methods for detectors that can be applied after the training independently of the underlying detector architecture (black-box). For simple classification tasks, a confidence map g is applied on top of a miscalibrated scoring classifier ˆp = h(x) to deliver a calibrated confidence score ˆq = g(h(x)). The key ingredient that can be used to improve the calibration of a detector is that we have additional information about the estimated bounding box ˆr. Therefore, the calibration map is not only a function of the confidence score, but also of ˆr. To define a general calibration map for binary problems, we use the logistic function and the combined input s =(ˆ p, ˆr) of size K by g(s) (determining the conversion rule (determining the calibration map (g(s) conversion rule) for the object prediction model (object detection prediction model) by carrying out the following steps). Kuppers, Section 3, 3.1, 3.2, p.1324-1325, section 5, p.1327-1328.)
determining a plurality of sampling values for assessing a performance of the object prediction model;
(i.e., 3.2. Multivariate Calibration Framework Our goal is to develop calibration methods for detectors that can be applied after the training independently of the underlying detector architecture (black-box). For simple classification tasks, a confidence map g is applied on top of a miscalibrated scoring classifier ˆp = h(x) to deliver a calibrated confidence score ˆq = g(h(x)). The key ingredient that can be used to improve the calibration of a detector is that we have additional information about the estimated bounding box ˆr. Therefore, the calibration map is not only a function of the confidence score, but also of ˆr. To define a general calibration map for binary problems, we use the logistic function and the combined input s =(ˆ p, ˆr) of size K by g(s) (determining a plurality of sampling values for assessing a performance (sampling values s are the combined inputs s=(p, r), normalized to [0,1] used to assess miscalibration performance via D-ECE or precision) of the object prediction model (object detection prediction model)). Kuppers, Section 3-3.2, p.1324-1325, Section 5, 5.1, p.1327-1328.)
for each sampling value of the plurality of sampling values, determining a corresponding statistical value based on ground-truth data and the plurality of confidence values, wherein the ground-truth data is associated with the sensor data; and
(i.e., Sections 3.1 Object Detectors… In this work we thus use precision as a surrogate for accuracy. That is to say, given 100 detections, each predicted with 0.9 confidence, 90 out of the 100 detections should be correctly classified. This is our measure to define calibration for black-box classifiers. Thus, an object detection model is perfectly calibrated if
PNG
media_image1.png
56
228
media_image1.png
Greyscale
(1) is fulfilled, where m=1denotes a correctly classified prediction that matches a ground-truth object and m=0 denotes a mismatch. Thus, P(m = 1) is the shorthand notation for approximating P(ˆy=y,ˆr=r) with a certain IoU threshold. (for each sampling value of the plurality of sampling values (each sampling value of the plurality of image sampling values s), determining a corresponding statistical value (precision statistical value) based on ground-truth data and the plurality of confidence values, wherein the ground-truth data is associated with the sensor data (based on match/mismatch ground truth and plurality of confidence p values, wherein ground truth is associated with sensor data images)) We use bounding box information relative to the image size r ∈ [0,1]J, where J is the dimension of the used box encoding. Similar to [3], we assume that a perfect calibration is not achievable by any known detection method. Kuppers, Section 3.1, p. 1324, Section 5.1 Evaluation Protocol, p. 1327-1328.)
determining the conversion rule for the object prediction model based on the plurality of sampling values and the plurality of corresponding statistical values.
(i.e., In this work we thus use precision (corresponding statistical values) as a surrogate for accuracy. That is to say, given 100 detections, each predicted with 0.9 confidence, 90 out of the 100 detections should be correctly classified. Kuppers, Section 3.1. p. 1324, 3.2 p. 1324-1325. To define a general calibration map for binary problems, we use the logistic function and the combined input s =(ˆ p, ˆr) of size K by
PNG
media_image2.png
34
114
media_image2.png
Greyscale
(2) Similar to the work presented in [9] regarding beta calibration, we interpret the logit z as the logarithm of the posterior odds
PNG
media_image3.png
32
280
media_image3.png
Greyscale
(3) For simplicity, we assume a uniform prior f(m) and use the approximation which in turn is the log-likelihood ratio ℓr(s). In contrast to classifier calibration, we can use multivariate probability density functions to model f(s | m) and therefore include the bounding box information. Even though it is possible to evaluate the distribution parameters directly, we use a discriminative approach to obtain a better calibration mapping [16]. Thus, the calibration process can then be posed as an optimization of a logistic regression problem (determining the conversion rule (calibration mapping conversion rule) for the object prediction model (object detection prediction model) based on the plurality of sampling values (s sampling values) and the plurality of corresponding statistical values (precision statistical values (determining the calibration mapping conversion rule for the object detection prediction model using logistic regression fitting on s sampling values and precision accuracy surrogates statistical values, sections 31-3.2 p.1324-1325)). Nevertheless, using different models f(s | m) results in an extremely varying expressiveness of the calibration mapping, as we explain in Fig. 3. Kuppers, Section 3.2. p. 1324-1325, 3-3.1 p. 1324.)
Kuppers does not explicitly disclose by computer hardware components.
However, Patel teaches in the field related to systems and methods for machine learning to draw an inference or make a prediction based on input data. Patel, Abstract, para 1. Patel is analogous to the claimed invention because Patel is directed to machine learning context based confidence calibration. Patel, Abstract, para 1, 2. In Figure 10 and corresponding description, Patel teaches and illustrates by computer hardware components. Patel, Abstract, Fig 10, para 57-62.
It would have been obvious to one of ordinary skill in the art to implement the method for determining calibration mapping conversion rules of Kuppers using the computer hardware components of Patel, with a reasonable expectation of success, in order to provide an operating environment for implementing aspects of the technology. Patel, Abstract, Fig 10, para 57-62, This would have provided the advantages of providing computer hardware components for implementation.
Regarding claim 2, which depends from claim 1 and recites:
wherein the plurality of sampling values corresponds to the plurality of confidence values.
Kuppers in view of Patel teaches the method of claim 1 from which claim 2 depends, including the plurality of sampling values. Kuppers teaches that, Therefore, the calibration map is not only a function of the confidence score, but also of ˆr. To define a general calibration map for binary problems, we use the logistic function and the combined input s =(ˆ p, ˆr) of size K by ..( plurality of sampling values corresponds to the plurality of confidence values (input samplings s values correspond to plurality of p confidence values)) Kuppers, section 3.2, p.1324.
Regarding claim 3, which depends from claim 1 and recites:
wherein each statistical value of the plurality of statistical values comprises a true-positive rate corresponding to the respective sampling value.
Kuppers in view of Patel teaches the method of claim 1 from which claim 3 depends, including each statistical value of the plurality of statistical values and sampling value of the plurality of sampling values. Kuppers teaches that, Instead, the common metrics precision and recall are used to evaluate the performance of an object detector. Precision is the fraction of correct detections among all detections (each statistical value of the plurality of statistical values comprises a true-positive rate (each statistical value comprises a precision true-positive rate) corresponding to the respective sampling value) while recall denotes the proportion of actual objects that are correctly identified as such among all ground-truth objects. … In this work we thus use precision as a surrogate for accuracy (each statistical value of the plurality of statistical values comprises a true-positive rate (using precision true-positive rate for calibration, each statistical value comprises a precision true-positive rate) corresponding to the respective sampling value). That is to say, given 100 detections, each predicted with 0.9 confidence, 90 out of the 100 detections should be correctly classified. Kuppers, section 3.1, p. 1324.
Regarding claim 4, which depends from claim 1 and further recites:
filtering the ground-truth data based on a condition, wherein the condition is at least one of an object class, a scene type, or bounding box properties; and for each sampling value of the plurality of sampling values, determining the corresponding statistical value based on the filtered ground-truth data and the plurality of confidence values.
Kuppers in view of Patel teaches the method of claim 1 from which claim 4 depends, including steps carried out by computer hardware components, ground-truth data, each sampling value of the plurality of sampling values, statistical value based on the ground-truth data and the plurality of confidence values. Kuppers teaches that, For this reason, we also integrate the box information into the ECEcalculation. Therefore, we define the detection ECE (D-ECE) as the expected deviation of the observed precision with respect to the given box properties. Using Eq. 1 we get
PNG
media_image4.png
26
232
media_image4.png
Greyscale
. (13) Thus, the D-ECE for a given class y depends on the IoU and on the number of additional box information (filtering the ground-truth data based on a condition, wherein the condition is at least one of an object class, a scene type, or bounding box properties (filtering the ground-truth data based on a condition, wherein the condition is at least one of bounding box properties condition (location, scale, Section 5 p. 1327-1328, Abstract, Fig 1))). If we calculate the expectation with respect to a subset of the variables, e.g. ˆp, calibration heatmaps can be generated (cp. Fig. 4) (for each sampling value of the plurality of sampling values, determining the corresponding statistical value based on the filtered ground-truth data and the plurality of confidence values (for each sampling value evaluating miscalibration filtered by ground truth bounding box properties (location, scale), Section 5 p.1327-1328, Abstract, Fig 1). … Consider the following example: Method A uses only the confidences for calibration without any additional box information (K =1)while method B also takes the cx position of the bounding boxes into account (K = 2). Method C calibrates with respect to all three quantities, confidence ˆp and cx, cy position (K =3). To compare these models, we need the D-ECEK (K =3with cx, cy position) because C uses the confidence and 2 additional box quantities for calibration. Kuppers, Figs 1, Abstract, Sections 4-5, p. 1327-1328.
Regarding claim 5, which depends from claim 1 and recites:
wherein the prediction values comprise data describing the respective prediction associated to an object class of a plurality of object classes.
Kuppers in view of Patel teaches the method of claim 1 from which claim 5 depends, including the prediction values. Kuppers teaches that, The detector can be interpreted as function h(x)=(ˆy, ˆp, ˆr), that takes the input images and generates predictions for the class, confidence and location. The training process for fitting the model parameters is based on a probabilistic model ˆπ(y, r| x), where the cross entropy between the training data πdata(x, y, r) (empirical distribution of π(x, y, r) given by the dataset D) and the model ˆπ(y, r |x) is minimized. The evaluation of a detector differs from classification in the sense that both, the class label and the regression output for the location, must match the ground-truth (prediction values comprise data describing the respective prediction associated to an object class of a plurality of object classes (detector prediction values describe respective detector predictions associated to object class label of a plurality of samples s dataset images object classes labels)…. In other words, the class prediction and a predefined threshold, e.g. IoU > 0.5, is used to classify whether a prediction is a true or false positive. … Thus, an object detection model is perfectly calibrated if
PNG
media_image5.png
52
234
media_image5.png
Greyscale
(1) is fulfilled, where m=1 denotes a correctly classified prediction that matches a ground-truth object and m=0 denotes a mismatch. Thus, P(m = 1) is the shorthand notation for approximating P(ˆy=y, ˆr=r) with a certain IoU threshold. Kuppers, Section 3.1, p.1324. Thus, the D-ECE for a given class y depends on the IoU and on the number of additional box information. Kuppers, Section 4. p. 1327. We apply our extended calibration methods to several pretrained object detectors available in [4]. … We use the COCO validation dataset 2017 [11] with images licensed for commercial use only to demonstrate the effectiveness of our approach. Table 1 Calibration scores for different networks with predictions including class, confidence, and bounding boxes. Kuppers, Section 5,5.1-5.2, Table 1, p. 1327-1328.
Regarding claim 6, which depends from claim 1 and recites:
wherein the prediction values comprise data describing the respective prediction associated to bounding box properties.
Kuppers in view of Patel teaches the method of claim 1 from which claim 6 depends, including the prediction values. Kuppers teaches that, Modern object detectors typically take an image as the input x and deliver a prediction in form of a class label y, a confidence score p and a bounding box r = (cx, cy, w, h), where (cx, cy) is the position of the box center and the scale is determined by the width and height (w, h). We interpret these quantities as random variables that follow a joint ground-truth distribution π(x, y, r) = π(y, r |x) π(x). The detector can be interpreted as function h(x)=(ˆy, ˆp, ˆr), that takes the input images and generates predictions for the class, confidence and location. The training process for fitting the model parameters is based on a probabilistic model ˆπ(y, r |x), where the cross entropy between the training data πdata(x, y, r) (empirical distribution of π(x, y, r) given by the dataset D) and the model ˆπ(y, r |x) is minimized. The evaluation of a detector differs from classification in the sense that both, the class label and the regression output for the location, must match the ground-truth. For the bounding box prediction ˆr it is nearly impossible to match the ground-truth perfectly, therefore, a certain overlap score (e.g. Intersection of Union (IoU)) is required to assign detections to ground truth annotations (prediction values comprise data describing the respective prediction associated to bounding box properties (prediction value is bounding box r with properties (c_x, c_y, w, h)). In other words, the class prediction and a predefined threshold, e.g. IoU > 0.5, is used to classify whether a prediction is a true or false positive. Kuppers, Section 3.1, p. 1324.
Regarding claim 8, which depends from claim 1 and recites:
wherein determining the conversion rule comprises a fitting of the conversion rule to the plurality of sampling values and the plurality of corresponding statistical values.
Kuppers in view of Patel teaches the method of claim 1 from which claim 8 depends, including determining the conversion tule and the plurality of sampling values. Kuppers teaches that To define a general calibration map for binary problems, we use the logistic function and the combined input s =(ˆ p, ˆr) of size K by
PNG
media_image6.png
30
106
media_image6.png
Greyscale
….In contrast to classifier calibration, we can use multivariate probability density functions to model f(s |m) and therefore include the bounding box information. Even though it is possible to evaluate the distribution parameters directly, we use a discriminative approach to obtain a better calibration mapping [16], Thus, the calibration process can then be posed as an optimization of a logistic regression problem (determining the conversion rule comprises a fitting of the conversion rule to the plurality of sampling values and the plurality of corresponding statistical values (fitting calibration mapping conversion rule g(s) via logistic regression of sampling values s and plurality of corresponding accuracy/precision surrogates statistical values). Kuppers, Section 3.1-3.2, p 1324.
Regarding claim 9, which depends from claim 8 and recites:
wherein the fitting of the conversion rule to the plurality of sampling values and the plurality of corresponding statistical values comprises using a regression method.
Kuppers in view of Patel teaches the method of claim 8 from which claim 9 depends, including fitting of the conversion rule to the plurality of sampling values and the plurality of corresponding statistical values. Kuppers teaches that, Kuppers in view of Patel teaches the method of claim 1 from which claim 8 depends, including determining the conversion tule and the plurality of sampling values. Kuppers teaches that To define a general calibration map for binary problems, we use the logistic function and the combined input s =(ˆ p, ˆr) of size K by
PNG
media_image6.png
30
106
media_image6.png
Greyscale
….In contrast to classifier calibration, we can use multivariate probability density functions to model f(s|m) and therefore include the bounding box information. Even though it is possible to evaluate the distribution parameters directly, we use a discriminative approach to obtain a better calibration mapping [16], Thus, the calibration process can then be posed as an optimization of a logistic regression problem (fitting of the conversion rule to the plurality of sampling values and the plurality of corresponding statistical values comprises using a regression method (fitting calibration mapping conversion rule g(s) via logistic regression of sampling values s and plurality of corresponding accuracy/precision surrogates statistical values). Kuppers, Section 3.1-3.2, p 1324.
Regarding claim 12, which depends from claim 1 and recites:
further comprising the following step carried out by computer hardware components: applying the conversion rule to an output of an object prediction model.
Kuppers in view of Patel teaches the method of claim 1 from which claim 12 depends, including steps carried out by computer hardware components, the conversion rule, and output of object prediction model. Kuppers teaches 3.2. Multivariate Calibration Framework For simple classification tasks, a confidence map g is applied on top of a miscalibrated scoring classifier ˆp = h(x) to deliver a calibrated confidence score ˆq = g(h(x)) (applying the conversion rule to an output of an object prediction model (applying the calibration mapping conversion rule g(s) to a prediction confidence output of an object detection prediction model)). The key ingredient that can be used to improve the calibration of a detector is that we have additional information about the estimated bounding box ˆr. Therefore, the calibration map is not only a function of the confidence score, but also of ˆr. To define a general calibration map for binary problems, we use the logistic function and the combined input s =(ˆ p, ˆr) of size K by
PNG
media_image6.png
30
106
media_image6.png
Greyscale
(2) … Even though it is possible to evaluate the distribution parameters directly, we use a discriminative approach to obtain a better calibration mapping [16]. Thus, the calibration process can then be posed as an optimization of a logistic regression problem. Kuppers, Section 3.2, p. 1324-1325, Section 4, p. 1327. For example, the framework proposed in this paper can either be used to integrate a dependence to the regression output or to further optimize the calibration in a post-processing step (applying the conversion rule to an output of an object prediction model (applying g(s) post detection). Kuppers, section 2, p. 1323, Section 4-5, p. 1327-1328.
Claim 15 recites a computer system that parallels the computer-implemented method of claim 1. Therefore, the analysis discussed above with respect to claim 1 also applies to claim 15. Accordingly, claim 15 is rejected based on substantially the same rational as set forth above with respect to claim 1. More specifically regarding A computer system comprising a plurality of computer hardware components, Kuppers does not explicitly disclose a computer system comprising a plurality of computer hardware components.
However, Patel teaches in the field related to systems and methods for machine learning to draw an inference or make a prediction based on input data. Patel, Abstract, para 1. Patel is analogous to the claimed invention because Patel is directed to machine learning context based confidence calibration. Patel, Abstract, para 1, 2. In Figure 10 and corresponding description, Patel teaches and illustrates a computer system comprising a plurality of computer hardware components. Patel, Abstract, Fig 10, para 57-62.
It would have been obvious to one of ordinary skill in the art to implement the method for determining calibration mapping conversion rules of Kuppers using the computer system comprising a plurality of computer hardware components of Patel, with a reasonable expectation of success, in order to provide an operating environment for implementing aspects of the technology. Patel, Abstract, Fig 10, para 57-62, This would have provided the advantages of providing computer hardware components for implementation.
Claim 16 recites A vehicle that includes the computer system of claim 15 and at least one sensor.
Kuppers in view of Patel teaches the computer system of claim 15, including sensor data. Kuppers does not explicitly disclose a vehicle included computer system and at least one sensor.
However, Patel teaches a vehicle computer system. Patel, Figs 10, 4, para 27, 37, 62. The method 400 at 410 includes obtaining an image frame (e.g., an input image 220). The image frame comprises any digitized form of image or composite of images of at least one object such as, but not limited to, document pages, drawing sheets, scanned pages, photographs, illustrations, or video frames of a video stream or file. … In some embodiments, the object instance are generated by the first machine learning model from elements appearing in photographs or video frames in the input image such as people, faces, animals, vehicles, buildings, signage, or other visual elements that the first machine learning model is trained to detect and/or recognize. For example, in some embodiments, the user device 102 comprises a self-driving vehicle (vehicle computer system) and the application 110 uses the machine learning models 112 with the context based confidence calibrator 114 to identify traffic control signs (e.g., stop signs, speed limit signs), other vehicles, and/or pedestrians (at least one sensor). Patel, para 37, 27, 62. The computing device 1000, in some embodiments, is be equipped with depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, and combinations of these, for gesture detection and recognition (at least one sensor). Patel, Figs 10, 4, para 62, 37, 27.
It would have been obvious to one of ordinary skill in the art to implement the method for determining calibration mapping conversion rules of Kuppers using the computer system comprising a plurality of computer hardware components, vehicle computer system, and at least one sensor of Patel, with a reasonable expectation of success, in order to provide an operating environment for implementing aspects of the technology and software application that can then consider whether or not to use the prediction from the machine learning model from the confidence score. Patel, Abstract, Figs 10, 4, para 1, 27, 37, 62, 57-62. This would have provided the advantages of providing computer hardware components for implementation and machine learning application predictions.
Claim 17 recites a computer system that parallels the computer-implemented method of claim 12. Therefore, the analysis discussed above with respect to claim 12 also applies to claim 17. Accordingly, claim 17 is rejected based on substantially the same rational as set forth above with respect to claim 12.
Claim 18 recites a non-transitory computer readable medium that parallels the computer-implemented method of claim 1. Therefore, the analysis discussed above with respect to claim 1 also applies to claim 18. Accordingly, claim 18 is rejected based on substantially the same rational as set forth above with respect to claim 1. More specifically regarding A non-transitory computer readable medium storing instructions that, when executed by a computer, cause the computer to, Kuppers does not explicitly disclose a computer system comprising a plurality of computer hardware components.
However, Patel teaches in the field related to systems and methods for machine learning to draw an inference or make a prediction based on input data. Patel, Abstract, para 1. Patel is analogous to the claimed invention because Patel is directed to machine learning context based confidence calibration. Patel, Abstract, para 1, 2. Patel teaches that embodiments of the present disclosure can include elements comprising program instructions resident on computer readable media which when implemented by such computer systems, enable them to implement the embodiments described herein. As used herein, the terms “computer readable media , “computer readable medium”, and “computer storage media” refer to tangible memory storage devices having non-transient physical forms and includes both volatile and nonvolatile, removable and non-removable media (non-transitory computer readable medium storing instructions that, when executed by a computer, cause the computer to). Such non-transient physical forms can include computer memory devices, such as but not limited to: punch cards, magnetic disk or tape, or other magnetic storage devices, any optical data storage system, flash read only memory (ROM), non-volatile ROM, programmable ROM (PROM), erasable-programmable ROM (E-PROM), Electrically erasable programmable ROM (EEPROM), random access memory (RAM), CD-ROM, digital versatile disks (DVD), or any other form of permanent, semi-permanent, or temporary memory storage system of device having a physical, tangible form. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media. Computer storage media does not comprise a propagated data signal. Program instructions include, but are not limited to, computer executable instructions executed by computer system processors and hardware description languages such as Very High Speed Integrated Circuit (VHSIC) Hardware Description Language (VHDL). Patel, Fig 10, para 65.
It would have been obvious to one of ordinary skill in the art to implement the method for determining calibration mapping conversion rules of Kuppers using the non-transitory computer readable medium storing instructions that, when executed by a computer, cause the computer to of Patel, with a reasonable expectation of success, in order to provide for implementing aspects of the technology. Patel, Abstract, Fig 10, para 65, 57-62. This would have provided the advantages of providing program instructions and computer readable media for implementation.
Claim 19 recites a non-transitory computer readable medium that parallels the computer-implemented method of claim 12. Therefore, the analysis discussed above with respect to claim 12 also applies to claim 19. Accordingly, claim 19 is rejected based on substantially the same rational as set forth above with respect to claim 12.
Claim(s) 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over Kuppers and Patel as applied to claim 8 above, and further in view of Krishnan et al. (Pub. No. US 2021/0117760 A1, published April 22, 2021.
Regarding claim 10, which depends from claim 8 and recites:
wherein the fitting of the conversion rule approximates a curve based on a plurality of scores, wherein each score of the plurality of scores represents a statistical value of the plurality of statistical values and the corresponding sampling value of the plurality of sampling values.
Kuppers in view of Patel teaches the method of claim 8 from which claim 10 depends, including the fitting of the conversion rule, statistical value of the plurality of statistical values and corresponding sampling value of the plurality of sampling values. Kuppers teaches that To define a general calibration map for binary problems, we use the logistic function and the combined input s =(ˆ p, ˆr) of size K by
PNG
media_image6.png
30
106
media_image6.png
Greyscale
….In contrast to classifier calibration, we can use multivariate probability density functions to model f(s |m) and therefore include the bounding box information. Even though it is possible to evaluate the distribution parameters directly, we use a discriminative approach to obtain a better calibration mapping [16], Thus, the calibration process can then be posed as an optimization of a logistic regression problem (determining the conversion rule comprises a fitting of the conversion rule to the plurality of sampling values and the plurality of corresponding statistical values (fitting calibration mapping conversion rule g(s) via logistic regression of sampling values s and plurality of corresponding accuracy/precision surrogates statistical values). Kuppers, Section 3.1-3.2, p 1324.
Kuppers does not specifically disclose approximates a curve based on a plurality of scores.
However, Krishnan teaches in the field related to deep neural networks, and, more particularly, to methods and apparatus to obtain well-calibrated uncertainty in deep neural networks. Krishnan, para 1. Krishnan, which is analogous to the claimed invention because Krishnan is directed to methods and apparatus to obtain well-calibrated uncertainty in deep neural networks, teaches that, In some examples, such image processing can include scene classification, object detection and localization, semantic segmentation, and/or facial recognition. Krishnan, Figs 5, 17, para 47, 49. A well-calibrated model is expected to provide a consistently higher AvU AUC score even at increased levels of data-shift. In the example of FIG. 17, boxplots summarize results across 16 different data-shift types (e.g., including showing minimum, maximum, and quartiles) at each shift intensity level (1.g., 1-5). … Such a method can be compute intensive during training as AvU is computed at different thresholds (e.g., u.sub.th=u.sub.min+(t (u.sub.max−u.sub.min)), where t∈[0,1]. In some examples, optimizing the area under the curve can be performed for training the model and/or post-hoc calibration (approximates a curve based on a plurality of scores (post-hoc calibration optimizes and approximates a curve based on a plurality of data samples uncertainty statistical value scores)) on SVI (e.g., SVI-AUAvUC and/or SVI-AUAvUTS). Krishnan, Figs 5, 17, para 108, 47, 49.
It would have been obvious to one of ordinary skill in the art to implement the method for determining calibration mapping conversion rules of Kuppers using the computer hardware components of Patel and approximates a curve based on a plurality of scores of Krishnan, with a reasonable expectation of success, in order to provide an operating environment for implementing aspects of the technology and in order to provide for obtaining reliable and accurate quantification of uncertainty estimates from deep neural networks and incorporating such quantification into decision-making is essential for AI-based applications where safety is critical, including applications related to autonomous vehicles, robotics, and medical diagnosis. Krishnan, para 2. Patel, Abstract, Fig 10, para 57-62, This would have provided the advantages of providing computer hardware components for implementation and the advantages of obtaining more reliable and accurate information for decision making in applications where safety is critical.
Regarding claim 11, which depends from claim 8 and recites:
wherein each score comprises a minimum distance to the curve.
Kuppers in view of Patel teaches the method of claim 8 from which claim 11 depends. Kuppers does not specifically disclose wherein each score comprises a minimum distance to the curve.
However, Krishnan teaches in the field related to deep neural networks, and, more particularly, to methods and apparatus to obtain well-calibrated uncertainty in deep neural networks. Krishnan, para 1. Krishnan, which is analogous to the claimed invention because Krishnan is directed to methods and apparatus to obtain well-calibrated uncertainty in deep neural networks, teaches that, In some examples, such image processing can include scene classification, object detection and localization, semantic segmentation, and/or facial recognition. Krishnan, Figs 5, 17, para 47, 49. A well-calibrated model is expected to provide a consistently higher AvU AUC score even at increased levels of data-shift. In the example of FIG. 17, boxplots summarize results across 16 different data-shift types (e.g., including showing minimum, maximum, and quartiles) at each shift intensity level (1.g., 1-5). … Such a method can be compute intensive during training as AvU is computed at different thresholds (e.g., u.sub.th=u.sub.min+(t (u.sub.max−u.sub.min)), where t∈[0,1]. In some examples, optimizing the area under the curve can be performed for training the model and/or post-hoc calibration (wherein each score (data samples uncertainty statistical value scores) comprises a minimum distance (optimized minimum distance) to the curve) on SVI (e.g., SVI-AUAvUC and/or SVI-AUAvUTS). Krishnan, Figs 5, 17, para 108, 47, 49.
It would have been obvious to one of ordinary skill in the art to implement the method for determining calibration mapping conversion rules of Kuppers using the computer hardware components of Patel and the feature wherein each score comprises a minimum distance to the curve of Krishnan, with a reasonable expectation of success, in order to provide an operating environment for implementing aspects of the technology and in order to provide for obtaining reliable and accurate quantification of uncertainty estimates from deep neural networks and incorporating such quantification into decision-making is essential for AI-based applications where safety is critical, including applications related to autonomous vehicles, robotics, and medical diagnosis. Krishnan, para 2. Patel, Abstract, Fig 10, para 57-62, This would have provided the advantages of providing computer hardware components for implementation and the advantages of obtaining more reliable and accurate information for decision making in applications where safety is critical.
Allowable Subject Matter
Claims 7, 13-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims and if the rejections as being indefinite are overcome and if the rejections as being directed to an abstract idea are overcome.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US-20190130580-A1, US-20190258878-A1, US-20220414382-A1, US-20230071760-A1, US-20230196832-A1, US-20230213646-A1, US-20240112454-A1, US-20240403728-A1, US-20250086934-A1, US-12573190-B2, US-20260120439-A1, US-20260154933-A1, US-20130013542-A1.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BARBARA LEVEL whose telephone number is (303)297-4748. The examiner can normally be reached Monday through Friday 8:00 AM - 5:00 PM MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BARBARA M LEVEL/ Examiner, Art Unit 2142
Prosecution Insights