DETAILED ACTION
This action is in response to the application filed on 05/29/2024. Claims 1-15 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The information disclosure statement (IDS) submitted on 05/29/2024, 07/09/2024, 01/22/2026, 04/03/2026, and 05/01/2026 was filed. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-20 rejected under 35 U.S.C. 101 because they are directed to an abstract idea that does not amount to significantly more.
Regarding claim 1:
Subject Matter of Eligibility Analysis Step 1:
Claim 1 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 1 recites
training data which comprise training input data and training target values are provided, and the training data are used to adjust parameters of the machine learning system (this limitation is a mental process as it encompasses a human mentally using input data and target data as training data to adjust parameters).
generates reconstruction data which represent a measure of a familiarity of the training data (this limitation is a mental process as it encompasses a human mentally creating data that is similar to the training data).
Therefore, claim 1 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 1 further recites additional elements of
training data which comprise training input data and training target values are provided, and the training data are used to adjust parameters of the machine learning system (this element is merely data gathering, which is an insignificant extra-solution activity (see MPEP 2106.05(g))).
such that the machine learning system, when the training input data are input, generates output data corresponding to the training target values (this element does not integrate the abstract idea into a practical application because it amounts to mere instructions to apply (see MPEP 2106.05(f))).
Therefore, claim 1 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 1 do not provide significantly more than the abstract idea itself, taken alone and in combination because
training data which comprise training input data and training target values are provided, and the training data are used to adjust parameters of the machine learning system is well understood, routine, and conventional. The court has ruled that “Receiving or transmitting data over a network, e.g., using the Internet to gather data” is recognized as a computer function that is well‐understood, routine, and conventional (buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)).
such that the machine learning system, when the training input data are input, generates output data corresponding to the training target values is an instruction to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f).
Therefore, claim 1 is subject-matter ineligible.
Regarding claim 2:
Subject Matter of Eligibility Analysis Step 1:
Claim 2 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 2 recites
the reconstruction data correspond to the training input data (this limitation is a mental process as it encompasses a human mentally creating data that corresponds to another data).
Therefore, claim 2 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 2 does not further recite any additional elements. Therefore, claim 2 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 2 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 2 is subject matter ineligible.
Regarding claim 3:
Subject Matter of Eligibility Analysis Step 1:
Claim 3 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 3 recites
the reconstruction data correspond to the training input data (this limitation is a mental process as it encompasses a human mentally creating data that corresponds to another data).
a deviation of the reconstruction data from the training target values is less than a deviation of the output data from the training target values (this limitation is mathematical concept since it recites a mathematical relationship between deviations)
Therefore, claim 3 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 3 does not further recite any additional elements. Therefore, claim [] is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 3 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 3 is subject matter ineligible.
Regarding claim 4:
Subject Matter of Eligibility Analysis Step 1:
Claim 4 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 4 is dependent on claim 1, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 1 is applied here. Therefore claim 4 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 4 further recites additional elements of
at least one of the training input data or the output data are used as an input to generate the reconstruction data (this element does not integrate the abstract idea into a practical application because it amounts to mere instructions to apply (see MPEP 2106.05(f))).
Therefore, claim 4 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 4 do not provide significantly more than the abstract idea itself, taken alone and in combination because
at least one of the training input data or the output data are used as an input to generate the reconstruction data is an instruction to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f).
Therefore, claim 4 is subject-matter ineligible.
Regarding claim 5:
Subject Matter of Eligibility Analysis Step 1:
Claim 5 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 5 is dependent on claim 1, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 1 is applied here. Therefore claim 5 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 5 further recites additional elements of
the machine learning system is a neural network (this element does not integrate the abstract idea into a practical application because it is a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f))).
Therefore, claim 5 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 5 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the machine learning system is a neural network uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f))
Therefore, claim 5 is subject-matter ineligible.
Regarding claim 6:
Subject Matter of Eligibility Analysis Step 1:
Claim 6 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 6 is dependent on claim 1, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 1 is applied here. Therefore claim 6 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 6 further recites additional elements of
the input data comprise sensor data (this element does not integrate an abstract idea because it recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h))).
Therefore, claim 6 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 6 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the input data comprise sensor data recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h).
Therefore, claim 6 is subject-matter ineligible.
Regarding claim 7:
Subject Matter of Eligibility Analysis Step 1:
Claim 7 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 7 is dependent on claim 1, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 1 is applied here.
Claim 7 further recites
a metric, the reconstruction data and data corresponding to the reconstruction data are used to generate a deviation value, wherein the deviation value is a measure of the familiarity of the training data for the input data, quantifies the uncertainty in the output data and is assigned to the output data (this limitation is a mental process as it encompasses a human mentally quantifying uncertainty by calculating a deviation value, if the equation was given).
Therefore, claim 7 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 7 further recites additional elements of
the machine learning system generates the output data from the input data (this element does not integrate the abstract idea into a practical application because it amounts to mere instructions to apply (see MPEP 2106.05(f))).
the machine learning system generates the reconstruction data (this element does not integrate the abstract idea into a practical application because it amounts to mere instructions to apply (see MPEP 2106.05(f))).
Therefore, claim 7 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 7 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the machine learning system generates the output data from the input data is an instruction to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f).
the machine learning system generates the reconstruction data is an instruction to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f).
Therefore, claim 7 is subject-matter ineligible.
Regarding claim 8:
Subject Matter of Eligibility Analysis Step 1:
Claim 8 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 8 recites
the data corresponding to the reconstruction data are the input data (this limitation is a mental process as it encompasses a human mentally creating data that corresponds to another data).
Therefore claim 8 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 8 does not further recite any additional elements. Therefore, claim 8 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 8 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 8 is subject matter ineligible.
Regarding claim 9:
Subject Matter of Eligibility Analysis Step 1:
Claim 9 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 9 is dependent on claim 7, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 8 is applied here. Therefore claim 9 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 9 further recites additional elements of
the data corresponding to the reconstruction data are the output data generated from the input data (this element does not integrate an abstract idea because it recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h))).
Therefore, claim 9 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 9 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the data corresponding to the reconstruction data are the output data generated from the input data recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h).
Therefore, claim 9 is subject matter ineligible.
Regarding claim 10:
Subject Matter of Eligibility Analysis Step 1:
Claim 10 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Claim 10 recites
at least one of the input data or the output data generated from the input data are used to generate the reconstruction data (this limitation is a mental process as it encompasses a human mentally creating data based on an input or output data).
Therefore, claim 10 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 10 does not further recite any additional elements. Therefore, claim 10 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
Since there are no additional elements, claim 10 does not provide significantly more than the abstract idea itself, taken alone or in combination. Therefore, claim 10 is subject matter ineligible.
Regarding claim 11:
Subject Matter of Eligibility Analysis Step 1:
Claim 11 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 11 is dependent on claim 7, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 7 is applied here. Therefore claim 11 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 11 further recites additional elements of
the metric is only applied to a part of the reconstruction data and the data corresponding to the reconstruction data (this element does not integrate the abstract idea into a practical application because it amounts to mere instructions to apply (see MPEP 2106.05(f))).
Therefore, claim 11 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 11 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the metric is only applied to a part of the reconstruction data and the data corresponding to the reconstruction data is an instruction to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f).
Therefore, claim 11 is subject-matter ineligible.
Regarding claim 12:
Subject Matter of Eligibility Analysis Step 1:
Claim 12 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 12 is dependent on claim 7, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 7 is applied here. Therefore claim 12 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 12 further recites additional elements of
an uncertainty warning is output for the output data, the uncertainty of which exceeds a predetermined value (this element is merely data gathering, which is an insignificant extra-solution activity (see MPEP 2106.05(g))).
Therefore, claim 12 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 12 do not provide significantly more than the abstract idea itself, taken alone and in combination because.
an uncertainty warning is output for the output data, the uncertainty of which exceeds a predetermined value is well understood, routine, and conventional. The court has ruled that “Receiving or transmitting data over a network, e.g., using the Internet to gather data” is recognized as a computer function that is well‐understood, routine, and conventional (buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014)).
Therefore, claim 12 is subject matter ineligible.
Regarding claim 13:
Subject Matter of Eligibility Analysis Step 1:
Claim 13 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 13 is dependent on claim 7, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 7 is applied here. Therefore claim 13 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 13 further recites additional elements of
the input data are stored for further training of the machine learning system for the output data, the uncertainty of which exceeds a predetermined value (this element is merely data storing, which is an insignificant extra-solution activity (see MPEP 2106.05(g))).
Therefore, claim 13 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 13 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the input data are stored for further training of the machine learning system for the output data, the uncertainty of which exceeds a predetermined value is well understood, routine, and conventional. The court has ruled that “Storing and retrieving information in memory” is recognized as a computer function that is well understood, routine, and conventional (Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015)).
Therefore, claim 13 is subject-matter ineligible.
Regarding claim 14:
Subject Matter of Eligibility Analysis Step 1:
Claim 14 recites a system, which is directed to a machine, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 14 is dependent on claim 7, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 7 is applied here. Therefore claim 14 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 14 further recites additional elements of
an input interface having one or more inputs for receiving input data (this element does not integrate the abstract idea into a practical application because it is a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f))).
a computer having one or more inputs coupled to one or more outputs of the input interface, the computer being configured to execute the method according to Claim 7 (this element does not integrate the abstract idea into a practical application because it is a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f))).
an output interface having one or more inputs coupled to one or more outputs of the computer and one or more outputs which for output the output data generated by the computer as well as the uncertainty in at least one of the output data or an uncertainty warning (this element does not integrate the abstract idea into a practical application because it is a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f))).
Therefore, claim 14 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim [] do not provide significantly more than the abstract idea itself, taken alone and in combination because
an input interface having one or more inputs for receiving input data uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f))
a computer having one or more inputs coupled to one or more outputs of the input interface, the computer being configured to execute the method according to Claim 7 uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f))
an output interface having one or more inputs coupled to one or more outputs of the computer and one or more outputs which for output the output data generated by the computer as well as the uncertainty in at least one of the output data or an uncertainty warning uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f))
Therefore, claim 14 is subject-matter ineligible.
Regarding claim 15:
Subject Matter of Eligibility Analysis Step 1:
Claim 15 recites a system, which is directed to a machine, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 15 is dependent on claim 14, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 14 is applied here. Therefore claim 15 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 15 further recites additional elements of
A vehicle, comprising a system for quantifying uncertainties in output data according to Claim 14 (this element does not integrate the abstract idea into a practical application because it is a generic computing component on which to perform the abstract idea (see MPEP 2106.05(f))).
Therefore, claim 15 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 15 do not provide significantly more than the abstract idea itself, taken alone and in combination because
A vehicle, comprising a system for quantifying uncertainties in output data according to Claim 14 uses a computer as a tool to perform the abstract idea and cannot provide significantly more (see MPEP 2106.05(f))
Therefore, claim 15 is subject-matter ineligible.
Regarding claim 16:
Subject Matter of Eligibility Analysis Step 1:
Claim 16 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 16 is dependent on claim 5, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 5 is applied here. Therefore claim 16 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 16 further recites additional elements of
the neural network comprises a convolutional neural network (this element does not integrate an abstract idea because it recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h))).
Therefore, claim 16 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 16 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the neural network comprises a convolutional neural network recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h).
Therefore, claim 16 is subject-matter ineligible.
Regarding claim 17:
Subject Matter of Eligibility Analysis Step 1:
Claim 17 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 17 is dependent on claim 6, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 6 is applied here. Therefore claim 17 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 17 further recites additional elements of
the sensor data comprises at least one of image data, radar data or lidar data (this element does not integrate an abstract idea because it recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h))).
Therefore, claim 17 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 17 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the sensor data comprises at least one of image data, radar data or lidar data recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h).
Therefore, claim 17 is subject-matter ineligible.
Regarding claim 18:
Subject Matter of Eligibility Analysis Step 1:
Claim 18 recites a method, which is directed to a process, and thus is one of the four statutory categories of patentable subject matter.
Subject Matter of Eligibility Analysis Step 2A Prong 1:
Because claim 18 is dependent on claim 6, the Subject Matter of Eligibility Analysis Step 2A Prong 1 from claim 6 is applied here. Therefore claim 18 recites an abstract idea.
Subject Matter of Eligibility Analysis Step 2A Prong 2:
Claim 18 further recites additional elements of
the sensor data comprises data from one or more vehicle sensors (this element does not integrate an abstract idea because it recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h))).
Therefore, claim 18 is not integrated into a practical application.
Subject Matter of Eligibility Analysis Step 2B:
The additional elements of claim 18 do not provide significantly more than the abstract idea itself, taken alone and in combination because
the sensor data comprises data from one or more vehicle sensors recites a field of use limitation to apply a judicial exception (see MPEP 2106.05(h).
Therefore, claim 18 is subject-matter ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ceccaldi et al. (US 10997717 B2) (hereafter referred to as Ceccaldi) in view of Mostajabi et al. (Regularizing Deep Networks) (hereafter referred to as Mostajabi).
Regarding claim 1, Ceccaldi teaches
training a machine learning system to quantify uncertainties in output data (Ceccaldi, paragraphs 0007-0008, “The second deep learning model may be termed a usability check model because it generates an output confidence score which is a check for the first deep learning model. Using a second, independent deep learning model to check the first deep learning model. As explained in more detail below, the second deep learning model may provide a confidence score linked to the training data or a confidence score linked more directly to the first deep learning model. Both types of confidence scores may also be used. Such confidence scores may perform better and be more stable than other standard metrics such as prediction metrics which are generated for deep learning models. Furthermore, the generation (i.e. calculation) of such confidence scores means that the user can be more confident that the results from the first deep learning model can be trusted if the confidence score is high and within a predefined confidence interval”).
generates reconstruction data which represent a measure of a familiarity of the training data (Ceccaldi, paragraph 0008, “The same input image is processed by a second deep learning model which may be termed a usability check model (S102). The usability check model deconstructs the input image and then reconstructs the input image from the deconstructed result to form a reconstructed image. The next step is then to compare the reconstructed image to the input image (S104)”).
Ceccaldi does not teach, but Mostajabi does teach
training input data and training target values are provided, and the training data are used to adjust parameters of the machine learning system (Mostajabi, Section 3.1, “Specifically, as shown in Figure 1,we train an autoencoder on the ground-truth label maps… Ground-truth semantic segmentation label maps are simpler than real images, so this autoencoder need not have as high of a capacity as a network operating on natural images” and “The PASCAL dataset [12] serves as our experimental testbed” and “We use cross-entropy loss on auxiliary regularization branches” (Mostajabi, Section 4). Examiner notes that the PASCAL dataset maps to the training input data and the ground-truth labels map to the training target values).
when the training input data are input, generates output data corresponding to the training target values (Mostajabi, Figure 1)
PNG
media_image1.png
343
675
media_image1.png
Greyscale
Examiner notes that in figure 1, the image maps to the training input data, the auxiliary output maps to the output data, and the ground-truth annotations of training examples map to the training target values.
Ceccaldi and Mostajabi are considered analogous to the claimed invention because they deal with image analysis. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Ceccaldi to use the ground-truth data and PASCAL dataset from Mostajabi in order to calculate a confidence score. One of the ordinary skill in the art would have known to apply the known technique of using ground-truth data when calculating a confidence score from an input. Therefore, applying Mostajabi’s technique would yield the predicable result of determining how accurate an output is based on real-world examples (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predicable results.
Regarding claim 2, Ceccaldi and Mostajabi teach the method of claim 1, Ceccaldi further teaches
the reconstruction data correspond to the training input data (Ceccaldi, paragraphs 0008, “The same input image is processed by a second deep learning model which may be termed a usability check model (S102). The usability check model deconstructs the input image and then reconstructs the input image from the deconstructed result to form a reconstructed image”)
Regarding claim 3, Ceccaldi and Mostajabi teach the method of claim 1, Ceccaldi does note teach, but Mostajabi does teach
the reconstruction data correspond to the training input data, and a deviation of the reconstruction data from the training target values is less than a deviation of the output data from the training target values (Mostajabi, Figure 1)
PNG
media_image1.png
343
675
media_image1.png
Greyscale
Examiner notes that in figure 1, the image maps to the training input data, the primary output maps to the output data, and the ground-truth annotations of training examples map to the training target values.
“Table 2 demonstrates the necessity of our two-phase training procedure. If we unfreeze the decoder and update its parameters in the second training phase, test performance of the primary output deteriorates. Likewise, if we skip the first phase ,and train from scratch with an unfrozen, randomly initialized decoder, the accuracy gain disappears” (Mostajabi, Section 4.2). Examiner notes that frozen decoder causes the higher test performance, which maps to a lower deviation.
Ceccaldi and Mostajabi are considered analogous to the claimed invention because they deal with image analysis. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Ceccaldi to use the ground-truth data and PASCAL dataset from Mostajabi in order to calculate a confidence score. One of the ordinary skill in the art would have known to apply the known technique of using ground-truth data when calculating a confidence score from an input. Therefore, applying Mostajabi’s technique would yield the predicable result of determining how accurate an output is based on real-world examples (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predicable results.
Regarding claim 4, Ceccaldi and Mostajabi teach the method of claim 1, Ceccaldi further teaches
at least one of the training input data or the output data are used as an input to generate the reconstruction data (Ceccaldi, paragraph 0008, “The same input image is processed by a second deep learning model which may be termed a usability check model (S102). The usability check model deconstructs the input image and then reconstructs the input image from the deconstructed result to form a reconstructed image”).
Regarding claim 5, Ceccaldi and Mostajabi teach the method of claim 1, Ceccaldi does not teach, but Mostajabi does teach
the machine learning system is a neural network (Mostajabi, Section 3.2, “Hence, we choose hypercolumn [14, 29] CNN architectures as a primary basis for experimentation, as they are are minimally separated from the established classification networks in design space”).
Ceccaldi and Mostajabi are considered analogous to the claimed invention because they deal with image analysis. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Ceccaldi to use the hypercolumn CNN from Mostajabi. Mostajabi teaches “they are minimally separated from the established classification networks in design space. They also offer the added advantage of having readily available ImageNet pretrained models, easing experimentation in this setting” (Mostajabi, Section 3.2) (See MPEP 2141 (III)(G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention).
Regarding claim 7, Ceccaldi and Mostajabi teach the method of claim 1, Ceccaldi further teaches
the machine learning system generates the output data from the input data (Ceccaldi, claim 1, “receiving an input image and process the input image with a first deep learning model to generate an output result for the input image”).
the machine learning system generates the reconstruction data (Ceccaldi, paragraph 0008, “The same input image is processed by a second deep learning model which may be termed a usability check model (S102). The usability check model deconstructs the input image and then reconstructs the input image from the deconstructed result to form a reconstructed image”).
a metric, the reconstruction data and data corresponding to the reconstruction data are used to generate a deviation value, wherein the deviation value is a measure of the familiarity of the training data for the input data, quantifies the uncertainty in the output data and is assigned to the output data (Ceccaldi, paragraph 0009, “The comparison may include calculating a confidence score based on the differences between the reconstructed and input images. As explained above, such a confidence score is indicative of the likelihood that the main task model is reliable. The more different the reconstructed image is from the original input image, the lower the confidence score and hence there is less confidence in the main task model to produce good results and vice versa. Merely as examples, the confidence score may be calculated using mean square error (MSE) or structural similarity (SSIM) and suitable equations are set out below. However, it will be appreciated that any suitable method that provides a confidence score may be used”)
Regarding claim 8, Ceccaldi and Mostajabi teach the method of claim 7, Ceccaldi further teaches
the data corresponding to the reconstruction data are the input data (Ceccaldi, paragraph 0008, “The same input image is processed by a second deep learning model which may be termed a usability check model (S102). The usability check model deconstructs the input image and then reconstructs the input image from the deconstructed result to form a reconstructed image”).
Regarding claim 9, Ceccaldi and Mostajabi teach the method of claim 7, Ceccaldi does not teach, but Mostajabi does teach
the data corresponding to the reconstruction data are the output data generated from the input data(Mostajabi, Figure 1)
PNG
media_image1.png
343
675
media_image1.png
Greyscale
Examiner notes that in figure 1, the image maps to the training input data, the auxiliary output maps to the output data, and the ground-truth annotations of training examples map to the training target values.
Ceccaldi and Mostajabi are considered analogous to the claimed invention because they deal with image analysis. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Ceccaldi to use the ground-truth data and PASCAL dataset from Mostajabi in order to calculate a confidence score. One of the ordinary skill in the art would have known to apply the known technique of using ground-truth data when calculating a confidence score from an input. Therefore, applying Mostajabi’s technique would yield the predicable result of determining how accurate an output is based on real-world examples (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predicable results.
Regarding claim 10, Ceccaldi and Mostajabi teach the method of claim 7, Ceccaldi further teaches
at least one of the input data or the output data generated from the input data are used to generate the reconstruction data (Ceccaldi, paragraph 0008, “The same input image is processed by a second deep learning model which may be termed a usability check model (S102). The usability check model deconstructs the input image and then reconstructs the input image from the deconstructed result to form a reconstructed image”).
Regarding claim 12, Ceccaldi and Mostajabi teach the method of claim 7, Ceccaldi further teaches
an uncertainty warning is output for the output data, the uncertainty of which exceeds a predetermined value (Ceccaldi, paragraph 0011, “When it is determined that the output confidence score is outside a confidence interval, an alert, e.g. an audio/visual signal, may be output to the user”).
Regarding claim 14, Ceccaldi and Mostajabi teach the method of claim 7, Ceccaldi further teaches
an input interface having one or more inputs receiving input data (Ceccaldi, claim 15, “An image processing system comprising an image capture device configured to capture an image”).
a computer having one or more inputs coupled to one or more outputs of the input interface, the computer configured to execute the method according to Claim 7 (Ceccaldi, claim 15, “an image processor configured to receive said image from the image capture device as an input image, and to process said input image with a first deep learning model to generate an output result for the input image, and apply a second deep learning model to the input image to generate an output confidence score that is indicative of the reliability of the output result from the first deep learning model for the input image”).
an output interface having one or more inputs coupled to one or more outputs of the computer and one or more outputs which for outputs the output data generated by the computer as well as the uncertainty in at least one of the output data or an uncertainty warning (Ceccaldi, claim 15, “a user interface configured to display at least one of the output confidence score and the output result that are generated by the image processor”).
Regarding claim 16, Ceccaldi and Mostajabi teach the method of claim 5, Ceccaldi does not teach, but Mostajabi does teach
the neural network comprises a convolutional neural network (Mostajabi, Section 3.2, “Hence, we choose hypercolumn [14, 29] CNN architectures as a primary basis for experimentation, as they are are minimally separated from the established classification networks in design space”).
Ceccaldi and Mostajabi are considered analogous to the claimed invention because they deal with image analysis. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Ceccaldi to use the hypercolumn CNN from Mostajabi. Mostajabi teaches “they are minimally separated from the established classification networks in design space. They also offer the added advantage of having readily available ImageNet pretrained models, easing experimentation in this setting” (Mostajabi, Section 3.2) (See MPEP 2141 (III)(G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention).
Claim(s) 6, 15, 17, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ceccaldi in view of Mostajabi and Stocco et al. (Misbehavior Prediction for Autonomous Driving Systems) (hereafter referred to as Stocco).
Regarding claim 6, Ceccaldi and Mostajabi teach the method of claim 1, Ceccaldi and Mostajabi do not teach, but Stocco does teach
the input data comprise sensor data (Stocco, Section 1, “To test such complex software systems, companies perform a limited number of expensive in-field tests, driving a car on real world streets, or within closed-course testing facilities [13]. This provides detailed sensor data of the vehicle that are recorded, played back, and recreated within a simulator to obtain comprehensive test scenarios”).
Ceccaldi, Mostajabi, and Stocco are considered analogous to the claimed invention because they deal with image analysis. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Ceccaldi and Mostajabi to use sensor data from vehicles from Stocco. Stocco teaches “Self-driving cars (SDC, hereafter) have benefited from many technological advancements both in hardware and in software. Data gathered by LIDAR sensors, cameras, and GPS are analyzed in real time by advanced DNNs which govern over the actual maneuvers of the car (i.e., steering, braking, acceleration). In order to manage a wide variety of driving scenarios, SDCs necessitate a large amount of driving data, combining nominal and adversarial scenarios” (Stocco, Section 2) (See MPEP 2141 (III)(G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention).
Regarding claim 15, Ceccaldi and Mostajabi teach the system of claim 14, Ceccaldi and Mostajabi do not teach, but Stocco does teach
A vehicle, comprising a system for quantifying uncertainties in output data according to Claim 14 (Stocco, Section 1, “In this paper, we tackle the self-assessment oracle problem for autonomous driving system, i.e., the problem of monitoring the confidence level of a DNN-based autonomous driving system in order to timely predict the occurrence of future misbehaviours”).
Ceccaldi, Mostajabi, and Stocco are considered analogous to the claimed invention because they deal with image analysis. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Ceccaldi and Mostajabi to use apply their methods on an autonomous driving system from Stocco. Stocco teaches “The problem is critical because a failure in detecting an unexpected condition may have severe consequences (i.e., a fatal crash), whereas false alarms, even if not dangerous, may cause driver’s discomfort and negatively affect the driving experience. Creating a self-assessment oracle that evaluates the confidence of a DNN at runtime, and predicts whether the system is within a low-confidence zone, is a largely unexplored research problem” (Stocco, Introduction) (See MPEP 2141 (III)(G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention).
Regarding claim 17, Ceccaldi and Mostajabi teach the method of claim 6, Ceccaldi and Mostajabi do not teach, but Stocco does teach
the sensor data comprises at least one of image data, radar data or lidar data (Stocco, Section 2, ““Self-driving cars (SDC, hereafter) have benefited from many technological advancements both in hardware and in software. Data gathered by LIDAR sensors, cameras, and GPS are analyzed in real time by advanced DNNs which govern over the actual maneuvers of the car (i.e., steering, braking, acceleration)”).
Ceccaldi, Mostajabi, and Stocco are considered analogous to the claimed invention because they deal with image analysis. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Ceccaldi and Mostajabi to use sensor data from vehicles from Stocco. Stocco teaches “In order to manage a wide variety of driving scenarios, SDCs necessitate a large amount of driving data, combining nominal and adversarial scenarios” (Stocco, Section 2) (See MPEP 2141 (III)(G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention).
Regarding claim 18, Ceccaldi and Mostajabi teach the method of claim 6, Ceccaldi and Mostajabi do not teach, but Stocco does teach
the sensor data comprises data from one or more vehicle sensors (Stocco, Section 2, ““Self-driving cars (SDC, hereafter) have benefited from many technological advancements both in hardware and in software. Data gathered by LIDAR sensors, cameras, and GPS are analyzed in real time by advanced DNNs which govern over the actual maneuvers of the car (i.e., steering, braking, acceleration)”).
Ceccaldi, Mostajabi, and Stocco are considered analogous to the claimed invention because they deal with image analysis. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Ceccaldi and Mostajabi to use sensor data from vehicles from Stocco. Stocco teaches “In order to manage a wide variety of driving scenarios, SDCs necessitate a large amount of driving data, combining nominal and adversarial scenarios” (Stocco, Section 2) (See MPEP 2141 (III)(G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ceccaldi in view of Mostajabi and Tommi et al. (US 20190258878 A1) (hereafter referred to as Tommi).
Regarding claim 11, Ceccaldi and Mostajabi teach the method of claim 7, Ceccaldi and Mostajabi do not teach, but Tommi does teach
the metric is only applied to a part of the reconstruction data and the data corresponding to the reconstruction data (Tommi, paragraph 0037, “To detect objects represented in an image, a conventional system may use a convolutional neural network (CNN) that provides coverage values and bounding boxes for a grid of spatial element regions of the image. Each coverage value may represent a likelihood that an object is depicted at least partially in a corresponding spatial element region...The detections may be aggregated to particular objects by clustering the detections, and a confidence value may be assigned to each aggregated detection”).
Ceccaldi, Mostajabi, and Tommi are considered analogous to the claimed invention because they deal with image analysis. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Ceccaldi and Mostajabi to apply their methods to different spatial element regions from Tommi. One of the ordinary skill in the art would have known to apply the known technique of generating a confidence score for different regions of the data. Therefore, applying Tommi’s technique would yield the predicable result of determining a machine learning model’s confidence level on different areas of an image (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predicable results.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ceccaldi in view of Mostajabi and Feng et al. (US 10417524 B2) (hereafter referred to as Feng).
Regarding claim 13, Ceccaldi and Mostajabi teach the method of claim 7, Ceccaldi and Mostajabi do not teach, but Feng does teach
the input data are stored for further training of the machine learning system for the output data, the uncertainty of which exceeds a predetermined value (Feng, claim 13, “determining an uncertainty measure for each classified image indicating a confidence of the classifier that the classified image is the positive image or the negative image” and “adding…the subset of the labeled classified images into the set of labeled images…and retraining the classifier with the updated set of labeled images” (Feng, claim 14)).
Ceccaldi, Mostajabi, and Feng are considered analogous to the claimed invention because they deal with image analysis. It would have been obvious to one having ordinary skill in the art prior to the effective filling date to have modified Ceccaldi and Mostajabi to retain the labeled classified image from Feng. One of the ordinary skill in the art would have known to apply the known technique of retraining models with the initial input. Therefore, applying Feng’s technique would yield the predicable result of machine learning model self-improving (See MPEP 2141 (III)(D) Applying a known technique to a known device ready for improvement to yield predicable results.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Gauerhof et al. (US 12125268 B2) discloses testing the robustness of an artificial neural network by training reconstruction system.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEVEN VO whose telephone number is (571)272-9622. The examiner can normally be reached Monday - Friday from 7-3 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.V./Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights