DETAILED ACTION
This action is responsive to the application filed on 11/06/2025. Claims 1-11, 13-25 are pending and have been examined. This action is Non-final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C.
120, 121, 365(c), or 386(c) is acknowledged.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/06/2025 has been entered.
Response to Arguments
Argument 1: The applicant argues that the claims are not mental processes because, in practice, a human could not possibly carry out the recited steps, especially while a vehicle is driving. They point to claim 1 as an example: it requires receiving dense sensor data from LIDAR and cameras, determining spatial locations of many surrounding objects using bounding boxes and semantic segmentation into classes like vehicles, pedestrians, and animals, assigning care or no-care attributes to each label based on combined radar and auxiliary data, generating model predictions, and applying a detailed loss function with positive and negative contributions and confidence thresholds. The applicant says there is far too much data and too many calculations for a person with pen and paper to perform in a useful time frame, so these steps must be done by a computer system in a moving vehicle. They also cite the specification’s statement that reliable labels cannot be derived from radar data directly by humans or by another algorithm, and that LIDAR-based labels are used as cross-domain ground truth, to show that the invention is tied to specific sensor processing in an operating vehicle, not mental reasoning. On this basis, the applicant asks that the 101 rejection be withdrawn and notes that new claims 21 to 23 add further subject-matter-eligible features beyond claims 1, 11, and 20.
Examiner Response to Argument 1: The examiner has considered the elements set forth above, however, applicant’s arguments do not overcome the rejection under 101 because, as mapped, the amended claims remain directed to mental processes and mathematical concepts implemented on generic sensors and computer hardware, with only field-of-use and data-gathering limitations. In claim 1 (and similarly in claims 11 and 20), the core steps are identifying labels based on dense auxiliary data and “at least one property of entities,” where that property is expressly defined as i) generating bounding boxes around objects and ii) performing semantic segmentation into object classes such as vehicle, pedestrian, and animal, and then assigning each label a care or no-care attribute by determining a perception capability and comparing a reference value to a threshold, and under the mapping this is evaluation, classification, and labeling of information using observation and rule-based decision-making that can, in principle, be done by a person with pen and paper (for example, drawing boxes around objects, naming them, and marking them care/no-care depending on whether a score exceeds a threshold), so it is a mental process. Likewise, defining a loss function that receives positive and negative loss contributions so that weights are increased or decreased depending on whether they contribute constructively or not, permitting negative contributions for all labels, permitting positive contributions for labels with a care attribute, and permitting positive contributions for labels with a no-care attribute only when a confidence value exceeds a predetermined threshold are mental processes and same for if/then conditions on numerical values (predictions, labels, weights, confidence and thresholds) that the mapping identifies as mental processes that could also be worked out on paper for a small set of values, and mental processes is also “generating model predictions for the labels”. The dependent claim mappings further show that the additional features consist of more mental and mathematical steps, such as determining a numerical reference value from radar energy in a spatial area (claim 3 / 14), computing ranges and angles from radar data and assigning them to spatial areas to decide care or no-care (claim 4 / 15), estimating expected range, range rate, and angle from dense auxiliary data and assigning those expected values to radar-derived values (claim 5 / 16), estimating range rate from a speed vector computed as differences of label positions over time (claim 6 / 17), selecting subsets of auxiliary data points in a spatial area, determining whether a direct line of sight exists, and assigning care when a ratio of counts exceeds a threshold (claim 7 / 10), regarding a point as having line of sight when it lies within a field of view of one of multiple radar sensors (claim 8 / 18), and projecting selected points to a cylinder or sphere, dividing the surface into pixel areas, marking the closest point in each pixel as visible, counting visible points, and comparing that count to a visibility threshold (claim 9 / 19), in each instance the mapping explains that these are human-mind-capable acts of selecting, calculating, projecting, counting and comparing that can be done with pen and paper, so they, too, are mental processes or mathematical concepts. Step 2A prong 2 and Step 2B in the mapping make clear that reciting that these abstract steps are “implemented within a vehicle while the host vehicle is driving” and that the data are captured by at least one radar primary sensor and at least one lidar / camera auxiliary sensor merely identifies the environment and the generic sources of data and is treated as insignificant extra-solution activity or field-of-use limitation, and similarly that performing the analysis “via a machine-learning algorithm” or on a generic processing unit is recited at a high level of generality, with no specific improvement to computer function, and therefore does not integrate the exception into a practical application or add significantly more. The mapping for claim 11 confirms that the system claim simply recites a primary radar sensor and auxiliary lidar/camera sensor plus a processing unit configured to perform the same abstract analysis, which is just a generic machine implementation of the method. Finally, the mappings for new claims 21-25 show that these claims add only further mathematical concepts and field-of-use language: basing actions “on the loss function,” adjusting training parameters based on the loss function, controlling adjustments via an error-based loss function comparing predictions and target outputs, and, in claims 21, 24, and 25, broadly stating that, based on the loss function, the method or system may assist in or autonomously drive the vehicle, these are all framed at a results-oriented, functional level with no added technical detail about how vehicle control is implemented, and the mapping therefore reasonably characterizes them as either mental / mathematical operations or merely limiting the abstract idea to the driving environment under MPEP 2106.05(f) and (h). With respect to applicant's remarks that claims 22 and 23 were added in response to a suggestion made in the October 16, 2025 interview and therefore "should contain statutory subject matter," this has been considered but is not persuasive. The suggestion concerned providing more detail on training. The language actually added in claims 22 and 23, however, merely describes generic supervised training: adjusting parameters of the machine-learning algorithm based on a loss function, and defining that loss function as an error signal from comparing predictions to a target output representative of the labels. Under their broadest reasonable interpretation and as mapped, these limitations recite the standard supervised-learning paradigm of computing an error between predictions and labels and updating parameters to reduce that error, which is itself a mathematical concept and mental process and does not add any particular technological implementation or improvement to computer functioning. Thus, although claims 22 and 23 respond to the interview suggestion by adding training-related language, they do not change the character of the claim set away from an abstract idea. Accordingly, even accepting applicant’s assertion that a human driver practically could not carry out all of these calculations in real time, the claims as written remain, under their broadest reasonable interpretation, directed to mental processes and mathematical concepts executed on generic sensors and processors with only insignificant data gathering and field-of-use limitations, so the rejection of all pending claims under 101 is properly maintained.
Argument 2: Applicant argues that Musk does not teach the claimed “perception capability” based care/no-care assignment because Musk’s sensor configuration and thresholding are different from what is recited in the claims. In the present claims, the primary sensor is explicitly a radar sensor and the auxiliary sensor is a LIDAR and/or camera, and the care or no-care attribute is assigned based on a reference value computed from sparse primary (radar) data for each spatial area and compared to a reference threshold. Applicant points out that in Musk the camera functions as the primary vision sensor and the radar and LIDAR act as auxiliary sensors whose data is associated with the camera image. According to applicant, Musk’s “threshold value” is applied to decide whether auxiliary sensor data is reliable enough to be used as ground truth for the camera-based model, and thus the threshold concerns auxiliary data, not a perception capability of a primary radar sensor per label or spatial area. For this reason, applicant contends that Musk does not teach the claimed step of determining a perception capability of the primary sensor from sparse primary radar data, computing a reference value per spatial area, and assigning care versus no-care based on that primary-sensor reference value.
Examiner Response to Argument 2: The examiner has considered the argument set forth above, but it is not persuasive. The examiner has considered the argument set forth above, but it is not persuasive. As an initial matter, applicant’s emphasis on “sparse” primary data and “dense” auxiliary data is not persuasive, because “sparse” and “dense” are relative terms of degree that do not provide a clear boundary for the scope of the claims. A rejection under 35 U.S.C. 112(b) will be made in this RCE for these terms as being indefinite, and, consistent with that, the “sparse” versus “dense” characterization does not carry patentable weight in the present 101 analysis. Applicant’s position also relies on importing the exemplary embodiment from paragraph 0022 of the specification (for example, “average of an intensity of the primary data”) into the claims. However, the claims merely recite that “the sparse primary data [is] usable to determine a reference value for a respective spatial area” and that a “perception capability” is used to assign care or no-care, without requiring any particular formula or that the reference value be an explicit average intensity. Under a broadest reasonable interpretation, a “perception capability” is simply a measure of how reliably the primary sensor can perceive a given label in its spatial area, and a “reference value” can be any scalar quality or certainty measure derived from the primary sensor data in that area. Musk teaches this same structure by using a radar sensor to emit radar and “identify the distance and direction of surrounding obstacles,” correlating these measurements to objects in the camera image, and then using “a threshold value… to determine whether to associate an object property as a ground truth of an identified object,” where “related data with a high degree of certainty is associated with an identified object while related data with a degree of certainty below a threshold value is not associated with the identified object.” For each object region in the camera image (that is, the spatial area or bounding box of the label), Musk derives from radar and other sensor data a certainty or reliability and compares it to a threshold to decide whether that object’s sensor measurement is used or ignored for training. The examiner interprets this certainty score as the claimed “reference value” for the spatial area and the threshold comparison as determining the primary sensor’s “perception capability” for that label, resulting in the same effect as assigning a “care” attribute when the reference value exceeds the threshold and a “no-care” attribute otherwise. Applicant further argues that Musk’s “primary” sensor is a camera and that radar and lidar are “auxiliary,” so Musk allegedly does not determine a perception capability of a primary radar sensor as claimed. This is also not persuasive. Musk expressly states that the “captured data includes vision data (such as video and/or still images) and additional auxiliary data such as radar, lidar, inertia, audio, odometry, location, and/or other forms of sensor data,” and further discloses that “in some embodiments, the related data may be conflicting sensor data. For example, ultrasonic and radar data output may conflict. In various embodiments, a threshold value is used to determine whether to associate an object property as a ground truth of an identified object” (citations omitted for brevity). Thus Musk teaches using both radar and lidar together and comparing different sensor modalities via a threshold to decide which measurement to trust. The labels “primary” and “auxiliary” in Musk are functional and arbitrary. It’s obvious to designate radar as the primary sensor and lidar and/or camera as auxiliary sensors while applying the same thresholding and conflict-resolution logic to determine which sensor has better perception capability for each object and spatial area. Moreover, even in the configuration where the camera is treated as “primary” in Musk, any threshold-based decision that compares radar and lidar confidence to decide whether to accept a given object property as ground truth is still, in substance, a determination about how well the sensor under evaluation can perceive that object in that area. In other words, measuring the auxiliary sensor’s confidence and using that to decide whether to trust or discard an object property associated with the primary image is still a determination of perception capability for the sensor whose performance in that region is being evaluated. Accordingly, under a broadest reasonable interpretation of “primary sensor” and “perception capability,” Musk teaches determining a perception capability from sensor data on a per-label spatial area and applying a threshold to that reference value, even if Musk does not use the exact terminology of the instant application or uses a different naming convention for primary versus auxiliary sensors. The examiner maintains their position.
Argument 3: Applicant argues that Northcutt does not teach the claimed care/no-care loss behavior. Northcutt appears to provide a different treatment for positive and negative contributions to a loss function, this treatment does not depend on a further attribute previously provided.
Examiner Response to Argument 3: The examiner has considered the argument set forth above, but it is not persuasive. The examiner asserts that it is unclear which specific limitation is being referred to, as none recites anything about “different treatment” or “treatment … depend on a further attribute previously provided”. The examiner further asserts that Northcutt does teach the limitation about positive and negative contributions to the loss function based on confidence (see mapping below), and arguments about Northcutt not teaching that the confidence is based on a perception capability of a sensor amounts to attacking references individually when a rejection is based on a combination of references. The examiner maintains their position.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AlA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
Claim 11: “a processing unit configured to be used by the machine-learning algorithm” invokes 112(f).
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AlA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
The specification mentions corresponding structure described in the specification. Processing unit 17 teaches the underlying structure of the “means configured to be used by the machine-learning algorithm” in fig. 2 paragraph [0048] “FIGS. 1 and 2 depict a host vehicle 11 which includes radar sensors 13 (see FIG. 2 ) and a LIDAR system 15 which are in communication with a processing unit 17. As shown in FIG. 1 , other vehicles are located in the environment of the host vehicle 11. The other vehicles are represented by bounding boxes 19 which are also referred to as labels 19 since these bounding boxes are provided based on data from the LIDAR system 15 for training a machine-learning algorithm. The training of the machine-learning algorithm is performed via the processing unit 17 (which also executes the algorithm itself) and uses primary data provided by the radar sensors 13.”
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AlA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AlA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AlA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing ou and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-25 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention
The claims recite, for example, “sparse primary data” and “dense auxiliary data” (see, ex. claim 1 and claims depending therefrom (sparse: 1,4,5, 10, 11, 15-16, and 20; Dense: 1, 5,6, 10, 11,16-20)). The terms “sparse” and “dense” are relative terms of degree. The claims do not provide any objective boundary for determining when sensor data is “sparse” versus “dense,” and the specification does not set forth any clear standard, threshold, or quantitative criterion that would allow one of ordinary skill in the art to determine with reasonable certainty whether a given instance of sensor data falls within or outside the scope of these terms. Instead, “sparse” and “dense” are used qualitatively and subjectively, and their meaning depends on unspecified factors such as sensor resolution, sampling rate, or point density, which can vary widely across implementations.
While relative terms can in some cases be definite when the specification or the state of the art provides an accepted standard, here there is no such objective standard identified for “sparse primary data” and “dense auxiliary data.” As a result, one of ordinary skill in the art cannot ascertain with reasonable certainty the metes and bounds of the claimed subject matter based on these terms. Accordingly, the recitation of “sparse primary data,” “dense auxiliary data,” and similar “sparse”/“dense” formulations renders the scope of the claims indefinite under 35 U.S.C. 112(b).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition
of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the
conditions and requirements of this title.
Claims 1-11, 13-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1, (similar to 11 and analogous to 20)
Step 1: The claim is directed to a method, which is considered to be a process, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“determine at least one property of entities in an environment of the at least one primary sensor… identifying labels based on the dense auxiliary data, the identifying labels comprising determining a respective spatial area to which each label is related;… assigning at least one of a care attribute or a no-care attribute to each identified label by determining a perception capability of the at least one primary sensor for the respective label…the primary data usable to determine a reference value for a respective spatial area and, for each label, the care attribute is assigned to the respective label if the reference value is greater than a reference threshold and the no-care attribute is assigned to the respective label if the reference value is smaller than or equal to the reference threshold;” – The limitation is directed to determining entities of an environment from a sensor, identifying labels from data, assigning an attribute to the label once determining the capability of the primary sensory, and further elements for which all that is recited above is directed to a process that can be performed in the human mind using evaluation, observation and/or judgment with the aid of pen and paper, and thus is directed to a mental process.
“defining a loss function for the model predictions, wherein the loss function receives a positive loss contribution for which weights of a model on which the machine-learning algorithm relies are increased if the weights contribute constructively to a prediction corresponding to the respective label and a negative loss contribution for which weights of the model are decreased if the weights contribute constructively to a prediction not corresponding to the respective label;”—This limitation is directed to defining a loss function for predictions, where a positive or negative loss contribution will be received once its determined that the weights will contribute the prediction constructively, then the weights are increased or decreased, which is directed to a process that can be performed in the human mind using evaluation, observation and/or judgment with the aid of pen and paper, and thus is directed to a mental process.
“permitting negative contributions to the loss function for all labels; permitting positive contributions to the loss function for labels having a care attribute; and permitting positive contributions to the loss function for labels having a no-care attribute only if a confidence value of the model prediction for the respective label is greater than a predetermined threshold.” – This limitation is directed to permitting negative or positive contributions to the loss function for the labels/attribute, and comparing attributes that have a confidence value to a predetermined threshold, which is all directed to a process that can be performed in the human mind using evaluation, observation and/or judgment with the aid of pen and paper, and thus is directed to a mental process.
“generating model predictions for the labels” – The limitation is directed to generating mode predictions for the labels, which is directed to a process that can be performed in the human mind using evaluation, observation and/or judgment with the aid of pen and paper, and thus is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“process primary data captured by at least one primary sensor…receiving dense auxiliary data from at least one auxiliary sensor” –This limitation recites processing data that is obtained and then receiving data from a sensor, which is all considered to be directed mere data gathering and obtaining of data to be manipulated, which is considered to be an insignificant-extra solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under step 2B, the act of receiving/sending data over a network is a well-understood, routine, and conventional activity (WURC), and cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“via a machine learning algorithm” -- This limitation recites that the mental process of generating model predictions (see prong 1) will be performed via a machine learning algorithm, however this limitation is mere instructions to implement the mental process on a generic computer, because machine learning algorithm is recited broadly at a high level of generality, and thus cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 1 non-patent eligible. Claims 11 and 20, are analogous to claim 1, and thus will face the same rejection as above granted that the limitations are very similar, aside from being different type of claims (method vs system vs non-transitory CRM).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition
of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the
conditions and requirements of this title.
Claims 1-11, 13-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1, (similar to 11 and analogous to 20)
Step 1: The claim is directed to a method, which falls under the category of a process. The claim satisfies Step 1.
Step 2A Prong 1:
“identifying labels based on the dense auxiliary data and the at least one property of entities, the identifying labels comprising determining a respective spatial area to which each label is related, wherein the at least one property of entities comprises i) a spatial location of objects surrounding the vehicle by generating bounding boxes which enclose the objects, respectively, and ii) a semantic segmentation including assignment of the objects surrounding the vehicle to respective object classes, the object classes including other vehicle, pedestrian, and animal; assigning at least one of a care attribute or a no-care attribute to each identified label by determining a perception capability of the at least one primary sensor for the respective label based on the sparse primary data captured by the at least one primary sensor and based on the dense auxiliary data captured by the at least one auxiliary sensor, the sparse primary data usable to determine a reference value for a respective spatial area and, for each label, the care attribute is assigned to the respective label if the reference value is greater than a reference threshold and the no-care attribute is assigned to the respective label if the reference value is smaller than or equal to the reference threshold;” -- The limitation is directed to analyzing information about objects in the environment (now expressly including generating bounding boxes and performing semantic segmentation into object classes such as vehicle, pedestrian, and animal), determining spatial areas for labels, and assigning each label a care or no-care attribute based on a perception capability and a comparison of a reference value to a threshold. Under a broadest reasonable interpretation, this is evaluation, classification, and labeling of information using observation, reasoning, and rule-based decision-making (e.g., deciding which objects in a scene are important based on a score and a threshold). Such steps can be performed in the human mind, with the aid of pen and paper (for example, looking at a scene, drawing boxes around objects, labeling them by type, and marking them as care/no-care depending on whether an associated value exceeds a threshold). Thus, this limitation is directed to a mental process.
“defining a loss function for the model predictions, wherein the loss function receives a positive loss contribution for which weights of a model on which the machine-learning algorithm relies are increased if the weights contribute constructively to a prediction corresponding to the respective label and a negative loss contribution for which weights of the model are decreased if the weights contribute constructively to a prediction not corresponding to the respective label;” – This limitation is directed to defining and evaluating a loss function for predictions and adjusting weights of a model based on whether they contribute constructively or not. This is a mathematical operation on numerical values (predictions, labels, and weights) and can ALSO be performed in the human mind or using pen and paper for a small set of weights. Thus, this limitation is directed to a mental process / mathematical concept.
“permitting negative contributions to the loss function for all labels; permitting positive contributions to the loss function for labels having a care attribute; and permitting positive contributions to the loss function for labels having a no-care attribute only if a confidence value of the model prediction for the respective label is greater than a predetermined threshold.” – This limitation is directed to applying logical conditions (if/then rules) to decide whether certain positive or negative contributions are included in the loss function based on label attributes and a comparison of a confidence value to a threshold. These are conditional mathematical operations and comparisons that can be performed in the human mind using evaluation, observation, and judgment with the aid of pen and paper. Thus, this limitation is directed to a mental process.
“generating model predictions for the labels” – This limitation is directed to generating predictions for labels, i.e., applying a model or rule to input information to obtain outputs. Conceptually, a person could perform this step by applying a decision rule or formula on paper to “predict” a label outcome. Thus, this limitation is also directed to mathematical concept. Further support is presented in the instant application, [0034] “to generate model predictions for the labels via the machine-learning algorithm, to define a loss function for the model predictions, to permit negative contributions to the loss function for all labels,”.
Step 2A Prong 2 and Step 2B:
“A method for training a machine-learning algorithm configured to process sparse primary data captured by at least one primary sensor in order to determine at least one property of entities in an environment of the at least one primary sensor, the method implemented within a vehicle while the host vehicle is driving and comprising: receiving dense auxiliary data from at least one auxiliary sensor, wherein the at least one auxiliary sensor comprises at least one of a light ranging and detection (LIDAR) sensor and a camera, and wherein the at least one primary sensor comprises at least one radar sensor;” – This limitation recites that the abstract analysis is carried out in the context of a vehicle while it is driving, using specific sensors (radar as the primary sensor and LIDAR/camera as auxiliary sensors) to capture sparse primary data and dense auxiliary data. The limitation is directed to an insignificant, extra-solution activity that cannot be integrated to a practical application (see MPEP 2106.05(g)). Furthermore, under Step 2B, the act of sending/receiving data over a network is a well-understood, routine, and conventional activity (WURC), that cannot provide significantly more than the judicial exception (see MPEP 2106.05(d)(II)).
“via a machine-learning algorithm” –The limitation recites that the model predictions are generated using a machine-learning algorithm, but the algorithm is recited at a high level of generality without any particular technical implementation or improvement to computer functionality, and does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
Therefore, claim 1 is non-patent eligible.
Regarding claim 2, (analogous to claim 13)
Step 1: The claim is directed to a method, which is considered to be a process, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“The method according to claim 1, wherein the predetermined threshold for the confidence value is zero.” – The limitation is directed to the predetermined threshold for the confidence value first introduced in claim 1 is set to zero, which can be evaluated and determined by the human mind to be valued at zero, and thus is directed to a mental process.
There are no elements to be evaluated under step 2A Prong 2 and Step 2B.
Thus, claim 2 non-patent eligible. Claim 13 is analogous to claim 2, and thus will face the same rejection as above granted that the limitations are very similar, aside from being different type of claims (method vs system).
Regarding claim 3, (analogous to claim 14)
Step 1:The claim is directed to a method, which is considered to be a process, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“wherein the reference value is determined based on radar energy detected by the radar sensor within the spatial area to which the respective label is related.” – The limitation is directed to determining a numerical reference value from radar energy detected in a spatial area associated with a label. Under a broadest reasonable interpretation, this is calculating or deriving a value from collected sensor data and associating that value with a region/label. Such operations (deriving a value from measurements and assigning it to a labeled region) can be performed in the human mind using evaluation, calculation, and judgment with the aid of pen and paper, and thus are directed to a mental process.
There no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 3 is non-patent eligible. Claim 14 is analogous to claim 3, and therefore will face the same rejection as above granted that the limitations are very similar, aside from being different type of claims (method vs system).
Regarding claim 4, (analogous to claim 15)
Step 1: The claim is directed to a method, which is considered to be a process, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“The method according to claim 3, wherein: ranges and angles at which radar energy is perceived are determined based on the sparse primary data captured by the radar sensor; and the ranges and angles are assigned to the spatial areas” – The limitation is directed to determining ranges and angles (for which can be calculated/generated using a pen and paper) that derives from data that was obtained by the radar sensor, and assigning the ranges and angles to spatial areas, for which can be done using observation and judgement, and thus is considered a mental process.
“the respective labels are related in order to determine the at least one of the care attribute or the no-care attribute for each label.” – The limitation is directed to related labels for determining if an attribute is classified as care or non-care per label, which is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 4 is non-patent eligible. Claim 15 is analogous to claim 4, and thus will face the same rejection as above granted that the limitations are very similar, aside from being different type of claims (method vs system).
Regarding claim 5, (analogous to claim 16)
Step 1: The claim is directed to a method, which is considered to be a process, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“an expected range, an expected range rate and an expected angle are estimated for each label based on the dense auxiliary data;” – The limitation is directed to estimating expected range, its rate and the angle for the labels based on aux data. Estimating expected values for labels based on data is directed to a process that can be performed in the human mind using pen and paper, and thus it’s directed to a mental process.
“the expected range, the expected range rate and the expected angle of the respective label are assigned to a range, a range rate and an angle derived from the sparse primary data of the radar sensor in order to determine the radar energy associated with the respective label.” – The limitation is directed to assigning the expected range, rate and angle with the respective labels to a what is derived from primary data of the radar sensor to determine the radar energy that is associated with a respective label. The act of assigning range rates and angles’ labels to a range, rate and label is directed to a process that can be performed in the human mind using evaluation, observation and/or judgment with the aid of pen and paper, and thus is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 5 is non-patent eligible. Claim 16 is analogous to claim 5, and thus will face the same rejection as above granted that the limitations are very similar, aside from being different type of claims (method vs system).
Regarding claim 6, (analogous claim 17)
Step 1: The claim is directed to a method, which is considered to be a process, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“The method according to claim 5, wherein the expected range rate is estimated for each label based on a speed vector which is estimated for a respective label” – The limitation is directed to estimating the range rate for the labels based on an estimated speed vector for the respective label. Estimating range rates based on an estimated speed vector for labels is directed a process that can be performed in the human mind using pen and paper, and thus it’s directed to a mental process.
“by using differences of label positions determined based on the dense auxiliary data at different points in time.” – The limitation is directed to determining label positions based on data in varied times, which under broadest reasonable interpretation (BRI), it is directed to a process that is capable of being performed using the human mind (with pen and paper as well), and thus the limitation is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 6 is non-patent eligible. Claim 17 is analogous to claim 6, and thus will face the same rejection as above granted that the limitations are very similar, aside from being different type of claims (method vs system).
Regarding claim 7, (similar to claim 10 and part of claim 1.)
Step 1: The claim is directed to a method, which is considered to be a process, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“The method according to claim 2, wherein: a subset of auxiliary data points is selected which are located within the spatial area related to the respective label; for each auxiliary data point of the subset, it is determined whether a direct line of sight exists between the at least one primary sensor and the auxiliary data point; and for each label, a care attribute is assigned to the respective label if a ratio of a number of auxiliary data points for which the direct line of sight exists to a total number of auxiliary data points of the subset is greater than a further predetermined threshold.” – The limitation is directed to selecting data points related to a respective label, determining if a direct line of sight will exist in the data points, and assigning a care attribute to a label if the ration value is greater than a predetermined threshold. The limitation is directed to a process that can be performed using the human mind using observation, evaluation, and judgement. Thus, the limitation is directed to a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 7 is non-patent eligible. Claim 10 and a part of claim 1 is analogous to claim 7, and thus will face the same rejection.
Regarding claim 8, (analogous to claim 18)
Step 1: The claim is directed to a method, which is considered to be a process, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“and the auxiliary data point is regarded as having a direct line of sight to the at least one primary sensor if the auxiliary data point is located within an instrumental field of view of at least one of the radar sensors and has a direct line of sight to at least one of the radar sensors.” – The limitation is directed to “regarding” data points to many factors like having a direct line of sight to a primary sensor of the if it is determined that the data point is located in the field of view of the radar sensors and will have a direct line of sight, which is directed to a process that can be performed in the human mind using evaluation, observation and/or judgment with the aid of pen and paper, and thus is directed to a mental process.
Step 2A Prong 2 and Step 2B:
“The method according to claim 7, wherein: the at least one primary sensor includes a plurality of radar sensors; -- The limitation recites that the primary sensor, at least one, would include a group of radar sensors, which is merely just limiting the primary sensor to a field of use, and cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 8 is non-patent eligible. Claim 18 is analogous to claim 8, and thus will face the same rejection as above granted that the limitations are very similar, aside from being different type of claims (method vs system).
Regarding claim 9, (analogous to claim 19).
Step 1: The claim is directed to a method, which is considered to be a process, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“for each pixel area, the auxiliary data point having a projection within the respective pixel area and being closest to the respective radar sensor is marked as visible; for each label, a number of visible auxiliary data points is determined which are located within the spatial area related to the respective label and which are marked as visible for at least one of the radar sensors; and the care attribute is assigned to the respective label if the number of visible auxiliary data points is greater than a visibility threshold.” – The limitation is directed to marking data points as visible and assigning care attributes to a label if the number of visible data points is greater than a determined visibility threshold, which is all directed to a process that can be performed using the human mind using observation, evaluation, and judgement. Thus, the limitation is directed to a mental process.
“for each of the radar sensors, a specific subset of the auxiliary data points is selected for which the auxiliary data points are related to a respective spatial area within an instrumental field of view of the respective radar sensor; -- The limitation is directed to “selecting” a subset of data points and then related to a spatial areas within a field of view of the radar sensor. The act of selecting data points to be designated to a certain area can be done in a mental process and is human mind-capable, and thus it is directed to a mental process.
“the auxiliary data points of the specific subset are projected to a cylinder or sphere surrounding the respective radar sensor; a surface of the cylinder or sphere is divided into pixel areas;” – The limitation is directed to a specific subset of data points to be projecting to a cylinder or sphere and dividing the geometric space into pixel areas. The act of projecting data points to a geometric space, such as a cylinder, is directed to a known mathematical concept, and thus is directed to math.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 9 is non-patent eligible. Claim 19 is analogous to claim 9, and thus will face the same rejection as above granted that the limitations are very similar, aside from being different type of claims (method vs system).
Regarding claim 11, The majority of the claim’s limitation is analogous to claims 1 and 20 (see rejection of claim 1 above and/or claim 20 below). Below is claim 11’s further limitations evaluated under 101:
Step 1: The claim is directed to a system, for which is directed to a machine, which is an allowable subject matter. The claim satisfies step 1.
There are no elements to be evaluated under step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“A system for training a machine-learning algorithm, the at least one primary sensor configured to capture sparse primary data and comprising at least one radar sensor: at least one auxiliary sensor configured to capture dense auxiliary data and comprising at least one of a light ranging and detection (LIDAR) sensor and a camera; and” – The limitation is directed to a system that comprise the further instructions to apply a sensor that will be configured to capture data, and another sensor to capture data as well, further amended to recite at least a aux sensor to comprise a LIDAR sensor and a camera, which cannot be integrated to a practical application, nor provide significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 11 is non-patent eligible.
Regarding claim 21,
Step 1: The claim is directed to a method, which is considered to be a process, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“based on the lost function,” -- The limitation is directed to performing tasks of the claim based on a lost function (a mathematical concept/calculation), and is directed to math.
Step 2A Prong 2 and Step 2B:
“The method of claim 1, further comprising…at least one of assisting in driving the vehicle and autonomously driving the vehicle.” -- The limitation recites further comprising assisting driving/auto driving the vehicle based on a lost function. The limitation amounts to no more than further limiting to a field of use/environment, and thus it does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 21 is non-patent eligible.
Regarding claim 22,
Step 1: The claim is directed to a method, which is considered to be a process, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“The method of claim 1, wherein, during training, parameters of the machine learning algorithm are adjusted based on the loss function.” -- The limitation is directed to adjusting parameters of an algorithm based on a loss function. The limitation is directed to mathematical concept, but is also human-mind capable using evaluation, observation, and judgment, and thus the limitation is directed to a mental process as well math.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 22 is non-patent eligible.
Regarding claim 23,
Step 1: The claim is directed to a system, which is considered to be a machine, and it is an allowable subject matter. The claim satisfies step 1.
Step 2A Prong 1:
“The method of claim 22, wherein the adjustment of the parameters of the machine-learning algorithm is controlled via the loss function, which is based on error signals due to comparison between predictions of the machine-learning algorithm and a target output representative of the labels.” -- The limitation is directed comparing an error between predictions and target outputs and using that error-based loss to control parameter updates. The limitation is directed to mathematical operations as well as a process that can be performed in the human mind using evaluation, observation, and judgement, thus the limitation is directed to math as well as a mental process.
There are no elements to be evaluated under Step 2A Prong 2 and Step 2B.
Thus, claim 23 is non-patent eligible.
Regarding claim 24,
Step 1: The claim is directed to a system, which is considered to be a machine, and it is an allowable subject matter. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The system of claim 11, wherein the processing unit is configured, based on the lost function, to at least one of assist in driving the vehicle and autonomously drive the vehicle.” -- The limitation is directed to a process that is configured to assist/auto assist to drive the vehicle base on the lost function. The limitation is recited in a high level of generality and merely applies to a computer, thus it does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(f)).
Thus, claim 24 is non-patent eligible.
Regarding claim 25,
Step 1: The claim is directed to a system, which is considered to be a machine, and it is an allowable subject matter. The claim satisfies step 1.
There are no elements to be evaluated under Step 2A Prong 1.
Step 2A Prong 2 and Step 2B:
“The system of claim 20, wherein the instructions further cause the processor, based on the lost function, to at least one of assist in driving the host vehicle and autonomously drive the host vehicle.” -- The limitation recites instructions to further have the processor assist in the host driving/auto host-driving of the vehicle. The limitation is directed to merely limiting the claim’s limitation to a field of use/environment, and thus it does not integrate to a practical application, nor provides significantly more than the judicial exception (see MPEP 2106.05(h)).
Thus, claim 25 is non-patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this
Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not
identically disclosed as set forth in section 102, if the differences between the claimed invention and the
prior art are such that the claimed invention as a whole would have been obvious before the effective filing
date of the claimed invention to a person having ordinary skill in the art to which the claimed invention
pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are
summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-11, 13-25are rejected under 35 U.S.C. 103 as being unpatentable over Musk et. al, US10956755B2 (referred herein as Musk) in view of Northcutt et. al, “Confident Learning: Estimating Uncertainty in Dataset Labels” (referred herein as Northcutt).
Regarding claim 1, Musk teaches:
A method for training a machine-learning algorithm configured to process sparse primary data captured by at least one primary sensor in order to determine at least one property of entities in an environment of the at least one primary sensor, the method implemented within a vehicle while the host vehicle is driving and comprising: [Musk, col 17 lines 30-35] “vehicle 501 is equipped with at least sensors 503 and 553 and captures sensor data used to measure object properties of neighboring vehicles”, [Musk, col. 5, lines 25-27] “[ The surround cameras] are affixed to a vehicle and provide 360 degrees of visibility around the vehicle” , [Musk, col. 6, lines 41-47] “radar sensors are able to capture surrounding details”. AND [Musk, col. 2, lines 42-44] “The collected sensor data is associated and used to generate training data and train a machine-learning model.”, wherein the examiner interprets this as the method for training the machine-learning model being implemented within the vehicle while the host vehicle is driving, since the sensor suite is mounted on the vehicle and captures data of the environment during vehicle operation; performing the same training steps within the in-vehicle AI processor while the vehicle is driving would have been an obvious implementation choice of the training technique described by Musk.)
receiving dense auxiliary data from at least one auxiliary sensor, wherein the at least one auxiliary sensor comprises at least one of a light ranging and detection (LIDAR) sensor and a camera; identifying labels based on the dense auxiliary data and the at least one property of entities, the identifying labels comprising determining a respective spatial area to which each label is related. ([Musk, col 2, lines 20-25] “auxiliary sensor data, such as radar and lidar results, the auxiliary data is associated with objects identified from the vision data” ([Musk, col. 2, line 67 - col. 3, line 4] “the captured image is used as an input to a machine learning model such as a model of a deep learning network running on the AI processor.”, wherein the examiner interprets the lidar results and vision (camera) data as dense auxiliary data received from auxiliary sensors that comprise at least one of a LIDAR sensor and a camera, and “auxiliary data… associated with objects identified from the vision data” as identifying labels based on the auxiliary data. Predicting the distance of each identified object from the image and sensor data corresponds to determining a respective spatial area in the environment to which each label (object) is related.)
the at least one property of entities comprises (i) a spatial location of objects surrounding the vehicle by generating bounding boxes which enclose the objects, respectively, ([Musk, col 15, line 22-30], “In various embodiments, bounding boxes are created for identified objects. The bounding boxes may be two-dimensional bounding boxes or three-dimensional bounding boxes, such as cuboids, that outline the exterior of the identified object”, wherein the examiner interprets “bounding boxes …outline the exterior of the identified object”, to be the same as "bounding boxes which enclose the objects” because they are both putting an enclosing box around the spatial location of an identified object.)
and (ii) a semantic segmentation including assignment of the objects surrounding the vehicle to respective object classes, the object classes including other vehicle, pedestrian, and animal. ([Musk, col. 3, lines 12-20], “a radar sensor mounted to a vehicle emits radar to identify the distance and direction of surrounding obstacles. The distances are then correlated to objects identified in a training image captured from the vehicle's camera. The associated training image is annotated with the distance measurements and used to train a machine learning model. In some embodiments, the model is used to predict additional properties such as an object's velocity” and used to train the machine-learning model, [Musk, col 18, line 26-29], “In the example shown, the set of distances and directions measured for each neighboring vehicle are approximated by distance vectors 513, 523, and 563. … bounding boxes approximate detected objects including detected neighboring vehicles 511, 521, and 561. The bounding boxes approximate the exterior of the detected objects.”, and [Musk, col 11 , line 30-41], “a detected vehicle can be labeled based on a predicted distance and direction as being in the left lane or right lane. In some embodiments, the detected vehicle can be labeled as being in a blind spot, as a vehicle that should be yielded to, or with another appropriate semantic label. In some embodiments, vehicles are assigned to roads or lanes in a map based on the determined ground truth. As additional examples, the determined ground truth can be used to label traffic lights, lanes, drivable space, or other features that assist autonomous driving.”, wherein the examiner interprets annotating each object in the training images with distance and direction and depicting distance vectors and bounding boxes for individual neighboring vehicles to be the same as representing spatial locations of objects surrounding the vehicle using regions that enclose each object (i.e., bounding boxes), and further interprets the use of ground truth to predict semantic labels such as lane position, blind-spot status, yield status, traffic lights, lanes, and drivable space to be the same as a semantic segmentation including assignment of objects and regions surrounding the vehicle to respective object classes, including at least other vehicles and pedestrians.)
assigning at least one of a care attribute or a no-care attribute to each identified label by determining a perception capability of the at least one primary sensor for the respective label based on the sparse primary data captured by the at least one primary sensor and based on the dense auxiliary data captured by the at least one auxiliary sensor, the sparse primary data usable to determine a reference value for a respective spatial area and, for each label, the care attribute is assigned to the respective label if the reference value is greater than a reference threshold and the no-care attribute is assigned to the respective label if the reference value is smaller than or equal to the reference threshold; ([Musk, col. 11, lines 16-21] “a threshold value is used to determine whether to associate an object property as a ground truth of an identified object. For example, related data with a high degree of certainty is associated with an identified object while related data with a degree of certainty below a threshold value is not associated with the identified object.”). ([Musk, col. 11, lines 16-21] “a threshold value is used to determine whether to associate an object property as a ground truth of an identified object. For example, related data with a high degree of certainty is associated with an identified object while related data with a degree of certainty below a threshold value is not associated with the identified object.”). From the application specification [0022], “The perception capability of the at least one primary sensor may therefore be determined by considering the spatial area related to the respective label, e.g., by considering a bounding box which represents the spatial area in which an object may be located. Such a bounding box may be determined e.g., based on dense data from a LIDAR system or a camera. The reference value for the spatial area of a label may be e.g., an average of an intensity of the primary data within the spatial area. Due to the relationship of the reference value for the primary data to the spatial area corresponding to the respective labels, the reliability for assigning the care or no-care attribute may be enhanced.”, Under a broadest reasonable interpretation, however, a “perception capability” is a measure of how reliably the primary sensor can perceive a given label in its spatial area, and a “reference value” can be any scalar quality or certainty measure derived from the primary sensor data in that area, not limited to the particular “average intensity” example in the specification. Musk teaches this same structure by using radar to emit radar signals and identify distance and direction of surrounding obstacles ([Musk, col. 3, lines 12–18]), correlating these radar measurements to objects identified in the camera image, and then using the threshold value above to decide, for each identified object region (i.e., its spatial area / bounding box), whether the radar-based object property is associated as ground truth. The examiner interprets the radar-derived certainty or reliability of the measurement within the spatial area of each label as the claimed “reference value” for that spatial area, and the threshold comparison as determining the primary sensor’s perception capability for that label. Associating data with a high degree of certainty as ground truth corresponds to assigning a care attribute when the reference value is greater than the reference threshold, while not associating data below the threshold corresponds to assigning a no-care attribute when the reference value is smaller than or equal to the reference threshold. The examiner hence interprets the radar-derived certainty or reliability of the measurement within the spatial area of each label as the claimed “reference value” for that spatial area, and the threshold comparison as determining the primary sensor’s perception capability for that label. Associating data with a high degree of certainty as ground truth corresponds to assigning a care attribute when the reference value is greater than the reference threshold, while not associating data below the threshold corresponds to assigning a no-care attribute when the reference value is smaller than or equal to the threshold.)
wherein the at least one primary sensor comprises at least one radar sensor. ([Musk, col. 3, lines 12-18] “a radar sensor mounted to a vehicle emits radar to identify the distance and direction of surrounding obstacles. The distances are then correlated to objects identified in a training image captured from the vehicle’s camera. The associated training image is annotated with the distance measurements and used to train a machine learning model.”, wherein the examiner interprets, under the broadest reasonable interpretation, this radar sensor and its distance/direction measurements as the primary sensor (and lidar to be “auxiliary/dense aux.”) that captures sparse primary data used to determine reference values for the spatial areas associated with the labels.)
Musk does not teach generating model predictions for the labels via a machine-learning algorithm; defining a loss function for the model predictions, wherein the loss function receives a positive loss contribution for which weights of a model on which the machine-learning algorithm relies are increased if the weights contribute constructively to a prediction corresponding to the respective label and a negative loss contribution for which weights of the model are decreased if the weights contribute constructively to a prediction not corresponding to the respective label; permitting negative contributions to the loss function for all labels; permitting positive contributions to the loss function for labels having a care attribute; and permitting positive contributions to the loss function for labels having a no-care attribute only if a confidence value of the model prediction for the respective label is greater than a predetermined threshold.
Northcutt teaches generating model predictions for the labels via a machine-learning algorithm; defining a loss function for the model predictions, wherein the loss function receives a positive loss contribution for which weights of a model on which the machine-learning algorithm relies are increased if the weights contribute constructively to a prediction corresponding to the respective label and a negative loss contribution for which weights of the model are decreased if the weights contribute constructively to a prediction not corresponding to the respective label; ([Northcutt, page 1381] “To train with errors removed, we account for missing data by reweighting the loss by
PNG
media_image1.png
76
212
media_image1.png
Greyscale
dividing by Q^_y,y’[i][i] normalizes out the count of clean training data and Q^y∗[i] re-normalizes…”, wherein the examiner interprets “reweighting the loss by 1/p^(y=i∣y∗=i)” as defining a loss function for the model predictions whose value depends on how each label’s prediction aligns with the latent clean label, and “normalizes out the count of clean training data” as allowing loss contributions from different labels to be scaled up or down (including effectively negative or reduced contributions) according to their reliability, corresponding to the claimed positive and negative contributions for constructive and non-constructive predictions.)
permitting negative contributions to the loss function for all labels; permitting positive contributions to the loss function for labels having a care attribute; and permitting positive contributions to the loss function for labels having a no-care attribute only if a confidence value of the model prediction for the respective label is greater than a predetermined threshold. ([Northcutt, page 1379] “if examples labeled i tend to have higher probabilities because the model is over-confident about class i, then ti will be proportionally larger; if some other class j tends toward low probabilities, tj will be smaller. These thresholds allow us to guess y* in spite of class-imbalance…”). Under a broadest reasonable interpretation, the “care” and “no-care” attributes in the claim simply distinguish more trusted labels, which always receive full positive reinforcement when predicted correctly, from less trusted labels, which only receive positive reinforcement when the model’s confidence exceeds a threshold, while negative contributions remain permitted for all labels. Northcutt teaches this same structural behavior by (i) estimating label reliability and confidence and (ii) introducing class-specific thresholds ti that govern how strongly each label contributes to the loss, such that high-confidence labels (larger ti ) receive stronger positive contributions, while low-confidence labels (smaller tj ) have reduced or effectively negative influence unless the model’s predicted probabilities exceed the corresponding threshold. The examiner hence interprets the more trusted classes with higher thresholds as “care” labels and the less trusted classes with smaller thresholds as “no-care” labels that contribute positively only when confidence is sufficiently high. Thus, the class-dependent thresholds taught by Northcutt implement the claimed behavior of permitting negative contributions for all labels, permitting positive contributions for labels having a care attribute, and permitting positive contributions for labels having a no-care attribute only if a confidence value of the model prediction for the respective label is greater than a predetermined threshold, even though Northcutt does not use the exact “care/no-care” terminology.)
Musk, Northcutt, and the instant application are analogous art because they are all directed to training machine-learning models using sensor data, auxiliary data, and labels to improve object detection and classification of an EV.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method for training a machine-learning algorithm disclosed by Musk to include the thresholding and confidence-based loss reweighting technique to “guess y* in spite of class-imbalance” taught by Northcutt. One would be motivated to do so to effectively improve the reliability of predictions and robustness of training by adjusting for class imbalance and noisy labels, as suggested by Northcutt ([Northcutt, page 1379] “These thresholds allow us to guess y* in spite of class-imbalance…”). Claims 11 and 20 are analogous to claim 1, and therefore are also rejected similarly with the same art as set forth above.
Regarding claim 2, Musk and Northcutt teaches The method according to claim 1, (see rejection of claim 1).
Northcutt further teaches wherein the predetermined threshold for the confidence value is zero. ([Northcutt, page 1379] “In practice with softmax, collisions sometimes occur for softmax outputs with higher temperature (more uniform probabilities), few collisions occur with lower temperature, and no collisions occur with a temperature of zero (one-hot prediction probabilities).”, wherein the examiner interprets “no collisions occur with a temperature of zero (one-hot prediction probabilities)” to be the same as “the predetermined threshold for the confidence value is zero” because setting the temperature of softmax to zero results in a deterministic confidence value, effectively establishing a threshold of zero for model predictions).
Musk, Northcutt, and the instant application are analogous art because they are all directed using confidence-based thresholds to refine predictions.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 1 disclosed by Musk and Northcutt to include the confidence threshold predetermination technique taught by Northcutt. One would be motivated to do so to effectively ensure deterministic and high-confidence predictions in the machine-learning model for cases of higher and lower temperatures in this example, as suggested by Northcutt ([Northcutt, page 1379] “softmax outputs with higher temperature … lower temperature…. no collisions occur with a temperature of zero (one-hot prediction probabilities)”). Claim 13 is analogous to claim 2, and therefore is also rejected similarly with the same art as set forth above.
Regarding claim 3, Musk and Northcutt teaches The method according to claim 2 (see rejection of claim 2).
Musk further teaches wherein the reference value is determined based on radar energy detected by the radar sensor within the spatial area to which the respective label is related. ([Musk, col. 3, lines 12-18] “For example, a radar sensor mounted to a vehicle emits radar to identify the distance and direction of surrounding obstacles. The distances are then correlated to objects identified in a training image captured from the vehicle's camera. The associated training image is annotated with the distance measurements and used to train a machine learning model.”, wherein the examiner interprets “a radar sensor mounted to a vehicle emits radar to identify the distance and direction of surrounding obstacles” as teaching that radar energy is transmitted and reflections are received from objects within the spatial area around the vehicle, and that the resulting radar measurements (distance and direction) are derived from the detected radar energy. The further disclosure that “the distances are then correlated to objects identified in a training image… [and] the associated training image is annotated with the distance measurements” is interpreted as associating, for each camera-defined object region (the spatial area to which the label is related), a scalar value derived from radar returns in that region. Under a broadest reasonable interpretation, this scalar radar-derived measurement (distance/direction or associated certainty) is a reference value determined based on radar energy detected by the radar sensor within the spatial area of the label, as recited in the claim.)Claim 14 is analogous to claim 3, and therefore is also rejected similarly with the same art as set forth above.
Regarding claim 4, Musk and Northcutt teaches The method according to claim 3, (see rejection of claim 3).
Musk further teaches wherein: ranges and angles at which radar energy is perceived are determined based on the sparse primary data captured by the radar sensor; and the ranges and angles are assigned to the spatial areas to which the respective labels are related in order to determine the at least one of the care attribute or the no-care attribute for each label. ([Musk, Col 18, lines 18-26] “In some embodiments, sensors 503 and 553 capture distance and direction measurements. Distance vector 513 depicts the distance and direction of neighboring vehicle 511, distance vector 523 depicts the distance and direction of neighboring vehicle 521, and distance vector 563 depicts the distance and direction of neighboring vehicle 561. In various embodiments, the actual distance and direction values captured are a set of values corresponding to the exterior surface detected by sensors 503 and 553”, wherein the examiner interprets “sensors 503 and 553 capture distance and direction measurements” to be the same as “ranges and angles at which radar energy is perceived are determined based on the primary data captured by the radar sensor” because distance and direction correspond to range and angle in radar detection). AND ([Musk, col 11, lines 30-40] “In some embodiments, the ground truth is determined to predict semantic labels. For example, a detected vehicle can be labeled based on a predicted distance and direction as being in the left lane or right lane. In some embodiments, the detected vehicle can be labeled as being in a blind spot, as a vehicle that should be yielded to, or with another appropriate semantic label. In some embodiments, vehicles are assigned to roads or lanes in a map based on the determined ground truth. As additional examples, the determined ground truth can be used to label traffic lights, lanes, drivable space, or other features that assist autonomous driving.”, wherein the examiner interprets “the ground truth is determined to predict semantic labels” to be the same as “the ranges and angles are assigned to the spatial areas to which the respective labels are related”, and “a detected vehicle can be labeled based on a predicted distance and direction” to be the same as “in order to determine the at least one of the care attribute or the no-care attribute for each label” because assigning labels based on predicted distance and direction aligns with assigning radar-based range and angle data to spatial areas for classification). Claim 15 is analogous to claim 4, and therefore is also rejected similarly with the same art as set forth above.
Regarding claim 5, Musk and Northcutt teaches The method according to claim 4, (see rejection of claim 4).
Musk further teaches wherein: an expected range, an expected range rate and an expected angle are estimated for each label based on the dense auxiliary data; the expected range, the expected range rate and the expected angle of the respective label are assigned to a range, a range rate and an angle derived from the sparse primary data of the radar sensor in order to determine the radar energy associated with the respective label. ([Musk, col 2, lines 40-51] “The trained machine learning model can be deployed to vehicles for accurately predicting object properties, such as distance, direction, and velocity, using only vision data. For example, once the machine learning model has been trained to be able to determine an object distance using images of a camera without a need of a dedicated distance sensor, it may become no longer necessary to include a dedicated distance sensor in an autonomous driving vehicle. When used in conjunction with a dedicated distance sensor, this machine learning model can be used as a redundant or a secondary distance data source to improve accuracy and/or provide fault tolerance.” AND [Musk, col 9, lines 50-54] “In various embodiments, the process of FIG. 2 is used to automatically label training data with corresponding ground truths.”, wherein the examiner interprets “accurately predicting object properties, such as distance, direction, and velocity” to be the same as “an expected range, an expected range rate and an expected angle are estimated for each label based on the auxiliary data”, and “automatically label training data with corresponding ground truths” to be the same as “the expected range, the expected range rate and the expected angle of the respective label are assigned to a range, a range rate and an angle derived from the primary data of the radar sensor in order to determine the radar energy associated with the respective label” because the machine learning model determines object properties, including distance, direction, and velocity, and associates them with labeled training data, which corresponds to assigning expected values derived from auxiliary data to radar-based measurements.) Claim 16 is analogous to claim 5, and therefore is also rejected similarly with the same art as set forth above.
Regarding claim 6, Musk and Northcutt teaches The method according to claim 5, (see rejection of claim 5).
Musk further teaches wherein the expected range rate is estimated for each label based on a speed vector which is estimated for a respective label by using differences of label positions determined based on the dense auxiliary data at different points in time. ([Musk, col 10, lines 35-55] “In some embodiments, the distances are for detected objects such as an obstacle, a barrier, a moving vehicle, a stationary vehicle, traffic control signals, pedestrians, etc. and used as the ground truth for training. In addition to distance, the ground truth for other object parameters such as direction, velocity, acceleration, etc. may be determined. For example, accurate distances and directions are determined as ground truths for identified objects. As another example, accurate velocity vectors are determined as ground truths for identified objects, such as vehicles and pedestrians. In various embodiments, vision data and related data are organized by timestamps and corresponding timestamps are used.”, wherein the examiner interprets “accurate velocity vectors are determined as ground truths for identified objects, such as vehicles and pedestrians” to be the same as “the expected range rate is estimated for each label based on a speed vector”, and “vision data and related data are organized by timestamps and corresponding timestamps are used” to be the same as “a speed vector which is estimated for a respective label by using differences of label positions determined based on the auxiliary data at different points in time” because the timestamps allow tracking object motion over time, which enables determining velocity changes based on position differences). Claim 17 is analogous to claim 6, and therefore is also rejected similarly with the same art as set forth above.
Regarding claim 7, Musk and Northcutt teaches The method according to claim 2, (see rejection of claim 2).
Musk further teaches wherein: a subset of auxiliary data points is selected which are located within the spatial area related to the respective label; for each auxiliary data point of the subset, it is determined whether a direct line of sight exists between the at least one primary sensor and the auxiliary data point; and for each label, a care attribute is assigned to the respective label if a ratio of a number of auxiliary data points for which the direct line of sight exists to a total number of auxiliary data points of the subset is greater than a further predetermined threshold. ([Musk, col 2, lines 20-28] “A machine learning training technique for generating highly accurate machine learning results from vision data is disclosed. Using auxiliary sensor data, such as radar and lidar results, the auxiliary data is associated with objects identified from the vision data to accurately estimate object properties such as object distance. In various embodiments, the collection and association of auxiliary data with vision data is done automatically and requires little, if any, human intervention.” AND [Musk, col 16, lines 65-68] “In the event the prediction and actual sensor data differ by more than a threshold amount, the image sensor data and related auxiliary data are transmitted and used to automatically generate training data.”, wherein the examiner interprets “the auxiliary data is associated with objects identified from the vision data to accurately estimate object properties such as object distance” to be the same as “a subset of auxiliary data points is selected which are located within the spatial area related to the respective label” because object identification is object labeling, and “in the event the prediction and actual sensor data differ by more than a threshold amount” to be the same as “a care attribute is assigned to the respective label if a ratio of a number of auxiliary data points for which the direct line of sight exists to a total number of auxiliary data points of the subset is greater than a further predetermined threshold” because the determination of prediction accuracy using auxiliary data involves a threshold comparison, which aligns with the assignment of attributes based on auxiliary data distribution and detection conditions). Claim 10 and a part of claim 1 is analogous to claim 7, and therefore is also rejected similarly with the same art as set forth above.
Regarding claim 8, Musk and Northcutt teaches The method according to claim 7, (see rejection of claim 7).
Musk further teaches wherein: the at least one primary sensor includes a plurality of radar sensors; and the auxiliary data point is regarded as having a direct line of sight to the at least one primary sensor if the auxiliary data point is located within an instrumental field of view of at least one of the radar sensors and has a direct line of sight to at least one of the radar sensors. ([Musk, col 17-18, lines 65 – 17] “In the example shown, field of views 509 and 559 of sensors 503 and 553, respectively, are depicted by dotted arcs between dotted arrows. The depicted fields of views 509 and 559 show the overhead perspective of the regions measured by sensors 503 and 553, respectively. Properties of objects in field of view 509 may be captured by sensor 503 and properties of objects in field of view 559 may be captured by sensor 553. For example, in some embodiments, distance, direction, and/or velocity measurements of objects in field of view 509 are captured by sensor 503. In the example shown, sensor 503 captures the distance and direction of neighboring vehicles 511 and 521. Sensor 503 does not measure neighboring vehicle 561 since neighboring vehicle 561 is outside the region of field of view 509. Instead, the distance and direction of neighboring vehicle 561 is captured by sensor 553. In various embodiments, objects not captured by one sensor may be captured by another sensor of a vehicle. Although depicted in FIG. 5 with only sensors 503 and 553, autonomous vehicle 501 may be equipped with multiple surround sensors (not shown) that provide 360 degrees of visibility around the vehicle.”, wherein the examiner interprets “autonomous vehicle 501 may be equipped with multiple surround sensors (not shown) that provide 360 degrees of visibility around the vehicle” to be the same as “the at least one primary sensor includes a plurality of radar sensors”, and “properties of objects in field of view 509 may be captured by sensor 503 and properties of objects in field of view 559 may be captured by sensor 553” to be the same as “the auxiliary data point is regarded as having a direct line of sight to the at least one primary sensor if the auxiliary data point is located within an instrumental field of view of at least one of the radar sensors and has a direct line of sight to at least one of the radar sensors” because the field of view of each sensor determines whether an object is detected, which is the same as determining whether an auxiliary data point has a direct line of sight based on its location in the sensor's instrumental field of view). Claim 18 is analogous to claim 8, and therefore is also rejected similarly with the same art as set forth above.
Regarding claim 9, Musk and Northcutt teaches The method according to claim 8, (see rejection of claim 8).
Musk further teaches:
wherein: for each of the radar sensors, a specific subset of the auxiliary data points is selected for which the auxiliary data points are related to a respective spatial area within an instrumental field of view of the respective radar sensor; the auxiliary data points of the specific subset are projected to a cylinder or sphere surrounding the respective radar sensor; a surface of the cylinder or sphere is divided into pixel areas; for each pixel area, the auxiliary data point having a projection within the respective pixel area and having the closest distance to the respective radar sensor is marked as visible; ([Musk, col 14, lines 37-47] “For example, in some embodiments, eight surround cameras are affixed to a vehicle and provide 360 degrees of visibility around the vehicle with a range of up to 250 meters. In some embodiments, camera sensors include a wide forward camera, a narrow forward camera, a rear view camera, forward-looking side cameras, and/or rearward-looking side cameras. In some embodiments, ultrasonic and/or radar sensors are used to capture surrounding details. For example, twelve ultrasonic sensors may be affixed to the vehicle to detect both hard and soft objects.”, wherein the examiner interprets “eight surround cameras are affixed to a vehicle and provide 360 degrees of visibility around the vehicle” to be the same as “for each of the radar sensors, a specific subset of the auxiliary data points is selected for which the auxiliary data points are related to a respective spatial area within an instrumental field of view of the respective radar sensor”, and “ultrasonic and/or radar sensors are used to capture surrounding details” to be the same as “the auxiliary data points of the specific subset are projected to a cylinder or sphere surrounding the respective radar sensor” because the use of multiple sensors with defined fields of view ensures that captured data is spatially organized and mapped for object detection, which aligns with the projection of auxiliary data points onto a structured coordinate system for determining object visibility).
for each label, a number of visible auxiliary data points is determined which are located within the spatial area related to the respective label and which are marked as visible for at least one of the radar sensors; and the care attribute is assigned to the respective label if the number of visible auxiliary data points is greater than a visibility threshold. ([Musk, col 8, lines 49-55] “In various embodiments, the process of FIG. 2 is used to automatically label training data with corresponding ground truths. The ground truth and image data are packaged as training data to predict properties of objects identified from the image data. In various embodiments, the sensor and related auxiliary data are captured using the deep learning system of FIG. 1.”, wherein the examiner interprets “automatically label training data with corresponding ground truths” to be the same as “for each label, a number of visible auxiliary data points is determined which are located within the spatial area related to the respective label and which are marked as visible for at least one of the radar sensors”, and “the ground truth and image data are packaged as training data to predict properties of objects identified from the image data” to be the same as “the care attribute is assigned to the respective label if the number of visible auxiliary data points is greater than a visibility threshold” because the process of labeling training data with ground truths involves associating detected objects with sensor data and using a threshold to determine their inclusion in training, which is the same as assigning a care attribute based on the number of visible auxiliary data points surpassing a threshold). Claim 19 is analogous to claim 9, and therefore is also rejected similarly with the same art as set forth above.
Regarding claim 10, Musk and Northcutt teaches The method according to claim 1, (see rejection of claim 1).
Musk further teaches wherein: identifying labels based on the dense auxiliary data includes determining a respective spatial area to which each label is related; a reference value for the respective spatial area is determined based on the sparse primary data; a subset of auxiliary data points is selected which are located within the spatial area related to the respective label; for each auxiliary data point of the subset, it is determined whether a direct line of sight exists between the at least one primary sensor and the auxiliary data point; and for each label, a care attribute is assigned to the respective label if the reference value is greater than a reference threshold and if a ratio of a number of auxiliary data points for which the direct line of sight exists to a total number of auxiliary data points of the subset is greater than a further predetermined threshold. ([Musk, col 2, lines 22-27] “Using auxiliary sensor data, such as radar and lidar results, the auxiliary data is associated with objects identified from the vision data to accurately estimate object properties such as object distance. In various embodiments, the collection and association of auxiliary data with vision data is done automatically and requires little, if any, human intervention.” AND [Musk, col 11, lines 15-20] “In various embodiments, a threshold value is used to determine whether to associate an object property as a ground truth of an identified object. For example, related data with a high degree of certainty is associated with an identified object while related data with a degree of certainty below a threshold value is not associated with the identified object.”, wherein the examiner interprets “the auxiliary data is associated with objects identified from the vision data” to be the same as “identifying labels based on the auxiliary data”, “to accurately estimate object properties such as object distance” to be the same as “determining a respective spatial area to which each label is related”, “a threshold value is used to determine whether to associate an object property as a ground truth of an identified object” to be the same as “a care attribute is assigned to the respective label if the reference value is greater than a reference threshold”, and “related data with a high degree of certainty is associated with an identified object while related data with a degree of certainty below a threshold value is not associated with the identified object” to be the same as “if a ratio of a number of auxiliary data points for which the direct line of sight exists to a total number of auxiliary data points of the subset is greater than a further predetermined threshold” because both describe making a classification decision based on a confidence threshold for data association).
Regarding claim 21, Musk and Northcutt teaches The method according to claim 1, (see rejection of claim 1).
Northcutt further teaches further comprising, based on the loss function, at least one of ([Northcutt, page 1381] “To train with errors removed, we account for missing data by reweighting the loss by
PNG
media_image2.png
59
204
media_image2.png
Greyscale
dividing by Q^_y,y’[i][i] normalizes out the count of clean training data and Q^y∗[i] re-normalizes…”, wherein the examiner interprets “reweighting the loss by 1/p^(y=i∣y∗=i)” as defining a loss function for the model predictions whose value depends on how each label’s prediction aligns with the latent clean label where “based on the loss function” can reasonably be read as “using a model trained according to that loss function to drive/assist.” The use of a loss function in supervised machine learning (ML) is a straightforward, predictable concept to develop an ML model in the context of autonomous vehicles.)
Northcutt does not teach assisting in driving the vehicle and autonomously driving the vehicle.
Musk further teaches assisting in driving the vehicle and autonomously driving the vehicle. ([Musk, col 2, lines 43-50 ] “The trained machine learning model can be deployed to vehicles for accurately predicting object properties, such as distance, direction, and velocity, using only vision data. For example, once the machine learning model has been trained to be able to determine an object distance using images of a camera without a need of a dedicated distance sensor, it may become no longer necessary to include a dedicated distance sensor in an autonomous driving vehicle. When used in conjunction with a dedicated distance sensor, this machine learning model can be used as a redundant or a secondary distance data source to improve accuracy and/or provide fault tolerance.” AND [Musk, col 11, lines 37-41 ] “In some embodiments, vehicles are assigned to roads or lanes in a map based on the determined ground truth. As additional examples, the determined ground truth can be used to label traffic lights, lanes, drivable space, or other features that assist autonomous driving.”, wherein the examiner interprets “autonomous driving vehicle” and “assist autonomous driving” as teaching “autonomously driving the vehicle” and “assisting in driving the vehicle,” respectively, and further interprets deployment of the trained model in the autonomous driving vehicle (and its use to assist autonomous driving) as performing, based on the loss function (which governed training as in Northcutt), at least one of assisting in driving the vehicle and autonomously driving the vehicle, as recited in the claim.)
Musk, Northcutt, and the instant application are analogous art because they are all directed to methods that train machine-learning models on vehicle sensor data using a loss function and then deploy those models in a vehicle to assist in driving or autonomously drive the vehicle.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method claim 1 disclosed by Musk and Northcutt to include the error correction using a loss function technique disclosed by Northcutt. One would be motivated to do so to effectively improve the robustness and accuracy of the trained model that is used to assist in driving or autonomously drive the vehicle, as suggested by Northcutt ([Northcutt, page 1381] “These thresholds allow us to guess y* in spite of class-imbalance...To train with errors removed, we account for missing data by reweighting the loss”).
Regarding claim 22, Musk and Northcutt teaches The method according to claim 1, (see rejection of claim 1).
Northcutt further teaches wherein, during training, parameters of the machine-learning algorithm are adjusted based on the loss function. ([Northcutt, page 1381] “To train with errors removed, we account for missing data by reweighting the loss by
PNG
media_image2.png
59
204
media_image2.png
Greyscale
dividing by Q^_y,y’[i][i] normalizes out the count of clean training data and Q^y∗[i] re-normalizes…”, wherein the examiner interprets “reweighting the loss” to the be same as “To train” as corresponding to “during training” in the claim, because for both ML training is performed based on the loss function. In any gradient-based supervised learning process, training “with errors removed” by defining and reweighting a loss inherently involves adjusting parameters of the machine-learning algorithm to reduce that loss. Thus, under a broadest reasonable interpretation, Northcutt teaches that, during training, parameters of the machine-learning algorithm are adjusted based on the loss function.)
Musk, Northcutt, and the instant application are analogous art because they are all directed to methods of training machine-learning models on labeled sensor data using loss functions that guide parameter updates.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method claim 1 disclosed by Musk and Northcutt to include the error correction using a loss function technique disclosed by Northcutt. One would be motivated to do so to effectively improve the quality and stability of the learned model parameters during training by using a loss-driven parameter update scheme that accounts for noisy or missing data, as suggested by Northcutt ([Northcutt, page 1381] “To train with errors removed, we account for missing data by reweighting the loss…To train with errors removed, we account for missing data by reweighting the loss”)
Regarding claim 23, Musk and Northcutt teaches The method of claim 22, (see rejection of claim 22).
Northcutt further teaches wherein the adjustment of the parameters of the machine-learning algorithm is controlled via the loss function, which is based on error signals due to comparison between predictions of the machine-learning algorithm and a target output representative of the labels. ([Northcutt, page 1381] “To train with errors removed, we account for missing data by reweighting the loss by
PNG
media_image3.png
69
204
media_image3.png
Greyscale
dividing by Q^_y,y’[i][i] normalizes out the count of clean training data and Q^y∗[i] re-normalizes…”, wherein the examiner interprets “p^(y~=i | y* =i)”, where y~ represents predicted labels and y* represents “latent true labels” are used in “reweighting the loss” along with thresholds ti, (predicted probabilities) to be the same as defining a loss function that is based on how the model’s predictions compare to target labels (the latent clean labels), i.e., an error signal between prediction and target. Because the training process in Northcutt is “to train with errors removed” via this loss, the parameter updates are controlled by this loss function, which itself is computed from the comparison between predictions and the target output representative of the labels.)
Musk, Northcutt, and the instant application are analogous art because they are all directed to supervised learning techniques in which model parameters are adjusted to reduce errors between predictions and labeled targets using a loss function.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method claim 22 disclosed by Musk and Northcutt to include the error correction using a loss function technique disclosed by Northcutt. One would be motivated to do so to effectively control the adjustment of the model parameters via a loss function that is based on prediction-versus-label error and is robust to class imbalance and noisy labels, as suggested by Northcutt ([Northcutt, page 1381] “These thresholds allow us to guess 𝑦* in spite of class-imbalance…To train with errors removed, we account for missing data by reweighting the loss”).
Regarding claim 24, Musk and Northcutt teaches, The system of claim 11, (see rejection of claim 1).
Musk further teaches wherein the processing unit is configured, … ([Musk, col. 2, line 65 - col. 3, line 4] “For example, the captured image is used as an input to a machine learning model such as a model of a deep learning network running on the AI processor. The model is used to predict the distance of objects identified in the image data.” wherein the examiner interprets “deep learning network running on the AI processor” to be the same as “processing unit” because they both involve using processors to run AI models.
to at least one of assist in driving the vehicle and autonomously drive the vehicle. ([Musk, col 2, lines 40-51] “The trained machine learning model can be deployed to vehicles for accurately predicting object properties, such as distance, direction, and velocity, using only vision data. For example, once the machine learning model has been trained to be able to determine an object distance using images of a camera without a need of a dedicated distance sensor, it may become no longer necessary to include a dedicated distance sensor in an autonomous driving vehicle. When used in conjunction with a dedicated distance sensor, this machine learning model can be used as a redundant or a secondary distance data source to improve accuracy and/or provide fault tolerance.” and ([Musk, col 11, lines 30-40] “In some embodiments, the ground truth is determined to predict semantic labels. For example, a detected vehicle can be labeled based on a predicted distance and direction as being in the left lane or right lane. In some embodiments, the detected vehicle can be labeled as being in a blind spot, as a vehicle that should be yielded to, or with another appropriate semantic label. In some embodiments, vehicles are assigned to roads or lanes in a map based on the determined ground truth. As additional examples, the determined ground truth can be used to label traffic lights, lanes, drivable space, or other features that assist autonomous driving.” wherein the examiner interprets “autonomous driving vehicle” as teaching “autonomously drive the vehicle,” and “assist autonomous driving” as teaching “assist in driving the vehicle.” Combined with Northcutt’s loss-based training, the processing unit running the trained model is therefore configured, based on the loss function, to at least one of assist in driving the vehicle and autonomously drive the vehicle.)
Musk does not teach based on the loss [lost] function.
Northcutt teaches based on the loss [lost] function ([Northcutt, page 1381] “To train with errors removed, we account for missing data by reweighting the loss by
PNG
media_image3.png
69
204
media_image3.png
Greyscale
dividing by Q^_y,y’[i][i] normalizes out the count of clean training data and Q^y∗[i] re-normalizes…”, wherein the examiner interprets “p^(y~=i | y* =i)”, wherein the examiner interprets “p^(y~=i | y* =i)”, where y~ represents predicted labels and y* represents “latent true labels” are used in “reweighting the loss” along with thresholds ti, (predicted probabilities) to be the same as defining a loss function that is based on how the model’s predictions compare to target labels (the latent clean labels), i.e., an error signal between prediction and target. Because the training process in Northcutt is “to train with errors removed” via this loss, the parameter updates are controlled by this loss function, which itself is computed from the comparison between predictions and the target output representative of the labels. Under a broadest reasonable interpretation, a processing unit that executes a model whose parameters have been learned by minimizing a loss function (as taught in Northcutt) is configured, based on the loss function, because the loss function determines the parameter values and thus the behavior of the processing unit. Thus, the processing unit is configured, based on the loss function.)
Musk, Northcutt, and the instant application are analogous art because they are all directed to software instructions that implement machine-learning models trained via a loss function on vehicle sensor data to assist in driving or autonomously drive the host vehicle.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the computer-readable medium / instructions claim 20 disclosed by Musk and Northcutt to include the use of loss function disclosed by Northcutt. One would be motivated to do so to effectively improve the robustness and accuracy of the host vehicle’s driver-assistance and autonomous-driving functions by using loss-based training that handles noisy or imbalanced labels, as suggested by Northcutt ([Northcutt, page 1381] “To train with errors removed, we account for missing data by reweighting the loss … These thresholds allow us to guess y* in spite of class-imbalance.”).
Regarding claim 25, Musk and Northcutt teaches, The system of claim 20 (see rejection of claim 1).
Musk further teaches wherein the instructions further cause the processor, ([Musk, col. 2, line 65 - col. 3, line 4] “For example, the captured image is used as an input to a machine learning model such as a model of a deep learning network running on the AI processor. The model is used to predict the distance of objects identified in the image data.” wherein the examiner interprets a “machine learning model” to be “running on the AI processor” to be the same as “instructions further cause the processor” because they both involve using processors to run AI models.
…to at least one of assist in driving the host vehicle and autonomously drive the host vehicle. ([Musk, col 2, lines 40-51] “The trained machine learning model can be deployed to vehicles for accurately predicting object properties, such as distance, direction, and velocity, using only vision data. For example, once the machine learning model has been trained to be able to determine an object distance using images of a camera without a need of a dedicated distance sensor, it may become no longer necessary to include a dedicated distance sensor in an autonomous driving vehicle. When used in conjunction with a dedicated distance sensor, this machine learning model can be used as a redundant or a secondary distance data source to improve accuracy and/or provide fault tolerance.”, [Musk, col 2, lines 22-27] “Using auxiliary sensor data, such as radar and lidar results, the auxiliary data is associated with objects identified from the vision data to accurately estimate object properties such as object distance. In various embodiments, the collection and association of auxiliary data with vision data is done automatically and requires little, if any, human intervention.”, and ([Musk, col 11, lines 30-40] “In some embodiments, the ground truth is determined to predict semantic labels. For example, a detected vehicle can be labeled based on a predicted distance and direction as being in the left lane or right lane. In some embodiments, the detected vehicle can be labeled as being in a blind spot, as a vehicle that should be yielded to, or with another appropriate semantic label. In some embodiments, vehicles are assigned to roads or lanes in a map based on the determined ground truth. As additional examples, the determined ground truth can be used to label traffic lights, lanes, drivable space, or other features that assist autonomous driving.”, wherein the examiner interprets the vehicle that utilizes “auxiliary sensor data, such as radar and lidar results … vision data to accurately estimate” to be the same as the “host vehicle” because both are the vehicle in which the system is installed on and is being controlled relative to the other vehicles/objects. The examiner further interprets “autonomous driving vehicle” as teaching “autonomously drive the vehicle,” and “assist autonomous driving” as teaching “assist in driving the vehicle.” Combined with Northcutt’s loss-based training, the processing unit running the trained model is therefore configured, based on the loss function, to at least one of assist in driving the vehicle and autonomously drive the vehicle.)
Musk does not teach based on the loss [lost] function.
Northcutt teaches based on the lost [lost] function, ([Northcutt, page 1381] “To train with errors removed, we account for missing data by reweighting the loss by
PNG
media_image3.png
69
204
media_image3.png
Greyscale
dividing by Q^_y,y’[i][i] normalizes out the count of clean training data and Q^y∗[i] re-normalizes…”, wherein the examiner interprets “p^(y~=i | y* =i)”, wherein the examiner interprets “p^(y~=i | y* =i)”, where y~ represents predicted labels and y* represents “latent true labels” are used in “reweighting the loss” along with thresholds ti, (predicted probabilities) to be the same as defining a loss function that is based on how the model’s predictions compare to target labels (the latent clean labels), i.e., an error signal between prediction and target. Because the training process in Northcutt is “to train with errors removed” via this loss, the parameter updates are controlled by this loss function, which itself is computed from the comparison between predictions and the target output representative of the labels. Under a broadest reasonable interpretation, a processing unit that executes a model whose parameters have been learned by minimizing a loss function (as taught in Northcutt) is configured, based on the loss function, because the loss function determines the parameter values and thus the behavior of the processing unit. Thus, the processing unit is configured, based on the loss function.)
Musk, Northcutt, and the instant application are analogous art because they are all directed to computer-readable instructions that implement machine-learning models trained via a loss function on vehicle sensor data to assist in driving or autonomously drive a host vehicle.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the computer-readable medium / instructions claim 20 disclosed by Musk and Northcutt to include the loss function disclosed by Northcutt. One would be motivated to do so to effectively improve the robustness and accuracy of the host vehicle’s driver-assistance and autonomous-driving capabilities by using a loss function that handles noisy or imbalanced labels, as suggested by Northcutt ([Northcutt, page 1381] “These thresholds allow us to guess y* in spite of class-imbalance … To train with errors removed, we account for missing data by reweighting the loss.”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVAN KAPOOR whose telephone number is (703)756-1434. The examiner can normally be reached Monday - Friday: 9:00AM - 5:00 PM EST (times may vary).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVAN KAPOOR/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights