DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The amendment filed July 1, 2026 has been entered. Independent claims 1, 8, and 15 and dependent claims 4, 6, 7, 11, 13, 14, 16, 18, and 20 have been amended. Claims 3, 10, and 17 are presently canceled. Claims 5, 12, and 19 were previously canceled. The remaining claims are in original or previously presented form. Therefore, claims 1, 2, 4, 6-9, 11, 13-16, 18, and 20 are pending in the application.
The applicant’s Remarks, filed July 1, 2026, has been fully considered. The applicant argues, under the heading “IV. Claim Rejections under 35 U.S.C. § 112,” that the claims have been amended to overcome the rejections given in the last detailed action, which was the Non-Final Rejection dated April 6, 2026. The examiner notes that at least claim 1 appears to not include some of the lines that existed in the previous claim 1, yet those lines are not lined through. Very little of claim 1 has been lined through, while most of it underlined. The examiner will examine the claims as filed. The examiner withdraws all the 35 U.S.C. 112 rejections made in the last detailed action due to the applicant’s amendments and arguments.
The applicant argues in the Remarks under the heading “V. Claim Rejections under 35 U.S.C. § 103,” that that Jiang et al. (US2019/0382003) does not teach “identifying a high-risk object in the surrounding area” as in present claim 1, but rather teaches identifying a high risk area of a road. Paragraph 0173 teaches that “The output of this risk analysis is risk data that describes an estimate whether collisions are likely in different portions of the roadway currently being traveled by the ego vehicle based in part on the estimated behaviors of the remote drivers and the ego driver (e.g., based on the first estimate and the second estimate).” Paragraph 0176 teaches that the system will perform “risk assessments for the remote drivers”. The paragraph states that the system “updates the twin data for different vehicles”. The entire disclosure of Jiang does teach determining a risk in different “portions of the roadway,” as the applicant argues, yet the implication of this is that the system knows the location at which a predicted collision will take place due to the risky driving of a nearby vehicle. So in paragraph 0176 there is a “risk assessments for the remote drivers” of the nearby vehicles. Paragraph 0080 also teaches “behavior metrics” of nearby drivers. These are at least close to a risk score for a driver. In any case, the new claims are substantially different and a new reference has been applied due to amendment. At least paragraphs 0176 and 0080 teach identifying a high-risk object. Kim et al. (US2019/0077402) also teaches this, as argued in the last detailed actions, and actions before that.
The applicant argues in the Remarks that “To whatever extent Kim could be said to calculate a risk level for a vehicle of interest, such risk level is calculated ‘based on the information about the driving record of the vehicle of interest’ (Kim, paragraph [0042])”. Kim teaches in paragraph 0443 “a risk level for the vehicle of interest”. This is based on the driving record of the vehicle, as the applicant argues.
The applicant has significantly changed at least the independent claims. Therefore, the grounds for rejection have changed. Please see the rejections in a separate section below. Before that section, however, the examiner offers the following notes on the present claims:
Present claim 1 recites “a machine learning model,” “a second machine learning model,” and “a third machine learned model.” Each has a different function. The “a machine learning model,” which the examiner will also refer to as the first machine learning model, predicts “a path of a vehicle based on vehicle telemetry data”. The second machine learning model, predicts “an intended movement of the recognized object based on the driving conditions and the object telemetry data”. And the third machine learning model calculates “a risk score for the recognized object based on the path of the vehicle and the predicted intended movement”.
Is there written description for three different machine learning models each with these functions? In the present specification, paragraph 0035 teaches that “In an embodiment, a supervised machine learning model may be trained to predict the path of a vehicle based on telemetry data from the vehicle or other sources.” Therefore, the first machine learning model has written description in the specification.
Paragraph 0040 teaches that “In an embodiment, a supervised machine learning model may be trained to predict the intended movements of a recognized object based on historical information about the movements of the object or about similar activity related to a recognized object or based on specific data that may be transmitted by the object, including telemetry or other data that may be relevant.” Nothing about making this prediction specifically based on “driving conditions” is taught here. But paragraph 0036 teaches toward the end that cameras and sensors may determine “driving conditions, e.g., object presence, relative location, and type along with vehicle performance data like speed or braking.” Paragraph 0037 teaches that these driving conditions may be used to create a 3D model of the surrounding. The paragraph adds that “In addition to detecting people and objects or vehicles in the surrounding area, driving conditions as described herein may also include information about weather conditions…such as rain, snow or ice. This can be used “in the prediction of the effect of objects and other factors in the surrounding area on the potential for an incident and may be recreated in the augmented reality environment. For instance, wet roads may cause a possible lack in traction and issues for the driver…and possibly exacerbate potential damage in an incident…” Although these paragraphs are referring to Fig. 2, step 204, it is reasonable to include the general idea in Fig. 2, step 206. The specific discussion of step 206 begins at paragraph 0039.
Paragraph 0040 does not differentiate this machine learning model from the one before it that predicts the path of the vehicle. Nor is a combination of models taught here.
Yet there a reasonable reading of the present disclosure in which the “a supervised machine learning model” in paragraph 0040 is different from the one mentioned in paragraph 0035. The argument could be made as follows: Paragraph 0035 is in the context of paragraph 0033-0034, which is all part of a discussion of Fig. 2, step 202. Note how the end of paragraph 0033 is similar to the first sentence of paragraph 0035. This section teaches “a supervised machine learning model,” as recited in paragraph 0035, that performs the function of step 202.
Paragraph 0040, which recites “a supervised machine learning model” that “may be trained to predict the intended movements of a recognized object,” is all part of a discussion of Fig. 2, step 206, which begins at paragraph 0039 and concludes at 0040. It is not explicitly clear in paragraph 0040 alone if this “a supervised machine learning model” is the same “a supervised machine learning model” mentioned in paragraph 0035 or not. But if so, why does paragraph 0040 go on to recite that the model can use “one or more of the following machine learning algorithms” and then list many different algorithms? Why repeat that same statement and list as in paragraph 0035 if the machine learning models were the same model? If they were one and the same model, such repetition would likely be absent.
The examiner will permit the language in present claim 1 regarding a first and second machine learning models. Considering these models separate has written description even if it could have been clearer.
Paragraph 0041 is more specific. It recites that “another supervised machine learning model, which may be separate from the above or combined, may be trained to calculate the risk score for the object relative to the vehicle…through an analysis of the predicted vehicle path and the intended movements of the recognized object.” This “another supervised machine learning model” that may be “separate” is what claim 1 calls “a third machine learning model.”
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claims contain subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for pre-AIA the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites in part:
identifying a recognized object of the one or more objects based on
the capturing and
the object telemetry data [.]
The capturing referred to here is “capturing, by one or more sensors, driving conditions from a surrounding area;” as recited in claim 1.
The present disclosure does not provide written description for “identifying a recognized object”. The disclosure does not teach identifying a recognized object based on the capturing and the object telemetry data. Rather, the disclosure teaches identifying a high-risk object. The disclosure also teaches recognizing an object. But “identifying a recognized object” goes beyond the scope of the disclosure. It seems that the phrase may have been intended to mean simply “recognizing an object…” For examination purposes that is how the phrase will be interpreted.
The reason the examiner is picky about this is because “identifying” could be construed to imply classifying an object as a person, cyclist, or vehicle, for example, whereas just the term “recognizing” as it is used in the present disclosure, does not carry that implication. Paragraph 0032 teaches that the host vehicle can obtain telemetry data from the vehicle. This apparently means from the host vehicle itself since the disclosure normal calls the host vehicle the “vehicle” and other vehicles (or pedestrians, etc) “objects.” The paragraph goes on to state that the system can “then capture driving conditions…including recognizing objects in the surrounding area that may be transmitting relevant data.” The paragraph states that host vehicle sensors may pick up “telemetry and other data from other vehicles”. And that “driving conditions in this context refers to specific activity in the surrounding area…such as objects obstructions, weather conditions” etc. “Objects in the surrounding area may then be identified as high-risk objects…” Paragraph 0038 discusses “recognizing and communicating with objects” including non-vehicle objects. Fig. 2, step 204 is “capture driving conditions form the surrounding area and recognize an object that is transmitting relevant data.”
Based on these references, in a broad reasonable interpretation, “recognizing” an object in the present disclosure means that the system is aware that the object is nearby, such as by its location and trajectory. Recognizing does not necessarily mean classifying the object as a person or a vehicle. Recognizing can simply mean acknowledging that there is an object nearby that is transmitting telemetry data.
Claim 1 also recites in part:
calculating, by a third machine learning model, a risk score for the recognized object based on
the path of the vehicle and
the predicted intended movement;
Claim 7 adds the following:
The computer-implemented method of claim 1, wherein
the risk threshold for the vehicle is determined using a fourth machine learning model that predicts vehicle risk based on historical vehicle data and the driving conditions.
Does the present disclosure provide written description for both this “fourth machine learning model” and a “vehicle risk”? Does the original disclosure support the claim that there a “fourth machine learning model”? And separately, is there support for claiming a separate quantity called “vehicle risk” apart from what claim 1 calls “a risk score for the recognized object”?
Claim 7 states that “a fourth machine learning model” calculates “vehicle risk” based on various data. But claim 1 already stated that “a third machine learning model” predicts “a risk score for the recognized object”. Is there a difference between “vehicle risk” in claim 7 and “a risk score for the recognized object”? Note that in the disclosure the “vehicle” is the host vehicle, and the “object” is another agent in the driving scene of the vehicle. The object can be another vehicle or a pedestrian or cyclist, for instance.
Paragraph 0041 of the present disclosure helps answer these questions. The paragraph teaches that “another supervised machine learning model…may be trained to calculate the risk score for the object relative to the vehicle, or predict the level of risk that a recognized object may pose to the vehicle.” It seems that the “risk score” and the “level of risk” are really the same. The risk score for the object is the level of risk that that object poses to the vehicle. The risk to the vehicle, or vehicle risk, is the risk that a nearby object poses. Really, this is a kind of mutual risk. The risk is a risk that the two agents will collide into each other given their current paths and other factors, such as whether there is rain or ice on the pavement.
Paragraph 0041 begins with a discussion of “another supervised machine learning model”. This one can “calculate the risk score for the object relative to the vehicle, or predict the level of risk”. A “risk score” is just that, while a “level of risk” is a classification, such as high risk or low risk. So this one machine learning model can do both. Paragraph 0041 then begins another sub-section beginning with “In an embodiment, an ensemble machine learning technique”. Here “the model” uses “multiple machine learning algorithms”. So according to this paragraph, there is still one “model,” yet it runs various “algorithms.” The paragraph then adds that “this machine learning model may also be used to compare the risk score to a risk threshold for the vehicle and also determine the risk threshold for the vehicle”. So “this [same] machine learning model,” that has been discussed since the beginning of paragraph 0041, can not only calculate a “risk score…or predict the level of risk that a recognized object may pose” but can also “compare the risk score to a risk threshold…and also determine the risk threshold”.
So when claim 7 recites “a fourth machine learning model” that “predicts vehicle risk” it lacks written description for yet another machine learning model. The description of the machine learning model that does what is claimed in claim 7 is found in paragraph 0041, and the model in that paragraph is the same model that claim 1 calls “a third machine learning model.” Therefore, the “a fourth machine learning model” lacks written description and is new matter.
The term in claim 7 of “vehicle risk” is really essentially the same, or very similar to, the “risk score” recited in claim 1. It is true that much of present claim 7 was in the original disclosure. Yet in the original disclosure, a machine learning model was not mentioned in independent claim 1. If anything, the “vehicle risk” in claim 7 is at least something very closely related to the “risk score for the object relative to the vehicle” recited in claim 1.
The examiner will not write an antecedent basis rejection for the term “vehicle risk” in claim 7 because the term was part of the original disclosure. But it is important to understand the metes and bounds of the claim. Claim 7 states that “the risk threshold for the vehicle is determined” using a machine learning model “that predicts vehicle risk based on historical vehicle data and the driving conditions.” Note that “the risk threshold for the vehicle” refers back to “a risk threshold for the vehicle” recited in claim 1. In claim 1, this threshold is used for classifying an object as a high-risk object when the “risk score for the object relative to the vehicle” is above the “risk threshold for the vehicle.” In other words, even though claim 7 states that this threshold is determined using a machine learning model “that predicts vehicle risk based on historical vehicle data and the driving conditions,” the threshold is still used determine if an object should be categorized as a high-risk object, which is based on “a risk score for the object relative to the vehicle,” as recited in claim 1. So the threshold determination in claim 7 is a narrowing of how that threshold is determined, not how risk for an object is initially determined. In other words, claim 7 gets into the specific details of how the threshold is determined, not how a risk score for an object is determined.
Claims 8 and 15 are substantially similar to claim 1 and rejected for the same reasons and will be interpreted in the same way for examination purposes.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 6-9, 13, 15, 16, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Parikh et al. (U.S. 12,415,549) in view of Lee (US2016/0339959 A1).
Regarding claim 1, Parikh teaches:
A computer-implemented method see Fig. 1 for computer 110. See Figs. 4 and 5 for a method.):
predicting, by a machine learning model, a path of a vehicle based on vehicle telemetry data (see Fig. 5, block 402 and col. 2, lines 7 for determining “what the vehicle plans to do next.” See col. 2, lines 32-35 for determining “a current and/or one or more future operations of the vehicle”. See col. 13, lines 40-63);
capturing, by one or more sensors, driving conditions from a surrounding area (see Fig. 3, block 310 and 312. See col. 3, lines 22-37);
receiving a plurality of object telemetry data from one or more objects in the surrounding area (see col. 11, line 50 through col. 12, line 12);
identifying a recognized object of the one or more objects based on
the capturing (see col. 13, line 64-col. 14, line 22 a system that can “detect object(s) in the environment surrounding the vehicle 202 (e.g., identify that an object exists), classify the object” etc. ) and
the object telemetry data (see col. 11, line 50 through col. 12, line 12.);
predicting, by a second machine learning model (see Fig. 4, block 404), an intended movement of the recognized object based on (see col. 8 lines 49-65 for “The prediction component 116 may use such data to a predict a future state, such as a signage state, position, orientation, velocity, acceleration, or the like, which collectively may be described as prediction data. For example, the prediction component may determine a prediction associated with vehicle object 128 indicating a predicted future position, orientation, velocity, acceleration, and/or state of object 128.” See also col. 13, line 64 through col. 14, line 35.)
the driving conditions (see Fig. 4, block 408) and
the object telemetry data (see col. 11, line 50 through col. 12, line 12.);
calculating, by a third machine learning model, a risk score for the recognized object (see Fig. 4, block 416 and 420. See Fig. 5, block 524) based on
the path of the vehicle (see Fig. 4. The path of the host vehicle is determined upstream of blocks 416 and 420 and used as input for those blocks. See col. 20, lines 52-col. 21, line 4. See col. 1, line 64-col. 2 line 38.) and
the predicted intended movement (see col. 1, line 64-col. 2 line 38. A pedestrian is predicted to be “about to cross a roadway in front of the vehicle”. The machine-learned model assigns a “relatively high score” to this can “mean a risk score that meets or exceeds a risk score threshold”);
responsive to the calculated risk score exceeding a risk threshold, classifying the recognized object as a high-risk object (see col. 1, line 64-col. 2, line 38. A pedestrian is predicted to be “about to cross a roadway in front of the vehicle”. The machine-learned model assigns a “relatively high score” to this can “mean a risk score that meets or exceeds a risk score threshold”).
Yet Parikh does not further teach:
generating an augmented reality display of the surrounding area using an augmented reality device,
wherein the augmented reality display of the surrounding area highlights the high-risk object, and
displays the predicted path of the vehicle and the intended movement of the high-risk object.
However, Lee teaches:
generating an augmented reality display of the surrounding area using an augmented reality device (see Lee paragraph 0118 for a display including a HUD. See paragraph 0396 for the HUD allowing the driver to recognize specific objects. See paragraph 0398 for displaying tracking information on a vehicle or obstacle. See paragraph 0399 for changing the color of the indicator when the obstacle becomes less than a first distance. The indicator can also flash if the obstacle gets even closer. See paragraph 0400 for displaying lanes and routes for the vehicle. See paragraph 0174-0175 for determining a risk and displaying it.),
wherein the augmented reality display of the surrounding area highlights the high-risk object (see paragraph 0399 for changing the color of the indicator when the obstacle becomes less than a first distance. The indicator can also flash if the obstacle gets even closer. See Fig. 19B for item 1021 and paragraph 0398), and
displays the predicted path of the vehicle and the intended movement of the high-risk object (see paragraph 0398 for displaying tracking information on a vehicle or obstacle. See paragraph 0399 for changing the color of the indicator when the obstacle becomes less than a first distance. The indicator can also flash if the obstacle gets even closer. See paragraph 0400 for displaying lanes and routes for the vehicle. See paragraph 0174-0175 for determining a risk and displaying it. See paragraph 0400 for generating a guide route for the host vehicle. See paragraph 0401 for also displaying information that guides a tracking object and information that guides a guide route. See Fig. 21B and paragraph 0418 for displaying a past, current, and “future movement path” of an object. See paragraph 0420 for this being displayed. See paragraph 0421 for displaying tracking information or simulation information regarding the falling object 952, which reasonably applies to other tracked objects such as vehicles.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Parikh, to add the additional features of generating an augmented reality display of the surrounding area using an augmented reality device, wherein the augmented reality display of the surrounding area highlights the high-risk object, and displays the predicted path of the vehicle and the intended movement of the high-risk object, as taught by Lee. The motivation for doing so would be to help the user (driver) intuitively recognizing the state around the host vehicle, as recognized by Lee (see paragraph 0401).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
This combination is especially obvious because Parikh at least strongly teaches toward what Lee more explicitly teaches. Parikh teaches a display screen in the vehicle in col. 11, lines 21-41. Parikh also teaches in Fig. 1, attached below, labeling an object with a score 136 in “representation 106”. See col. 7, lines 36-57 for “Representation 134 further includes an arrow depicting a current and/or predicted heading, position, velocity, and/or acceleration of object 128 that may also be part of the object detection determined in association with object 128. Object 128 may be stopped. The representation further includes depictions of object detections associated with a number of other pedestrians (smaller three-dimensional cuboids) and vehicles and their respective current and/or predicted headings, positions velocities, and/or accelerations that are unlabeled for clarity.” See col. 6, lines 40-44 for “FIG. 1 depicts an example of such a trajectory 124, represented as an arrow indicating a heading, velocity, and/or acceleration, although the trajectory itself may comprise instructions for a controller, which may, in turn, actuate a drive system of the vehicle 102.”
PNG
media_image1.png
542
826
media_image1.png
Greyscale
Regarding claim 2, Parikh and Lee teach the computer-implemented method of claim 1.
Yet Parikh does not further teach:
The computer-implemented method of claim 1, further comprising
transmitting a notification about the high-risk object to the vehicle, wherein the notification is a voice prompt warning.
However, Lee teaches:
transmitting a notification about the high-risk object to the vehicle, wherein the notification is a voice prompt warning (in the present disclosure, see paragraph 0044 for a notification including “a text message displayed on the augmented reality screen” or “a voice prompt”. With that in mind, see Lee paragraph 0420 for providing not only a box or flashing box display, as in paragraphs 0398-0399, but see paragraph 0420 for displaying text regarding an obstacle and paragraph 0422 for displaying flashing regarding an obstacle. See also Fig. 19B for text message 1101.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Parikh and Lee, to add the additional features wherein the augmented reality display of transmitting a notification about the high-risk object to the vehicle, as taught by Lee. The motivation for doing so would be to help the user (driver) intuitively recognizing the state around the host vehicle, as recognized by Lee (see paragraph 0401).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 6, Parikh and Lee teach the computer-implemented method of claim 1.
Parikh further teaches:
The computer-implemented method of claim 1, wherein
thethird machine learning model the risk score by determining a probability of an incident between the vehicle and the recognized object based on prior detected movements of the recognized object and the driving conditions (the present claim has been amended to state that the object related to the probability of an incident is a recognized object. This again returns to the discussion of what “recognized” means. In the examiner’s view, if an object can be detected by sensors, the object is recognized by the host vehicle because recognition and classification are different. The present claim is supported by at least paragraph 0008.
With that in mind, see Parikh col. 1, line 64-col. 2 line 38. A pedestrian is predicted to be “about to cross a roadway in front of the vehicle”. In other words, there is about to be a collision. The machine-learned model assigns a “relatively high score” to this can “mean a risk score that meets or exceeds a risk score threshold”. See co. 21, lines 5-55 for generating a score using the third machine-learned model based on a predicted action of an object.).
Regarding claim 7, Parikh and Lee teach the computer-implemented method of claim 1.
Parikh further teaches:
The computer-implemented method of claim 1, wherein
the risk threshold for the vehicle is determined using a fourth machine learning model that predicts vehicle risk based on historical vehicle data and the driving conditions (see Parikh col. 21, lines 5-35).
Regarding claim 8, Parikh teaches:
A computer system, the computer system comprising (see col. 12, line 60-col. 13, lines 12):
one or more processors, one or more computer readable memories, one or more computer readable storage media, and program instructions stored on at least one of the one or more computer readable storage medium for execution by at least one of the one or more processors via at least one of the one or more memories, wherein the computer system is capable of performing a method comprising (see col. 12, line 60-col. 13, lines 12):
predicting, by a machine learning model, a path of a vehicle based on vehicle telemetry data (for the remainder of the rejection see the analogous bullet points in the rejection of claim 1 which is substantially similar);
capturing, by one or more sensors, driving conditions from a surrounding area;
receiving a plurality of object telemetry data from one or more objects in the surrounding area;
identifying a recognized object of the one or more objects based on the capturing and the object telemetry data;
predicting, by a second machine learning model, an intended movement of the recognized object based on
the driving conditions and
the object telemetry data;
calculating, by a third machine learning model, a risk score for the recognized object based on the path of the vehicle and the predicted intended movement;
responsive to the calculated risk score exceeding a risk threshold, classifying the recognized object as a high-risk object; and
generating an augmented reality display of the surrounding area using an augmented reality device, wherein the augmented reality display of the surrounding area highlights the high-risk object, and displays the predicted path of the vehicle and the intended movement of the high-risk object.
Regarding claim 9, see the rejection of claim 2 which is substantially similar.
Regarding claim 13, see the rejection of claim 6 which is substantially similar.
Regarding claim 15, Parikh teaches:
A computer program product see col. 12, line 60-col. 13, lines 12):
one or more computer[[-]]_readable storage media; program instructions, stored on at least one of the one or more computer[[-]]_readable storage media, the program instructions executable by a processor to cause the processor to perform a method comprising (see col. 12, line 60-col. 13, lines 12):
predicting, by a machine learning model, a path of a vehicle based on vehicle telemetry data (for the remainder of the rejection see the analogous bullet points in the rejection of claim 1 which is substantially similar);
capturing, by one or more sensors, driving conditions from a surrounding area;
receiving a plurality of object telemetry data from one or more objects in the surrounding area;
identifying a recognized object of the one or more objects based on the capturing and the object telemetry data;
predicting, by a second machine learning model, an intended movement of the recognized object based on
the driving conditions and
the object telemetry data;
calculating, by a third machine learning model, a risk score for the recognized object based on the path of the vehicle and the intended movement;
responsive to the calculated risk score exceeding a risk threshold, classifying the recognized object as a high-risk object; and
generating an augmented reality display of the surrounding area using an augmented reality device, wherein the augmented reality display of the surrounding area highlights the high-risk object, and displays the predicted path of the vehicle and the predicted intended movement of the high-risk object.
Regarding claim 16, see the rejection of claim 2 which is substantially similar.
Regarding claim 20, see the rejection of claim 6 which is substantially similar.
Claims 4, 11, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Parikh in view of Lee in further view of Rubin et al. (US2013/0281141).
Regarding claim 4, Parikh and Lee teach the computer-implemented method of claim 1.
Yet Parikh and Lee do not further teach:
The computer-implemented method of claim 1, wherein the identifying the high-risk object further comprises:
responsive to determining that [[a]]the recognized object is not transmitting theobject telemetry data,
However, Rubin teaches:
The computer-implemented method of claim 1, wherein the identifying the high-risk object further comprises:
responsive to determining that [[a]]the recognized object is not transmitting theobject telemetry data,in the present disclosure, paragraph 0038 teaches that V2V, among “further technologies” such as “sensors or cameras” can be used to “source both the telemetry data that may be used by the module 150 to learn about the vehicle and the surrounding area, including recognizing and communicating with objects in the surrounding area. Paragraph 0039 teaches that “In recognizing objects…the vehicle…may also monitor relevant data, such as telemetry for the object…that may be transmitted by the recognized object…and the absence of relevant data from the object may cause the module 150 [of the host vehicle] to immediately classify the object as a high-risk object”. It seems from this paragraph that an object has been recognized, perhaps through a camera, but there is an “absence of relevant data,” such as telemetry data, coming “from the object”. The paragraph does not say explicitly whether or not the object had been transmitting telemetry data via V2V and then stopped, or whether the object was recognized some other way and then the system simply noticed an “absence of relevant data” such as telemetry data. In either case, the object could be recognized and then “the absence of relevant data” causes the host vehicle to “immediately classify the object as a high-risk object to be highlighted in the augmented reality display to the driver.”
With that in mind, see Rubin paragraph 0570 for determining if a vehicle has a “failed V2V transmitter” and that information being recorded and “the risk broadcast”. See paragraph 0737 for displaying a vehicle “with the highest risk” such as a vehicle that is running a stop sign. See paragraph 0736 for assigning a color to a vehicle to make it identifiable in the display of the host vehicle. The paragraph also teaches displaying nearby vehicles along with “any other identifying information” that can be “updated as necessary on the display”. See paragraph 0738 for displaying this information using a HUD. The paragraph teaches displaying highlighting and blinking around the vehicle. Overall, Rubin teaches a HUD that can display high-risk vehicles and can be updated as necessary. This could reasonably include vehicles with a failed V2V transmitter, since those are considered high-risk to Rubin. It is reasonable to interpret this to mean that the system of Rubin may recognize a vehicle, then the vehicle’s transmitter may fail, telemetry data is not longer being transmitted. This is considered high risk. In response, this high risk vehicle is highlighted in the HUD of the cabin.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Parikh and Lee to add the additional features as indicated as being taught by Rubin. The motivation for doing so would be allow a driver or vehicle occupant to “rapidly and easily identify a particular vehicle,” as recognized by Rubin (see paragraphs 0737-0738).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claims 11 and 18, see the rejection of claim 4 which is substantially similar.
Additional Art
The prior art made of record here, though not relied upon, is considered pertinent to the present disclosure.
Arnicar (U.S. 10,332,292), a Zoox disclosure, teaches a great deal about displaying images for a driver. In many ways, the reference can be read as teaching displaying everything shown in Parikh Fig. 1, which is also a Zoox disclosure.
PNG
media_image2.png
546
794
media_image2.png
Greyscale
PNG
media_image3.png
542
752
media_image3.png
Greyscale
One close prior art is Esna Ashari Esfahani et al. (U.S. 12,365,335), hereinafter, Esfahani. Esfahani teaches generating a lot of scores related to dangerous driving of other objects. This scoring can be done using machine learning. “The machine learning classifier model has been trained to output the plurality of mutual interaction violation classification scores based on the plurality of measurements of the first remote vehicle and the plurality of measurements of the second remote vehicle.” See Fig. 2 below.
Esfahani, claim 1 recites (with some relevant parts in bold):
1. A system for detecting hazards for a vehicle, the system comprising: a vehicle sensor for determining information about an environment surrounding the vehicle; a global navigation satellite system (GNSS) for determining a geographical location, heading, and orientation of the vehicle; a controller in electrical communication with the vehicle sensor and the GNSS, wherein the controller is programmed to: perform a plurality of measurements of a first remote vehicle using the vehicle sensor to determine a position, heading, and velocity of the first remote vehicle; determine a plurality of classification scores of the first remote vehicle based at least in part on the plurality of measurements of the first remote vehicle, wherein to determine the plurality of classification scores,
the controller is further programmed to:
determine a plurality of position and speed violation classification scores of the first remote vehicle using a first machine learning classifier model;
determine a plurality of mutual interaction violation classification scores of the first remote vehicle using a second machine learning classifier model;
determine an anomaly detection score of the first remote vehicle using a machine learning anomaly detection model;
determine a plurality of traffic rule violation classification scores of the first remote vehicle using a third machine learning classifier model; and
determine a plurality of visual hazard classification scores of the first remote vehicle using a fourth machine learning classifier model;
determine an overall hazard score of the first remote vehicle based at least in part on the plurality of classification scores of the first remote vehicle,
wherein the overall hazard score of the first remote vehicle is a weighted exponential moving average of the plurality of position and speed violation classification scores, the plurality of mutual interaction violation classification scores, the anomaly detection score, the plurality of traffic rule violation classification scores, and the plurality of visual hazard classification scores; and
control the vehicle using an automated routing system to guide the vehicle away from the first remote vehicle in response to determining that the overall hazard score of the first remote vehicle is above a predetermined overall hazard score threshold.
One difference between Esfahani and the present claim 1 is that present claim 1 performs “predicting” while Esfahani is concerned with classifying or scoring what has already occurred. That is to say, Esfahani is concerned with scoring the past while the present disclosure is focused, at least in part, with scoring the future. Present claim 1 recites in part “predicting, by a machine learning model, a path of a vehicle based on vehicle telemetry data;”. Even if this can be interpreted to mean that the system knows, or predicts, where the host vehicle is going to go based on where the vehicle is currently headed, there is a still a predicting aspect. Thus when present claim 1 later recites calculating a risk score based on “the path of the vehicle” and the predicted intended movement, this “path of the vehicle” is in fact the predicted path of the vehicle.
Present claim 1 also recites “predicting, by a second machine learning model, an intended movement of the recognized object”. Here again, predicting a future movement is involved.
Furthermore, the machine learning models in Esfahani do not have the same functions as those of present claim 1.
PNG
media_image4.png
734
550
media_image4.png
Greyscale
Another close prior art is Yoon (US2020/0010081), an LG disclosure. Claim 1 recites:
10. The autonomous vehicle of claim 1, the recognizer, comprising:
an object recognizer configured to recognize the first information [that is state information on at least one object near the vehicle, according to claim 1] on the basis of a first algorithm model based on an artificial neural network;
a space recognizer configured to recognize the second information [that is state information on at least one space near the vehicle, according to claim 1] on the basis of a second algorithm model based on artificial neural network; and
a line recognizer configured to recognize the third information [that is state information on at least one line near the vehicle, according to claim 1] on the basis of a third algorithm model based on an artificial neural network, and
wherein each of the first algorithm model, the second algorithm model and the third algorithm model includes an input layer comprised of input nodes, an output layer comprised of output nodes, and one or more hidden layers disposed between the input layer and the output layer and comprised of hidden nodes, and weights of edges that connect nodes, and biases of nodes are updated through learning.
Yet Yoon does not teach predicting paths of the vehicle and object, in contrast to present claim 1.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL M. ROBERT whose telephone number is (571)270-5841. The examiner can normally be reached M-F 7:30-4:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached at 571-272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL M. ROBERT/Primary Examiner, Art Unit 3665
Prosecution Insights