DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/05/2026 has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1, 2, 11-17 are canceled.
Claims 3-10 and 18-21 are pending and have been examined.
This action is in reply to the papers filed on 07/28/2026 (effective filing date 06/01/2023).
Information Disclosure Statement
The information disclosure statement(s) submitted: 06/01/2023, has/have been considered by the Examiner and made of record in the application file.
Amendment
The present Office Action is based upon the original patent application filed on 06/01/2023 as modified by the amendment filed on 01/05/2026.
Reasons For Allowance
Prior-Art Rejection withdrawn
Claims xxx are allowed. The closest prior art (See PTO-892, Notice of References Cited) does not teach the claimed:
The closest prior-art (xxx) teach the features as disclosed in Non-final Rejection (xxxx), however, these cited references do not teach and the prior-art does not teach at least the following:
Claim Rejections - 35 USC §101 - Withdrawn
Per Applicant’s amendments and arguments and considering new guidance in the MPEP, the rejections are withdrawn. Specifically, in Applicant’s Remarks (dated 08/29/2025, pgs. 16-19), Applicant traverses the 35 USC §101 rejections arguing that the amended claims recite new limitations that are not abstract, amount to significantly more, are directed to a practical application, etc…
Claim Rejections - 35 USC § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 3-10 and 18-21 are rejected under 35 U.S.C. § 101 as being directed to non-statutory subject matter because the claimed invention is directed to an abstract idea without significantly more. These claims recite a method and system for recommending roadway infrastructure maintenance tasks.
Claim 19 recites [a] method, comprising: retrieving map data associated with an environment, the map data indicates a mapped infrastructure element; collecting sensor data from a vehicle sensor system, the sensor data indicates a motorist perception of the mapped infrastructure element; inferring, from the sensor data and the map data, a perception state of the mapped infrastructure element; retrieving an infrastructure record that indicates a status of the mapped infrastructure element; and recommending a maintenance task to be performed based on the perception state of the mapped infrastructure element and the infrastructure record.
The claims are being rejected according to the 2019 Revised Patent Subject Matter Eligibility Guidance (Federal Register, Vol. 84, No. 5, p. 50-57 (Jan. 7, 2019)).
Step 1: Does the Claim Fall within a Statutory Category?
Yes. Claims 18-21 recite a method and, therefore, are directed to the statutory class of a process. Claims 3-10 recite a system/apparatus and, therefore, are directed to the statutory class of machine.
Step 2A, Prong One – Claim 19 Recites a Judicial Exception
Claim 19 recites a judicial exception in the form of a mental process, i.e., concepts performed in the human mind, including observations, evaluations, judgments, and opinions.
Under the broadest reasonable interpretation, the claim encompasses collecting and obtaining information, evaluating that information, making a determination based on the evaluation, and recommending an action based on the determination. These are mental processes identified as abstract ideas in the MPEP § 2106.04(a)(2).
Specifically, claim 19 recites the following limitations:
Claim Limitation
Mental Process Grouping
"retrieving map data associated with an environment, the map data indicates a mapped infrastructure element"
Obtaining information for evaluation (observation)
"collecting sensor data from a vehicle sensor system, the sensor data indicates a motorist perception of the mapped infrastructure element"
Collecting or observing information
"inferring, from the sensor data and the map data, a perception state of the mapped infrastructure element"
Evaluating information and making a judgment or determination
"retrieving an infrastructure record that indicates a status of the mapped infrastructure element"
Obtaining additional information for consideration
"recommending a maintenance task to be performed based on the perception state of the mapped infrastructure element and the infrastructure record"
Forming an opinion or recommendation based on the evaluation
Collectively, these limitations recite the abstract process of gathering information from multiple sources, evaluating that information to determine a perception state of an infrastructure element, considering additional information regarding the infrastructure element, and recommending a maintenance task.
For example, a person could observe how motorists perceive a roadway sign or other infrastructure element, consult a map identifying that infrastructure element, review maintenance records indicating its current status, determine whether motorists are correctly perceiving the infrastructure element, and recommend an appropriate maintenance activity. These acts involve observations, evaluations, judgments, and recommendations, which are mental processes.
Although the claim recites collecting sensor data using a vehicle sensor system, the vehicle sensor system merely gathers information that is subsequently evaluated. Likewise, the recited retrieval of map data and infrastructure records merely obtains information for use in the evaluation. Under the broadest reasonable interpretation, these additional limitations simply identify the sources of the information that is analyzed and do not alter the character of the claim, which is directed to evaluating information and recommending an action based on that evaluation.
Accordingly, the claim recites steps that, under their broadest reasonable interpretation, encompass mental observations, evaluations, judgments, and recommendations. The recited vehicle sensor system, map data, and infrastructure record merely provide information that is evaluated, while the focus of the claim remains the determination of a perception state and the recommendation of a maintenance task. Therefore, claim 19 recites a mental process, which is an abstract idea under Step 2A, Prong One of the 2019 Revised Patent Subject Matter Eligibility Guidance.
Step 2A, Prong Two
The additional elements beyond the judicial exception are:
map data;
a vehicle sensor system;
sensor data;
an infrastructure record; and
a mapped infrastructure element.
These additional elements do not integrate the judicial exception into a practical application. The claimed vehicle sensor system merely serves as a source of data. The claim does not recite any improvement to sensor technology, vehicle operation, map generation, communication protocols, image processing, or computer functionality. Likewise, retrieving map data and infrastructure records merely obtains information for use in the abstract analysis. The inference step merely uses the collected information to determine a perception state, while the recommending step merely provides advice regarding a maintenance task.
The claim does not:
improve the functioning of a computer;
improve vehicle sensors;
improve map generation;
improve infrastructure maintenance equipment;
effect any transformation of an article;
control machinery;
automatically perform maintenance;
alter operation of a vehicle; or
otherwise apply the judicial exception in a meaningful technological manner.
Instead, the additional elements merely gather data from conventional sources and use that data as inputs to the abstract analytical process. Accordingly, the additional elements merely link the judicial exception to a technological environment and do not impose any meaningful limit on the judicial exception. Therefore, claim 1 is directed to the abstract idea itself and does not integrate the judicial exception into a practical application.
Step 2B
Because the claim is directed to a judicial exception without integration into a practical application, it is necessary to determine whether the claim includes additional elements that amount to significantly more than the judicial exception.
The additional elements include:
retrieving map data;
collecting sensor data using a vehicle sensor system;
retrieving an infrastructure record; and
recommending a maintenance task.
These elements, individually and in combination, merely implement the abstract idea using generic computer technology and generic data sources performing their ordinary functions.
Specifically,
retrieving data is a generic computer function;
collecting data from sensors is a routine function of conventional vehicle sensor systems;
retrieving stored records is a routine database function;
inferring information from collected data represents the abstract analysis itself;
recommending a maintenance task merely communicates the analytical result.
The claim does not recite:
a particular improvement in machine learning,
a new sensor architecture,
improved image processing,
improved map generation,
improved computer operation,
specialized hardware,
unconventional data structures,
unconventional data collection techniques,
or any technological mechanism that performs more than the ordinary use of generic computer components.
Considering the claim as an ordered combination likewise fails to amount to significantly more. The sequence of gathering information, analyzing the information, consulting existing records, and providing a recommendation merely automates a process that could otherwise be performed mentally or manually by an infrastructure inspector using conventional computer tools. Accordingly, the additional elements amount to no more than instructions to implement the abstract idea using generic computer components performing their well-understood, routine, and conventional functions. Therefore, claim 1 does not include significantly more than the judicial exception.
Conclusion - 35 U.S.C. § 101
Claim 1 is directed to the abstract idea of collecting information regarding infrastructure, analyzing the information to infer a perception state, and recommending a maintenance task, which is a mental process. The additional elements do not integrate the judicial exception into a practical application and merely use generic computer technology to implement the abstract idea. Furthermore, the additional elements, individually and as an ordered combination, do not amount to significantly more than the judicial exception because they merely perform well-understood, routine, and conventional computer functions.
Accordingly, claim 1 is rejected under 35 U.S.C. § 101.
Additionally, pursuant to the requirement under Berkheimer, the following citations are provided to demonstrate that the additional elements, identified as extra-solution activity, amount to activities that are well-understood, routine, and conventional. See MPEP 2106.05(d).
Capturing an image (code) with an RFID reader. Ritter, US Patent No. 7734507 (Col. 3, Lines 56-67); “RFID: Riding on the Chip” by Pat Russo. Frozen Food Age. New York: Dec. 2003, vol. 52, Issue 5; page S22.
Receiving or transmitting data over a network. Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014).
Storing and retrieving information in memory. Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93.
Outputting/Presenting data to a user. Mayo, 566 U.S. at 79, 101 USPQ2d at 1968; OIP Techs., Inc. v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1092-93 (Fed. Cir. 2015); MPEP 2106.05(g)(3).
Using a machine learning model to determine user segment characteristics for an ad campaign. https://whites.agency/blog/how-to-use-machine-learning-for-customer-segmentation/.
Thus, taken alone and in combination, the additional elements do not amount to significantly more than the above-identified judicial exception (the abstract idea), and are ineligible under 35 USC 101.
Independent system claims 3, 5, 6, 7, 9 and independent method claim 20 also contains the identified abstract ideas, with the additional elements of a processor and storage medium, which are a generic computer components, and thus not significantly more for the same reasons and rationale above.
Dependent claims 4, 8, 10 ,18, 21 further describe the abstract idea. The additional elements of the dependent claims fail to integrate the abstract idea into a practical application and do not amount to significantly more than the abstract idea. Thus, as the dependent claims remain directed to a judicial exception, and as the additional elements of the claims do not amount to significantly more, the dependent claims are not patent eligible.
As such, the claims are not patent eligible.
Invention Could be Performed Manually
It is conceivable that the invention could be performed manually without the aid of machine and/or computer. For example, Applicant claims retrieving map data, collecting sensor data, inferring a perception state, and recommending a maintenance task to be performed, etc… Each of these features could be performed manually and/or with the aid of a simple generic computer to facilitate the transmission of data.
See also Leapfrog Enterprises, Inc. v. Fisher-Price, Inc., and In re Venner, which stand for the concept that automating manual activity and/or applying modern electronics to older mechanical devices to accomplish the same result is not sufficient to distinguish over the prior art. Here, applicant is merely claiming computers to facilitate and/or automate functions which used to be commonly performed by a human.
Leapfrog Enterprises, Inc. v. Fisher-Price, Inc., 485 F.3d 1157, 82 USPQ2d 1687 (Fed. Cir. 2007) "[a]pplying modern electronics to older mechanical devices has been commonplace in recent years…"). The combination is thus the adaptation of an old idea or invention using newer technology that is commonly available and understood in the art.
In In re Venner, 262 F.2d 91, 95, 120 USPQ 193, 194 (CCPA 1958), the court held that broadly providing an automatic or mechanical means to replace manual activity which accomplished the same result is not sufficient to distinguish over the prior art. MPEP 2144.04, III Automating a Manual Activity.
MPEP 2144.04 III - Automating a Manual Activity and In re Venner, 262 F.2d 91, 95, 120 USPQ 193, 194 (CCPA 1958) further stand for and provide motivation for using technology, hardware, computer, or server to automate a manual activity.
Therefore, the Office finds no improvements to another technology or field, no improvements to the function of the computer itself, and no meaningful limitations beyond generally linking the use of an abstract idea to a particular technological environment. Therefore, based on the two-part Alice Corp. analysis, there are no limitations in any of the claims that transform the exception (i.e., the abstract idea) into a patent eligible application.
Claim Rejections - Not an Ordered Combination
None of the limitations, considered as an ordered combination provide eligibility, because taken as a whole, the claims simply instruct the practitioner to implement the abstract idea with routine, conventional activity.
Claim Rejections - Preemption
Allowing the claims, as presently claimed, would preempt others from recommending roadway infrastructure maintenance tasks. Furthermore, the claim language only recites the abstract idea of performing this method, there are no concrete steps articulating a particular way in which this idea is being implemented or describing how it is being performed.
Claim Rejections - 35 USC §112
Per Applicants’ amendments/arguments, the rejection(s)/objection(s) is/are withdrawn.
The following is a quotation of 35 U.S.C. §112(a):
IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 3-10 and 18-21 are rejected under 35 U.S.C. §112(a) as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor had possession of the claimed invention.
New Matter Rejection - Regarding Claim 3, Applicant’s originally filed specification (PGPub. 2024/0403834) fails to disclose the amended claimed features as follows: recommend a maintenance task to be performed based on the perception state of the mapped infrastructure element and a detected discrepancy between the map data and the images of the environment…
New Matter Rejection - Regarding Independent Claims 3, 5, 6, 7, 9, 19, Applicant’s originally filed specification (PGPub. 2024/0403834) fails to disclose the amended claimed features as follows: “infer a perception state”. The originally filed claims and specification (filed 06/01/2023) disclose “infer a perceived infrastructure element…” and “inferring… a perceived infrastructure element…”. However, as originally filed, there is no disclose of “infer a perception state” and/or “inferring a perception state”. Furthermore, the words “perceived” and “perception” are not the same (not synonymous); "perceived" is the past tense of the verb "perceive," while "perception" is a noun that refers to the interpretation of sensory information.
Applicant is directed to address each rejected claim individually by specifically indicating support, in the originally filed specification (i.e., page, paragraph, and line), for the rejected claimed feature(s).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Rezvan Behbahani et al. 2021/0339741; in view of Nagata et al. 2020/0302193; in view of Barzelay et al. 2019/0188521.
18/327,312 – Claim 3. (Previously Presented) Leonard et al. 2019/0283756 teaches A system, comprising: a processor; and a memory storing machine-readable instructions that, when executed by the processor (Leonard et al. 2019/0283756 [Abstract; 0004; 0022-0024]), cause the processor to: retrieve map data associated with an environment, wherein the map data (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data][0016 - an object detection system … a location determination system]) indicates a mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment][0028 - in FIG. 2, the object detection system 130 may detect obstacle 210 located along a predicted trajectory 212 of the vehicle][0038 - the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); collect sensor data comprising images of the environment from a vehicle sensor system (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.), the sensor data indicates a motorist perception of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like.); infer a perception state of the mapped infrastructure element based on the sensor data and the map data (Leonard et al. 2019/0283756 [0012 - system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like. [0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); identify that the mapped infrastructure element is not captured in an image of the environment of the vehicle; and recommend a maintenance task to be performed based on the perception state of the mapped infrastructure element (Leonard et al. 2019/0283756 [0005 - developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors…] In yet another embodiment, a method includes receiving a signal indicating a driver of a vehicle is attempting to make a turn at an intersection. The method further includes accessing, by a computer, a behavioral profile that is representative of the intersection. The method also includes developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors and the behavioral profile of the intersection. Finally, the method includes calculating at least one maneuver to be performed by one or more vehicle systems of the vehicle based at least in part on at least one of the perception of the driver and the risk of the intersection. [0038 - perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.) and a detected discrepancy between the map data and the images of the environment (Leonard et al. 2019/0283756 [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment] As noted above, the logic stored on the one or more memory modules 106 may include object recognition logic. The object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment. Example object recognition algorithms include, but are not limited to, edge detection algorithms, corner detection algorithms, blob detection algorithms, and feature description algorithms (e.g., scale-invariant feature transform (“SIFT”), speeded up robust features (“SURF”), gradient location and orientation histogram (“GLOH”), and the like). The logic stored on the electronic control unit may also include speech recognition logic used to detect the words spoken by the driver and/or passengers within the vehicle 100. Any known or yet-to-be-developed speech recognition algorithms may be used for the speech recognition logic.).
Leonard et al. 2019/0283756 may not expressly disclose the “identify that the mapped infrastructure element is not captured in an image of the environment of the vehicle” features, however, Nagata et al. 2020/0302193 teaches (Nagata et al. 2020/0302193 [0081 -determines whether an unidentified obstacle, which is an obstacle included in a plurality of obstacles that are not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle, is present based on the obtained first information and vehicle information, and map information indicating a map of the vicinity of specific vehicle] First determiner 41 determines whether an unidentified obstacle, which is an obstacle included in a plurality of obstacles that are not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle, is present based on the obtained first information and vehicle information, and map information indicating a map of the vicinity of specific vehicle 20. FIG. 3 is a schematic illustration illustrating a relationship between specific vehicle 20 and an unidentified obstacle in information processing system 1 according to Embodiment 1. An unidentified obstacle includes not only an obstacle that is not visible from specific vehicle 20, but also an obstacle that is difficult to see, that is, an obstacle that is not at all visible from specific vehicle 20. [0083 - map information here indicates a map of the vicinity of specific vehicle…] The map information here indicates a map of the vicinity of specific vehicle 20 and includes, for example, an internal structure (floor map) of a parking facility, for example. [0087 - The map information may be pre-stored in a recording medium included in information processing device] The map information may be pre-stored in a recording medium included in information processing device 40 or may be obtained from, for instance, other server device via a network. [0095 - obstacle when viewed from specific vehicle 20 and is not visible from specific vehicle 20, is present based on the first information, the vehicle information, and map information…] Information processing device 40 determines whether an unidentified obstacle, which is included in obstacles each being hidden behind other obstacle when viewed from specific vehicle 20 and is not visible from specific vehicle 20, is present based on the first information, the vehicle information, and map information. Specifically, first determiner 41 in information processing device 40 specifies one or more moving obstacles (one example of the first obstacles) among obstacles in the vicinity of specific vehicle 20, based on the first information and the vehicle information. First determiner 41 determines, for each of one or more moving obstacles, whether other obstacle indicated in the first information or the map information is present between specific vehicle 20 and the moving obstacle. When such other obstacle is present between specific vehicle 20 and the moving obstacle, first determiner 41 further determines, for each of the second obstacles indicated in the second information received from specific vehicle 20, whether the second obstacle is identical to a corresponding one of one or more moving obstacles. When determining that the second obstacle is identical to the corresponding one of one or more moving obstacles, first determiner 41 determines that the second obstacle is not an unidentified obstacle. Stated differently, first determiner 41 does not determine, as an unidentified obstacle, a moving obstacle that is indicated in the first information and is identical to a corresponding one of the second obstacles indicated in the second information. First determiner 41 thus determines, as an unidentified obstacle, at least one first obstacle (moving obstacle) remaining after the second obstacles have been excluded from the first obstacles, and determines that at least one unidentified obstacle is present (S13). When Yes at step S13, first determiner 41 transmits information indicating at least one unidentified obstacle to specific vehicle 20. [0101 -determines whether an unidentified obstacle, which is included in a plurality of obstacles detected by surveillance camera 30 and is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20, is present based on the first information regarding the plurality of obstacles detected by surveillance camera 30 and the map information indicating a map of the vicinity of specific vehicle] As described above, in information processing system 1 and the information processing method, first determiner 41 determines whether an unidentified obstacle, which is included in a plurality of obstacles detected by surveillance camera 30 and is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20, is present based on the first information regarding the plurality of obstacles detected by surveillance camera 30 and the map information indicating a map of the vicinity of specific vehicle 20. When first determiner 41 determines that at least one unidentified obstacle is present, first communicator 42 outputs information indicating at least one unidentified obstacle to specific vehicle 20. This enables specific vehicle 20 to obtain the information indicating at least one unidentified obstacle that is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20. Specific vehicle 20 is thus capable of performing an operation based on the information indicating at least one unidentified obstacle.[0117; 0140; 0175]). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Nagata et al. 2020/0302193. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a maintenance task to be performed” features, however Rezvan Behbahani et al. 2021/0339741 teaches (Rezvan Behbahani et al. 2021/0339741 [0093 - recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades] The requirements system 852 may then select a set of requirements for the scenario/vehicle type (e.g., set of vehicles having similar equipment or capabilities) combination to send to the vehicles 802 as the requirement data 842. In some cases, the requirements system 852 may also recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Rezvan Behbahani et al. 2021/0339741. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “detected discrepancy” features, however, Barzelay et al. 2019/0188521 teaches (Barzelay et al. 2019/0188521 [0007 - comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object] According to an aspect of the present invention, there is a method, computer program product and/or computer system for performing the following operations (not necessarily in the following order): (i) receiving a first video image that includes a first initial version infrastructure object image showing a first infrastructure object with the first initial version infrastructure object image being characterized by a first viewing vector; (ii) receiving a second video image that includes a second initial version infrastructure object image showing the first infrastructure object with the second initial version infrastructure object image being characterized by a second viewing vector that is at least approximately parallel to the first viewing vector; (iii) selecting the first initial version infrastructure image from the first video; (iv) analyzing, by machine logic, the second video image to determine that the second initial version infrastructure object image is a match with an identical instance of the first initial version infrastructure object image, with the analysis of the second video image including: (a) constructing, by machine logic, a three dimensional (3D) data model of at least a portion of the environment around the first infrastructure object based, at least in part, upon the first and second video images, and (b) determining the match based, at least in part, upon the respective relationships of the first and second initial infrastructure object images to the 3D data model; (v) adjusting, by machine logic, at least one of the first and second initial version infrastructure object image to obtain a plurality of adjusted infrastructure image objects respectively corresponding to the first and second initial version infrastructure object images, with the plurality of adjusted infrastructure object images showing the first infrastructure object aligned with itself across the plurality of adjusted infrastructure object images; (vi) comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Barzelay et al. 2019/0188521. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks including neural network/machine learning tools which should prove to improve user experience, maximize profits, and optimize revenue.
Claims 3, 4, 8, 10 are rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Agarwal et al. 2022/0198842; in view of Nagata et al. 2020/0302193; in view of Barzelay et al. 2019/0188521.
18/327,312 – Claim 3. (Previously Presented) Leonard et al. 2019/0283756 teaches A system, comprising: a processor; and a memory storing machine-readable instructions that, when executed by the processor (Leonard et al. 2019/0283756 [Abstract; 0004; 0022-0024]), cause the processor to: retrieve map data associated with an environment, wherein the map data (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data]) indicates a mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment][0028 - in FIG. 2, the object detection system 130 may detect obstacle 210 located along a predicted trajectory 212 of the vehicle][0038 - the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); collect sensor data comprising images of the environment from a vehicle sensor system (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.), the sensor data indicates a motorist perception of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like.); infer a perception state of the mapped infrastructure element based on the sensor data and the map data (Leonard et al. 2019/0283756 [0012 - system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like. [0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); identify that the mapped infrastructure element is not captured in an image of the environment of the vehicle; and recommend a maintenance task to be performed based on the perception state of the mapped infrastructure element (Leonard et al. 2019/0283756 [0005 - developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors…] In yet another embodiment, a method includes receiving a signal indicating a driver of a vehicle is attempting to make a turn at an intersection. The method further includes accessing, by a computer, a behavioral profile that is representative of the intersection. The method also includes developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors and the behavioral profile of the intersection. Finally, the method includes calculating at least one maneuver to be performed by one or more vehicle systems of the vehicle based at least in part on at least one of the perception of the driver and the risk of the intersection. [0038 - perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.) and a detected discrepancy between the map data and the images of the environment (Leonard et al. 2019/0283756 [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment] As noted above, the logic stored on the one or more memory modules 106 may include object recognition logic. The object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment. Example object recognition algorithms include, but are not limited to, edge detection algorithms, corner detection algorithms, blob detection algorithms, and feature description algorithms (e.g., scale-invariant feature transform (“SIFT”), speeded up robust features (“SURF”), gradient location and orientation histogram (“GLOH”), and the like). The logic stored on the electronic control unit may also include speech recognition logic used to detect the words spoken by the driver and/or passengers within the vehicle 100. Any known or yet-to-be-developed speech recognition algorithms may be used for the speech recognition logic.).
Leonard et al. 2019/0283756 may not expressly disclose the “identify that the mapped infrastructure element is not captured in an image of the environment of the vehicle” features, however, Nagata et al. 2020/0302193 teaches (Nagata et al. 2020/0302193 [0081 - determines whether an unidentified obstacle, which is an obstacle included in a plurality of obstacles that are not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle, is present based on the obtained first information and vehicle information, and map information indicating a map of the vicinity of specific vehicle] First determiner 41 determines whether an unidentified obstacle, which is an obstacle included in a plurality of obstacles that are not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle, is present based on the obtained first information and vehicle information, and map information indicating a map of the vicinity of specific vehicle 20. FIG. 3 is a schematic illustration illustrating a relationship between specific vehicle 20 and an unidentified obstacle in information processing system 1 according to Embodiment 1. An unidentified obstacle includes not only an obstacle that is not visible from specific vehicle 20, but also an obstacle that is difficult to see, that is, an obstacle that is not at all visible from specific vehicle 20. [0083 - map information here indicates a map of the vicinity of specific vehicle…] The map information here indicates a map of the vicinity of specific vehicle 20 and includes, for example, an internal structure (floor map) of a parking facility, for example. [0087 - The map information may be pre-stored in a recording medium included in information processing device] The map information may be pre-stored in a recording medium included in information processing device 40 or may be obtained from, for instance, other server device via a network. [0095 - obstacle when viewed from specific vehicle 20 and is not visible from specific vehicle 20, is present based on the first information, the vehicle information, and map information…] Information processing device 40 determines whether an unidentified obstacle, which is included in obstacles each being hidden behind other obstacle when viewed from specific vehicle 20 and is not visible from specific vehicle 20, is present based on the first information, the vehicle information, and map information. Specifically, first determiner 41 in information processing device 40 specifies one or more moving obstacles (one example of the first obstacles) among obstacles in the vicinity of specific vehicle 20, based on the first information and the vehicle information. First determiner 41 determines, for each of one or more moving obstacles, whether other obstacle indicated in the first information or the map information is present between specific vehicle 20 and the moving obstacle. When such other obstacle is present between specific vehicle 20 and the moving obstacle, first determiner 41 further determines, for each of the second obstacles indicated in the second information received from specific vehicle 20, whether the second obstacle is identical to a corresponding one of one or more moving obstacles. When determining that the second obstacle is identical to the corresponding one of one or more moving obstacles, first determiner 41 determines that the second obstacle is not an unidentified obstacle. Stated differently, first determiner 41 does not determine, as an unidentified obstacle, a moving obstacle that is indicated in the first information and is identical to a corresponding one of the second obstacles indicated in the second information. First determiner 41 thus determines, as an unidentified obstacle, at least one first obstacle (moving obstacle) remaining after the second obstacles have been excluded from the first obstacles, and determines that at least one unidentified obstacle is present (S13). When Yes at step S13, first determiner 41 transmits information indicating at least one unidentified obstacle to specific vehicle 20. [0101 - determines whether an unidentified obstacle, which is included in a plurality of obstacles detected by surveillance camera 30 and is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20, is present based on the first information regarding the plurality of obstacles detected by surveillance camera 30 and the map information indicating a map of the vicinity of specific vehicle] As described above, in information processing system 1 and the information processing method, first determiner 41 determines whether an unidentified obstacle, which is included in a plurality of obstacles detected by surveillance camera 30 and is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20, is present based on the first information regarding the plurality of obstacles detected by surveillance camera 30 and the map information indicating a map of the vicinity of specific vehicle 20. When first determiner 41 determines that at least one unidentified obstacle is present, first communicator 42 outputs information indicating at least one unidentified obstacle to specific vehicle 20. This enables specific vehicle 20 to obtain the information indicating at least one unidentified obstacle that is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20. Specific vehicle 20 is thus capable of performing an operation based on the information indicating at least one unidentified obstacle.[0117; 0140; 0175]). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Nagata et al. 2020/0302193. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommending roadway infrastructure maintenance tasks” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “detected discrepancy” features, however, Barzelay et al. 2019/0188521 teaches (Barzelay et al. 2019/0188521 [0007 - comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object] According to an aspect of the present invention, there is a method, computer program product and/or computer system for performing the following operations (not necessarily in the following order): (i) receiving a first video image that includes a first initial version infrastructure object image showing a first infrastructure object with the first initial version infrastructure object image being characterized by a first viewing vector; (ii) receiving a second video image that includes a second initial version infrastructure object image showing the first infrastructure object with the second initial version infrastructure object image being characterized by a second viewing vector that is at least approximately parallel to the first viewing vector; (iii) selecting the first initial version infrastructure image from the first video; (iv) analyzing, by machine logic, the second video image to determine that the second initial version infrastructure object image is a match with an identical instance of the first initial version infrastructure object image, with the analysis of the second video image including: (a) constructing, by machine logic, a three dimensional (3D) data model of at least a portion of the environment around the first infrastructure object based, at least in part, upon the first and second video images, and (b) determining the match based, at least in part, upon the respective relationships of the first and second initial infrastructure object images to the 3D data model; (v) adjusting, by machine logic, at least one of the first and second initial version infrastructure object image to obtain a plurality of adjusted infrastructure image objects respectively corresponding to the first and second initial version infrastructure object images, with the plurality of adjusted infrastructure object images showing the first infrastructure object aligned with itself across the plurality of adjusted infrastructure object images; (vi) comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Barzelay et al. 2019/0188521. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks including neural network/machine learning tools which should prove to improve user experience, maximize profits, and optimize revenue.
18/327,312 – Claim 4. (Previously Presented) Leonard et al. 2019/0283756 further teaches The system of claim 3, wherein: the sensor data further comprises vehicle operational data (Leonard et al. 2019/0283756 [0004 – a vehicle includes a plurality of sensors configured to output a plurality of operational signals][0016 – a vehicle operating condition sensor system][0035 – vehicle operating condition sensor system may include any device configured to detect one or more operating conditions of the vehicle … the vehicle operating condition sensor system may detect vehicle speed, direction, acceleration, braking, position of the steering wheel, and the like. As such, the vehicle operating condition sensor system may include an operation condition sensor configured to output an operational signal indicative of one or more operations of the vehicle. The operation condition sensor may include, but is not limited to, a speed sensor, an acceleration sensor, a braking sensor, a steering angle sensor, and the like.]); and the machine-readable instruction to recommend the maintenance task further comprises a machine-readable instruction that, when executed by the processor, causes the processor to recommend the maintenance task based on the map data (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data]), the images of the environment (Leonard et al. 2019/0283756 [0027; 0033; 0038; 0055]), and the vehicle operational data (Leonard et al. 2019/0283756 [0004; 0016; 0035]).
Leonard et al. 2019/0283756 may not expressly disclose the “recommend the maintenance task” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
18/327,312 – Claim 8. (Previously Presented) Leonard et al. 2019/0283756 further teaches The system of claim 3, wherein the machine-readable instruction to recommend the maintenance task further comprises a machine-readable instruction that, when executed by the processor (Leonard et al. 2019/0283756 [Abstract; 0004; 0022-0024]), causes the processor to recommend a repair of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data][0032 - lampposts, signs, crosswalks, and other objects]).
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a repair” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
18/327,312 – Claim 10. (Previously Presented) Leonard et al. 2019/0283756 further teaches The system of claim 3, wherein the machine-readable instruction to recommend the maintenance task further comprises a machine-readable instruction that, when executed by the processor, causes the processor to recommend a maintenance task to a non-road surface infrastructure element (Leonard et al. 2019/0283756 [Abstract; 0004; 0022-0024]).
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a maintenance task to a non-road surface infrastructure element” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Rezvan Behbahani et al. 2021/0339741; in view of Agarwal et al. 2022/0198842; in view of van den BERG et al. 2021/0201329.
18/327,312 – Claim 5. (Previously Presented) Leonard et al. 2019/0283756 further teaches A system, comprising: a processor; and a memory storing machine-readable instructions that, when executed by the processor (Leonard et al. 2019/0283756 [Abstract; 0004; 0022-0024]), cause the processor to: retrieve map data associated with an environment (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data]), wherein the map data indicates a mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment][0028 - in FIG. 2, the object detection system 130 may detect obstacle 210 located along a predicted trajectory 212 of the vehicle][0038 - the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); collect sensor data from a vehicle sensor system (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.), the sensor data indicates a motorist perception of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like.); infer a perception state of the mapped infrastructure element based on the sensor data and the map data (Leonard et al. 2019/0283756 [0012 - system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like. [0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); and recommend a maintenance task to be performed based on the perception state of the mapped infrastructure element (Leonard et al. 2019/0283756 [0005 - developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors…] In yet another embodiment, a method includes receiving a signal indicating a driver of a vehicle is attempting to make a turn at an intersection. The method further includes accessing, by a computer, a behavioral profile that is representative of the intersection. The method also includes developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors and the behavioral profile of the intersection. Finally, the method includes calculating at least one maneuver to be performed by one or more vehicle systems of the vehicle based at least in part on at least one of the perception of the driver and the risk of the intersection. [0038 - perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.) based on: a similarity between the sensor data and historical sensor data collected from other vehicles passing through the environment; and a previously performed maintenance task associated with the historical sensor data (Leonard et al. 2019/0283756 [0044 - intersections encountered by other vehicles that are in wireless communication with the vehicle 100 may also be used to build the substitute behavioral profile] Sometimes a behavioral profile may not be readily available for a particular intersection. In other words, the electronic control unit 102 is unable to access a behavioral profile from either the memory modules 106 or the servers 188. In response to determining there is no behavioral profile available for a particular intersection, the electronic control unit 102 may calculate a substitute behavioral profile based on data pertaining to a plurality of intersections encountered by the vehicle 100 over time. Alternatively, intersections encountered by other vehicles that are in wireless communication with the vehicle 100 may also be used to build the substitute behavioral profile as well. More specifically, machine learning may be used to recognize patterns at intersections based on specific characteristics such as, but not limited to, number of road segments, the presence of any traffic signals, weather conditions based on the season, time of day, and geographical location. For instance, all intersections in a particular geographical area may include one or more specific characteristics in common, and may be considered when determining a substitute behavioral profile.).
Leonard et al. 2019/0283756 may not expressly disclose the “previously performed maintenance task” features, however van den BERG et al. 2021/0201329 teaches (van den BERG et al. 2021/0201329 [0009 - vehicles moving around a location use their on-board sensors to gather data about the environment, themselves and other vehicles; the data is used to understand the effect of vehicle usage on the environment, and allows the impact of a specific vehicle and journey to be determined … impact their travel has on the environment, for example based on the cost of previous repairs…] Aspects of the present disclosure include a method that include the following operations: vehicles moving around a location use their on-board sensors to gather data about the environment, themselves and other vehicles; the data is used to understand the effect of vehicle usage on the environment, and allows the impact of a specific vehicle and journey to be determined; a tourist travelling in a vehicle may therefore be billed for only the impact their travel has on the environment, for example based on the cost of previous repairs; and when the fee is paid a blockchain asset is created which may be tracked, allowing the final use of the money on environmental repairs to be checked and traced back to the tourist.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by van den BERG et al. 2021/0201329. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a maintenance task to be performed” features, however Rezvan Behbahani et al. 2021/0339741 teaches (Rezvan Behbahani et al. 2021/0339741 [0093 - recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades] The requirements system 852 may then select a set of requirements for the scenario/vehicle type (e.g., set of vehicles having similar equipment or capabilities) combination to send to the vehicles 802 as the requirement data 842. In some cases, the requirements system 852 may also recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Rezvan Behbahani et al. 2021/0339741. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommending roadway infrastructure maintenance tasks” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Rezvan Behbahani et al. 2021/0339741; in view of Agarwal et al. 2022/0198842; in view of Tays et al. 2010/0057512.
18/327,312 – Claim 6. (Previously Presented) Leonard et al. 2019/0283756 teaches A system, comprising: a processor; and a memory storing machine-readable instructions that, when executed by the processor (Leonard et al. 2019/0283756 [Abstract; 0004; 0022-0024]), cause the processor to: retrieve map data associated with an environment (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data]), wherein the map data indicates a mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment][0028 - in FIG. 2, the object detection system 130 may detect obstacle 210 located along a predicted trajectory 212 of the vehicle][0038 - the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); collect sensor data from a vehicle sensor system (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.), the sensor data indicates a motorist perception of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like.); infer a perception state of the mapped infrastructure element based on the sensor data and the map data (Leonard et al. 2019/0283756 [0012 - system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like. [0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); and retrieve an infrastructure record that indicates a status of the mapped infrastructure element; and recommend a maintenance task to be performed based on the perception state of the mapped infrastructure element and the infrastructure record (Leonard et al. 2019/0283756 [0005 - developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors…] In yet another embodiment, a method includes receiving a signal indicating a driver of a vehicle is attempting to make a turn at an intersection. The method further includes accessing, by a computer, a behavioral profile that is representative of the intersection. The method also includes developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors and the behavioral profile of the intersection. Finally, the method includes calculating at least one maneuver to be performed by one or more vehicle systems of the vehicle based at least in part on at least one of the perception of the driver and the risk of the intersection. [0038 - perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.).
Leonard et al. 2019/0283756 may not expressly disclose the “retrieve an infrastructure record that indicates a status of the infrastructure element” features, however Tays et al. 2010/0057512 teaches (Tays et al. 2010/0057512 [0102 - tool to record a condition… in connection with a component of the [] infrastructure, a tool [] to view all outstanding conditions in connection with a certain component of the [] infrastructure and a tool [] to select maintenance activities in connection with…] The tools to manage conditions 1934 include a tool to retrieve components 1942 in the database of components, a tool to create a component inventory report 1944, a tool 1946 allowing to modify a component of the railway infrastructure, such as a track segment or feature, a tool to record a condition 1948 in connection with a component of the railway infrastructure, a tool 1950 to view all outstanding conditions in connection with a certain component of the railway infrastructure and a tool 1952 to select maintenance activities in connection with a condition.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Tays et al. 2010/0057512. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a maintenance task to be performed” features, however Rezvan Behbahani et al. 2021/0339741 teaches (Rezvan Behbahani et al. 2021/0339741 [0093 - recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades] The requirements system 852 may then select a set of requirements for the scenario/vehicle type (e.g., set of vehicles having similar equipment or capabilities) combination to send to the vehicles 802 as the requirement data 842. In some cases, the requirements system 852 may also recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Rezvan Behbahani et al. 2021/0339741. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommending roadway infrastructure maintenance tasks” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Rezvan Behbahani et al. 2021/0339741; in view of Agarwal et al. 2022/0198842; in view of Fastner et al. 2022/0413669.
18/327,312 – Claim 7. (Previously Presented) Leonard et al. 2019/0283756 further teaches A system, comprising: a processor; and a memory storing machine-readable instructions that, when executed by the processor (Leonard et al. 2019/0283756 [Abstract; 0004; 0022-0024]), cause the processor to: retrieve map data associated with an environment (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data]), wherein the map data indicates a mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment][0028 - in FIG. 2, the object detection system 130 may detect obstacle 210 located along a predicted trajectory 212 of the vehicle][0038 - the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); collect sensor data from a vehicle sensor system (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.), the sensor data indicates a motorist perception of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like.); infer a perception state of the mapped infrastructure element based on the sensor data and the map data (Leonard et al. 2019/0283756 [0012 - system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like. [0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); and retrieve an annotation regarding a historical maintenance task performed within the environment; and recommend a maintenance task to be performed based on the perception state of the mapped infrastructure element and the annotation (Leonard et al. 2019/0283756 [0005 - developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors…] In yet another embodiment, a method includes receiving a signal indicating a driver of a vehicle is attempting to make a turn at an intersection. The method further includes accessing, by a computer, a behavioral profile that is representative of the intersection. The method also includes developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors and the behavioral profile of the intersection. Finally, the method includes calculating at least one maneuver to be performed by one or more vehicle systems of the vehicle based at least in part on at least one of the perception of the driver and the risk of the intersection. [0038 - perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.).
Leonard et al. 2019/0283756 may not expressly disclose the “annotation” features, however Fastner et al. 2022/0413669 teaches (Fastner et al. 2022/0413669 [0139 – maintenance history of an asset; 0142 – asset history information … an image of an asset or textual notes on a condition of an asset…; 0145 - A threshold hold age can be a baseline useful life value for an asset, or it may be dynamically calculated from a baseline useful life value adjusted based on additional information relating to the asset, such as history of the asset (e.g., previous condition)]). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Fastner et al. 2022/0413669. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a maintenance task to be performed” features, however Rezvan Behbahani et al. 2021/0339741 teaches (Rezvan Behbahani et al. 2021/0339741 [0093 - recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades] The requirements system 852 may then select a set of requirements for the scenario/vehicle type (e.g., set of vehicles having similar equipment or capabilities) combination to send to the vehicles 802 as the requirement data 842. In some cases, the requirements system 852 may also recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Rezvan Behbahani et al. 2021/0339741. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommending roadway infrastructure maintenance tasks” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Rezvan Behbahani et al. 2021/0339741; in view of Agarwal et al. 2022/0198842; in view of Dahlin 2012/0320204.
18/327,312 – Claim 9. (Previously Presented) Leonard et al. 2019/0283756 further teaches A system, comprising: a processor; and a memory storing machine-readable instructions that, when executed by the processor (Leonard et al. 2019/0283756 [Abstract; 0004; 0022-0024]), cause the processor to: retrieve map data associated with an environment (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data]), wherein the map data indicates a mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment][0028 - in FIG. 2, the object detection system 130 may detect obstacle 210 located along a predicted trajectory 212 of the vehicle][0038 - the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); collect sensor data from a vehicle sensor system (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.), the sensor data indicates a motorist perception of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like.); infer a perception state of the mapped infrastructure element based on the sensor data and the map data (Leonard et al. 2019/0283756 [0012 - system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like. [0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); and recommend a maintenance task to be performed based on the perception state of the mapped infrastructure element, wherein the maintenance task is an environmental repair, wherein the environmental repair increases a visibility (Leonard et al. 2019/0283756 [0002; 0018; 0029; 0041; 0049 - visibility]) of the mapped infrastructure element (Leonard et al. 2019/0283756 [0005 - developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors…] In yet another embodiment, a method includes receiving a signal indicating a driver of a vehicle is attempting to make a turn at an intersection. The method further includes accessing, by a computer, a behavioral profile that is representative of the intersection. The method also includes developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors and the behavioral profile of the intersection. Finally, the method includes calculating at least one maneuver to be performed by one or more vehicle systems of the vehicle based at least in part on at least one of the perception of the driver and the risk of the intersection. [0038 - perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.).
Leonard et al. 2019/0283756 may not expressly disclose the “environmental repair, wherein the environmental repair increases a visibility” features, however, Dahlin 2012/0320204 teaches (Dahlin 2012/0320204 [0023 - To increase the visibility of roadway signs and other assets to drivers, the majority of signs incorporate retroreflective sheeting materials, which reflect incoming light towards its source to provide improved visibility during the night.][0025 - each asset would preferably be assessed for other characteristics to determine whether it needs to be replaced, such as if the asset has been damaged or knocked over, and is no longer visible][ [0026 - The asset assessment system of the present invention provides an automatic and cost efficient means for gathering information about a collection of assets, in particular, a variety of assets located at different geographical locations. Such assessment information can be analyzed in the field or stored in a database for post processing. For example, the asset assessment system can collect information and analyze the presence or absence of the asset, its current condition, such as whether it is located at the correct location or whether it is orientated correctly, whether it requires maintenance, such as whether it is bent or damaged, and/or whether it meets certain requirements, such as retroreflectivity requirements, etc. These assets could include roadway signs, guard rails, or other assets or objects residing in areas accessible to mobile units that contain the asset assessment system of the present invention.]). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Dahlin 2012/0320204. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a maintenance task to be performed” features, however Rezvan Behbahani et al. 2021/0339741 teaches (Rezvan Behbahani et al. 2021/0339741 [0093 - recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades] The requirements system 852 may then select a set of requirements for the scenario/vehicle type (e.g., set of vehicles having similar equipment or capabilities) combination to send to the vehicles 802 as the requirement data 842. In some cases, the requirements system 852 may also recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Rezvan Behbahani et al. 2021/0339741. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommending roadway infrastructure maintenance tasks” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Rezvan Behbahani et al. 2021/0339741; in view of Agarwal et al. 2022/0198842; in view of Shah et al. 2019/0342739.
18/327,312 – Claim 19. (Previously Presented) Leonard et al. 2019/0283756 teaches A method, comprising: retrieving map data associated with an environment (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data]), the map data indicates a mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment][0028 - in FIG. 2, the object detection system 130 may detect obstacle 210 located along a predicted trajectory 212 of the vehicle][0038 - the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); collecting sensor data from a vehicle sensor system (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.), the sensor data indicates a motorist perception of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like.); inferring, from the sensor data and the map data, a perception state of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like. [0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); retrieving an infrastructure record that indicates a status of the mapped infrastructure element; and recommending a maintenance task to be performed based on the perception state of the mapped infrastructure element and the infrastructure record (Leonard et al. 2019/0283756 [0005 - developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors…] In yet another embodiment, a method includes receiving a signal indicating a driver of a vehicle is attempting to make a turn at an intersection. The method further includes accessing, by a computer, a behavioral profile that is representative of the intersection. The method also includes developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors and the behavioral profile of the intersection. Finally, the method includes calculating at least one maneuver to be performed by one or more vehicle systems of the vehicle based at least in part on at least one of the perception of the driver and the risk of the intersection. [0038 - perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.).
Leonard et al. 2019/0283756 may not expressly disclose the “infrastructure record that indicates a status of the … infrastructure element” features, however, Shah et al. 2019/0342739 teaches (Shah et al. 2019/0342739 [0039 - The map data 256 may indicate available travel infrastructure for the geofenced area 252 and the conditions of such travel infrastructure, and may be transmitted to the vehicles 102 to help guide vehicle 102 occupants through and out of the geofenced area…][0063 - reporting an incident (e.g., infrastructure damage, injury)][0070 - reporting other types of incidents (e.g., infrastructure damage, property damage, endangered person)][0093 - the application logic 264 may retrieve map data 256 relevant to the navigation request. The map data 256 may include a record of the travel infrastructure available in the geofenced area 252, and may include data indicative of the conditions of such travel infrastructure, such as if a given road of the travel infrastructure includes a hazardous travel condition (e.g., flooding, unstable or broken materials, blocked by debris) caused by the disaster]). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Shah et al. 2019/0342739. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a maintenance task to be performed” features, however Rezvan Behbahani et al. 2021/0339741 teaches (Rezvan Behbahani et al. 2021/0339741 [0093 - recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades] The requirements system 852 may then select a set of requirements for the scenario/vehicle type (e.g., set of vehicles having similar equipment or capabilities) combination to send to the vehicles 802 as the requirement data 842. In some cases, the requirements system 852 may also recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Rezvan Behbahani et al. 2021/0339741. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommending roadway infrastructure maintenance tasks” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Rezvan Behbahani et al. 2021/0339741; in view of Agarwal et al. 2022/0198842; in view of Shah et al. 2019/0342739; in further view of Barzelay et al. 2019/0188521.
18/327,312 – Claim 18. (Previously Presented) Leonard et al. 2019/0283756 teaches The method of claim 19, wherein: the sensor data comprises images of the environment (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); and recommending the maintenance task further comprises recommending the maintenance task based on a detected discrepancy between the map data and the images of the environment (Leonard et al. 2019/0283756 [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment] As noted above, the logic stored on the one or more memory modules 106 may include object recognition logic. The object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment. Example object recognition algorithms include, but are not limited to, edge detection algorithms, corner detection algorithms, blob detection algorithms, and feature description algorithms (e.g., scale-invariant feature transform (“SIFT”), speeded up robust features (“SURF”), gradient location and orientation histogram (“GLOH”), and the like). The logic stored on the electronic control unit may also include speech recognition logic used to detect the words spoken by the driver and/or passengers within the vehicle 100. Any known or yet-to-be-developed speech recognition algorithms may be used for the speech recognition logic.).
Leonard et al. 2019/0283756 may not expressly disclose the “detected discrepancy” features, however, Barzelay et al. 2019/0188521 teaches (Barzelay et al. 2019/0188521 [0007 - comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object] According to an aspect of the present invention, there is a method, computer program product and/or computer system for performing the following operations (not necessarily in the following order): (i) receiving a first video image that includes a first initial version infrastructure object image showing a first infrastructure object with the first initial version infrastructure object image being characterized by a first viewing vector; (ii) receiving a second video image that includes a second initial version infrastructure object image showing the first infrastructure object with the second initial version infrastructure object image being characterized by a second viewing vector that is at least approximately parallel to the first viewing vector; (iii) selecting the first initial version infrastructure image from the first video; (iv) analyzing, by machine logic, the second video image to determine that the second initial version infrastructure object image is a match with an identical instance of the first initial version infrastructure object image, with the analysis of the second video image including: (a) constructing, by machine logic, a three dimensional (3D) data model of at least a portion of the environment around the first infrastructure object based, at least in part, upon the first and second video images, and (b) determining the match based, at least in part, upon the respective relationships of the first and second initial infrastructure object images to the 3D data model; (v) adjusting, by machine logic, at least one of the first and second initial version infrastructure object image to obtain a plurality of adjusted infrastructure image objects respectively corresponding to the first and second initial version infrastructure object images, with the plurality of adjusted infrastructure object images showing the first infrastructure object aligned with itself across the plurality of adjusted infrastructure object images; (vi) comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Barzelay et al. 2019/0188521. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks including neural network/machine learning tools which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a maintenance task to be performed” features, however Rezvan Behbahani et al. 2021/0339741 teaches (Rezvan Behbahani et al. 2021/0339741 [0093 - recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades] The requirements system 852 may then select a set of requirements for the scenario/vehicle type (e.g., set of vehicles having similar equipment or capabilities) combination to send to the vehicles 802 as the requirement data 842. In some cases, the requirements system 852 may also recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Rezvan Behbahani et al. 2021/0339741. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommending roadway infrastructure maintenance tasks” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Rezvan Behbahani et al. 2021/0339741; in view of Agarwal et al. 2022/0198842; in view of Fastner et al. 2022/0413669.
18/327,312 – Claim 20. (Previously Presented) Leonard et al. 2019/0283756 teaches A method, comprising: retrieving map data associated with an environment (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data]), the map data indicates a mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment][0028 - in FIG. 2, the object detection system 130 may detect obstacle 210 located along a predicted trajectory 212 of the vehicle][0038 - the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); collecting sensor data from a vehicle sensor system (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.), the sensor data indicates a motorist perception of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like.); inferring, from the sensor data and the map data, a perception state of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like. [0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); retrieving an annotation regarding a historical maintenance task performed within the environment; and recommending a maintenance task to be performed based on the perception state of the mapped infrastructure element and the annotation (Leonard et al. 2019/0283756 [0005 - developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors…] In yet another embodiment, a method includes receiving a signal indicating a driver of a vehicle is attempting to make a turn at an intersection. The method further includes accessing, by a computer, a behavioral profile that is representative of the intersection. The method also includes developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors and the behavioral profile of the intersection. Finally, the method includes calculating at least one maneuver to be performed by one or more vehicle systems of the vehicle based at least in part on at least one of the perception of the driver and the risk of the intersection. [0038 - perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.).
Leonard et al. 2019/0283756 may not expressly disclose the “annotation” features, however Fastner et al. 2022/0413669 teaches (Fastner et al. 2022/0413669 [0139 – maintenance history of an asset; 0142 – asset history information … an image of an asset or textual notes on a condition of an asset…; 0145 - A threshold hold age can be a baseline useful life value for an asset, or it may be dynamically calculated from a baseline useful life value adjusted based on additional information relating to the asset, such as history of the asset (e.g., previous condition)]). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Fastner et al. 2022/0413669. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a maintenance task to be performed” features, however Rezvan Behbahani et al. 2021/0339741 teaches (Rezvan Behbahani et al. 2021/0339741 [0093 - recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades] The requirements system 852 may then select a set of requirements for the scenario/vehicle type (e.g., set of vehicles having similar equipment or capabilities) combination to send to the vehicles 802 as the requirement data 842. In some cases, the requirements system 852 may also recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Rezvan Behbahani et al. 2021/0339741. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommending roadway infrastructure maintenance tasks” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Rezvan Behbahani et al. 2021/0339741; in view of Agarwal et al. 2022/0198842; in view of Fastner et al. 2022/0413669; in view of Barzelay et al. 2019/0188521.
18/327,312 – Claim 21. (Previously Presented) Leonard et al. 2019/0283756 teaches The method of claim 20, wherein: the sensor data comprises images of the environment (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); and recommending the maintenance task further comprises recommending the maintenance task based on a detected discrepancy between the map data and the images of the environment (Leonard et al. 2019/0283756 [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment] As noted above, the logic stored on the one or more memory modules 106 may include object recognition logic. The object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment. Example object recognition algorithms include, but are not limited to, edge detection algorithms, corner detection algorithms, blob detection algorithms, and feature description algorithms (e.g., scale-invariant feature transform (“SIFT”), speeded up robust features (“SURF”), gradient location and orientation histogram (“GLOH”), and the like). The logic stored on the electronic control unit may also include speech recognition logic used to detect the words spoken by the driver and/or passengers within the vehicle 100. Any known or yet-to-be-developed speech recognition algorithms may be used for the speech recognition logic.).
Leonard et al. 2019/0283756 may not expressly disclose the “detected discrepancy” features, however, Barzelay et al. 2019/0188521 teaches (Barzelay et al. 2019/0188521 [0007 - comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object] According to an aspect of the present invention, there is a method, computer program product and/or computer system for performing the following operations (not necessarily in the following order): (i) receiving a first video image that includes a first initial version infrastructure object image showing a first infrastructure object with the first initial version infrastructure object image being characterized by a first viewing vector; (ii) receiving a second video image that includes a second initial version infrastructure object image showing the first infrastructure object with the second initial version infrastructure object image being characterized by a second viewing vector that is at least approximately parallel to the first viewing vector; (iii) selecting the first initial version infrastructure image from the first video; (iv) analyzing, by machine logic, the second video image to determine that the second initial version infrastructure object image is a match with an identical instance of the first initial version infrastructure object image, with the analysis of the second video image including: (a) constructing, by machine logic, a three dimensional (3D) data model of at least a portion of the environment around the first infrastructure object based, at least in part, upon the first and second video images, and (b) determining the match based, at least in part, upon the respective relationships of the first and second initial infrastructure object images to the 3D data model; (v) adjusting, by machine logic, at least one of the first and second initial version infrastructure object image to obtain a plurality of adjusted infrastructure image objects respectively corresponding to the first and second initial version infrastructure object images, with the plurality of adjusted infrastructure object images showing the first infrastructure object aligned with itself across the plurality of adjusted infrastructure object images; (vi) comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Barzelay et al. 2019/0188521. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks including neural network/machine learning tools which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a maintenance task to be performed” features, however Rezvan Behbahani et al. 2021/0339741 teaches (Rezvan Behbahani et al. 2021/0339741 [0093 - recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades] The requirements system 852 may then select a set of requirements for the scenario/vehicle type (e.g., set of vehicles having similar equipment or capabilities) combination to send to the vehicles 802 as the requirement data 842. In some cases, the requirements system 852 may also recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Rezvan Behbahani et al. 2021/0339741. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommending roadway infrastructure maintenance tasks” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Examiner’s Response to Arguments
Per Applicants’ amendments/arguments, the rejections are withdrawn.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Examiner’s Response: Claim Rejections – 35 USC §112
Per Applicants’ amendments/arguments, the rejections are withdrawn.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Applicant traverses the rejection arguing that the specification discloses or supports the disputed claim language. The Office maintains the rejection. The examiner does not agree that the specification supports the claim language in a manner such that one of ordinary skill in the art would find support in the specification for the claimed subject matter. The claims remain rejected and Applicant is encouraged to amend the claims to more closely conform to that which the specification discloses including using language as disclosed in the specification.
Examiner’s Response: Claim Rejections – 35 USC §101
Per Applicants’ amendments/arguments, the rejections are withdrawn. See notes above for additional reasoning and rationale for dropping 35 USC 101 rejection including Applicant’s amendments, arguments, lack of abstract idea, and practical integration.
Regarding Claims 3-10 and 18-21, on page(s) 8-10 of Applicant’s Remarks (dated 07/28/2026), Applicants traverse the 35 USC §101 rejections arguing the following: claims are not abstract (e.g., not mental processes and/or certain methods of organizing human activity) and that the claims are integrated into a practical application by using vehicle sensors to obtain information regarding a physical roadway environment, correlating that information with map data identifying physical infrastructure elements, inferring the perception state of those infrastructure elements, and using that determination to identify appropriate maintenance.
Respectfully, the Office disagrees, and the claim remain rejected as follows. See revised rejection above and further detailed herein below.
Applicant claims retrieving map data associated with an environment. However, Applicant does not indicate how this data is obtained. It is conceivable that a person can obtain said map data from a physical paper or electronic map which is a mental process. Next Applicant claims collecting sensor data and inferring a perception state of an infrastructure element from the sensor and map data and recommending a maintenance task. However, Applicant does not disclose nor claim how this is accomplished. Rather, Applicant merely claims the results of inferring a perception state and recommending a maintenance task. Thus, Applicant is claiming collecting data which is abstract, inferring some kind of perception state (abstract) and recommending a result (maintenance task - abstract). This language is result oriented and does not provide nor disclose how the recommended result is achieved. Consequently, the claim remains rejected as abstract and not integrated into a practical application. Furthermore, but for the vehicle sensor system, there is no structure claimed for performing the claimed invention. Since most of the steps are abstract (e.g., retrieving data, inferring a perception state, retrieving a record, and recommending a maintenance task) the claim remains rejected as abstract, not integrated into a practical application, and patent ineligible / subject matter ineligible.
A method, comprising: retrieving map data associated with an environment, the map data indicates a mapped infrastructure element; collecting sensor data from a vehicle sensor system, the sensor data indicates a motorist perception of the mapped infrastructure element; inferring, from the sensor data and the map data, a perception state of the mapped infrastructure element; retrieving an infrastructure record that indicates a status of the mapped infrastructure element; and recommending a maintenance task to be performed based on the perception state of the mapped infrastructure element and the infrastructure record.
Examiner’s Response: Claim Rejections – 35 USC § 103
Per Applicants’ amendments/arguments, the rejections are withdrawn. See notes above for additional reasoning and rationale for dropping prior-art rejection including Applicant’s amendments and arguments and unique combination of features and elements not taught by the prior-art without hindsight reasoning.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Regarding Claims 3-10 and 18-21, on page(s) 12-18 of Applicant’s Remarks (dated 07/28/2026), Applicants traverse the 35 USC §103 rejections arguing the following: the cited claims do not teach the claimed features. Respectfully, the Office disagrees and maintains the rejections in their entirety. Under the Broadest Reasonable Interpretation (BRI), the following references teach the claimed features as follows:
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Rezvan Behbahani et al. 2021/0339741; in view of Nagata et al. 2020/0302193; in view of Barzelay et al. 2019/0188521.
18/327,312 – Claim 3. (Previously Presented) Leonard et al. 2019/0283756 teaches A system, comprising: a processor; and a memory storing machine-readable instructions that, when executed by the processor (Leonard et al. 2019/0283756 [Abstract; 0004; 0022-0024]), cause the processor to: retrieve map data associated with an environment, wherein the map data (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data][0016 - an object detection system … a location determination system]) indicates a mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment][0028 - in FIG. 2, the object detection system 130 may detect obstacle 210 located along a predicted trajectory 212 of the vehicle][0038 - the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); collect sensor data comprising images of the environment from a vehicle sensor system (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.), the sensor data indicates a motorist perception of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like.); infer a perception state of the mapped infrastructure element based on the sensor data and the map data (Leonard et al. 2019/0283756 [0012 - system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like. [0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); identify that the mapped infrastructure element is not captured in an image of the environment of the vehicle; and recommend a maintenance task to be performed based on the perception state of the mapped infrastructure element (Leonard et al. 2019/0283756 [0005 - developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors…] In yet another embodiment, a method includes receiving a signal indicating a driver of a vehicle is attempting to make a turn at an intersection. The method further includes accessing, by a computer, a behavioral profile that is representative of the intersection. The method also includes developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors and the behavioral profile of the intersection. Finally, the method includes calculating at least one maneuver to be performed by one or more vehicle systems of the vehicle based at least in part on at least one of the perception of the driver and the risk of the intersection. [0038 - perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.) and a detected discrepancy between the map data and the images of the environment (Leonard et al. 2019/0283756 [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment] As noted above, the logic stored on the one or more memory modules 106 may include object recognition logic. The object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment. Example object recognition algorithms include, but are not limited to, edge detection algorithms, corner detection algorithms, blob detection algorithms, and feature description algorithms (e.g., scale-invariant feature transform (“SIFT”), speeded up robust features (“SURF”), gradient location and orientation histogram (“GLOH”), and the like). The logic stored on the electronic control unit may also include speech recognition logic used to detect the words spoken by the driver and/or passengers within the vehicle 100. Any known or yet-to-be-developed speech recognition algorithms may be used for the speech recognition logic.).
Leonard et al. 2019/0283756 may not expressly disclose the “identify that the mapped infrastructure element is not captured in an image of the environment of the vehicle” features, however, Nagata et al. 2020/0302193 teaches (Nagata et al. 2020/0302193 [0081 -determines whether an unidentified obstacle, which is an obstacle included in a plurality of obstacles that are not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle, is present based on the obtained first information and vehicle information, and map information indicating a map of the vicinity of specific vehicle] First determiner 41 determines whether an unidentified obstacle, which is an obstacle included in a plurality of obstacles that are not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle, is present based on the obtained first information and vehicle information, and map information indicating a map of the vicinity of specific vehicle 20. FIG. 3 is a schematic illustration illustrating a relationship between specific vehicle 20 and an unidentified obstacle in information processing system 1 according to Embodiment 1. An unidentified obstacle includes not only an obstacle that is not visible from specific vehicle 20, but also an obstacle that is difficult to see, that is, an obstacle that is not at all visible from specific vehicle 20. [0083 - map information here indicates a map of the vicinity of specific vehicle…] The map information here indicates a map of the vicinity of specific vehicle 20 and includes, for example, an internal structure (floor map) of a parking facility, for example. [0087 - The map information may be pre-stored in a recording medium included in information processing device] The map information may be pre-stored in a recording medium included in information processing device 40 or may be obtained from, for instance, other server device via a network. [0095 - obstacle when viewed from specific vehicle 20 and is not visible from specific vehicle 20, is present based on the first information, the vehicle information, and map information…] Information processing device 40 determines whether an unidentified obstacle, which is included in obstacles each being hidden behind other obstacle when viewed from specific vehicle 20 and is not visible from specific vehicle 20, is present based on the first information, the vehicle information, and map information. Specifically, first determiner 41 in information processing device 40 specifies one or more moving obstacles (one example of the first obstacles) among obstacles in the vicinity of specific vehicle 20, based on the first information and the vehicle information. First determiner 41 determines, for each of one or more moving obstacles, whether other obstacle indicated in the first information or the map information is present between specific vehicle 20 and the moving obstacle. When such other obstacle is present between specific vehicle 20 and the moving obstacle, first determiner 41 further determines, for each of the second obstacles indicated in the second information received from specific vehicle 20, whether the second obstacle is identical to a corresponding one of one or more moving obstacles. When determining that the second obstacle is identical to the corresponding one of one or more moving obstacles, first determiner 41 determines that the second obstacle is not an unidentified obstacle. Stated differently, first determiner 41 does not determine, as an unidentified obstacle, a moving obstacle that is indicated in the first information and is identical to a corresponding one of the second obstacles indicated in the second information. First determiner 41 thus determines, as an unidentified obstacle, at least one first obstacle (moving obstacle) remaining after the second obstacles have been excluded from the first obstacles, and determines that at least one unidentified obstacle is present (S13). When Yes at step S13, first determiner 41 transmits information indicating at least one unidentified obstacle to specific vehicle 20. [0101 -determines whether an unidentified obstacle, which is included in a plurality of obstacles detected by surveillance camera 30 and is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20, is present based on the first information regarding the plurality of obstacles detected by surveillance camera 30 and the map information indicating a map of the vicinity of specific vehicle] As described above, in information processing system 1 and the information processing method, first determiner 41 determines whether an unidentified obstacle, which is included in a plurality of obstacles detected by surveillance camera 30 and is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20, is present based on the first information regarding the plurality of obstacles detected by surveillance camera 30 and the map information indicating a map of the vicinity of specific vehicle 20. When first determiner 41 determines that at least one unidentified obstacle is present, first communicator 42 outputs information indicating at least one unidentified obstacle to specific vehicle 20. This enables specific vehicle 20 to obtain the information indicating at least one unidentified obstacle that is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20. Specific vehicle 20 is thus capable of performing an operation based on the information indicating at least one unidentified obstacle.[0117; 0140; 0175]). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Nagata et al. 2020/0302193. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommend a maintenance task to be performed” features, however Rezvan Behbahani et al. 2021/0339741 teaches (Rezvan Behbahani et al. 2021/0339741 [0093 - recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades] The requirements system 852 may then select a set of requirements for the scenario/vehicle type (e.g., set of vehicles having similar equipment or capabilities) combination to send to the vehicles 802 as the requirement data 842. In some cases, the requirements system 852 may also recommend system upgrades or maintenance for particular type of vehicles (e.g., sensors with improved accuracy, updated software for the perception system 820, the prediction system 822, the planning system 824, etc.) among other types of upgrades.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Rezvan Behbahani et al. 2021/0339741. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “detected discrepancy” features, however, Barzelay et al. 2019/0188521 teaches (Barzelay et al. 2019/0188521 [0007 - comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object] According to an aspect of the present invention, there is a method, computer program product and/or computer system for performing the following operations (not necessarily in the following order): (i) receiving a first video image that includes a first initial version infrastructure object image showing a first infrastructure object with the first initial version infrastructure object image being characterized by a first viewing vector; (ii) receiving a second video image that includes a second initial version infrastructure object image showing the first infrastructure object with the second initial version infrastructure object image being characterized by a second viewing vector that is at least approximately parallel to the first viewing vector; (iii) selecting the first initial version infrastructure image from the first video; (iv) analyzing, by machine logic, the second video image to determine that the second initial version infrastructure object image is a match with an identical instance of the first initial version infrastructure object image, with the analysis of the second video image including: (a) constructing, by machine logic, a three dimensional (3D) data model of at least a portion of the environment around the first infrastructure object based, at least in part, upon the first and second video images, and (b) determining the match based, at least in part, upon the respective relationships of the first and second initial infrastructure object images to the 3D data model; (v) adjusting, by machine logic, at least one of the first and second initial version infrastructure object image to obtain a plurality of adjusted infrastructure image objects respectively corresponding to the first and second initial version infrastructure object images, with the plurality of adjusted infrastructure object images showing the first infrastructure object aligned with itself across the plurality of adjusted infrastructure object images; (vi) comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Barzelay et al. 2019/0188521. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks including neural network/machine learning tools which should prove to improve user experience, maximize profits, and optimize revenue.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over: Leonard et al. 2019/0283756; in view of Agarwal et al. 2022/0198842; in view of Nagata et al. 2020/0302193; in view of Barzelay et al. 2019/0188521.
18/327,312 – Claim 3. (Previously Presented) Leonard et al. 2019/0283756 teaches A system, comprising: a processor; and a memory storing machine-readable instructions that, when executed by the processor (Leonard et al. 2019/0283756 [Abstract; 0004; 0022-0024]), cause the processor to: retrieve map data associated with an environment, wherein the map data (Leonard et al. 2019/0283756 [0012 – map data; 0060 – map data; Claims 8 & 19 – map data]) indicates a mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment][0028 - in FIG. 2, the object detection system 130 may detect obstacle 210 located along a predicted trajectory 212 of the vehicle][0038 - the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); collect sensor data comprising images of the environment from a vehicle sensor system (Leonard et al. 2019/0283756 [0027 - object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle… object detection system 130 may include an object detection sensor…] The object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104. The object detection system 130 may include any device configured to detect the presence of an object within the vicinity of the vehicle 100. The object detection system 130 may include an object detection sensor 132. The object detection sensor 132 may be configured to output an output signal indicative of the presence of one or more objects within a vicinity of the intersection. Based on the output signal of the object detection sensor 132, the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. The object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132. [0033 - sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle … object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like)] The location determination system 140 may include a location sensor 142 configured to output an output signal indicative of the location of the vehicle 100. Based on the output signal of the location determination system 140, the electronic control unit 102 may execute logic to determine a vehicle location. The location sensor 142 may include, but is not limited to, a camera, a GPS unit, and the like. In embodiments where the location sensor 142 includes a camera, the electronic control unit 102 may execute object recognition logic to determine based on objects within the environment of the vehicle 100, the location of the vehicle 100. For example, the one or more processors 105 of the electronic control unit 102 may execute object recognition logic, such that the electronic control unit 102 may read signs and/or recognize objects that may indicate a location of the vehicle 100 (e.g., on ramps, highways, sidewalks, storefronts, houses, and the like).[0035 – various sensors][0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment … of the vehicle] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.), the sensor data indicates a motorist perception of the mapped infrastructure element (Leonard et al. 2019/0283756 [0012 - the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like.); infer a perception state of the mapped infrastructure element based on the sensor data and the map data (Leonard et al. 2019/0283756 [0012 - system may determine the driver is about to make a turn based on map data] As described in detail below, embodiments of the present disclosure assist a driver in performing a turn at an intersection. Embodiments of the present disclosure may first determine that the driver is attempting to make a turn at an intersection. The vehicle system may determine that the driver is about to make the turn based on actions such as, but not limited to, activation of a turn signal, a direction of the driver's gaze, or the position of a steering wheel. In one embodiment, the system may determine the driver is about to make a turn based on map data. In response to determining the driver is attempting to make a turn, the vehicle control system then determines a risk associated with the intersection, a perception associated with the driver, and a perception of the vehicle. The perception of the driver may be used to determine if the driver is distracted, and the risk is used to determine if is likely that the vehicle may contact another vehicle, be driven off the road, or another undesirable outcome. [0018 - perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle] As explained in greater detail below, the perception of the driver and the perception of the vehicle are based on a behavioral profile associated with the intersection and data gathered by a plurality of sensors of the vehicle. The behavioral profile includes one or more characteristics that are representative of the intersection, and the sensors indicate conditions at the intersection such as visibility, cross-traffic, obstructions, driver attentiveness, and the like. [0038 - perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.); identify that the mapped infrastructure element is not captured in an image of the environment of the vehicle; and recommend a maintenance task to be performed based on the perception state of the mapped infrastructure element (Leonard et al. 2019/0283756 [0005 - developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors…] In yet another embodiment, a method includes receiving a signal indicating a driver of a vehicle is attempting to make a turn at an intersection. The method further includes accessing, by a computer, a behavioral profile that is representative of the intersection. The method also includes developing a perception as well as a risk associated with the intersection based on a plurality of operational signals from a plurality of sensors and the behavioral profile of the intersection. Finally, the method includes calculating at least one maneuver to be performed by one or more vehicle systems of the vehicle based at least in part on at least one of the perception of the driver and the risk of the intersection. [0038 - perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system] In addition to the driver's perception, the perception of the vehicle 100 may also be determined. The perception of the vehicle 100 may be determined based on objects located within or around the vicinity of the intersection 200 (FIG. 2) detected by the object detection system 130. More specifically, the perception of the vehicle is based on the information collected by various cameras and sensors of the object detection system 130 to view the surrounding environment (e.g., the intersection) of the vehicle 100. The perception of the vehicle 100 may also be determined based on one more operating conditions of the vehicle 100 detected by the vehicle operating condition sensor system 160.) and a detected discrepancy between the map data and the images of the environment (Leonard et al. 2019/0283756 [0025 - object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment] As noted above, the logic stored on the one or more memory modules 106 may include object recognition logic. The object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment. Example object recognition algorithms include, but are not limited to, edge detection algorithms, corner detection algorithms, blob detection algorithms, and feature description algorithms (e.g., scale-invariant feature transform (“SIFT”), speeded up robust features (“SURF”), gradient location and orientation histogram (“GLOH”), and the like). The logic stored on the electronic control unit may also include speech recognition logic used to detect the words spoken by the driver and/or passengers within the vehicle 100. Any known or yet-to-be-developed speech recognition algorithms may be used for the speech recognition logic.).
Leonard et al. 2019/0283756 may not expressly disclose the “identify that the mapped infrastructure element is not captured in an image of the environment of the vehicle” features, however, Nagata et al. 2020/0302193 teaches (Nagata et al. 2020/0302193 [0081 - determines whether an unidentified obstacle, which is an obstacle included in a plurality of obstacles that are not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle, is present based on the obtained first information and vehicle information, and map information indicating a map of the vicinity of specific vehicle] First determiner 41 determines whether an unidentified obstacle, which is an obstacle included in a plurality of obstacles that are not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle, is present based on the obtained first information and vehicle information, and map information indicating a map of the vicinity of specific vehicle 20. FIG. 3 is a schematic illustration illustrating a relationship between specific vehicle 20 and an unidentified obstacle in information processing system 1 according to Embodiment 1. An unidentified obstacle includes not only an obstacle that is not visible from specific vehicle 20, but also an obstacle that is difficult to see, that is, an obstacle that is not at all visible from specific vehicle 20. [0083 - map information here indicates a map of the vicinity of specific vehicle…] The map information here indicates a map of the vicinity of specific vehicle 20 and includes, for example, an internal structure (floor map) of a parking facility, for example. [0087 - The map information may be pre-stored in a recording medium included in information processing device] The map information may be pre-stored in a recording medium included in information processing device 40 or may be obtained from, for instance, other server device via a network. [0095 - obstacle when viewed from specific vehicle 20 and is not visible from specific vehicle 20, is present based on the first information, the vehicle information, and map information…] Information processing device 40 determines whether an unidentified obstacle, which is included in obstacles each being hidden behind other obstacle when viewed from specific vehicle 20 and is not visible from specific vehicle 20, is present based on the first information, the vehicle information, and map information. Specifically, first determiner 41 in information processing device 40 specifies one or more moving obstacles (one example of the first obstacles) among obstacles in the vicinity of specific vehicle 20, based on the first information and the vehicle information. First determiner 41 determines, for each of one or more moving obstacles, whether other obstacle indicated in the first information or the map information is present between specific vehicle 20 and the moving obstacle. When such other obstacle is present between specific vehicle 20 and the moving obstacle, first determiner 41 further determines, for each of the second obstacles indicated in the second information received from specific vehicle 20, whether the second obstacle is identical to a corresponding one of one or more moving obstacles. When determining that the second obstacle is identical to the corresponding one of one or more moving obstacles, first determiner 41 determines that the second obstacle is not an unidentified obstacle. Stated differently, first determiner 41 does not determine, as an unidentified obstacle, a moving obstacle that is indicated in the first information and is identical to a corresponding one of the second obstacles indicated in the second information. First determiner 41 thus determines, as an unidentified obstacle, at least one first obstacle (moving obstacle) remaining after the second obstacles have been excluded from the first obstacles, and determines that at least one unidentified obstacle is present (S13). When Yes at step S13, first determiner 41 transmits information indicating at least one unidentified obstacle to specific vehicle 20. [0101 - determines whether an unidentified obstacle, which is included in a plurality of obstacles detected by surveillance camera 30 and is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20, is present based on the first information regarding the plurality of obstacles detected by surveillance camera 30 and the map information indicating a map of the vicinity of specific vehicle] As described above, in information processing system 1 and the information processing method, first determiner 41 determines whether an unidentified obstacle, which is included in a plurality of obstacles detected by surveillance camera 30 and is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20, is present based on the first information regarding the plurality of obstacles detected by surveillance camera 30 and the map information indicating a map of the vicinity of specific vehicle 20. When first determiner 41 determines that at least one unidentified obstacle is present, first communicator 42 outputs information indicating at least one unidentified obstacle to specific vehicle 20. This enables specific vehicle 20 to obtain the information indicating at least one unidentified obstacle that is not visible from specific vehicle 20 among obstacles each being hidden behind other obstacle when viewed from specific vehicle 20. Specific vehicle 20 is thus capable of performing an operation based on the information indicating at least one unidentified obstacle.[0117; 0140; 0175]). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Nagata et al. 2020/0302193. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “recommending roadway infrastructure maintenance tasks” features, however Agarwal et al. 2022/0198842 teaches (Agarwal et al. 2022/0198842 [0136 - techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle … information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. … information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.] The predictive techniques described here have applications beyond generating a predictive maintenance schedule for a vehicle. In an embodiment, information received from vehicles (e.g., sensor data, HD map data, etc.) and the correlations established by the predictive model (e.g., the predictive model 1406) are used in reverse to identify geographical areas causing an outsized impact on vehicle maintenance. In an embodiment, this information is used for risk-based routing, such as by selecting a route for a vehicle that minimizes the impact on the health of vehicle components, or prioritizing which vehicles in a fleet are sent to high impact areas based on the vehicle's health. For example, if an area of a city is known to have rough roads which cause deterioration, a route can be selected for the vehicle to avoid the rough roads, or a fleet operator could prioritize sending vehicles that are past their useful life to travel on the rough roads (or vehicles having components that are healthy such that the vehicles can withstand the rough roads). In an embodiment, information about areas causing an outsized impact on vehicle maintenance are used to generate an infrastructure report identifying roads or other infrastructure that need repair or maintenance. The infrastructure report can be provided to, for example, government officials to assist in prioritizing infrastructure improvements.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Agarwal et al. 2022/0198842. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks which should prove to improve user experience, maximize profits, and optimize revenue.
Leonard et al. 2019/0283756 may not expressly disclose the “detected discrepancy” features, however, Barzelay et al. 2019/0188521 teaches (Barzelay et al. 2019/0188521 [0007 - comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object] According to an aspect of the present invention, there is a method, computer program product and/or computer system for performing the following operations (not necessarily in the following order): (i) receiving a first video image that includes a first initial version infrastructure object image showing a first infrastructure object with the first initial version infrastructure object image being characterized by a first viewing vector; (ii) receiving a second video image that includes a second initial version infrastructure object image showing the first infrastructure object with the second initial version infrastructure object image being characterized by a second viewing vector that is at least approximately parallel to the first viewing vector; (iii) selecting the first initial version infrastructure image from the first video; (iv) analyzing, by machine logic, the second video image to determine that the second initial version infrastructure object image is a match with an identical instance of the first initial version infrastructure object image, with the analysis of the second video image including: (a) constructing, by machine logic, a three dimensional (3D) data model of at least a portion of the environment around the first infrastructure object based, at least in part, upon the first and second video images, and (b) determining the match based, at least in part, upon the respective relationships of the first and second initial infrastructure object images to the 3D data model; (v) adjusting, by machine logic, at least one of the first and second initial version infrastructure object image to obtain a plurality of adjusted infrastructure image objects respectively corresponding to the first and second initial version infrastructure object images, with the plurality of adjusted infrastructure object images showing the first infrastructure object aligned with itself across the plurality of adjusted infrastructure object images; (vi) comparing, by machine logic, the adjusted infrastructure object images with each other to determine a difference data set corresponding to a set of differences between at least two of the plurality of adjusted infrastructure object images; and (vii) analyzing, by machine logic, the difference data set to determine that a potential maintenance condition exists regarding the first infrastructure object.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Leonard et al. 2019/0283756 to include the features as taught by Barzelay et al. 2019/0188521. One of ordinary skill in the art would have been motivated to do so to incorporate well-known features for recommending roadway infrastructure maintenance tasks including neural network/machine learning tools which should prove to improve user experience, maximize profits, and optimize revenue.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion
PERTINENT PRIOR ART – Patent Literature
The prior-art made of record and considered pertinent to applicant's disclosure.
Doutre et al. 2023/0177827 [0009 - infrastructure capable of being scanned by a camera mounted to a vehicle]
Kaku et al. 2021/0374432 [0044 - sensor calibration module 123 can be configured to receive data from the sensor system 120 and/or any other type of system capable of capturing information relating to the vehicle 100 and/or the external environment of the vehicle ]
Krehl et al. 2024/0312218 [0004 - computing system may then process the resultant hybrid BEV representation of the surrounding environment of the vehicle to, for example, derive a fused representation of the surrounding environment, derive various aspects of the road infrastructure on which the vehicle operates (e.g., lane markings, road topology, lane topology, crosswalks, etc.)]
Komori et al. 2024/0112149 [0009] (2): In the above-described aspect (1), the manager causes the provider to provide information for promoting maintenance of the road when a difference between a position of a road included in an image captured by the sensor device and a position of the road included in the reference image is greater than or equal to a threshold value.
Dickson et al. 2023/0306573 [0056 - neural network][0024 - one or more machine learning techniques may be provided for identifying maintenance candidates. For instance, a machine learning model may be provided that analyzes sensor data collected from infrastructure assets, and outputs one or more candidates for a selected type of maintenance (e.g., reparative and/or preventive maintenance). Additionally, or alternatively, the machine learning model may output one or more images and/or portions of images that are identified as having a selected characteristic (e.g., exhibiting a selected type of damage), for example, to explain why the selected type of maintenance is recommended. Such a machine learning model may be trained using sensor data]
PERTINENT PRIOR ART – Non-Patent Literature (NPL)
The NPL prior-art made of record and considered pertinent to applicant's disclosure.
Toyota's high-tech city project set for launch: Woven City a 'test track' for smart homes, robotics, autonomous vehicles. In: ISE: Industrial & Systems Engineering at Work, Mar2025.
K. Wong, Y. Gu and S. Kamijo, "Mapping for Autonomous Driving: Opportunities and Challenges," in IEEE Intelligent Transportation Systems Magazine, vol. 13, no. 1, pp. 91-106, Spring 2021, doi: 10.1109/MITS.2020.3014152.
E. Yurtsever, J. Lambert, A. Carballo and K. Takeda, "A Survey of Autonomous Driving: Common Practices and Emerging Technologies," in IEEE Access, vol. 8, pp. 58443-58469, 2020, doi: 10.1109/ACCESS.2020.2983149.
THIS ACTION IS MADE FINAL
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW T. SITTNER whose telephone number is (571) 270-7137 and email: matthew.sittner@uspto.gov. The examiner can normally be reached on Monday-Friday, 8:00am - 5:00pm (Mountain Time Zone).
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sarah M. Monfeldt can be reached on (571) 270-1833.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW T SITTNER/
Primary Examiner, Art Unit 3629b
Prosecution Insights