DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
This first final action is in response to applicant's amendments on Feb. 26, 2026. Claims 1-20 are pending and have been considered as follows.
Examiner's response
Applicant's amendments/arguments with respect to the rejection of claims 1-20 under 35 U.S.C. 101 have been fully considered and are persuasive. The rejection of claims 1-20 under 35 U.S.C. 101 has been withdrawn.
Applicant’s arguments with respect to the claim rejections under 35 USC 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. 103 as being obvious over by Szczerba (US 20230290156 A1) in view of Yu (US 20200378779 A1) in view of Verghese (US 20210195112 A)
Regarding claim 1, Szczerba teaches a method comprising: receiving sensor data of a sensor of a vehicle as the vehicle travels along a road( Szczerba, Figs. 1-3, [0020] The controller is configured to receive the roadway data from at least one of the plurality of sensors and receive vehicle-location data from at least one of the plurality of sensors);
determining a confidence of sensor of the vehicle is below a threshold based on the sensor data ([0020] providing a vehicle user with lane information when the vehicle approaches a road junction or when the visibility is poor. [0050] these virtual images 50 are particularly helpful in situations of poor visibility; [0056] At block 104, the controller 34 determines the location of the vehicle 10 and the environmental driving conditions around the vehicle 10, such as weather causing poor visibility; [0060] The method 100 also includes block 105, which is performed after block 104. At block 105, the controller 34 determines, using the image data (e.g., images), visibility (or legibility) of one or more road signs 53 (FIG. 2) along the route of the vehicle 10 is less than a predetermined minimum-confidence threshold; Step 105 in Fig. 7); and
displaying the view of the road within the vehicle via augmented reality in spatial alignment with the road while the vehicle travels along the road ( [0062] At block 112, the controller 34 determines the specific virtual images 50 to be displayed on the display 29 (e.g., the dual-focal plane augmented reality display) based on the lane information, which is part of the roadway data previously received by the controller 34. the controller 34 may select virtual images 50 that indicate that the lanes 61 lead to a particular street or a particular interstate. Also, for instance, the controller 34 may select virtual images 50 that indicate lane rules with arrows).
Szczerba does not explicitly teach but Yu teaches the limitation of determining a view of the road based on execution of an artificial intelligence (AI) model on the previously-captured sensor data to compensate for the confidence of the sensor being below the threshold (Yu, Some embodiments of the hazard detection system may comprise one or more neural networks, including fully-connected neural networks, convolutional neural networks, recurrent neural networks, or attention models. The hazard detection system 116 may detect, identify, and track objects in image input such as still photos or video captured by camera 110. The hazard detection system 116 may additionally predict whether any of the tracked objects are hazardous based on attribute data such as type, size, location, distance, or movement of the tracked objects. Hazard detection system 116 may predict whether an object is hazardous by applying object classification to classify objects into hazardous and non-hazardous categories. The hazard detection system 116 may be trained on labeled training data comprising examples of hazardous and non-hazardous objects with associated images. [0039] a computer vision algorithm or machine learning model may be used to segment and identify regions in the video feed which correspond to elements of the road such as the current lane, lane markers, other lanes, and shoulder of the road. In addition, attributes such as up-hill or down-hill orientation, curvature, and distance of the road may be detected. A 3D model of the road surface may be created using the elements and attributes identified from the video feed. Overlay 8 may be projected on to the 3D model of the road surface and drawn in the video feed. [0042] one or more machine learning models may be used to predict the display position of the maneuver from the input data. In some embodiments, the display position of the maneuver location may be directly identified in the video feed. A computer vision algorithm or machine learning model may be applied which takes the video feed as input and outputs the display position of the maneuver location. Fig. 4 and [0053] Hazard view 410 may display a live video feed collected from camera 110 along with optional augmented reality elements to warn the user of potential hazards. [0056] Hazard route overlay 43 is an augmented reality overlay which alters the appearance of route overlay 8 to alert the user to the existence of a potential hazard; the limitation of “to compensate for the confidence of the sensor being below the threshold” is intent to use ).
It would have been obvious to one of ordinary skill in the art before the effective date of the present invention to modify, displaying lane information on an augmented reality display includes receiving roadway data, as taught by Szczerba, determining a view of the road based on execution of an artificial intelligence (AI) model on the previously-captured sensor data, as taught by Yu, as Szczerba and Yu are directed to an augmented reality display on a vehicle, as Szczerba and Yu are directed to vehicle control (same field of endeavor), and one of ordinary skill in the art would have recognized the established utility using determining a view of the road based on execution of an artificial intelligence (AI) model on the previously-captured sensor data and predictably applied it to Szczerba’s teaching to makes complex decisions in real-time.
While Szczerba as modified by Yu teaches ([0062] At block 112, the controller 34 determines the specific virtual images 50 to be displayed on the display 29 (e.g., the dual-focal plane augmented reality display) based on the lane information, which is part of the roadway data previously received by the controller 34; the virtual images 50 may be indicated of the lane rules, lane name, the name of the road that the lane 61 leads to or other relevant lane information. For example, as shown in FIG. 2, the controller 34 may select virtual images 50 that indicate that the lanes 61 lead to a particular street or a particular interstate), Szczerba as modified by Yu does not explicitly teach but Verghese teaches extracting previously-captured sensor data of the road, wherein the previously-captured sensor data is captured by sensors of other vehicles travelling along the road in condition in which sensor confidence of the other vehicle exceeds the threshold (a sensor ( lidar sensor, radar sensor, and/or camera) is expected to detect objects outside the autonomous vehicle at a minimum confidence level and using data acquired with the one or more sensors, such as clear-weather past image 602 and/or lidar data for one or more objects in the past image 602. ([0019], [0006], For example, a processor associated with the particular sensor can be configured to associate a higher confidence level (e.g., higher than a predefined confidence threshold level) to objects or other information detected; a change from a first environmental condition (e.g., clear weather) to a second environmental condition (e.g., foggy or snowy weather) can cause at least one of such parameters to degrade; [0129]. Also, it is well established that a vehicle may use other vehicles’ sensor data using on V2V (vehicle to vehicle ) communication).
It would have been obvious to one of ordinary skill in the art before the effective date of the present invention to modify, displaying lane information on an augmented reality display includes receiving roadway data, as taught by Szczerba as modified by Yu, using sensor data whose confidence exceeds the threshold as Verghese, Szczerba, Verghese and Yu are directed to vehicle control (same field of endeavor). One of ordinary skill in the art would have recognized the established utility using sensor data whose confidence exceeds the threshold and predictably applied it to teachings of Szczerba as modified by Yu to improve sensor data reliability.
Regarding claims 8 and 15, please see the rejection above with regarding claim 1.
Regarding claim 2, While Szczerba teaches the limitation of the displaying comprises displaying the AR view of the road on one or more of a windshield of the vehicle and a display screen of the vehicle as the vehicle travels along the road (Fig. 4), Szczerba does not explicitly teach but Yu teaches the limitation of wherein the determining the view comprises generating an augmented reality (AR) view of the road based on execution of the AI model (Yu, [0024],[0039], [0042], [0053], [0056]). It would have been obvious to one of ordinary skill in the art before the effective date of the present invention to modify, displaying lane information on an augmented reality display includes receiving roadway data, as taught by Szczerba, determining a view of the road based on execution of an artificial intelligence (AI) model on the previously-captured sensor data, as taught by Yu, as Szczerba and Yu are directed to an augmented reality display on a vehicle, (same field of endeavor), and one of ordinary skill in the art would have recognized the established utility using determining a view of the road based on execution of an artificial intelligence (AI) model on the previously-captured sensor data and predictably applied it to Szczerba’s teaching to makes complex decisions in real-time.
Regarding claims 9 and 16, please see the rejection above with regarding claim 2.
Regarding claim 3, Szczerba does not explicitly teach but Yu teaches the limitation of wherein the determining the view comprises generating an augmented reality (AR) view of the road based on execution of the AI model (Yu, [0024],[0039], [0042], [0053], [0056]), and the displaying comprises displaying the AR view of the road on a display screen of a remote terminal configured to remotely operate the vehicle (Yu, [0026] the electronic device 102 may be a smartphone that is aimed at the road in the direction of travel, such as by mounting on a dashboard or being held in a user's hand).
It would have been obvious to one of ordinary skill in the art before the effective date of the present invention to modify, displaying lane information on an augmented reality display includes receiving roadway data, as taught by Szczerba, determining a view of the road based on execution of an artificial intelligence (AI) model on the previously-captured sensor data, as taught by Yu, as Szczerba and Yu are directed to an augmented reality display on a vehicle, (same field of endeavor), and one of ordinary skill in the art would have recognized the established utility using determining a view of the road based on execution of an artificial intelligence (AI) model on the previously-captured sensor data and predictably applied it to Szczerba’s teaching to makes complex decisions in real-time.
Regarding claims 10 and 17, please see the rejection above with regarding claim 3.
Regarding claim 4, Szczerba teaches the limitation of wherein the determining that the confidence of the sensor is below the threshold comprises determining that visibility of an environment around the vehicle has deteriorated based on at least one of a weather condition, a time of day, and an object in the road, and the extracting comprises extracting the previously-captured sensor data of the road from sensor data captured in conditions in which the visibility of the environment around the vehicle had not deteriorated ([0020] providing a vehicle user with lane information when the vehicle approaches a road junction or when the visibility is poor. [0050] these virtual images 50 are particularly helpful in situations of poor visibility; [0056] At block 104, the controller 34 determines the location of the vehicle 10 and the environmental driving conditions around the vehicle 10, such as weather causing poor visibility; [0060] The method 100 also includes block 105, which is performed after block 104. At block 105, the controller 34 determines, using the image data (e.g., images), visibility (or legibility) of one or more road signs 53 (FIG. 2) along the route of the vehicle 10 is less than a predetermined minimum-confidence threshold).
Regarding claims 11 and 18, please see the rejection above with regarding claim 4.
Regarding claim 5, Szczerba teaches the limitation of wherein the determining the view comprises determining locations of lane lines within the road, and the displaying comprises displaying the locations of the lane lines in augmented reality within the vehicle ( Fig. 3-5 and corresponding paragraphs).
Regarding claims 12 and 19, please see the rejection above with regarding claim 5.
Regarding claim 6, Szczerba does not explicitly teach but Yu teaches the limitation of wherein the method further comprises augmenting the view of the road with additional data to fill-in missing parts of the road based on execution of the AI model, and the displaying comprises displaying the view of the road with the additional data (Yu,[0024] [0056]).
It would have been obvious to one of ordinary skill in the art before the effective date of the present invention to modify, displaying lane information on an augmented reality display includes receiving roadway data, as taught by Szczerba, determining a view of the road based on execution of an artificial intelligence (AI) model on the previously-captured sensor data, as taught by Yu, as Szczerba and Yu are directed to an augmented reality display on a vehicle, (same field of endeavor), and one of ordinary skill in the art would have recognized the established utility using determining a view of the road based on execution of an artificial intelligence (AI) model on the previously-captured sensor data and predictably applied it to Szczerba’s teaching to makes complex decisions in real-time.
Regarding claims 13 and 20, please see the rejection above with regarding claim 6.
Regarding claim 7, Szczerba teaches the limitation of wherein the determining the view further comprises iteratively updating the view of the road at predetermined intervals based on additional sensor data received as the vehicle continues to travel along the road, and iteratively displaying the updated view of the road within the vehicle at the predetermined intervals ([0064] [0064] At block 116, the controller 34 determines, in real time, the location, type, size, shape, and color of the virtual images 50 (FIG. 2) to be displayed on the display 29 (e.g., the dual-focal plane augmented reality display) based on the location of the eyes 66 and/or the head 69 of the user of the vehicle 10 and the lane information).
Regarding claim 14, please see the rejection above with regarding claim 7.
Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. For example, Beek (JP 2022524920 A) teaches (page 1 of Beek English Translation, Claim) filters the sensor data by determining the reliability level of the sensor data and discarding the sensor data in response to the determination that the sensor data is less than or equal to the reliability threshold).
Viswanathan (US 20200189390 Al) teaches provide using vision-based map-ping augmented reality of objects in the environment of the vehicle in response to objects being obscured ([0030] signs are occluded by snow, foliage, or decay, the navigation decision may fall fully on a human driver that lacks sufficient information to make an informed decision with respect to turns or appropriate lanes, [0047] Also, remote sensing, such as aerial or satellite photography and/or LiDAR, can be used to generate map geometries directly or through machine learning as described herein). Another example, Lee (US 20180031384A1) teaches an augmented road line display system.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Anne Antonucci can be reached on (313)446-6519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-100.
/JINGLI WANG/ Examiner, Art Unit 3666
/ANNE MARIE ANTONUCCI/ Supervisory Patent Examiner, Art Unit 3666
Prosecution Insights