DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The office action is in response to Applicant’s amendment filed 08/31/2026 which has been entered and made of record. Claims 1, 6-8, 11, 13-15, 17 and 19-20 have been amended. No claim has been newly added. Claims 1-20 are pending in the application. Applicant's amendments to the claim language have overcome each and every objection previously set forth in the Non-Final Office Action mailed 06/03/2026.
Response to Arguments
Applicant’s arguments, filed 08/31/2026, with respect to the rejection(s) under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Hummelshoj and Toyoda as fully explained below.
Applicant argues Hummelshoj does not teach the newly amended independent claims.
Examiner agrees. However, a new ground of rejection is made in view of Hummelshoj and Toyoda. First, Toyoda teaches a user profile based on the skill level of the driver, a selected mode, historic encounters by the driver; a hazard threshold is defined based on the profile, the threshold is used to decide if displaying a warning sign, paragraph [0048-0049] “ the engagement module 230 determines which potential hazards identified at 320 will be rendered in the AR system 180 as display scenarios…… the engagement module 230 filters or otherwise assesses which of the potential hazards are to be rendered according to, for example, a hazard threshold and/or additional information such as driver skill level. …… the engagement module 230 can select which potential hazards to display according to preference, risk, threat level …… the engagement module 230 can adjust the hazard threshold according to, in one embodiment, a skill level of the driver, a selected mode (e.g., training), current awareness/engagement of the driver on driving tasks, historic encounters by the driver with similar potential hazards (e.g., failure to properly navigate a hazard), and so on. In this way, the engagement module 230 can customize how many and/or which potential hazards are ultimately displayed to the driver in order to induce the driver to engage with driving tasks while informing the driver of the potential hazards.”,
Second Toyoda teaches analyzing the driver state information based on vehicle status to decide on potential hazard, if the hazard is over the threshold, a graphics element is rendered, paragraph [0056] “the engagement module 230 analyzes the driver state information collected by the monitoring module 220 to determine whether the driver is responding to the display scenario by engaging the driving tasks and the surrounding environment in an expected manner. For example, the engagement module 230, in one embodiment, can analyze the driver state information to determine whether the driver is visually scanning a region of the surrounding environment for the potential hazard, whether the driver is adjusting a speed and/or path of the vehicle 100 to account for the potential hazard, whether the driver is obeying laws”, paragraph [0048] “ the hazard threshold defines a likelihood of risk that is to be satisfied by the potential hazards to qualify for display. That is, when a potential hazard satisfies the hazard threshold (e.g., 30% likelihood of risk or greater), then the engagement module 230 renders graphical elements for the potential hazard.”.
Conclusions: The rejections set in the previous Office Action are shown to have been proper, and the claims are rejected below. New citations and parenthetical remarks can be considered new grounds of rejection and such new grounds of rejection are necessitated by the Applicant's amendments to the claims. Therefore, the present Office Action is made final.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7-12, 14-18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hummelshoj et al. (US 20190228118 A1), hereinafter as Hummelshoj, in view of Toyoda et al. (US 20180322783 A1), hereinafter as Toyoda.
Regarding claim 1, Hummelshoj teaches A computer-implemented method for generating vehicular-based visualizations (Hummelshoj paragraph [0016] “FIG. 3 illustrates one embodiment of a method associated with varying an amount and type of information that is provided to a vehicle operator to ascertain how the vehicle operator perceives a surrounding environment.”), the method comprising:
receiving, by a computing device, a plurality of vehicular parameters associated with a vehicle (Hummelshoj teaches sensor data as a plurality of vehicular parameters, paragraph [0033-0034] “the scene module 220 selects a set of previously gathered sensor data (e.g., video, LIDAR data, etc.) that the scene module 220 transforms into the simulation…… the sensor data can include, for example, scan data that embodies observations of one or more objects in a surrounding environment proximate to the vehicle 100 and/or other aspects about the surroundings.”, paragraph [0073] “The example sensors may be part of the one or more environment sensors 122 and/or the one or more vehicle sensors 121”);
analyzing, by the computing device, the plurality of vehicular parameters based on a user profile associated with at least one occupant of the vehicle (Hummelshoj teaches log 260 as the user profile related to the electronic inputs from the vehicle operator, Figure 3, paragraph [0052-0054] “At 350, the interface module 230 monitors for electronic inputs of the vehicle 100 that are generated by an operator …… as the interface module 230 detects the inputs, the interface module 230 logs the inputs into the log 260 as correlating with a particular portion of the driving scene and a particular one of the visualization algorithms 250……. the interface module 230 scores the inputs by comparing the inputs with recorded inputs from when the sensor data was collected. That is, the interface module 230 can compare the inputs from the operator with original inputs to determine how closely the present inputs are in relation to the original inputs.”);
generating, by the computing device, a virtual environment visualization associated with the vehicle based on the analysis (Hummelshoj Figure 3, paragraph [0055] “At 370, the interface module 230 determines whether the control inputs are adequate. In one embodiment, the interface module 230 indicates when the control inputs are out of bounds (e.g., insufficient) to the scene module 220 so that the scene module 220 can adapt which of the algorithms 250 are used to modify the driving scene. In this way, the perception system 170 can customize the visualization such that the operator is provided with various displays of information for the driving scene and a best-fit can be determined.”); ……
Hummelshoj is not relied on for the below claim language …… and selecting and integrating, by the computing device, at least one virtual element into the virtual environment visualization based on one or more correlations between at least one occupant preference specified by the user profile and at least one vehicular parameter of the plurality of vehicular parameters. Toyoda teaches …… and selecting and integrating, by the computing device, at least one virtual element into the virtual environment visualization based on one or more correlations between at least one occupant preference specified by the user profile and at least one vehicular parameter of the plurality of vehicular parameters (Toyoda teaches using skill level of the driver, current awareness/engagement of the driver as part of the user profile, a hazard threshold is defined based on the preference, further teaches analyzing the driver state information to decide on engagement level based on monitoring vehicle speed and path, and display potential hazard if satisfying hazard threshold, paragraph [0048-0049] “ the engagement module 230 determines which potential hazards identified at 320 will be rendered in the AR system 180 as display scenarios…… the engagement module 230 filters or otherwise assesses which of the potential hazards are to be rendered according to, for example, a hazard threshold and/or additional information such as driver skill level. …… the engagement module 230 can select which potential hazards to display according to preference, risk, threat level …… the engagement module 230 can adjust the hazard threshold according to, in one embodiment, a skill level of the driver, a selected mode (e.g., training), current awareness/engagement of the driver on driving tasks, historic encounters by the driver with similar potential hazards (e.g., failure to properly navigate a hazard), and so on. In this way, the engagement module 230 can customize how many and/or which potential hazards are ultimately displayed to the driver in order to induce the driver to engage with driving tasks while informing the driver of the potential hazards.”, paragraph [0056] “the engagement module 230 analyzes the driver state information collected by the monitoring module 220 to determine whether the driver is responding to the display scenario by engaging the driving tasks and the surrounding environment in an expected manner. For example, the engagement module 230, in one embodiment, can analyze the driver state information to determine whether the driver is visually scanning a region of the surrounding environment for the potential hazard, whether the driver is adjusting a speed and/or path of the vehicle 100 to account for the potential hazard, whether the driver is obeying laws”).
Hummelshoj and Toyoda are in the same field of endeavor, namely generating vehicle related virtual experience. Toyoda teaches displaying potential hazards using graphical elements to improve driver engagement (paragraph [0020] “the disclosed systems and methods improve driver engagement on driving tasks and awareness of surroundings through motivating the self-engagement of the driver.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Toyoda with the method of Hummelshoj to improve user interaction.
Regarding claim 2, Hummelshoj in view of Toyoda teaches The computer-implemented method of claim 1, and further teaches wherein analyzing the plurality of vehicular parameters further comprises:
analyzing, by the computing device, a plurality of contextual information associated with the at least one occupant (Hummelshoj paragraph [0021] “By modifying how the driving scene is displayed, the perception system can log characteristics about how the operator perceives the surrounding environment to provide controls to the vehicle.”);
analyzing, by the computing device, a plurality of feedback information associated with the virtual environment visualization and the at least one occupant (Hummelshoj teaches grading based on user input as the feedback information, paragraph [0007] “in one embodiment, modifies the visualization to provide a reduced/modified view of the driving scenario to the operator. Accordingly, as the operator provides driving inputs while perceiving the modified view, the perception system can grade or otherwise assess whether the operator is controlling the vehicle in a safe or generally acceptable manner for the present driving conditions.”, paragraph [0009] “the perception system dynamically varies which visualization algorithm modifies the visualization of the driving scene as a function of changing driving conditions, feedback about driving inputs, and/or according to other criteria.”); and updating, by the computing device, the user profile based on the plurality of feedback information (Hummelshoj teaches a loop from checking Control Inputs Sufficient 370 to Select Visualization Algorithm 320 in Figure 3, paragraph [0050-0055] “the scene module 220 uses the visualization algorithm to apply one or more techniques to modify which aspects of the driving scene are displayed to the operator. As a general approach, the algorithms 250 focus on select aspects of the simulated environment in order to improve processing efficiency through processing a subset of available sensor data…… as the interface module 230 detects the inputs, the interface module 230 logs the inputs into the log 260 as correlating with a particular portion of the driving scene and a particular one of the visualization algorithms 250…… At 370, the interface module 230 determines whether the control inputs are adequate. In one embodiment, the interface module 230 indicates when the control inputs are out of bounds (e.g., insufficient) to the scene module 220 so that the scene module 220 can adapt which of the algorithms 250 are used to modify the driving scene.”).
Regarding claim 3, Hummelshoj in view of Toyoda teaches The computer-implemented method of claim 1, and further teaches wherein the virtual environment visualization is depicted to the at least one occupant via a computer-mediated reality device (CMR) communicatively coupled to at least one display of the vehicle (Hummelshoj paragraph [0022] “ the simulator is implemented within a functional vehicle by integrating display screens within the vehicle, providing for virtual reality interfaces, providing for augmented reality interfaces, and so on.”, paragraph [0026] “ the vehicle 100 is generally a vehicle simulator that includes a display positioned in a manner so as to simulate views of a surrounding environment of the vehicle 100 to the operator. Of course, in further embodiments, the display may instead be a virtual reality head-mounted device (HMD) or augmented reality HMD…… The simulation system 180 can be integrated with a standard functional vehicle or may be a stand-alone system that is independent of an actual vehicle.”).
Regarding claim 4, Hummelshoj in view of Toyoda teaches The computer-implemented method of claim 1, wherein generating the virtual environment visualization comprises: and further teach utilizing, by the computing device, one or more machine learning models to generate one or more predictions associated with a vehicular experience based on the plurality of vehicular parameters (Toyoda paragraph [0045] “the monitoring module 220 identifies the potential hazards by analyzing the sensor data according to, for example, the hazard model 250. That is, in one embodiment, the monitoring module 220 characterizes aspects of the surrounding environment as provided in the sensor data to determine whether the combination of objects, and other factors constitute one or more potential hazards to the vehicle 100. Additionally, the monitoring module 220 may accept the sensor data as electronic inputs and process the electronic inputs according to a machine learning algorithm and learned aspects embodied in the hazard model 250 to identify the potential hazards.”); and integrating, by the computing device, a plurality of virtual elements derived from the one or more predictions into the virtual environment visualization (Toyoda paragraph [0048-0050] “the engagement module 230 determines which potential hazards identified at 320 will be rendered in the AR system 180 as display scenarios…… when a potential hazard satisfies the hazard threshold (e.g., 30% likelihood of risk or greater), then the engagement module 230 renders graphical elements for the potential hazard…… the engagement module 230 renders graphical elements that correlate with the potential hazards within one or more AR displays of the AR system 180. In one embodiment, the engagement module 230 renders graphical elements within the AR system 180 by rendering overlays on objects/regions within the surrounding environment (e.g., animations, avatars, etc.),”).
Hummelshoj and Toyoda are in the same field of endeavor, namely generating vehicle related virtual experience. Toyoda teaches displaying potential hazards using graphical elements to improve driver engagement (paragraph [0020] “the disclosed systems and methods improve driver engagement on driving tasks and awareness of surroundings through motivating the self-engagement of the driver.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Toyoda with the method of Hummelshoj to improve user interaction.
Regarding claim 5, Hummelshoj in view of Toyoda teach The computer-implemented method of claim 4, and further teach further comprising:
correlating, by the computing device, the plurality of vehicular parameters with the plurality of virtual elements based on the user profile (Toyoda teaches using driver state information collected by monitoring module as part of the user profile, paragraph [0056] “ the engagement module 230 analyzes the driver state information collected by the monitoring module 220 to determine whether the driver is responding to the display scenario by engaging the driving tasks and the surrounding environment in an expected manner. For example, the engagement module 230, in one embodiment, can analyze the driver state information to determine whether the driver is visually scanning a region of the surrounding environment for the potential hazard, whether the driver is adjusting a speed and/or path of the vehicle 100 to account for the potential hazard, whether the driver is obeying laws”);
and updating, by the computing device, the virtual environment visualization based on the correlation (Toyoda paragraph [0057] “The engagement module 230, in one embodiment, analyzes the driver state information according to the hazard model 250 and/or a separate driver model that characterizes aspects of the driver behavior in order to determine engagement by the driver. Accordingly, the engagement module 230 can adjust the rendering of the display scenario, at 350, that depicts the potential hazard by further embellishing the graphical elements (e.g., decreasing opacity, increasing brightness, adding additional graphics, etc.)”).
Hummelshoj and Toyoda are in the same field of endeavor, namely generating vehicle related virtual experience. Toyoda teaches displaying potential hazards using graphical elements to improve driver engagement (paragraph [0020] “the disclosed systems and methods improve driver engagement on driving tasks and awareness of surroundings through motivating the self-engagement of the driver.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Toyoda with the method of Hummelshoj to improve user interaction.
Regarding claim 7, Hummelshoj in view of Toyoda teaches The computer-implemented method of claim 1, and further teaches wherein generating the virtual environment visualization comprises:
integrating, by the computing device, the vehicle and a subset of the plurality of vehicular parameters into the virtual environment visualization (Hummelshoj paragraph [0050] “the algorithms 250 focus on select aspects of the simulated environment in order to improve processing efficiency through processing a subset of available sensor data.”, paragraph [0073] “ The example sensors may be part of the one or more environment sensors 122 and/or the one or more vehicle sensors 121);
and updating, by the computing device, the virtual environment visualization based on the user profile (Hummelshoj Figure 3, paragraph [0055] “At 370, the interface module 230 determines whether the control inputs are adequate. In one embodiment, the interface module 230 indicates when the control inputs are out of bounds (e.g., insufficient) to the scene module 220 so that the scene module 220 can adapt which of the algorithms 250 are used to modify the driving scene. In this way, the perception system 170 can customize the visualization such that the operator is provided with various displays of information for the driving scene and a best-fit can be determined.”).
Regarding claim 8, it recites similar limitations of claim 1 but in a computer program product form. The rationale of claim 1 rejection is applied to reject claim 8. In addition, Hummelshoj teaches A computer program product for generating vehicular-based visualizations, the computer program product comprising one or more computer readable storage media and program instructions collectively stored on the one or more computer readable storage media, the stored program instructions comprising (Hummelshoj paragraph [0090] “arrangements described herein may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied, e.g., stored, thereon.”).
Regarding claim 9, claim 9 has similar limitations as claim 2, therefore it is rejected under the same rationale as claim 2.
Regarding claim 10, claim 10 has similar limitations as claim 3, therefore it is rejected under the same rationale as claim 3.
Regarding claim 11, claim 11 has similar limitations as claim 4, therefore it is rejected under the same rationale as claim 4.
Regarding claim 12, claim 12 has similar limitations as claim 5, therefore it is rejected under the same rationale as claim 5.
Regarding claim 14, claim 14 has similar limitations as claim 7, therefore it is rejected under the same rationale as claim 7.
Regarding claim 15, it recites similar limitations of claim 1 but in a computer system form. The rationale of claim 1 rejection is applied to reject claim 15. In addition, Hummelshoj teaches A computer system for generating vehicular-based visualizations, the computer system comprising: one or more processors; one or more computer-readable memories; program instructions stored on at least one of the one or more computer-readable memories for execution by at least one of the one or more processors, the program instructions comprising (Hummelshoj paragraph [0082] “The vehicle 100 can include one or more modules, at least some of which are described herein. The modules can be implemented as computer-readable program code that, when executed by a processor 110, implement one or more of the various processes described herein. One or more of the modules can be a component of the processor(s) 110, or one or more of the modules can be executed on and/or distributed among other processing systems to which the processor(s) 110 is operatively connected. The modules can include instructions (e.g., program logic) executable by one or more processor(s) 110. Alternatively, or in addition, one or more data store 115 may contain such instructions.”).
Regarding claim 16, claim 16 has similar limitations as claim 2, therefore it is rejected under the same rationale as claim 2.
Regarding claim 17, claim 17 has similar limitations as claim 4, therefore it is rejected under the same rationale as claim 4.
Regarding claim 18, claim 18 has similar limitations as claim 5, therefore it is rejected under the same rationale as claim 5.
Regarding claim 20, claim 20 has similar limitations as claim 7, therefore it is rejected under the same rationale as claim 7.
Claim(s) 6, 13 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hummelshoj et al. (US 20190228118 A1), hereinafter as Hummelshoj, in view of Toyoda et al. (US 20180322783 A1), hereinafter as Toyoda, further in view of Sokolov et al. (US 20220242450 A1), hereinafter as Sokolov.
Regarding claim 6, Hummelshoj in view of Toyoda teaches The computer-implemented method of claim 1, but is not relied on for the below claim language wherein the virtual environment visualization comprises one or more interactive virtual objects configured to receive at least one input from the user. Sokolov teaches wherein the virtual environment visualization comprises one or more interactive virtual objects configured to receive at least one input from a user (Sokolov paragraph [0031] “the fusion of all physical and virtual spatial data into the Metaverse World Model is the foundation for interaction between physical and virtual objects…… virtual world objects such as trucks, buses, cars, vans, motorbikes as well as vulnerable road users such as pedestrians, cyclists and animals can be injected into the metaverse world model requiring physical cars to avoid them”).
Hummelshoj, Toyoda and Sokolov are in the same field of endeavor, namely generating vehicle related virtual experience. Sokolov teaches the interaction of virtual objects and real objects to improve accuracy (paragraph [0147] “The Metaverse implementation provides a set of methods designed for various sensor data formats and their data processing systems allowing seamless and accurate infusion of artificial virtual objects and conditions into normal sensor data, reflecting real physical objects and conditions.”, and paragraph [0168] “Software algorithm upgrades and changes can be rapidly provided and tested out in these hybrid real-world and virtual world scenarios, greatly increasing the rapidity of algorithm improvement and testing against a far broader range of scenarios than would be possible with testing limited to real-world only testing.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Sokolov with the method of Hummelshoj and Toyoda to improve interaction and accuracy.
Regarding claim 13, claim 13 has similar limitations as claim 6, therefore it is rejected under the same rationale as claim 6.
Regarding claim 19, claim 19 has similar limitations as claim 6, therefore it is rejected under the same rationale as claim 6.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAOMING WEI whose telephone number is (571)272-3831. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
/XIAOMING WEI/ Examiner, Art Unit 2611
Prosecution Insights