DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority under 35 U.S.C. 119(a)-(d). The certified copy of European patent application number EP24174118 filed on 05/06/2024 has been received and made of record.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/02/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“a perception module”, “a visualization module” in claim 1;
“an interpretation module” in claim 5.
“a perception module”, “a visualization module”; and “an interpretation module” in claim 16.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-2 and 14-17 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by European PGPubs EP3859390 to Torschmied et al..
Regarding claim 1, Torschmied et al. teach a computer-implemented method for improving vision of a vehicle's environment for one or more users of the vehicle (abstract, par 0001, “method for rendering a representation of an environment of a vehicle, a system for rendering a representation of an environment of a vehicle, and a vehicle comprising a system for rendering a representation of an environment of the vehicle”), the method comprising:
aggregating, by a perception module of the vehicle, sensor data provided by a plurality of sensors of the vehicle (par 0050, “By using an image processing technique such as for example SfM ("Structure from Motion"), depth information, i.e. distance information, can be obtained from the captured images. For this purpose, parameters of the motion of the vehicle 10, such as the travelled distance, are taken into account. The motion parameters can be obtained using one or more sensors including, for example, a GNSS location sensor (GPS), a motion sensor, an acceleration sensor, an odometer, etc.”);
constructing, by the perception module, a three-dimensional (3D) visual world model, based on the aggregated sensor data, the 3D visual world model representing the vehicle and the vehicle's environment (par 0017-0018, “This information may be extracted from the color information of the pixels associated with the object in the captured image(s). By obtaining color and/or texture information, a more realistic three-dimensional model of the object may be rendered. …a rendering step of the method includes extracting points which belong to an object from the point cloud. By extracting those points of the (three-dimensional) point cloud, a three-dimensional model of the object may be generated”, par 0051-0052, “As a result, a three-dimensional model 3 of the object 2 may be rendered including distance information, dimensions, shape, color, and texture of the object. This model 3 may be displayed to an occupant of the vehicle 10.”);
rendering, by a visualization module of the vehicle, a user's view of the vehicle's environment, based on the constructed 3D visual world model (par 0009, “In today’s applications including Advanced Driver Assistance Systems (ADAS), information on static or dynamic objects may be detected and displayed to an occupant of a vehicle. These objects can include static lanes (such as road surface markings) and/or three-dimensional objects that may be rendered into a virtual scene which is displayed to the occupant. Rendering of three-dimensional objects can be performed based on a fusion of sensor data provided by one or more sensors of the vehicle including LiDAR, a radar system, and one or more cameras”, par 0021, “the rendering step includes rendering, for each detected object, a three-dimensional virtual object in a three-dimensional scene based on the extracted object distances …. The rendered objects may be arranged at positions relative to each other and relative to the vehicle such that a scene corresponding to the real world is created. This step may also comprise a step of adapting the perspective of the generated scene from a viewpoint of the camera unit to a viewpoint of an occupant of the vehicle”, par 0066, “In another process S6, a three-dimensional virtual object 3 of each object 2 is rendered in a three-dimensional scene 4 based on the extracted object distances”); and
providing, by the visualization module, the rendered view to one or more displays for use by the one or more users of the vehicle (par 0010, “The information comprised in such a rendered virtual scene may comprise lane and line information, wherein road markings and the ego-lane that the vehicle is currently driving on may be displayed together with one or more adjacent lane(s). Furthermore, static objects such as barriers, road shoulders, buildings and/or other obstacles may be displayed. The scene may also display dynamic objects such as other road users including other vehicles or and/or pedestrians. Furthermore, the virtual scene may be integrated with additional information obtained from a map based on a digital terrain model (DTM) or three-dimensional city data adding information on static objects or landmarks such as buildings, elevation, mountains, bridges, etc.”, par 0067, “in a last visualization step E of the method, the rendered three-dimensional scene 4 is output to a display unit 13 such that an occupant of the vehicle 10 may view the scene 4 on a screen of the display unit 13.”).
Regarding claim 2, Torschmied et al. teach all the limitation of claim 1, and further teach wherein: the aggregated sensor data includes one or more of: current sensor data, previously recorded sensor data obtained from a sensor data history buffer, and generated sensor data for complementing the aggregated sensor data, and the generated sensor data includes data regarding portions of the vehicle's environment not visible to the plurality of sensors (par 0050, “Fig. 1a) illustrates a motion in forward direction of the vehicle 10. As an example of a stationary object 2, a tree on a side of the road the vehicle 10 is driving on is illustrated. The camera 14 captures an image at each of two points in time between which the vehicle travels a certain distance”, par 0057, “each pixel or each predefined subset of pixels of the image is associated with depth information which may be obtained directly from a stereo image provided by the stereo camera 14. Alternatively, depth information may be provided by using an SfM ("Structure from Motion") algorithm, wherein pixels of an image are analyzed by comparing a current image with an image captured at an earlier point in time, wherein the camera 14 (i.e. the vehicle) has moved a known distance with a known heading.”).
Regarding claim 14, Torschmied et al. teach all the limitation of claim 1, and further teach wherein rendering the user's view of the vehicle's environment includes rendering, for each of the one or more users of the vehicle, a respective view of the vehicle's environment (par 0021, “the three-dimensional graphical representation of the scene including the rendered three-dimensional models of the detected object(s) may be generated. In particular, the rendered objects may be arranged at positions relative to each other and relative to the vehicle such that a scene corresponding to the real world is created. This step may also comprise a step of adapting the perspective of the generated scene from a viewpoint of the camera unit to a viewpoint of an occupant of the vehicle.”).
Regarding claim 15, Torschmied et al. teach a non-transitory computer-readable medium comprising instructions (par 0044). The remaining limitations of the claim are similar in scope to claim 1 and rejected under the same rationale.
Regarding claim 16, Torschmied et al. teach a system for improving vision of a vehicle's environment for one or more users of the vehicle, the system comprising: a perception module including a plurality of sensors; a visualization module; an interpretation module; and one or more displays (Fig 2, par 0055), wherein the plurality of sensors includes at least one of: a LiDAR sensor, a camera, a radar sensor, and an ultrasonic sensor (par 0009, par 0050). The remaining limitations of the claim are similar in scope to claim 1 and rejected under the same rationale.
Regarding claim 17, Torschmied et al. teach a vehicle comprising the system of claim 16 (Fig 1(a) and 2, par 0050, par 0055). The remaining limitations of the claim are similar in scope to claim 16 and rejected under the same rationale.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3-5 is/are rejected under 35 U.S.C. 103 as being unpatentable over European PGPubs EP3859390 to Torschmied et al. in view of U.S. PGPubs 2021/0406559 to Efland et al..
Regarding claim 3, Torschmied et al. teach all the limitation of claim 1, but keep silent for teaching wherein the constructed 3D visual world includes positional information and visual property information regarding the vehicle's environment determined based on the aggregated sensor data.
In related endeavor, Efland et al. teach wherein the constructed 3D visual world includes positional information and visual property information regarding the vehicle's environment determined based on the aggregated sensor data (par 0045-0049, “vehicle 121 may capture LiDAR data that provides a 3D representation of the spatial properties (e.g., shape, location, etc.) of various features within real-world environment 120 from the perspective of vehicle 121”, par 0055-0056, “The different types of map data that may be included in these layers may take various forms, examples of which may include (i) road-network data that provides information regarding the road network found at a real-world location, (ii) geometric data that provides information regarding the physical geometry of the real-world location (e.g., the physical geometry of the ground and any other objects found at that real-world location), (iii) semantic data that provides information about semantic elements found at the real-world location (e.g., lanes, traffic lights, traffic signs, etc.), (iv) “priors” data that provides information about observed prior behavior at the real-world location (e.g., prior vehicle trajectories that have been observed within one or more lanes found at the real-world location), and (v) real-time data that provides real-time information about the real-world location (e.g., current traffic conditions)”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Torschmied et al. to include wherein the constructed 3D visual world includes positional information and visual property information regarding the vehicle's environment determined based on the aggregated sensor data as taught by Efland et al. to perform functions such as localization, object detection, prediction, and path planning using a variety of data, including but not limited to sensor data captured by the vehicle and map data related to the vehicle's surrounding environment to provide information regarding the physical geometry of the geographic region (e.g., the geometry of the ground surfaces within the geographic region), semantic data that provides information regarding semantic elements within the geographic region (e.g., lanes, traffic lights, traffic signs, crosswalks, etc.), and behavior data that provides information regarding observed prior behavior within the geographic region (e.g., prior trajectories of agents such as vehicles, cyclists, and pedestrians), among other possibilities.
Regarding claim 4, Torschmied et al. as modified by Efland et al. teach all the limitation of claim 3, and Torschmied et al. further teach further comprising: detecting, by the perception module, one or more objects in a vicinity of the vehicle based on the aggregated sensor data, wherein constructing the 3D visual world model is further based on the detected objects, and wherein rendering the view of the vehicle's environment further includes rendering visual representations of the detected objects in the rendered view (par 0012, “objects in a captured image may be detected and a three-dimensional scene may be rendered by arranging and displaying generic three-dimensional models corresponding to the detected objects”, par 0020-0021, “the rendering step includes rendering, for each detected object, a three-dimensional virtual object in a three-dimensional scene based on the extracted object distances. In this step, the three-dimensional graphical representation of the scene including the rendered three-dimensional models of the detected object(s) may be generated. In particular, the rendered objects may be arranged at positions relative to each other and relative to the vehicle such that a scene corresponding to the real world is created. This step may also comprise a step of adapting the perspective of the generated scene from a viewpoint of the camera unit to a viewpoint of an occupant of the vehicle”, par 0024, “the detected distance information is fused with the captured image to extract object distances. In particular, distance information may be fused with the image data such that distance information may be assigned to each pixel or each of a subset of pixels such that a three-dimensional point cloud may be generated”).
Regarding claim 5, Torschmied et al. as modified by Efland et al. teach all the limitation of claim 4, and further teach further comprising identifying, by an interpretation module, one or more of the detected objects as objects relevant for the one or more users of the vehicle (Torschmied et al.: “A more realistically rendered model of the object has the advantage that it may be recognized more easily by human occupants. An object direction may include the direction the object, such as another vehicle, is currently moving or about to move. The direction an object is about to move may be derived from an intention which may be detected by recognizing patterns including activation of turn signals, steering movements, and/or lane markings. Such features may be analyzed using artificial intelligence including, for example, neural networks. The object type may be obtained by applying image recognition algorithms. This way the detected object(s) may be classified into one or more object type(s), for example one of: a vehicle, a motorcycle, a lorry, a bicycle, a pedestrian, a tree, a building, a road surface marking, a barrier, a bridge, a sign (post), etc.”, Efland et al.: par 0056, ““The different types of map data that may be included in these layers may take various forms, examples of which may include (i) road-network data that provides information regarding the road network found at a real-world location, (ii) geometric data that provides information regarding the physical geometry of the real-world location (e.g., the physical geometry of the ground and any other objects found at that real-world location), (iii) semantic data that provides information about semantic elements found at the real-world location (e.g., lanes, traffic lights, traffic signs, etc.), (iv) “priors” data that provides information about observed prior behavior at the real-world location (e.g., prior vehicle trajectories that have been observed within one or more lanes found at the real-world location), and (v) real-time data that provides real-time information about the real-world location (e.g., current traffic conditions)”, par 0126-0128, “Because there may not yet be any additional sensor data indicating that the newly observed vehicle trajectory 153 is indicative of a more permanent change, the off-vehicle computing platform may initially designate the observed trajectory 153 as an update to the real-time layer 205. Accordingly, the off-vehicle computing platform may effect an update to the map that includes an indication of the new prior in the real-time layer 205”).
Claim(s) 6-8 is/are rejected under 35 U.S.C. 103 as being unpatentable over European PGPubs EP3859390 to Torschmied et al. in view of U.S. PGPubs 2021/0406559 to Efland et al., further in view of U.S. PGPubs 2025/0329257 to Tian.
Regarding claim 6, Torschmied et al. as modified by Efland et al. teach all the limitation of claim 5, but keep silent for teaching wherein identifying, by the interpretation module, the one or more of the detected objects as objects relevant for the one or more users of the vehicle includes: determining a set of driving parameters for the vehicle, wherein the set of driving parameters includes at least one of velocity and steering angle; and based on the determined set of driving parameters and the positional information and visual property information included in the 3D visual world model, determining objects as relevant for the one or more users of the vehicle by evaluating, for each of the detected objects, if the respective detected object is one or more of: an object with reduced visibility such that the object is only visible to the user to a limited extent, a collision risk static object such that there is a risk of a collision between the vehicle and the collision risk static object, and a collision risk moving object such that there is a risk of a collision between the vehicle and the collision risk moving object.
In related endeavor, Tian teaches wherein identifying, by the interpretation module, the one or more of the detected objects as objects relevant for the one or more users of the vehicle includes: determining a set of driving parameters for the vehicle, wherein the set of driving parameters includes at least one of velocity and steering angle (par 0035, “The first state information of the ego vehicle is used to characterize a real-time driving state and physical information of the ego vehicle. The real-time driving state may include but is not limited to a speed, acceleration, a yaw angle, a yaw rate, and a vehicle control state (such as a position of an accelerator pedal, a position of a brake pedal, an angle of a steering wheel, angular velocity of the steering wheel, a vehicle gear, and a vehicle mode)”); and based on the determined set of driving parameters and the positional information and visual property information included in the 3D visual world model, determining objects as relevant for the one or more users of the vehicle by evaluating, for each of the detected objects, if the respective detected object is one or more of: an object with reduced visibility such that the object is only visible to the user to a limited extent, a collision risk static object such that there is a risk of a collision between the vehicle and the collision risk static object, and a collision risk moving object such that there is a risk of a collision between the vehicle and the collision risk moving object (par 0027, par 0031-0032, “The collision risk between the ego vehicle and the target object may be determined based on the first state information of the ego vehicle 11 and the second state information of the target object 12. Moreover, the target time to collision may be determined when it is determined that a collision will occur, so that the ego vehicle may be controlled to take corresponding collision avoidance measures, such as deceleration and emergency braking. According to this disclosure, the target time to collision that can better reflect an actual scenario may be determined based on the first predicted time to collision and the collision scenario that are preliminarily determined, which improves robustness and accuracy of a time to collision prediction scheme, thereby helping improve safety and effectiveness of vehicle driving control”, par 00034-0041, “The first state information of the ego vehicle is used to characterize a real-time driving state and physical information of the ego vehicle. The real-time driving state may include but is not limited to a speed, acceleration, a yaw angle, a yaw rate, and a vehicle control state (such as a position of an accelerator pedal, a position of a brake pedal, an angle of a steering wheel, angular velocity of the steering wheel, a vehicle gear, and a vehicle mode) ….The second state information of the target object is used to characterize motion state information and physical information of the target object, which may include but is not limited to a type, a length, and a width of the target object, a lateral/longitudinal distance of the target object in a vehicle coordinate system of the ego vehicle, lateral/longitudinal velocity, lateral/longitudinal acceleration, a yaw angle, a yaw rate, and other information…..Determining a collision risk and a first predicted time to collision between the ego vehicle and the target object based on the first state information and the second state information. The collision risk is used to indicate a risk of collisions between the ego vehicle and the target object, and may indicate that a collision will occur or will not occur. The first predicted time to collision is used to indicate a time when a collision may occur between the ego vehicle and the target object, as determined based on the first state information and the second state information. In this embodiment, the first predicted time to collision may be calculated based on a motion trajectory of a center point of the ego vehicle and a motion trajectory of a center point of the target object”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Torschmied et al. as modified by Efland et al. to include wherein identifying, by the interpretation module, the one or more of the detected objects as objects relevant for the one or more users of the vehicle includes: determining a set of driving parameters for the vehicle, wherein the set of driving parameters includes at least one of velocity and steering angle; and based on the determined set of driving parameters and the positional information and visual property information included in the 3D visual world model, determining objects as relevant for the one or more users of the vehicle by evaluating, for each of the detected objects, if the respective detected object is one or more of: an object with reduced visibility such that the object is only visible to the user to a limited extent, a collision risk static object such that there is a risk of a collision between the vehicle and the collision risk static object, and a collision risk moving object such that there is a risk of a collision between the vehicle and the collision risk moving object as taught by Tian to predict, in a real-time manner, a trajectory of an ego vehicle and a trajectory of a target object (such as a pedestrian or a vehicle) around to which collisions may occur, and calculate time to collision (TTC) to evaluate a collision risk based on the calculated TTC to determine time to collision that can better reflect an actual collision scenario based on the first predicted time to collision and the collision scenario that are preliminarily determined, which improves robustness and accuracy of a time to collision prediction scheme, thereby helping improve safety of vehicle driving control.
Regarding claim 7, Torschmied et al. as modified by Efland et al. and Tian teach all the limitation of claim 6, and Tian further teaches wherein evaluating, for each of the detected objects, if the respective detected object is a collision risk static object includes: determining a time to collision of the vehicle with the respective detected object based on the determined set of driving parameters and the 3D visual world model (Fig 2, par 00034-0041, “The first state information of the ego vehicle is used to characterize a real-time driving state and physical information of the ego vehicle. The real-time driving state may include but is not limited to a speed, acceleration, a yaw angle, a yaw rate, and a vehicle control state (such as a position of an accelerator pedal, a position of a brake pedal, an angle of a steering wheel, angular velocity of the steering wheel, a vehicle gear, and a vehicle mode) ….The second state information of the target object is used to characterize motion state information and physical information of the target object, which may include but is not limited to a type, a length, and a width of the target object, a lateral/longitudinal distance of the target object in a vehicle coordinate system of the ego vehicle, lateral/longitudinal velocity, lateral/longitudinal acceleration, a yaw angle, a yaw rate, and other information…..Determining a collision risk and a first predicted time to collision between the ego vehicle and the target object based on the first state information and the second state information. The collision risk is used to indicate a risk of collisions between the ego vehicle and the target object, and may indicate that a collision will occur or will not occur. The first predicted time to collision is used to indicate a time when a collision may occur between the ego vehicle and the target object, as determined based on the first state information and the second state information. In this embodiment, the first predicted time to collision may be calculated based on a motion trajectory of a center point of the ego vehicle and a motion trajectory of a center point of the target object”).
Regarding claim 8, Torschmied et al. as modified by Efland et al. and Tian teach all the limitation of claim 6, and Tian further teaches wherein evaluating, for each of the detected objects, if the respective detected object is a collision risk moving object includes: predicting a trajectory of the respective detected object; and based on the predicted trajectory, the determined set of driving parameters and the 3D visual world model, determining a time to collision of the vehicle with the respective detected object (par 00034-0041, “The first state information of the ego vehicle is used to characterize a real-time driving state and physical information of the ego vehicle. The real-time driving state may include but is not limited to a speed, acceleration, a yaw angle, a yaw rate, and a vehicle control state (such as a position of an accelerator pedal, a position of a brake pedal, an angle of a steering wheel, angular velocity of the steering wheel, a vehicle gear, and a vehicle mode) ….The second state information of the target object is used to characterize motion state information and physical information of the target object, which may include but is not limited to a type, a length, and a width of the target object, a lateral/longitudinal distance of the target object in a vehicle coordinate system of the ego vehicle, lateral/longitudinal velocity, lateral/longitudinal acceleration, a yaw angle, a yaw rate, and other information…..Determining a collision risk and a first predicted time to collision between the ego vehicle and the target object based on the first state information and the second state information. The collision risk is used to indicate a risk of collisions between the ego vehicle and the target object, and may indicate that a collision will occur or will not occur. The first predicted time to collision is used to indicate a time when a collision may occur between the ego vehicle and the target object, as determined based on the first state information and the second state information. In this embodiment, the first predicted time to collision may be calculated based on a motion trajectory of a center point of the ego vehicle and a motion trajectory of a center point of the target object”, Fig 3, par 0050-0069, “Determining a first predicted trajectory within a preset time period for the ego vehicle based on the first state information ….Determining a second predicted trajectory within the preset time period for the target object based on the second state information. ….Determining at least one second predicted time to collision between the ego vehicle and the target object based on the first predicted trajectory and the second predicted trajectory. ….Determining the collision risk between the ego vehicle and the target object based on at least one of the second predicted time to collision”).
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over European PGPubs EP3859390 to Torschmied et al. in view of U.S. PGPubs 2021/0406559 to Efland et al., further in view of U.S. PGPubs 2025/0329257 to Tian, further in view of U.S. PGPubs 2023/0230334 to Ko et al..
Regarding claim 9, Torschmied et al. as modified by Efland et al. and Tian teach all the limitation of claim 7, and Tian further teaches further comprising: sorting the identified relevant objects based on the determined time to collision of the vehicle with the respective detected object (par 0064, “Since vehicle 22 is approaching vehicle 21 and there is a visual link, the radar sensor 13 integrated in vehicle 21 can detect vehicle 22. The system according to the invention in vehicle 21 therefore decides that vehicle 22 is visible to the driver. Since vehicle 22 is furthermore in a different lane, vehicle 22 presents no risk of collision for vehicle 21. The system according to the invention therefore does not highlight vehicle 22 from the presentation for the driver. Since the junction is surrounded by buildings 25, 26, 27 and 28, there is no direct visual link from vehicle 21 to vehicle 23 or to vehicle 24, however. Since vehicle 23 has already passed through the junction a few moments beforehand and has been identified by the radar sensor, however, the ambient information data capture device 11 knows both the direction of movement and the speed of vehicle 23. Moreover, there is a WLAN connection from vehicle 21 to vehicle 23, which furthermore transmits position data, speed data, and direction of movement data to vehicle 21. Model data for producing a realistic, three-dimensional model of vehicle 23 in vehicle 21 are also transmitted by means of the WLAN connection. The system according to the invention in vehicle 21 therefore presents the driver with a realistic, three-dimensional model of vehicle 23 in a two-dimensional projection on the windshield. Since vehicle 23 is not visible to the driver of vehicle 21, because it is concealed by building 28, the information presentation device presents vehicle 23 to the driver as though he could see through building 28. Since the information presentation device presents all the presented objects in a manner that is true to location and size, the driver of vehicle 21 can perceive the presented model of vehicle 23 at that location in the windshield at which he would perceive it if his view were not blocked by building 28. In addition, he perceives the model of vehicle 23 from that angle which would correspond to his actual perception. Since the system identifies, on the basis of the known direction of movement of vehicle 23, that there is no danger of collision between vehicle 21 and vehicle 23, it does not particularly highlight vehicle 23 from the surroundings so as not to unnecessarily direct the attention of the driver to an object which does not present a hazard”), but keep silent for teaching removing less relevant objects from the sorted identified relevant objects to maintain only a subset of most relevant objects.
In related endeavor, Ko et al. teach further comprising: sorting the identified relevant objects based on the determined time to collision of the vehicle with the respective detected object (par 0070, “the importance of an external object may be classified based on a first importance and a second importance”); and removing less relevant objects from the sorted identified relevant objects to maintain only a subset of most relevant objects (par 0072, “the processor 220 may control the display 231 to display at least one augmented reality object or at least one virtual reality object in the first area in response to the importance of the external object which is less than the first importance. The processor 220 may control the display 231 to display at least one augmented reality object or at least one virtual reality object in the first area even when information on an external object is not obtained. At this time, the processor 220 may control the display 231 not to display the augmented reality object or the virtual reality object in the second area.”, par 0078, “in operation 370, the processor 220 may control the display 231 such that, in response to the importance of the external object which is equal to or greater than the second importance, the indicator is displayed in a second size larger than the first size and at least a part of at least one augmented reality object or at least one virtual reality object being displayed in the first area is removed according to priority”, par 0097, “the processor 220 may control the display 231 such that at least some of the plurality of augmented reality objects being displayed in the first area 421 are removed according to priority, thereby displaying augmented reality objects as shown in reference number 431. For example, the processor 220 may control the display 231 such that the augmented reality object including the destination information 433 and the movement direction information 434 is continuously displayed in the first area 431 and the augmented reality object including the time information indicating an arrival time 415 and information on nearby shops 416 is removed due to the low priority thereof”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Torschmied et al. as modified by Efland et al. and Tian to include removing less relevant objects from the sorted identified relevant objects to maintain only a subset of most relevant objects as taught by Ko et al. to control the display to display an indicator in the second area so that the indicator has a first size and removes at least a part of the at least one augmented reality object or the at least one virtual reality object according to priorities.
Claim(s) 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over European PGPubs EP3859390 to Torschmied et al. in view of U.S. PGPubs 2021/0406559 to Efland et al., further in view of U.S. PGPubs 2013/0181823 to Stahlin et al..
Regarding claim 10, Torschmied et al. as modified by Efland et al. teach all the limitation of claim 4, but keep silent for teaching wherein rendering the view of the vehicle's environment further includes one or more of: augmenting one or more of the visual representations of the detected objects in the rendered view with additional visual information; and augmenting the rendered view by adding at least one object to the rendered view.
In related endeavor, Stahlin et al. teach wherein rendering the view of the vehicle's environment further includes one or more of: augmenting one or more of the visual representations of the detected objects in the rendered view with additional visual information; and augmenting the rendered view by adding at least one object to the rendered view (Fig 3, par 0067-0068, “The system uses a camera sensor in the ambient information data capture device to identify a stop sign 34 and concludes therefrom that the driver does not have traffic priority at the junction. This information is additionally verified by means of the digital map material which is present in the system. In order to point this out to the driver, the system visually highlights stop sign 34 by presenting stop sign 34 in flashing yellow on the windshield … The ambient information data evaluation device analyzes the situation and establishes that there is a danger of collision between the system's own vehicle and vehicle 33. For this reason, not only is a model of vehicle 33 presented to the driver as though he can see through building 32, but also the model of vehicle 33 is presented in red and flashing”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Torschmied et al. as modified by Efland et al. to include wherein rendering the view of the vehicle's environment further includes one or more of: augmenting one or more of the visual representations of the detected objects in the rendered view with additional visual information; and augmenting the rendered view by adding at least one object to the rendered view as taught by Stahlin et al. to insert with additional information matched to the surroundings such as navigation advice, markers for particular buildings or other two-dimensional elements into the perception of the surroundings to provide warning a driver about hazards which are not visible to the driver himself or useful information for driver to allow the driver to react more quickly.
Regarding claim 11, Torschmied et al. as modified by Efland et al. and Stahlin et al. teach all the limitation of claim 10, Stahlin et al. further teach wherein augmenting the one or more of the visual representations of the detected objects in the rendered view with additional visual information includes at least one of: augmenting the one or more of the visual representations of the detected objects in the rendered view with a predicted trajectory of the vehicle; and augmenting the one or more of the visual representations of the detected objects in the rendered view with markings indicating an extent of the vehicle as a boundary line along the predicted trajectory (par 0064, “The system according to the invention in vehicle 21 therefore presents the driver with a realistic, three-dimensional model of vehicle 23 in a two-dimensional projection on the windshield. Since vehicle 23 is not visible to the driver of vehicle 21, because it is concealed by building 28, the information presentation device presents vehicle 23 to the driver as though he could see through building 28. Since the information presentation device presents all the presented objects in a manner that is true to location and size, the driver of vehicle 21 can perceive the presented model of vehicle 23 at that location in the windshield at which he would perceive it if his view were not blocked by building 28. In addition, he perceives the model of vehicle 23 from that angle which would correspond to his actual perception. Since the system identifies, on the basis of the known direction of movement of vehicle 23, that there is no danger of collision between vehicle 21 and vehicle 23, it does not particularly highlight vehicle 23 from the surroundings so as not to unnecessarily direct the attention of the driver to an object which does not present a hazard”, Fig 3, par 0067-0068, “The system uses a camera sensor in the ambient information data capture device to identify a stop sign 34 and concludes therefrom that the driver does not have traffic priority at the junction. This information is additionally verified by means of the digital map material which is present in the system. In order to point this out to the driver, the system visually highlights stop sign 34 by presenting stop sign 34 in flashing yellow on the windshield … The ambient information data evaluation device analyzes the situation and establishes that there is a danger of collision between the system's own vehicle and vehicle 33. For this reason, not only is a model of vehicle 33 presented to the driver as though he can see through building 32, but also the model of vehicle 33 is presented in red and flashing”).
Claim(s) 12-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over European PGPubs EP3859390 to Torschmied et al. in view of WO2019086222 to Meysel et al..
Regarding claim 12, Torschmied et al. teach all the limitation of claim 1, but keep silent for teaching wherein rendering the user's view of the vehicle's environment further includes: determining at least one of the user's position in relation to the vehicle's position in the environment based on a position and an orientation of the user's head and a viewing direction of the user; based on the determining, adjusting a virtual position of the user in the 3D visual world model; and rendering the user's view based on the adjusted virtual position of the user.
In related endeavor, Meysel et al. teach wherein rendering the user's view of the vehicle's environment further includes: determining at least one of the user's position in relation to the vehicle's position in the environment based on a position and an orientation of the user's head and a viewing direction of the user; based on the determining, adjusting a virtual position of the user in the 3D visual world model; and rendering the user's view based on the adjusted virtual position of the user (abstract, “a method and a device for the use of a virtual reality function of a virtual reality device by a user in a vehicle, involving the following steps: determining the movement of the vehicle itself; determining the position and the orientation of the user's head in the vehicle; overlaying optical visual features onto the virtual space created by the virtual reality device, wherein the optical visual features correspond with the movement of the vehicle itself, as well as with the position and the orientation of the user's head in the vehicle”, Figs 1-3 and pages 8-10….adjust virtual content based on the viewing direction of user, head position and orientation of user and vehicle position).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Torschmied et al. to include wherein rendering the user's view of the vehicle's environment further includes: determining at least one of the user's position in relation to the vehicle's position in the environment based on a position and an orientation of the user's head and a viewing direction of the user; based on the determining, adjusting a virtual position of the user in the 3D visual world model; and rendering the user's view based on the adjusted virtual position of the user as taught by Meysel et al. to receive intuitively experienceable visual feedback on the state of motion of the vehicle, such as standstill, start or drive, without having to give up immersion in the virtual environment.
Regarding claim 13, Torschmied et al. as modified by Meysel et al. teach all the limitation of claim 12, and Meysel et al. further teach wherein adjusting the user's virtual position includes at least one of: laterally shifting the user's virtual position in relation to the vehicle's position, longitudinally shifting the user's virtual position in relation to the vehicle's position, and rotating the user's virtual position in relation to the vehicle's position (page 7, “a device for generating visual features and their superimposition in the virtual space generated by the virtual reality device, wherein the visual features are a function of the vehicle's own motion and the position and orientation of the user's head”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US2023/0230334 disclose acquire information on an external object through the sensor, determines the importance of the external object through the information, and controls the display to display at least one augmented reality object or at least one virtual reality object in the first area in response to at least one of determining that the importance is less than a first importance and that the information on the external object has not been acquired.
US2023/0230330 disclose generating a visualization of a three-dimensional (3D) world view from the perspective of a camera of a vehicle wherein the visualization provides a real-world image from the perspective of the camera of the vehicle with one or more graphical overlays of the 3D content.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jin Ge whose telephone number is (571)272-5556. The examiner can normally be reached 8:00 to 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIN . GE
Examiner
Art Unit 2619
/JIN GE/Primary Examiner, Art Unit 2619
