DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 04/15/2026 has been entered.
Response to Arguments
Applicant's arguments, see Remarks pages 7-10, filed 04/15/2026, with respect to the rejections of claims 1, 9, & 17 under 35 U.S.C. 103 have been fully considered but they are not persuasive.
On pages 8-9 of Remarks, Applicant argues:
PNG
media_image1.png
970
668
media_image1.png
Greyscale
Examiner respectfully disagrees.
Paragraph 0051 of Bagon discloses “Regardless of where the REM map data is stored and/or accessed, the AV map data may include a geographic location of known landmarks that are readily identifiable in the navigated environment in which the vehicle 100 travels…in addition to the use of location-based sensors such as GNSS, the database of landmarks provided by the REM map data enables the vehicle 100 to identify the landmarks using the one or more image acquisition devices 104. Once identified, the vehicle 100 may implement other sensors such as LIDAR, accelerometers, speedometers, etc. or images from the image acquisitions device 104, to evaluate the position and location of the vehicle 100 with respect to the identified landmark positions”. Wherein images captured by the vehicle’s cameras are utilized for the localization of the vehicle through comparisons with the stored map data landmarks. In addition, paragraphs 0056 and 0099 of Bagon disclose the determination of the vehicle’s ego-motion based on the image’s captured by the vehicle’s cameras, and their usage in determining the vehicle’s current location, position, and orientation relative to the landmarks in the AV map.
Paragraphs 0123-0125 of Bagon further disclose “The process flow 600 may further include the one or more processors computing (block 604) a vehicle FoV that is associated with the position and orientation of the vehicle at the determined geographic location. This may include, for instance, the completion of the vehicle localization process performed in block 402 as discussed herein with reference to FIG. 4. For example, the vehicle FoV may be determined by correlating an angular range identified with the front of the vehicle using the orientation and position of the vehicle. Moreover, the angular range determined in this way may be further mapped to the particular geographic location of the vehicle referenced to the AV map data…
The process flow 600 may further include the one or more processors identifying (block 608) one or more features and/or objects from the AV map data that are contained within the vehicle FoV. For example, because the vehicle FoV is already calculated, the objects and/or features contained within this vehicle FoV may be identified from the AV map data. For example, the angular range of the vehicle FoV may be correlated to the AV map data based upon the geographic location of the vehicle 100 to determine which objects, features, etc. from the AV map data are contained within the vehicle FoV”. Wherein the vehicle’s images captured and processed for vehicle localization, are subsequently processed for the determination of the vehicle’s FOV and further mapped to the AV map data and objects/features present within.
Thus, Bagon discloses the limitation “receive an image of a scene that captures the TFOV of the imaging system at an image capture time, the image comprising one or more of a point cloud, a camera image, or a video image”.
Paragraphs 0088-0089 of Bagon disclose “the applicable device (e.g. the AR device 301) may receive the position and orientation of the vehicle 100 from the safety system 200, which was computed as part of the vehicle localization calculations. The applicable device may also receive the AV map data or, alternatively, the identified graphical representations as identified via the safety system 200, and their corresponding locations mapped to the vehicle FoV. That is, the corresponding locations of the graphical representations are mapped with respect to the position and orientation of the vehicle 100… The AR device 301 may then, in this example, determine which of the graphical representations to include in the generated AR display frame for the user’s FOV, and their corresponding locations within the user’s FoV. This may include, for instance, translating the location of each graphical representation within the vehicle FoV to an updated location with respect to the user’ s FoV. This process may utilize any suitable techniques, including known techniques, to perform this coordinate transformation based upon the vehicle and occupant localization processes and an analysis of their respective position and orientation in three-dimensional space. Thus, and as shown in FIG. 4, the computational delay represents the time between when the vehicle localization is initially performed (i.e. the position and orientation of the vehicle 100 is calculated) and when the AR display frame containing the graphical representations is generated”. Wherein the AV map data’s graphical representations mapped to the vehicle’s FOV, determined based off the position and orientation of the vehicle, comprise the images captured by the vehicle’s cameras. Wherein paragraph 0093 of Bagon further discloses the usage of coordinate transformations and predicted position change to shift the AV map data’s graphical representations mapped to the vehicle’s FOV.
Thus, Bagon further discloses the limitations “wherein to translate the image, the controller is configured to: determine a first section of the image capturing the TFOV that provides a first IFOV of the scene at the image capture time from the TFOV, and
determine, based on a coordinate transformation and the predicted position change, a second section of the image capturing the TFOV that provides a second IFOV of the scene to display on the display device at the predicted display time, wherein the second IFOV is different from the first IFOV, and wherein the coordinate transformation and the predicted position change determine a portion of the TFOV to allocate to the second IFOV”.
Although, Bagon fails to disclose the claim 1 limitations, “determine, based on a sliding factor and the predicted position change, a second section of the image capturing the TFOV that provides a second IFOV of the scene to display on the display device at the predicted display time, wherein the second IFOV is different from the first IFOV, and wherein the sliding factor and the predicted position change determine a portion of the TFOV to allocate to the second IFOV”.
Thus, as is further disclosed below in the rejection of claim 1 under 35 U.S.C. 103, since Manfred, in paragraphs 0117-0118, discloses a method of generating a motion compensated image by applying sliding factors onto the image coordinates of an input image, Bagon in view of Manfred discloses the claim 1 limitations “receive an image of a scene that captures the TFOV of the imaging system at an image capture time, the image comprising one or more of a point cloud, a camera image, or a video image…
determine a first section of the image capturing the TFOV that provides a first IFOV of the scene at the image capture time from the TFOV, and determine, based on a sliding factor and the predicted position change, a second section of the image capturing the TFOV that provides a second IFOV of the scene to display on the display device at the predicted display time, wherein the second IFOV is different from the first IFOV, and wherein the sliding factor and the predicted position change determine a portion of the TFOV to allocate to the second IFOV”.
Therefore, the rejection of claim 1 under 35 U.S.C. 103 is maintained.
As per claim(s) 9 & 17, arguments made in rejecting claim(s) 1 are analogous.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-6, 9-14, and 17-19, is/are rejected under 35 U.S.C. 103 as being unpatentable over Bagon et al. (WO2023037347A1) hereinafter referenced as Bagon, in view of Manfred et al. (US2021201853A1) hereinafter referenced as Manfred.
Regarding claim 1, Bagon discloses: A vehicle comprising: an imaging system comprising a plurality of imaging sensors, the imaging system configured to capture a total field of view (TFOV) (Bagon: 0104: “the safety system 200 may identify…one or more features and objects that are contained within the vehicle FoV that are detected via one or more vehicle sensors. This may include the safety system 200 performing object detection using sensor data from cameras as well as other sensors that may operate in a non-visible spectrum, such as LIDAR and RADAR sensors, for instance.”; Wherein the captured sensor data pertains to the vehicle’s field of view.) that is larger than an instantaneous field of view (IFOV) (Bagon: 0066: “the safety system 200 may therefore identify one or more features and objects included in the AV map data that are contained within the vehicle FoV based upon the vehicle ego-motion...the safety system 200 may also determine the relative location of the identified features and objects with respect to the geographic location of the vehicle using the AV map data. Finally, once the features and objects and their relative positions with respect to the vehicle 100 are determined, the safety system 200 may generate an AR display frame by filtering the identified features and objects contained within the vehicle FoV to present those that are contained within the occupant’s FoV. ”; Wherein the vehicle’s field of view is filtered in order to create the AR display frame which comprises the occupant’s field of view.) for display in the vehicle;
a display device configured to display the IFOV (Bagon: 0066-0067: “the safety system 200 may generate an AR display frame by filtering the identified features and objects contained within the vehicle FoV to present those that are contained within the occupant’s FoV...The occupant FoV computed in this manner is then used to project the graphical representations onto the vehicle windshield of the vehicle 100.”; Wherein the IFOV is projected onto the vehicle windshield.);
and a controller (Bagon: Figure 1; 0017: “Regardless of the particular implementation of the vehicle 100 and the accompanying safety system 200 as shown in FIG. 1 and FIG. 2, the safety system 200 may include one or more processors 102, one or more image acquisition devices 104 such as, e.g., one or more vehicle cameras or any other suitable sensor configured to perform image acquisition over any suitable range of wavelengths…one or more user interfaces 206 (such as, e.g., a display, a touch screen, a microphone, a loudspeaker, one or more buttons and/or switches, and the like)”) configured to:
receive an image of a scene that captures the TFOV of the imaging system at an image capture time, the image comprising one or more of a point cloud, a camera image, or a video image (Bagon: 0051: “ Regardless of where the REM map data is stored and/or accessed, the AV map data may include a geographic location of known landmarks that are readily identifiable in the navigated environment in which the vehicle 100 travels…in addition to the use of location-based sensors such as GNSS, the database of landmarks provided by the REM map data enables the vehicle 100 to identify the landmarks using the one or more image acquisition devices 104. Once identified, the vehicle 100 may implement other sensors such as LIDAR, accelerometers, speedometers, etc. or images from the image acquisitions device 104, to evaluate the position and location of the vehicle 100 with respect to the identified landmark positions.”
0056: “The one or more processors 102 may process sensory information (such as images, radar signals, depth information from LIDAR or stereo processing of two or more images) of the environment of the vehicle 100 together with position information, such as GPS coordinates, the vehicle's ego-motion, etc., to determine a current location, position, and/or orientation of the vehicle 100 relative to the known landmarks by using information contained in the AV map.”;
0123-0125: “ the vehicle FoV may be determined by correlating an angular range identified with the front of the vehicle using the orientation and position of the vehicle. Moreover, the angular range determined in this way may be further mapped to the particular geographic location of the vehicle referenced to the AV map data… The process flow 600 may further include the one or more processors identifying (block 608) one or more features and/or objects from the AV map data that are contained within the vehicle FoV. For example, because the vehicle FoV is already calculated, the objects and/or features contained within this vehicle FoV may be identified from the AV map data. For example, the angular range of the vehicle FoV may be correlated to the AV map data based upon the geographic location of the vehicle 100 to determine which objects, features, etc. from the AV map data are contained within the vehicle FoV.”;
Wherein the images captured using image acquisition devices 104 are used to determine the vehicle’s FOV, which has AV map data mapped onto it.);
predict a vehicle location at a predicted image display time; translate the image to a predicted image having an estimated view of the scene from the vehicle at the predicted vehicle location based on a predicted render time, a predicted display time, an amount of predicted position change between vehicle position at the image capture time and predicted vehicle position at the predicted image display time (Bagon: 0091: “For other implementations in which the computational delay is known a priori, or not expected to deviate significantly over time, the computation delay may be estimated via other means, e.g. via calibration or other suitable testing process.”; 0093: “because a considerable amount of the computational delay is the result of the identification and rendering of the graphical representations to be presented in the AR view, the updated (i.e. current) position and orientation data for the vehicle 100 and the occupant may be applied (block 410) to further shift (e.g. via coordinate transformation) the relative position and orientation of each of the graphical representations to align with the actual physical locations of features, objects, etc. that are presented in the AR view.”; Wherein the render time and positions are determined/predicted in order to shift the AR view to display the AR frame such that it is aligned with the user’s view, which constitutes the translation of an image to a predicted vehicle location at a predicted time the frame will be aligned with the user, based on a predicted rendering time and predicted positional changes.);
wherein to translate the image, the controller is configured to: determine a first section of the image capturing the TFOV that provides a first IFOV of the scene at the image capture time from the TFOV (Bagon: 0088-0089: “the applicable device (e.g. the AR device 301) may receive the position and orientation of the vehicle 100 from the safety system 200, which was computed as part of the vehicle localization calculations. The applicable device may also receive the AV map data or, alternatively, the identified graphical representations as identified via the safety system 200, and their corresponding locations mapped to the vehicle FoV. That is, the corresponding locations of the graphical representations are mapped with respect to the position and orientation of the vehicle 100…The AR device 301 may then, in this example, determine which of the graphical representations to include in the generated AR display frame for the user’s FOV, and their corresponding locations within the user’s FoV.”; Wherein, based on the data captured during vehicle localization, a user FOV AR display frame is generated),
and determine, based on a coordinate transformation and the predicted position change, a second section of the image capturing the TFOV that provides a second IFOV of the scene to display on the display device at the predicted display time, wherein the second IFOV is different from the first IFOV, and wherein the coordinate transformation and the predicted position change determine a portion of the TFOV to allocate to the second IFOV (Bagon: 0093: “because a considerable amount of the computational delay is the result of the identification and rendering of the graphical representations to be presented in the AR view, the updated (i.e. current) position and orientation data for the vehicle 100 and the occupant may be applied (block 410) to further shift (e.g. via coordinate transformation) the relative position and orientation of each of the graphical representations to align with the actual physical locations of features, objects, etc. that are presented in the AR view.”; Wherein the first generated AR display frame is shifted and transformed based on a time delay and the vehicle’s positional change in order for its contents to be aligned at display time.)
and display the translated image on the display device at the predicted image display time (Bagon: Figures 3&4; 0094: “Once the motion compensation has been applied in this manner, the AR display frame is then presented (block 412) in the AR view as discussed herein.”; Wherein the AR view is a vehicle windshield);
wherein the vehicle is configured for an operator to navigate or avoid obstacles using the translated image on the display device at the predicted image display time (Bagon: 0057-0058: “The aspects described herein further leverage the use of the REM map data to identify road features and objects as noted above, and optionally other types of information as noted herein, to enhance driving safety and convenience by selectively displaying such features and objects to a user (e.g. an occupant of the vehicle such as the driver or, alternatively, another passenger)...the AV map data as discussed herein is described primarily with respect to the use of geographic locations of known landmarks and other types of information that may be identified with those landmarks. However, this is by way of example and not limitation, and the AV map may be identified with any suitable content that can be linked to an accurate geographic location. In this way, the presentation of graphical representations of various features, objects, and other information as further discussed herein, which utilize localization and vehicle and user FoV tracking, may include third party content or other suitable content that may comprise part of the AV map data.”).
Bagon does not disclose expressly: determine, based on a sliding factor and the predicted position change, a second section of the image capturing the TFOV that provides a second IFOV of the scene to display on the display device at the predicted display time, wherein the second IFOV is different from the first IFOV, and wherein the sliding factor and the predicted position change determine a portion of the TFOV to allocate to the second IFOV.
Thus, Bagon does not disclose expressly: the determination of the motion compensated AR display frame based on a sliding factor.
Manfred discloses: a method of generating a motion compensated image by applying sliding factors onto the image coordinates of an input image to generate a motion compensated output image (Manfred: Figure 11; 0113: “FIG. 12 is a diagram illustrating the correspondence between the tracking information of a mobile body or the like and the latency compensation based on the tracking information. Here, six-axis motion information is assumed to be obtained as the tracking information.”; 0117-0118: “A horizontal shift Δx is a shift in the x direction, and causes a horizontal shift of the real object that is viewed by the viewer through the screen. Therefore the method of latency compensation is to perform a horizontal shift operation. A vertical shift Δy is a shift in the direction, and causes a vertical shift of the real object that is viewed by the viewer through the screen. Therefore the method of latency compensation is to perform a vertical shift operation.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique of translating an image based on a shift factor for latency compensation disclosed by Manfred into the graphical representations alignment process disclosed by Bagon by translating the graphical representations within the AR frame based upon a shift factor determined based on initial data and predicted data after the predicted computational delay. The suggestion/motivation for doing so would have been “the latency compensation regarding the rotation or scaling may be performed in addition to the latency compensation regarding the shift displacement. The scaling is an operation to reduce or zoom a virtual object. According to the present embodiment, the rotation error or the scaling error, or both of them, of a virtual object due to latency can be compensated, and therefore the AR display having high trackability can be realized.” (Manfred: 0092; Wherein compensation of shift displacement allows for the AR display to have a higher trackability). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Bagon with Manfred to obtain the invention as specified in claim 1.
Regarding claim 2, Bagon in view of Manfred discloses: The vehicle of claim 1, wherein to translate the image, the controller is configured to translate the image based on a sliding factor that is based on an estimated distance travelled by the vehicle between a vehicle position at the image capture time and a predicted vehicle position at the predicted display time. (Bagon: 0093: “because a considerable amount of the computational delay is the result of the identification and rendering of the graphical representations to be presented in the AR view, the updated (i.e. current) position and orientation data for the vehicle 100 and the occupant may be applied (block 410) to further shift (e.g. via coordinate transformation) the relative position and orientation of each of the graphical representations to align with the actual physical locations of features, objects, etc. that are presented in the AR view.”; Wherein the coordinate transformation includes horizontal and vertical shift factors as taught by Manfred.).
Bagon in view of Manfred does not disclose expressly: the controller is configured to translate the image based on a scaling factor that is based on an estimated distance travelled by the vehicle between a vehicle position at the image capture time and a predicted vehicle position at the predicted display time.
Manfred further discloses: the translation of an image, for compensating latency caused by processing, based on scaling an image based on a change in predicted distance traveled by a vehicle and objects in its environment, based on a difference in tracking data determined during initial capture and right before display time (Manfred: Figure 12; 0113 & 0119: “FIG. 12 is a diagram illustrating the correspondence between the tracking information of a mobile body or the like and the latency compensation based on the tracking information. Here, six-axis motion information is assumed to be obtained as the tracking information… A front-back shift Δz is a shift in the z direction, and causes a reduction or an enlargement of the real object that is viewed by the viewer through the screen. Therefore the method of latency compensation is to perform a reduction or a zooming operation.”; 0137: “Tracking information that is at least one of first tracking information of a mobile body in which the head up display is mounted, second tracking information of a viewer of the head up display, and third tracking information of the real object”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique further taught by Manfred of scaling an image based on a difference of predicted distances traveled into the graphical representation alignment process disclosed by Bagon in view of Manfred by scaling the AR frame based upon the estimated distance traveled during the estimated computational delay. The suggestion/motivation for doing so would have been “According to the present embodiment, the rotation error or the scaling error of the virtual object due to latency, or both of the items can be compensated, and therefore an AR display having higher trackability can be realized.” (Manfred: 0152; Wherein scaling for latency compensation allows for the AR display to have a higher trackability). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Bagon in view of Manfred with the further teaching of Manfred to obtain the invention as specified in claim 2.
Regarding claim 3, Bagon in view of Manfred discloses: The vehicle of claim 2, wherein the FOV of the vehicle, based on a correlated angular range mapped to a geographic location (Bagon: 0123: “the vehicle FoV may be determined by correlating an angular range identified with the front of the vehicle using the orientation and position of the vehicle. Moreover, the angular range determined in this way may be further mapped to the particular geographic location of the vehicle referenced to the AV map data.”), and the vehicle occupant, based on a determined gaze direction (Bagon: 0124: “The occupant FoV may be calculated, for instance, by identifying the orientation and position of the occupant’s head to determine a gaze direction.”), are determined.
Bagon in view of Manfred does not disclose expressly: wherein to translate the image, the controller is configured to translate the image based on an amount of scenery angle change between a scenery angle at the image capture time and a scenery angle at the predicted image display time.
Manfred further discloses: the translation of an image, for compensating latency caused by processing, based on a change in angle predicted by a difference in tracking data determined during initial capture and right before display time (Figure 12; 0114-0116: “A yaw displacement Δα is a rotational displacement in which an axis parallel to the y direction, which is a vertical direction, is the rotation axis…A pitch displacement Δβ is a rotational displacement in which an axis parallel to the x direction, which is a horizontal direction, is the rotation axis…A roll displacement Δγ is a rotational displacement in which an axis parallel to the z direction, which is a front-back direction of the mobile body, is the rotation axis.”; 0126: “When the distance between the pitch rotation center PTC and the screen 34 is denoted as DCF, the pitch displacement Δβ of the mobile body 32 causes the screen 34 to vertically shift by DCF×Δβ”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique disclosed by Manfred of translating an image based on a difference of predicted scenery angles into the graphical representation alignment process disclosed by Bagon in view of Manfred by translating the AR frame based upon determined angular changes before and after the predicted computational delay. The suggestion/motivation for doing so would have been “According to the present embodiment, the rotation error or the scaling error of the virtual object due to latency, or both of the items can be compensated, and therefore an AR display having higher trackability can be realized.” (Manfred: 0152; Wherein compensation of rotational difference allows for the AR display to have a higher trackability). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Bagon in view of Manfred with the further teaching of Manfred to obtain the invention as specified in claim 3.
Regarding claim 4, Bagon in view of Manfred discloses: The vehicle of claim 3, wherein to translate the image, the controller is configured to calculate a scenery angle change factor, wherein the scenery angle change factor is based on an amount of angular change between a scenery angle at the image capture time and a scenery angle at the predicted image display time (Bagon: 0095: “the processing circuitry of the safety system 200 and/or the applicable device, as the case may be, is configured to compensate for changes in the position and orientation of the vehicle and user during the delay period by tracking the ego-motion of the vehicle and the position and orientation of the user’s head.”; Wherein Bagon discloses the image translation based on data initially captured and data after a predicted computational delay. Wherein, Bagon translates an image based on a change in scenery angles as taught by Manfred.)
Regarding claim 5, Bagon in view of Manfred discloses: The vehicle of claim 1, wherein the sliding factor is proportional to the predicted position change (Manfred: 0127: “The distance between the head of the viewer 52 and the real object 12 is denoted as DPT, and the distance between the screen 34 and the real object 12 is denoted as DFT. If it is assumed that the screen 34 and the real object 12 do not move, when the viewer moves up by +Δyp, for example, the position of the real object 12 on the screen 34 viewed from the viewer 52 relatively moves up by +(DFT/DPT)×Δyp relative to the screen 34 . If it is assumed that the distance to the real object 12 is sufficiently large, DFT/DPT can be approximated to 1, and therefore the vertical shift amount of the real object 12 is Δyp. Therefore, in this case, the latency compensation parameter is m 23 =Δyp.”; Wherein the predicted position change is determined as taught by Bagon.).
Regarding claim 6, Bagon in view of Manfred discloses: The vehicle of claim 1, wherein the imaging system has an image capture rate (Bagon: 0099: “the ego-motion may be computed via the safety system 200 in accordance with an image frame rate, e.g. when the vehicle cameras are used for this purpose. Thus, the frequency of this frame rate may be one example of the data acquisition parameters that may be adjusted in this manner. That is, the frequency of this frame rate may be further increased to reduce the delay between when the ego-motion is computed, thus further reducing the computational delay.”), and the controller is configured to translate an image to an estimated image at a translation rate that is equal to or lower than the image capture rate (Bagon: 0093: “the vehicle ego-motion may be computed in a continuous manner, and this data may be readily available. The ego-motion data may thus be generated and accessed with significantly less delay compared to the computational delay.”; Wherein the continuous computation of ego-motion, allowing for less delay compared to the rendering of the AR frame, constitutes the image translation rate being lower than the image capture rate).
As per claim(s) 9, arguments made in rejecting claim(s) 1 are analogous.
As per claim(s) 10, arguments made in rejecting claim(s) 2 are analogous.
As per claim(s) 11, arguments made in rejecting claim(s) 3 are analogous.
As per claim(s) 12, arguments made in rejecting claim(s) 4 are analogous.
Regarding claim 13, Bagon in view of Manfred discloses: The method of claim 9, wherein the sliding factor is determined using a linear function (Manfred: 0127: “ The distance between the head of the viewer 52 and the real object 12 is denoted as DPT, and the distance between the screen 34 and the real object 12 is denoted as DFT. If it is assumed that the screen 34 and the real object 12 do not move, when the viewer moves up by +Δyp, for example, the position of the real object 12 on the screen 34 viewed from the viewer 52 relatively moves up by +(DFT/DPT)×Δyp relative to the screen 34 . If it is assumed that the distance to the real object 12 is sufficiently large, DFT/DPT can be approximated to 1, and therefore the vertical shift amount of the real object 12 is Δyp. Therefore, in this case, the latency compensation parameter is m 23 =Δyp.”; Wherein the sliding factor for the AR display frame is determined based on a linear function including the horizontal/vertical object shift.).
As per claim(s) 14, arguments made in rejecting claim(s) 6 are analogous.
As per claim(s) 17, arguments made in rejecting claim(s) 1 are analogous. In addition, 0071 of Bagon discloses “The memory 303 is configured to store data and/or instructions such that, when the instructions are executed by the processors 302, cause the AR device 301 to perform the various functions as described herein…the memory 303 may be implemented as a non- transitory computer readable medium storing one or more executable instructions such as, for example, logic, algorithms, code, etc.”.
As per claim(s) 18, arguments made in rejecting claim(s) 2 are analogous.
As per claim(s) 19, arguments made in rejecting claim(s) 3 are analogous.
Claim(s) 7-8, and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Bagon in view of Manfred and further in view of Li et al. (Flow-Grounded Spatial-Temporal Video Prediction from Still Images) hereinafter referenced as Li.
Regarding claim 7, Bagon in view of Manfred discloses: The vehicle of claim 1, wherein the imaging system has an adjustable image capture rate (Bagon: 0097: “These delay parameters may include any suitable parameters that are used as part of the delay compensation techniques as discussed herein, such as e.g. a predetermined threshold value for the computation delay, image frame rate frequency, the sampling rate with respect to the sensor data acquired via the safety system 200, the AR device 301, etc.”).
Bagon in view of Manfred does not disclose expressly: translate an image to an estimated image at a translation rate that is higher than the image capture rate.
Li discloses: the translation of a single captured image into a series of predicted future images. (Li: Figure 6; 5 Conclusion: “we propose a video prediction algorithm that synthesizes a set of likely future frames in multiple time steps from one single still image.”)
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique disclosed by Li of translating an image into a series of future images into Bagon in view of Manfred by translating the rendered image into a series of computationally delayed compensated AR frames. The suggestion/motivation for doing so would have been in order to reduce the computational processing caused by rendering, and reduce the average computational delay. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Bagon in view of Manfred with Li to obtain the invention as specified in claim 7.
Regarding claim 8, Bagon in view of Manfred discloses: The vehicle of claim 1, wherein the imaging system has an adjustable image capture rate (Bagon: 0097: “These delay parameters may include any suitable parameters that are used as part of the delay compensation techniques as discussed herein, such as e.g. a predetermined threshold value for the computation delay, image frame rate frequency, the sampling rate with respect to the sensor data acquired via the safety system 200, the AR device 301, etc.”).
Bagon in view of Manfred does not disclose expressly: translate an image to an estimated image at a translation rate that is higher than the image capture rate, and the image that is translated is a previously estimated image.
Li discloses: the translation of a single captured image into a series of predicted future images, wherein for the generation of a predicted frame, the algorithm uses a previous frame, including a predicted one (Li: Figure 4: “Starting from the first frame and first flow, we iteratively run warping or the proposed Flow2rgb model based on the previous result and next flow to obtain the sequence.”; 3 Proposed Algorithm: “We formulate the video prediction as two phases: flow prediction and flow-to-frame generation. The flow prediction phase, triggered by a noise, directly predicts a set of consecutive flow maps conditioned on the observed first frame. Then the flow-to-frame phase iteratively synthesizes future frames with the previous frame and the corresponding predicted flow map, starting from the first given frame and first predicted flow map.”)
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique disclosed by Li of iteratively translating a captured image into a series of future images using a previous frame into Bagon in view of Manfred by translating the rendered image into a series of computationally delayed compensated AR frames. The suggestion/motivation for doing so would have been in order to reduce the computational processing caused by rendering, and reduce the average computational delay. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Bagon in view of Manfred with Li to obtain the invention as specified in claim 8.
As per claim(s) 15, arguments made in rejecting claim(s) 7 are analogous.
As per claim(s) 16, arguments made in rejecting claim(s) 8 are analogous.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bagon in view of Manfred, and further in view of Inukai et al. (US2022319366A1) hereinafter referenced as Inukai.
Regarding claim 20, Bagon in view of Manfred discloses: The non-transitory computer readable media of claim 17, wherein the sliding factor is determined using a linear function (Manfred: 0127: “ The distance between the head of the viewer 52 and the real object 12 is denoted as DPT, and the distance between the screen 34 and the real object 12 is denoted as DFT. If it is assumed that the screen 34 and the real object 12 do not move, when the viewer moves up by +Δyp, for example, the position of the real object 12 on the screen 34 viewed from the viewer 52 relatively moves up by +(DFT/DPT)×Δyp relative to the screen 34 . If it is assumed that the distance to the real object 12 is sufficiently large, DFT/DPT can be approximated to 1, and therefore the vertical shift amount of the real object 12 is Δyp. Therefore, in this case, the latency compensation parameter is m 23 =Δyp.”; Wherein the sliding factor for the AR display frame is determined based on a linear function including the horizontal/vertical object shift.).
Bagon in view of Manfred does not disclose expressly: wherein the sliding factor is determined using a non-linear function.
Inukai discloses: wherein the sliding factor is determined using a non-linear function (Inukai: Figure 5:
PNG
media_image2.png
203
259
media_image2.png
Greyscale
;
0105-0107: “A specific example of the correction processing by corrector 15 will be described with reference to FIGS. 4 and 5. FIG. 5 is a graph for describing the correction processing by corrector 15 in rendering system 100 in embodiment 1… In FIG. 5, the ordinate indicates the position of vehicle 30 along the X-axis, and the abscissa indicates time. As shown in FIG. 5, for example, Xc 2 is the position of vehicle 30 along the X-axis at time t 0 +2 Ts at which detector 400 detects object 6, whereas X′c 2 is the position of vehicle 30 along the X-axis at time t 0 +2 Ts+Td at which renderer 13 renders display image 8. The displacement amount ΔXs 2 in the position of object 6 along the X-axis relative to vehicle 30 over the delay period Td is calculated as the difference between X′c 2 and Xc 2. Corrector 15 calculates the coordinate X′ 2 of object 6 along the X-axis at time t 0 +2 Ts+Td by adding the calculated displacement amount ΔXs 2 to the coordinate X 2 of object 6 along the X-axis at time t 0 +2 Ts obtained by first obtainer 11 at time t 0 +2 Ts+Td.”;
Wherein the displacement along the x-axis is calculated using a non-linear equation, as shown in Figure 5.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the non-linear displacement calculations taught by Inukai for the horizontal and vertical shift calculations disclosed by Bagon in view of Manfred. The suggestion/motivation for doing so would have been “As described above, corrector 15 can correct the displacement of the display position of content 7 in display image 8 by calculating the displacement amount in the position of vehicle 30 over the delay period Td…The inventor of the present application has found out that estimating and updating the delay period Td by estimator 14 improves the accuracy of correcting the displacement of the display position of content 7 in display image 8 by corrector 15.” (Inukai: 0110-0111). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Bagon in view of Manfred with Inukai to obtain the invention as specified in claim 20.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights