DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claims 7-8, objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claims 16,18-20 are allowed.
Response to Arguments
Applicant’s arguments, with respect to the rejection(s) of claim(s) 1 and 9 under 35 USC § 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Yu(EP-4369177-A1).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1,9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williams(US-20150029218-A1) in view of Smith(US-20190387168-A1) and Yu(EP-4369177-A1)
Regarding claim 1, Williams discloses A heads-up display device comprising: a display(para.[0023] “ The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD)”); a camera(para.[0031] “One embodiment of mobile device 19 includes a network interface 145, processor 146, memory 147, camera 148, sensors 149, and display 150, all in communication with each other.”); and a display controller comprising a processor and memory storing non-transitory instructions executable by the processor(para.[0030] “Processor 156 allows server 15 to execute computer readable instructions stored in memory 157 in order to perform processes discussed herein. ”) to exhibit one or more virtual objects via the display(para.[0047] “FIG. 3A depicts one embodiment of a system for generating and displaying images associated with a virtual object (or more than one virtual object) at a frame rate that is greater than a rendering frame rate for a core rendering pipeline.” Explicit disclosure of the reference exhibiting virtual objets through the display.), the one or more virtual objects generated via the processor(para.[0044] “In one embodiment, eye c 216 may comprise a see-through display, whereby images generated by processing unit 236 may be projected and/or displayed on the see-through display.” Disclosure by the reference of the rendered objects being generated by the processor.), and where positions of the one or more virtual objects are adjusted in each of the displayed images each time the displayed images are updated(para.[0024] “ The updated image may comprise an image rotation, translation, resizing (e.g., stretching or shrinking), shifting, or tilting of at least a portion of the pre-rendered image in order to correct for differences between the predicted pose and the updated pose (e.g., to compensate for an incorrect pose prediction when generating the pre-rendered image).” The reference clearly teaches adjusting the position of the virtual object in updated displayed images.).
However, Williams alone does not fully disclose the one or more virtual objects exhibited in displayed images via the display at a rate faster than new images are captured via the camera. The combination of Williams and Smith discloses the one or more virtual objects exhibited in displayed images via the display at a rate faster than new images are captured via the camera(Williams teaches displaying virtual-object imagery at an update rate fater than the based image-generation rate. In para.[0039] “In some embodiments, an HMD, such as mobile device 19, may display images of virtual objects within an augmented reality (AR) environment at a frame rate that is greater than a rendering frame rate for the core rendering pipeline or rendering GPU.” Thus, Williams teaches the faster display side of the claimed limitation. Smith teaches the camera frame rate side of the limitation in para.[0273] “Likewise, an assessment of relative motion between the head mounted display and one or more features in the environment can be combined with other structures, processes or process steps or features, for example, to alter the frame rate of the camera from a first frame rate to a second frame rate and/or adjust the amount of processing on the frames obtained by the camera that are processed;” based on motion or comparison of virtual image content locations with the camera viewing zone. Smith further teaches that the camera frame rate may be increased or decreased depending on circumstances, including that the second frame rate may be greater than the first frame rate, and in some embodiments that one frame rate of about 5-20 fps. Thus, Smith teaches that camera image acquisition occurs at a controllable, potentially reduced frame rate in an AR display system that also displays virtual image content.) It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Smith into the teachings of Williams in order to have a better balance with image acquisition and processing while maintaining stable and responsive content. However, the combination still does not disclose to reduce a perceptible stepwise motion of the one or more virtual objects
Yu does disclose to reduce a perceptible stepwise motion of the one or more virtual objects(para[0061] “ The HUD image stabilization module is configured to smooth a position output of the tracking and fusion module through sliding window filtering, to reduce a jitter degree of the tracking and fusion module, and improve stability of a 3D detection box.” The claimed “perceptible stepwise motion” and Yu’s taught ‘jitter’ are addressing the same general visual problem. An AR overlay tied to a real-world object changes position in a visibly discontinuous way. Yu smooths the displayed position of the AR detection so that the overlay remains more stable and visually continuous.) It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Smith into the teachings of Williams in order to have a better balance with image acquisition and processing while maintaining stable and responsive content.
Regarding claim 9, Williams discloses A method for operating a heads-up display device, the method comprising: capturing a first image via a camera (para. [0038] “In some embodiments, an HMD, such as mobile device 19, may use images of an environment captured from an outward facing camera in order to determine a six degree of freedom (6DOF) pose corresponding with the images relative to a 3D map of the environment.” Explicit disclosure of using images captured from a camera.); identifying an object in the first image (para.[0037] “In some embodiments, a mobile device, such as mobile device 19, may be in communication with a server in the cloud, such as server 15, and may provide to the server location information (e.g., the location of the mobile device via GPS coordinates) and/or image information (e.g., information regarding objects detected within a field of view of the mobile device) associated with the mobile device. In response, the server may transmit to the mobile device one or more virtual objects based upon the location information and/or image information provided to the server.”); However Williams alone does not fully disclose and generating a display image via a heads-up display, the display image including a virtual object that is placed in the display image based on a motion of the object.
The combination of Williams and Smith does disclose and generating a display image via a heads-up display, the display image including a virtual object that is placed in the display image based on a motion of the object(Smith discloses in para.[0615] “assess relative motion between said head mounted display and one or more features in said environment, said assessment of relative motion comprising determining whether the head mounted display has moved, is moving or is expected to move with respect to one or more features in the environment and/or determining whether one or more features in the environment have moved, are moving or are expected to move relative to the head mounted display; based on said assessment of relative motion between said head mounted display and one or more features in said environment, alter the frame rate of the camera from a first frame rate to a second frame rate and/or adjust the amount of processing on the frames obtained by the camera that are processed.” Smith discloses a display showing virtual image content, assessing motion and using motion related information to determine where render able virtual image content would appear to adjust a rendering location of the virtual objects. This is analogous to generating a display image with a virtual object placed based on motion.) It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Smith into the teachings of Williams in order to have a better balance with image acquisition and processing while maintaining stable and responsive content. However, the combination does not disclose wherein the virtual object is exhibited in displayed images via the display at a rate faster than new images captured via the camera to reduce a perceptible stepwise motion of the virtual object.
Yu does disclose wherein the virtual object is exhibited in displayed images via the display at a rate faster than new images captured via the camera to reduce a perceptible stepwise motion of the virtual object(para.[0149] “For example, in a process of detecting a target object on a road, after an update position of the target object is not obtained within threshold duration (that is, after it is determined that the detection is blocked), a detection system directly uses, as an output, a predicted value that is of a next-moment position of the target object and that is obtained by using a prediction model. A tracking and output rate is greatly improved.” This passage shows the same concept behind the claimed limitation. The HUD need not wait for a newly completed camera/detection result before updating the displayed AR information. Instead, Yu teaches predicted positions to maintain or accelerate the output and reduce visual jitter/latency. This is analogous to displaying virtual objects more frequently than new camera-derived positions become available. This can be combined with Williams teachings where they state at para.[0039] “In some embodiments, an HMD, such as mobile device 19, may display images of virtual objects within an augmented reality (AR) environment at a frame rate that is greater than a rendering frame rate for the core rendering pipeline or rendering GPU”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Yu into the teachings of Williams in order to have a better balance with image acquisition and processing while maintaining stable and responsive content
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williams as modified by Smith and Yu as applied to claim 1 above, and further in view of Zhang(US-10748340-B1).
Regarding claim 2, the combination of Williams, Smith and Yu discloses all the elements of claim 1 as discussed above. However, the combination does not disclose wherein the new images are images are captured at a fixed rate.
Zhang does disclose wherein the new images are images are captured at a fixed rate (col. 4 lines 35-40, “In the example of FIG. 5, the frame rate of display 26 is 120 Hz. In the example of FIG. 6, the frame rate of display 26 is 96 Hz, so frame duration TP is lengthened relative to frame duration TP of FIG. 5 and output light pulse duration TW is lengthened relative to output light pulse duration TW of FIG. 5.” Zhang provides explicit disclosure of fixed frame rates being used.) It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Zhang into the combination of teachings of Williams, Smith and Yu in order to have cleaner synchronization with display updates, and provide lower jitter and stability in the display.
Claim(s) 3,4,10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williams as modified by Smith and Yu as applied to claim 1 above, and further in view of Tsuji (US-6327536-B1).
Regarding claim 3, the combination of Williams, Smith and Yu disclose all the elements of claim 1 as discussed above. However, the combination does not fully disclose wherein a position of the one or more virtual objects is based on a position of an identified object in an image captured via the camera, and wherein the one or more virtual objects are configured to make a real-world object more noticeable.
Tsuji does disclose wherein a position of the one or more virtual objects is based on a position of an identified object in an image captured via the camera (col. 2 lines 50-51 “Preferably, the imaging means comprises two infrared cameras capable of detecting infrared rays.” Col. 5 lines 30-35 “In the right image and the left image, an identical object is displayed as images at respective locations horizontally displaced from each other, so that it is possible to calculate a distance from the vehicle 10 to the object based on the displacement (parallax).” Here Tsuji teaches the system using camera images to detect and locate objects.), and wherein the one or more virtual objects are configured to make a real-world object more noticeable(col. 13 lines 38-47, “At step S45, a voice alarm is generated by the speaker 3, and as shown in FIG. 21B, an image obtained e.g. by the camera 1R is displayed on the screen 4a of the HUD 4 such that a closing object is emphatically displayed (for instance, enclosed in a frame for emphasis). FIG. 21A shows a state where the screen 4a is not displayed, while FIG. 21B shows a state where the screen 4a is displayed. This enables the driver to positively recognize an object having a high possibility of collision against the vehicle 10.” The “frame for emphasis” taught by Tsuji is analogous to aiding in making an object more “noticeable.”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Tsuji into the combination of teachings of Williams, Smith and Yu in order to have more accurate object registration and recognition.
Regarding claim 4, the combination of Williams, Smith and Yu discloses all the elements of claim 3 as discussed above. However, the combination does not disclose where the one or more virtual objects appear to surround the real-world object.
Tsuji does disclose where the one or more virtual objects appear to surround the real-world object (col. 13 lines 38-43 “At step S45, a voice alarm is generated by the speaker 3, and as shown in FIG. 21B, an image obtained e.g. by the camera 1R is displayed on the screen 4a of the HUD 4 such that a closing object is emphatically displayed (for instance, enclosed in a frame for emphasis).” The real-world object enclosed in a frame is a display element that appears to surround the object aligning with the claim element.) It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Tsuji into the combination of teachings of Williams, Smith and Yu in order to have more accurate object registration and recognition.
Regarding claim 10, the combination of Williams, Smith and Yu disclose all the elements of claim 9 as discussed above. However, the combination does not fully disclose where the motion of the object is based on a trajectory of the object, and further comprising: capturing a second image via the camera and identifying the object in the second image(col. 1 lines 52-67 “The vehicle environment monitoring system is characterized by comprising:relative position-detecting means for detecting a relative position of the object to the automotive vehicle from the image obtained by the imaging means to obtain position data; movement vector-calculating means for calculating positions of the object in a real space based on a plurality of time series items of the position data detected on the object by the relative position-detecting means, and calculating a movement vector of the object based on the positions in the real space; and determining means for determining whether or not the object has a high possibility of collision against the automotive vehicle based on the movement vector.” Further, Tsuji discloses at col. 4 lines 41-51 “Referring first to FIG. 1, there is shown the arrangement of a vehicle environment monitoring system, according to the embodiment of the invention, which has two right and left infrared cameras 1R, 1L capable of detecting far-infrared rays, a yaw rate sensor 5 for detecting yaw rate of the vehicle, a vehicle speed sensor 6 for detecting traveling speed (vehicle speed) VCAR of the vehicle, a brake sensor 7 for detecting an operation amount of a brake, not shown, an image-processing unit 2 for detecting an object, such as an animal or the like, ahead of the vehicle based on image data obtained by the above cameras 1R, 1L...” Tsuji expressly teaches that the object’s motion is determined from a movement vector, and that the movement vector is calculated from a plurality of time-series items of position data for the object.) It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Tsuji into the combination of teachings of Williams, Smith and Yu in order to improve the motion estimation accuracy.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williams as modified by Smith, Yu, and Tsuji as applied to claim 3 above, and further in view of Seder (US-20100253593-A1).
The combination of Williams, Smith and Tsuji disclose all the elements of claim 3 as discussed above. However, they do not disclose where the one or more virtual objects are placed to appear proximate to the real-world object via the heads-up display device.
Seder does disclose where the one or more virtual objects are placed to appear proximate to the real-world object via the heads-up display device(para.[0181] “In order to bring the operator's attention from the area of distraction sign 254 to the critical information of vehicle 208, a textual alert and accompanying arrow are displayed proximately to the operator's gaze location. In this way, the operator's attention can be drawn to the critical information as quickly as possible.” Seder discusses placing additional display elements nearby to direct attention to that object, aligning with the claim element). It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Seder into the combination of teachings of Williams, Smith, Yu and Tsuji in order to a improve visibilities of certain objects in the display.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williams as modified by Smith and Yu as applied to claim 1 above, and further in view of Emura et al. (US-2015/0103174 A1).
Regarding claim 6, the combination of Williams, Smith and Yu disclose all the elements of claim 1 as discussed above. However, the combination does not disclose: additional instructions to position the one or more virtual objects generated via the processor via the heads-up display device based on a trajectory of a first identified object captured in a first image via the camera and the first identified object captured in a second image via the camera.
Emura does disclose additional instructions to position the one or more virtual objects generated via the processor via the heads-up display device based on a trajectory of a first identified object captured in a first image via the camera and the first identified object captured in a second image via the camera (Emura in figure 16 and para. [0145] “Referring to FIG. 16, a frame 1201 indicates the position and the size of a pedestrian (an example of the first object) who is walking on the left side in the traveling direction of the vehicle and a frame 1202 indicates the position and the size of a pedestrian (an example of the second object) who is walking on the right side in the traveling direction of the vehicle (refer to rectangles in FIG. 16). The pedestrians are examples of the moving objects”.
Emura in figures 4A-D shows that the position of the virtual object (virtual symbol around the one or more pedestrian) is based on trajectory over time from a plurality of images (e.g. first and second images) via the camera).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Emura into the combination of teachings of Williams, Smith and Yu in order to better illustrate the warning or alter to the user about nearby pedestrians walking near the vehicle.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williams as modified by Smith, Yu and Tsuji as applied to claim 10 above, and further in view of Lu et al. (US-2021/0245773 A1).
Regarding claim 11, the combination of Williams, Smith, Yu and Tsuji disclose all the elements of claim 10 as discussed above. However, the combination does not disclose: estimating the motion of the object based on the first image and the second image.
Lu does disclose estimating the motion of the object based on the first image and the second image. (para. [0014] “… The collaborative computing process uses the image data to: (1) estimate the walking path of the pedestrian based on images captured by the pedestrian device and/or images captured by a subset of vehicles included in a group of vehicles”). It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Lu into the combination of teachings of Williams, Smith and Yu in order to better track pedestrians crossing the street and use this information to alert the driver of the vehicle.
Claim(s) 12-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williams as modified by Smith, Yu, Tsuji, and Lu as applied to claim 11 above, and further in view of Emura et al. (US-2015/0103174 A1).
Regarding claims 12-14, the combination of Williams, Smith, Yu Tsuji, and Lu disclose all the elements of claim 11 as discussed above. However, the combination does not disclose the features from claims 12-14.
Emura does disclose adjusting a position of the virtual object in the display image based on the motion of the object (Emura in figure 16 and para. [0145] “Referring to FIG. 16, a frame 1201 indicates the position and the size of a pedestrian (an example of the first object) who is walking on the left side in the traveling direction of the vehicle and a frame 1202 indicates the position and the size of a pedestrian (an example of the second object) who is walking on the right side in the traveling direction of the vehicle (refer to rectangles in FIG. 16). The pedestrians are examples of the moving objects”.
Emura in figures 4A-D shows that the position of the virtual object (virtual symbol around the one or more pedestrian) is adjusted over time and based on the motion of the object (the pedestrians walking)) and further comprising adjusting a size of the virtual object based on a speed of a vehicle, (Emura in claim 8 “determining a size of the virtual graphic of the certain shape on the basis of at least one of a size of the moving body, a distance from the moving body to the vehicle, a speed of the vehicle, and a relative speed between any one of the moving bodies and the vehicle”) and where the virtual object is configured to enhance visual identification of the object (Emura in figure 16 and para. [0145] where the virtual graphics enhance visual identification of the pedestrian cross the street (the object)).
It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Emura into the combination of teachings of Williams, Smith, Yu and Lu in order to better illustrate the warning or alter to the user about nearby pedestrians walking near the vehicle.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williams as modified by Smith and Yu as applied to claim 9 above, and further in view of Seder (US-20100253593-A1).
Regarding claim 15, the combination of Williams, Smith and Yu disclose all the elements of claim 9 as discussed above. However, the combination does not disclose where placing the virtual object in the display image based on the motion of the object includes placing the virtual object in the display image based on a location the object is expected to be, the location the object is expected to be being based on a position of the object in the first image.
Seder does disclose where placing the virtual object in the display image based on the motion of the object includes placing the virtual object in the display image based on a location the object is expected to be, the location the object is expected to be being based on a position of the object in the first image (para. [0118] “FIG. 17 schematically illustrates an exemplary image fusion module, in accordance with the present disclosure. The fusion module of FIG. 17 monitors as inputs range sensor data comprising object tracks and camera data. The object track information is used to extract an image patch or a defined area of interest in the visual data corresponding to object track information. Next, areas in the image patch are analyzed and features or patterns in the data indicative of an object in the patch are extracted. The extracted features are then classified according to any number of classifiers. An exemplary classification can include classification as a fast moving object, such a vehicle in motion, a slow moving object, such as a pedestrian, and a stationary object, such as a street sign. Data including the classification is then analyzed according to data association in order to form a vision fused based track. These tracks and associated data regarding the patch are then stored for iterative comparison to new data and for prediction of relative motion to the vehicle suggesting a likely or imminent collision event. Additionally, a region or regions of interest, reflecting previously selected image patches, can be forwarded to the module performing image patch extraction, in order to provide continuity in the analysis of iterative vision data. In this way, range data or range track information is overlaid onto the image plane to improve collision event prediction or likelihood analysis.” Seder expressly teaches using object-track information and camera data to form a fused track, storing that track for iterative comparison to new data and for prediction of relative motion, and then overlaying the resulting track information onto the image plane.) It would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to incorporate the teachings of Seder into the combination of teachings of Williams, Smith and Yu in order to allow for placing display indicators to improve warning usefulness.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRIS ALEJANDRO PUNTIER whose telephone number is (703)756-1893. The examiner can normally be reached M-F 7:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHRIS ALEJANDRO PUNTIER/ Examiner, Art Unit 2616
/DANIEL F HAJNIK/ Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights