DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1, 3-17 and 19-21 are pending in the instant application. Claims 1, 4, 16 and 20 are amended and claims 2 and 18 are canceled.
Response to Arguments
Applicant's arguments filed 01/30/2026 have been considered but are moot in view of the new ground(s) of rejection.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/17/2026 and 03/17/2026 are being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 4, 16-17, and 19-21 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 4, recites “the head-mounted display unit is configured to warp each rendered frame a threshold number of times based on respective orientation information, the threshold number being two or more”. Applicant’s disclosure is fails to describe the claimed the threshold number being two or more. Therefore, the claim fails to comply with the written description requirement.
Claim 16, recites “warping, by the head-mounted display unit at a second frame rate higher than the first frame rate, each rendered frame a threshold number of times based on orientation information associated with the system, the threshold number being two or more”. Applicant’s disclosure is fails to describe the claimed the threshold number being two or more. Therefore, the claim fails to comply with the written description requirement.
Claim 20, recites “warping, by the head-mounted display unit at a second frame rate higher than the first frame rate, each rendered frame a threshold number of times based on orientation information associated with the head-mounted display system, the threshold number being two or more”. Applicant’s disclosure is fails to describe the claimed the threshold number being two or more. Therefore, the claim fails to comply with the written description requirement.
Claims 17 and 21 depend directly from a claim rejected under 35 U.S.C. 112(a). Therefore claims 17 and 21 are also rejected under 35 U.S.C. 112(a) for failing to comply with the written description requirement.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Quach et al. (US 9858637 B1, hereinafter referenced as Quach) in view of Chen et al. (US 20190149809 A1, hereinafter referenced as Chen), further in view of Hunt et al. (US 10785471 B1, hereinafter referenced as Hunt).
Regarding Claim 1, Quach teaches a head-mounted display system (see Figs. 1-2) comprising:
head mounted display unit (see Fig. 2, col. 3 lines 8-9, col. 3 lines 62-67, col. 4 lines 1-8, head-mount virtual reality display); and
a local processing system communicatively coupled to the head-mounted display unit, the local processing system being configured to generate rendered frames at a first frame rate to be output to the head-mounted display via the data link to be warped at the head-mounted display unit to generate warped rendered frames (see abstract, col. 1 lines 39-53, col. 2 lines 39-59, col. 3 lines 62-67, col. 4 lines 1-8, col.4 lines 38-55, col. 5 lines 28-36, Figs. 1-2. The computer system includes a virtual reality application executed by a processing device. A graphic processing unit (GPU) displays rendered images associated with the virtual reality application. The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth);
wherein the head-mounted display unit (see Fig. 2, col. 3 lines 8-9, col. 3 lines 62-67, col. 4 lines 1-8, head-mount virtual reality display) configured to output the warped rendered frames as virtual content to a user of the head-mounted display system at (abstract, col. 2 lines 46-59, col. 3 lines 62-67, col. 4 lines 1-8, col.4 lines 38-55, col. 5 lines 28-36. It should be appreciated that the system 100 may be implemented in various types of virtual reality systems (e.g., headsets, goggles, eyewear, external VR display(s), projection systems, etc.). As illustrated in the embodiment of FIG. 2, the system 100 may be incorporated in an integrated headmount display (HMD) 200 that is worn by a user. A graphic processing unit (GPU) displays rendered images associated with the virtual reality application. The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth) a second frame rate higher than the first frame rate (col. 1 lines 39-53. Time warp involves warping the rendered image before sending it to the display to correct for the user's movement that occurred after the rendering. Time warp may reduce latency and increase or maintain frame rate (i.e., the number of frames display per second (fps)). This process takes an already rendered image, modifies it with the predicted positional information based on the collected positional information obtained from sensors (e.g., sensor(s) housed in a HMD), and then displays the modified image on the VR display. Note: By increasing the frame rate, a second frame rate would be higher than a first frame rate), wherein the head-mounted display unit comprises:
an orientation sensor, the orientation sensor configured to detect orientation information associated with an orientation of the head-mounted display unit (see col. 2 lines 46-60, col. 4 lines 9-38. As illustrated in FIG. 2, the sensor(s) enable the system 100 to track the rotational and translational movement of a user (e.g., the user's head, eyes, and/or other body parts) within three-dimensional space using six degrees of freedom (6DOF). As known in the art, the term 6DOF refers to the freedom of movement of the user in three-dimensional space. The sensors track motion as forward/backward (surge), up/down (heave), left/right (sway) translation in three perpendicular axes, combined with changes in orientation through rotation about three perpendicular axes (i.e., pitch, yaw, and roll). One or more accelerometer sensors 126 may sense changes in the magnitude and/or direction of the proper acceleration of the user's head and/or other body parts. One or more gyroscope sensors 128 may be used to track angular motion of the user's head and/or other body parts);
a display, the display configured to output light to present the virtual content to the user (abstract, col. 2 lines 33-45, col. 3 lines 62-67, col. 4 lines 1-8, col. 5 lines 36-50. As known in the art, the GPU command buffer 314 provides commands to the GPU 108 for displaying the rendered images on the stereo display 302. Inherently the images include visible light); and
one or more head-mounted display unit processors, the one or more head-mounted display unit processors (see Fig. 1, Fig. 3, CPU 106, GPU 108), col. 4 lines 38-45) configured to:
receive the rendered frames generated by the local processing system at (col. 2 lines 49-52, col. 5 lines 30-57. A graphic processing unit (GPU) displays rendered images associated with the virtual reality application) at a first frame rate (col. 1 lines 39-53. Time warp involves warping the rendered image before sending it to the display to correct for the user's movement that occurred after the rendering. Time warp may reduce latency and increase or maintain frame rate (i.e., the number of frames display per second (fps)). This process takes an already rendered image, modifies it with the predicted positional information based on the collected positional information obtained from sensors (e.g., sensor(s) housed in a HMD), and then displays the modified image on the VR display. Note: the first frame rate corresponds to the frame rate before is increased at the time warp, which is lower than the frame rate after time warp);
obtain orientation information associated with the orientation of the head- mounted display unit (see col. 2 lines 46-60, col. 4 lines 9-38, col. 6 lines 28-43. As illustrated in FIG. 2, the sensor(s) enable the system 100 to track the rotational and translational movement of a user (e.g., the user's head, eyes, and/or other body parts) within three-dimensional space using six degrees of freedom (6DOF). As known in the art, the term 6DOF refers to the freedom of movement of the user in three-dimensional space. The sensors track motion as forward/backward (surge), up/down (heave), left/right (sway) translation in three perpendicular axes, combined with changes in orientation through rotation about three perpendicular axes (i.e., pitch, yaw, and roll). One or more accelerometer sensors 126 may sense changes in the magnitude and/or direction of the proper acceleration of the user's head and/or other body parts. One or more gyroscope sensors 128 may be used to track angular motion of the user's head and/or other body parts);
warp the rendered frames generated by the local processing system (col. 2 lines 46-60, col. 5 lines 28-50, col. 6 lines 7-29. The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth)
output, via the display, the warp rendered frames (see col. 2, lines 33-60, col.5, lines 37-50, claim 1. A warp engine implemented by the GPU, the GPU configured to update a rendered image before sending to a virtual reality display based on one or more of the updated position, the speed, and the acceleration. As known in the art, the GPU command buffer 314 provides commands to the GPU 108 for displaying the rendered images on the stereo display 302. If the GPU 108 includes a hardware warp engine, the sensor data processing offload hardware component 130 may bypass the command buffer 314 by directly feeding the updated positional information, the speed data, and/or the acceleration data to the GPU 108 (reference numeral 335)) at the second frame rate that is higher than the first frame rate (col. 1 lines 39-53. Time warp involves warping the rendered image before sending it to the display to correct for the user's movement that occurred after the rendering. Time warp may reduce latency and increase or maintain frame rate (i.e., the number of frames display per second (fps)). This process takes an already rendered image, modifies it with the predicted positional information based on the collected positional information obtained from sensors (e.g., sensor(s) housed in a HMD), and then displays the modified image on the VR display. Note: By increasing the frame rate, a second frame rate would be higher than a first frame rate).
Quach does not explicitly teach the local processing system external to the head-mounted display unit and communicatively coupled to the head-mounted display unit via a data link, the local processing system being configured to generate rendered frames to be output to the head-mounted display via the data link to be warped at the head-mounted display unit to generate warped rendered frames, the head-mounted display unit configured to output the warped rendered frames as virtual content to a user of the head-mounted display system; one or more head-mounted display unit processors, the one or more head-mounted display unit processors configured to: receive the rendered frames generated by the local processing system, which is external to the head-mounted display unit; and warp the rendered frames generated by the local processing system, which is external to the head-mounted display unit, to generate the warped rendered frames, and output, via the display, the rendered frames at the first frame rate, and output, via the display, the warped rendered frames and such that a plurality of warped rendered frames are presented to the user between the rendered frames.
However, Chen teaches the local processing system external to the head-mounted display unit (see para. [0036], para. [0038] and Fig. 1. The computing device 110 may be, for example a personal computer or a laptop with computing capability and includes a processor. The processor is, for example, a central processing unit (CPU), a graphics processing unit (GPU) or any other programmable microprocessor for general or special use, a digital signal processor (DSP), a programmable controller, an application specific integrated circuit (ASIC), a programmable logic device (PLD) or other similar devices, or a combination of these devices. The computing device 110 is disposed separately from the HMD 120) and communicatively coupled to the head-mounted display unit via a data link (see Fig. 1, para. [0038], para. [0054]. The computing device 110 is disposed separately from the HMD 120, and coupled to the HMD 120 in a wired or a wireless manner for data transmission), the local processing system being configured to generate rendered frames to be output to the head-mounted display via the data link (see para. [0015], para. [0040], para. [0046]. The computing device is configured to generate a plurality of image frames to be displayed on the HMD according to a frame rate. The computing device is configured to transfer the generated image frames to the HMD. The computing device 110 generates a plurality of image frames to be displayed on the HMD 120 according to a frame rate (step S202)) to be warped at the head-mounted display unit to generate warped rendered frames (see para. [0015], para. [0026]. The HMD is configured to restore the second frames with the second resolution among the image frames to a regular resolution of the first display and the second display, and to interleaved display the first frames and the restored second frames on the first display and the second display. The application provides a display system for adaptive interleaved image warping), the head-mounted display unit configured to output the warped rendered frames as virtual content to a user of the head-mounted display system (see Fig. 2, para. [0015], para. [0030], para. [0039], para. [0049]. The HMD is configured to restore the second frames with the second resolution among the image frames to a regular resolution of the first display and the second display, and to interleaved display the first frames and the restored second frames on the first display and the second display); head-mounted display unit configured to: receive the rendered frames generated by the local processing system, which is external to the head-mounted display unit (see Fig. 1, para. [0015], para. [0038], para. [0040], para. [0046]. After generating the image frames, the computing device 110 transfers the generated image frames to the HMD 120 (step S204). Then, the HMD 120 restores the second frames with the second resolution among the image frames to a regular resolution of the first display 121 and the second display 122 (step S206)); and warp the rendered frames generated by the local processing system, which is external to the head-mounted display unit, to generate the warped rendered frames (see Figs. 1-2, para. [0015], para. [0030], para. [0038]-[0039], para. [0049]. The HMD is configured to restore the second frames with the second resolution among the image frames to a regular resolution of the first display and the second display, and to interleaved display the first frames and the restored second frames on the first display and the second display. The computing device 110 is disposed separately from the HMD 120).
Quach and Chen are related to head mounted displays and virtual reality, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the display system disclosed by Quach with Chen’s teachings, since it would have reduced the computation loading of the computing device and the data size for transmission while keeping the quality of the image frames (Chen para. [0026]).
Quach and Chen do not explicitly disclose and output, via the display, the rendered frames at the first frame rate, and output, via the display, the warped rendered frames and such that a plurality of warped rendered frames are presented to the user between the rendered frames.
However, Hunt teaches output, via the display, the rendered frames at the first frame rate, and output, via the display, the warped rendered frames at the second frame rate that is higher than the first frame rate and such that a plurality of warped rendered frames are presented to the user between the rendered frames (see claim 1, col. 3 lines 3-18, col. 3 lines 44-56, col. 6 lines 48-67, col. 7 lines 1-3 . Generate one or more synthetic frames using one or more of the frames from the processed content and the warping parameters, upsample the content from the first frame rate to a second frame rate by inserting the one or more synthetic frames among the frames received from the console at the first frame rate to generate augmented content, wherein the second frame rate is faster than the first frame rate, and present the augmented content at the second frame rate via an electronic display. A synthetic frame is an image frame created by warping and/or shifting features of an original content frame. The number of synthetic frames to generate between adjacent original frames is (Thigh/Tlow)−1. For example, 100 fps corresponds to a frame every 10 ms, and 500 fps corresponds to a frame every 2 ms. Accordingly, to achieve 500 fps or, in other words, a frame every 2 ms, four (10 ms/2 ms−1) synthetic frames should be generated for every 10 ms time period. And one period of augmented content would have an original frame followed by the four additional synthetic frames).
Quach, Chen and Hunt are related to head mounted displays and virtual reality, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the display system disclosed by Quach and Chen with Hunt’s teachings, since it would have mitigated motion blur that could otherwise occur using a slower frame rate when there is movement of the HMD. In addition, increasing to a fast frame rate allows these intervals to occur more often, and assuming a same display intensity as some slower frame rate, the fast frame rate would result in an increase in brightness of the displayed content. And a brighter display allows for use with HMDs that include optics that attenuate light from the display. Moreover, in some cases, it takes less energy to render a synthetic frame than an original frame (e.g., an I-frame). Accordingly, power requirements are substantially less for rendering augmented content at a given frame rate—than for rendering only full frames at the given frame rate (see Hunt, col. 3, lines 19-41).
Regarding Claim 3, Quach, Chen and Hunt teach the head-mounted display system of claim 1.
Chen further teaches wherein the data link comprises a cable connecting the local processing system and the head-mounted display unit (see Fig. 1, para. [0038]. The computing device 110 is disposed separately from the HMD 120, and coupled to the HMD 120 in a wired or a wireless manner for data transmission), wherein a bandwidth of the data link limits the first frame rate (see para. [0040], para. [0052] and claim 4. the computing device 110 generates a plurality of image frames to be displayed on the HMD 120 according to a frame rate (step S202). The computing devices 110 may detect a transmission bandwidth for transferring the generated image frames and adaptively adjust the second resolution according to the transmission bandwidth, so as to reduce the data size to be transferred and prevent delay of data transmission. Note: bandwidth is a property of the wire).
Quach, Chen and Hunt are related to head mounted displays and virtual reality, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the display system disclosed by Quach, Chen and Hunt with Chen’s teachings, since it would have reduced the computation loading of the computing device and the data size for transmission while keeping the quality of the image frames (Chen para. [0026]). Moreover, it would have aided in reduce the data size to be transferred and prevent delay of data transmission (Chen par. [0052]).
Regarding Claim 14, Quach, Chen and Hunt teach the head-mounted display system of claim 1.
Quach further teaches wherein the one or more head-mounted processors are configured to warp the rendered frames based on a determined gaze of the user of the head-mounted display unit (see col. 4, lines 30-37, col. 5 lines 30-37. One or more cameras 124 may track eye movements. The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth).
Claims 4, 16-17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Quach (US 9858637 B1) in view of Chen (US 20190149809 A1) in view of Hunt (US 10785471 B1), further in view of Baran et al. (US 20170085867 A1, hereinafter referenced as Baran).
Regarding Claim 4, Quach, Chen and Hunt teach the head-mounted display system of claim 1.
Quach, Chen and Hunt do not explicitly teach wherein the head-mounted display unit is configured to warp each rendered frame a threshold number of times based on respective orientation information, the threshold number being two or more.
However, Baran teaches wherein the head-mounted display unit is configured to warp each rendered frame a threshold number of times based on respective orientation information, the threshold being two or more (see para. [0090], para. [0129], para. [0133], para. [0273]. The multi-view display apparatus may be part of a wearable (e.g., virtual reality) headset worn by a viewer. The error views may be used to determine how to update the values of the first set of actuation signals to obtain a second set of actuation signals (in order to reduce the error between the display views and the scene views). The second set of actuation signals are then used to determine a second set of display views that would be generated by a multi-view display if it were driven by the second set of actuation signals, and a second set of error views is generated by comparing the second set of display views with the scene views. The second set of error views is then used to determine how to update the values of the second set of actuation signals to obtain a third set of actuation signals in order further reduce the error between the display vies and the scene views. This iterative process may be repeated until the error between the display views and the scene views falls below a predetermined threshold, a threshold number of iterations has been performed, a threshold amount of time has elapsed, or any other suitable stopping criteria has been satisfied).
Quach, Chen, Hunt and Baran are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen and Hunt with Baran’s teachings, since it would have enhanced the head mounted display system by allowing performing operations a proper number of times to reduce display errors.
Regarding Claim 16, Quach teaches a system (see Figs. 1-2) comprising:
a head-mounted display unit (see Fig. 2, col. 3 lines 8-9, col. 3 lines 62-67, col. 4 lines 1-8, head-mount virtual reality display);
a local processing system communicatively coupled to the head-mounted display unit (see abstract, col. 1 lines 39-53, col. 2 lines 39-59, col. 3 lines 62-67, col. 4 lines 1-8, col.4 lines 38-55, col. 5 lines 28-36, Figs. 1-2) The computer system includes a virtual reality application executed by a processing device. A graphic processing unit (GPU) displays rendered images associated with the virtual reality application. The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second);
one or more processors (see abstract, col. 1 lines 39-53, col. 2 lines 39-59, col. 3 lines 62-67, col. 4 lines 1-8, col.4 lines 38-55, col. 5 lines 28-36. The computer system includes a virtual reality application executed by a processing device. A graphic processing unit (GPU) displays rendered images associated with the virtual reality application. The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth); and
one or more computer storage media (see Fig. 1, col. 4 lines 9-15, col. 4 lines 38-45. As illustrated in FIG. 1, the system 100 comprises a system on chip (SoC) 102 electrically coupled to one or more sensors and a memory system via a memory bus. In the embodiment of FIG. 1, the memory system comprises a dynamic random access memory (DRAM) 104 coupled to the SoC 102 via a random access memory (RAM) bus (e.g., a double data rate (DDR) bus 105)) storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations (see col. 4 lines 46-55, col. 6 lines 11-27, col. 7 lines 64-66. The virtual reality application(s) 120 may comprise software executing on the CPU 106 and the GPU 108. It should be appreciated that one or more of the method steps described herein may be stored in the memory as computer program instructions) comprising:
generating, by the local processing system at a first frame rate, rendered frames of virtual content to be displayed by the head-mounted display unit of the system (see abstract, Figs. 1-2, col. 2 lines 46-60, col. 3 lines 62-67, col. 4 lines 1-8, col.4 lines 38-55, col. 5 lines 28-50, col. 6 lines 7-29. A graphic processing unit (GPU) displays rendered images associated with the virtual reality application. The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth);
providing, by the local processing system, the rendered frames of the virtual content (see col. 2 lines 49-52, col. 5 lines 30-57. A graphic processing unit (GPU) displays rendered images associated with the virtual reality application) to the head-mounted display unit of the system (see Fig. 2, col. 3 lines 8-9, col. 3 lines 62-67, col. 4 lines 1-8. It should be appreciated that the system 100 may be implemented in various types of virtual reality systems (e.g., headsets, goggles, eyewear, external VR display(s), projection systems, etc.). As illustrated in the embodiment of FIG. 2, the system 100 may be incorporated in an integrated headmount display (HMD) 200 that is worn by a user. In other embodiments, some of components in the system 100 may be integrated into the HMD 200 while others may be provided by an external processing system (e.g., a portable computing device) or external display (e.g., a computer display, a projection display, etc.)), the rendered frames being provided at the first frame rate (see col. 1 lines 39-53, col. 2 lines 46-60, col. 5 lines 28-50, col. 6 lines 7-29. Time warp involves warping the rendered image before sending it to the display to correct for the user's movement that occurred after the rendering. Time warp may reduce latency and increase or maintain frame rate (i.e., the number of frames display per second (fps)). The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth);
warping, by the head-mounted display unit, at a second frame rate higher than the first frame rate (see col. 1 lines 39-53, col. 2 lines 46-60, col. 5 lines 28-50, col. 6 lines 7-29. Time warp involves warping the rendered image before sending it to the display to correct for the user's movement that occurred after the rendering. Time warp may reduce latency and increase or maintain frame rate (i.e., the number of frames display per second (fps)). The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth. Note: By increasing the frame rate, a second frame rate would be higher than a first frame rate), each rendered frame based on orientation information associated with the system (see abstract, col. 2 lines 46-60, col. 4 lines 9-38. Receiving sensor data from one or more sensors tracking translational and rotational motion of a user for a virtual reality application. An updated position of the user is computed based on the received sensor data. The speed and acceleration of the user movement may be computed based on the sensor data. The updated position, the speed, and the acceleration may be provided to a warp engine configured to update a rendered image before sending to a virtual reality display based on one or more of the updated position, the speed, and the acceleration. As illustrated in FIG. 2, the sensor(s) enable the system 100 to track the rotational and translational movement of a user (e.g., the user's head, eyes, and/or other body parts) within three-dimensional space using six degrees of freedom (6DOF). As known in the art, the term 6DOF refers to the freedom of movement of the user in three-dimensional space. The sensors track motion as forward/backward (surge), up/down (heave), left/right (sway) translation in three perpendicular axes, combined with changes in orientation through rotation about three perpendicular axes (i.e., pitch, yaw, and roll). One or more accelerometer sensors 126 may sense changes in the magnitude and/or direction of the proper acceleration of the user's head and/or other body parts. One or more gyroscope sensors 128 may be used to track angular motion of the user's head and/or other body parts); and
outputting, via a display of the head-mounted display unit, the warped frames at (see abstract, col. 1 lines 39-53, col. 2 lines 46-60, col. 5 lines 28-50, col. 6 lines 7-29. Time warp involves warping the rendered image before sending it to the display to correct for the user's movement that occurred after the rendering. Time warp may reduce latency and increase or maintain frame rate (i.e., the number of frames display per second (fps)). The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth. Note: By increasing the frame rate, a second frame rate would be higher than a first frame rate) at the second frame rate that is higher than the first frame rate (col. 1 lines 39-53. Time warp involves warping the rendered image before sending it to the display to correct for the user's movement that occurred after the rendering. Time warp may reduce latency and increase or maintain frame rate (i.e., the number of frames display per second (fps)). This process takes an already rendered image, modifies it with the predicted positional information based on the collected positional information obtained from sensors (e.g., sensor(s) housed in a HMD), and then displays the modified image on the VR display. Note: By increasing the frame rate, a second frame rate would be higher than a first frame rate).
Quach does not explicitly disclose a local processing system external to the head-mounted display unit and communicatively coupled to the head-mounted display unit via a hardware connection, providing, by the local processing system via the hardware connection, the rendered frames virtual content to the head-mounted display unit of the system; and warping each rendered frame a threshold number of times, the threshold number being two or more, outputting, via a display of the head-mounted display unit, the rendered frames at the first frame rate, and outputting, via the display of the head-mounted display unit, the warped frames at the second frame rate and such that a plurality of warped rendered frames are presented to a user between the rendered frames, and wherein the display is configured to output the threshold number of warped frames associated with a first rendered frame followed by the threshold number of warped frames associated with a second subsequent rendered frame.
However, Chen teaches the local processing system external to the head-mounted display unit (see para. [0036], para. [0038] and Fig. 1. The computing device 110 may be, for example a personal computer or a laptop with computing capability and includes a processor. The processor is, for example, a central processing unit (CPU), a graphics processing unit (GPU) or any other programmable microprocessor for general or special use, a digital signal processor (DSP), a programmable controller, an application specific integrated circuit (ASIC), a programmable logic device (PLD) or other similar devices, or a combination of these devices. The computing device 110 is disposed separately from the HMD 120) and communicatively coupled to the head-mounted display unit via a hardware connection (see Fig. 1, para. [0038], para. [0054]. The computing device 110 is disposed separately from the HMD 120, and coupled to the HMD 120 in a wired or a wireless manner for data transmission), providing, by the local processing system via the hardware connection, the rendered frames virtual content to the head-mounted display unit of the system (see para. [0015],para. [0038], para. [0040], para. [0046]. The computing device is configured to generate a plurality of image frames to be displayed on the HMD according to a frame rate. The computing device is configured to transfer the generated image frames to the HMD. The computing device 110 generates a plurality of image frames to be displayed on the HMD 120 according to a frame rate (step S202)) warping, by the head-mounted display unit, each rendered frame (see para. [0015], para. [0026]. The HMD is configured to restore the second frames with the second resolution among the image frames to a regular resolution of the first display and the second display, and to interleaved display the first frames and the restored second frames on the first display and the second display. The application provides a display system for adaptive interleaved image warping), outputting, via a display of the head-mounted display unit, the warped frames (see Fig. 2, para. [0015], para. [0030], para.[0038]-[0039], para. [0049]. The HMD is configured to restore the second frames with the second resolution among the image frames to a regular resolution of the first display and the second display, and to interleaved display the first frames and the restored second frames on the first display and the second display).
Quach and Chen are related to head mounted displays and virtual reality, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the display system disclosed by Quach with Chen’s teachings, since it would have reduced the computation loading of the computing device and the data size for transmission while keeping the quality of the image frames (Chen para. [0026]).
Quach and Chen do not explicitly disclose warping each rendered frame a threshold number of times, the threshold number being two or more, outputting, via a display of the head-mounted display unit, the rendered frames at the first frame rate, and outputting, via the display of the head-mounted display unit, the warped frames at the second frame rate and such that a plurality of warped rendered frames are presented to a user between the rendered frames, and wherein the display is configured to output the threshold number of warped frames associated with a first rendered frame followed by the threshold number of warped frames associated with a second subsequent rendered frame.
However, Hunt teaches outputting, via a display of the head-mounted display unit, the rendered frames at the first frame rate, and outputting, via the display of the head-mounted display unit, the warped frames at the second frame rate that is higher than the first frame rate and such that a plurality of warped rendered frames are presented to a user between the rendered frames (see claim 1, col. 3 lines 3-18, col. 3 lines 44-56, col. 6 lines 48-67, col. 7 lines 1-3 . Generate one or more synthetic frames using one or more of the frames from the processed content and the warping parameters, upsample the content from the first frame rate to a second frame rate by inserting the one or more synthetic frames among the frames received from the console at the first frame rate to generate augmented content, wherein the second frame rate is faster than the first frame rate, and present the augmented content at the second frame rate via an electronic display. A synthetic frame is an image frame created by warping and/or shifting features of an original content frame. The number of synthetic frames to generate between adjacent original frames is (Thigh/Tlow)−1. For example, 100 fps corresponds to a frame every 10 ms, and 500 fps corresponds to a frame every 2 ms. Accordingly, to achieve 500 fps or, in other words, a frame every 2 ms, four (10 ms/2 ms−1) synthetic frames should be generated for every 10 ms time period. And one period of augmented content would have an original frame followed by the four additional synthetic frames).
Quach, Chen and Hunt are related to head mounted displays and virtual reality, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the display system disclosed by Quach and Chen with Hunt’s teachings, since it would have mitigated motion blur that could otherwise occur using a slower frame rate when there is movement of the HMD. In addition, increasing to a fast frame rate allows these intervals to occur more often, and assuming a same display intensity as some slower frame rate, the fast frame rate would result in an increase in brightness of the displayed content. And a brighter display allows for use with HMDs that include optics that attenuate light from the display. Moreover, in some cases, it takes less energy to render a synthetic frame than an original frame (e.g., an I-frame). Accordingly, power requirements are substantially less for rendering augmented content at a given frame rate—than for rendering only full frames at the given frame rate (see Hunt, col. 3, lines 19-41).
Quach, Chen and Hunt do not explicitly disclose warping each rendered frame a threshold number of times, the threshold number being two or more, and wherein the display is configured to output the threshold number of warped frames associated with a first rendered frame followed by the threshold number of warped frames associated with a second subsequent rendered frame.
However, Baran teaches warping each rendered frame a threshold number of times, the threshold number being two or more, wherein the display is configured to output the threshold number of warped frames associated with a first rendered frame followed by the threshold number of warped frames associated with a second subsequent rendered frame (see para. [0090], para. [0129], para. [0133], para. [0273]. The multi-view display apparatus may be part of a wearable (e.g., virtual reality) headset worn by a viewer. A subset of pixels can be updated at a rate that is higher than the equivalent full-refresh frame rate of the display element. The error views may be used to determine how to update the values of the first set of actuation signals to obtain a second set of actuation signals (in order to reduce the error between the display views and the scene views). The second set of actuation signals are then used to determine a second set of display views that would be generated by a multi-view display if it were driven by the second set of actuation signals, and a second set of error views is generated by comparing the second set of display views with the scene views. The second set of error views is then used to determine how to update the values of the second set of actuation signals to obtain a third set of actuation signals in order further reduce the error between the display vies and the scene views. This iterative process may be repeated until the error between the display views and the scene views falls below a predetermined threshold, a threshold number of iterations has been performed, a threshold amount of time has elapsed, or any other suitable stopping criteria has been satisfied).
Quach, Chen, Hunt and Baran are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen and Hunt with Baran’s teachings, since it would have enhanced the head mounted display system by allowing performing operations a proper number of times to reduce display errors.
Regarding Claim 17, Quach, Chen, Hunt and Baran teach the system of claim 16.
Chen further teaches wherein the hardware connection comprises a cable connecting the local processing system and the head-mounted display unit (see Fig. 1, para. [0038]. The computing device 110 is disposed separately from the HMD 120, and coupled to the HMD 120 in a wired or a wireless manner for data transmission).
Quach, Chen, Hunt and Baran are related to head mounted displays and virtual reality, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the display system disclosed by Quach, Chen, Hunt and Baran with Chen’s teachings, since it would have reduced the computation loading of the computing device and the data size for transmission while keeping the quality of the image frames (Chen para. [0026]). Moreover, it would have aided in reduce the data size to be transferred and prevent delay of data transmission (Chen par. [0052]).
Regarding Claim 20, Quach teaches method (see abstract. An exemplary method involves receiving sensor data from one or more sensors tracking translational and rotational motion of a user for a virtual reality application) implemented by a head-mounted display system (see Fig. 2), the head-mounted display system comprising a local processing system and head- mounted display unit and in communication with the head-mounted display unit (see Figs. 1-2, col. 2 lines 46-60), the method comprising:
generating, by the local processing system at a first frame rate, rendered frames of virtual content to be displayed via the head-mounted display unit (see abstract, Figs. 1-2, col. 2 lines 46-60, col. 3 lines 62-67, col. 4 lines 1-8, col.4 lines 38-55, col. 5 lines 28-50, col. 6 lines 7-29. A graphic processing unit (GPU) displays rendered images associated with the virtual reality application. The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth);
providing, by the local processing system (see col. 2 lines 49-52, col. 5 lines 30-57. A graphic processing unit (GPU) displays rendered images associated with the virtual reality application), the rendered frames to the head- mounted display unit (see Figs. 1-2, col. 3 lines 8-9, col. 3 lines 62-67, col. 4 lines 1-8. It should be appreciated that the system 100 may be implemented in various types of virtual reality systems (e.g., headsets, goggles, eyewear, external VR display(s), projection systems, etc.). As illustrated in the embodiment of FIG. 2, the system 100 may be incorporated in an integrated headmount display (HMD) 200 that is worn by a user. In other embodiments, some of components in the system 100 may be integrated into the HMD 200 while others may be provided by an external processing system (e.g., a portable computing device) or external display (e.g., a computer display, a projection display, etc.)), the rendered frames being provided at the first frame rate (see col. 1 lines 39-53, col. 2 lines 46-60, col. 5 lines 28-50, col. 6 lines 7-29. Time warp involves warping the rendered image before sending it to the display to correct for the user's movement that occurred after the rendering. Time warp may reduce latency and increase or maintain frame rate (i.e., the number of frames display per second (fps)). The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth);
warping, by the head-mounted display unit at a second frame rate higher than the first frame rate (see col. 1 lines 39-53, col. 2 lines 46-60, col. 5 lines 28-50, col. 6 lines 7-29. Time warp involves warping the rendered image before sending it to the display to correct for the user's movement that occurred after the rendering. Time warp may reduce latency and increase or maintain frame rate (i.e., the number of frames display per second (fps)). The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth. Note: By increasing the frame rate, a second frame rate would be higher than a first frame rate), each rendered frame based on orientation information associated with the head-mounted display system (see abstract, col. 2 lines 46-60, col. 4 lines 9-38. Receiving sensor data from one or more sensors tracking translational and rotational motion of a user for a virtual reality application. An updated position of the user is computed based on the received sensor data. The speed and acceleration of the user movement may be computed based on the sensor data. The updated position, the speed, and the acceleration may be provided to a warp engine configured to update a rendered image before sending to a virtual reality display based on one or more of the updated position, the speed, and the acceleration. As illustrated in FIG. 2, the sensor(s) enable the system 100 to track the rotational and translational movement of a user (e.g., the user's head, eyes, and/or other body parts) within three-dimensional space using six degrees of freedom (6DOF). As known in the art, the term 6DOF refers to the freedom of movement of the user in three-dimensional space. The sensors track motion as forward/backward (surge), up/down (heave), left/right (sway) translation in three perpendicular axes, combined with changes in orientation through rotation about three perpendicular axes (i.e., pitch, yaw, and roll). One or more accelerometer sensors 126 may sense changes in the magnitude and/or direction of the proper acceleration of the user's head and/or other body parts. One or more gyroscope sensors 128 may be used to track angular motion of the user's head and/or other body parts); and
outputting, via a display of the head-mounted display unit, the warped frames (see abstract, col. 1 lines 39-53, col. 2 lines 46-60, col. 5 lines 28-50, col. 6 lines 7-29. Time warp involves warping the rendered image before sending it to the display to correct for the user's movement that occurred after the rendering. Time warp may reduce latency and increase or maintain frame rate (i.e., the number of frames display per second (fps)). The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth. Note: By increasing the frame rate, a second frame rate would be higher than a first frame rate) at the second frame rate that is higher than the first frame rate (col. 1 lines 39-53. Time warp involves warping the rendered image before sending it to the display to correct for the user's movement that occurred after the rendering. Time warp may reduce latency and increase or maintain frame rate (i.e., the number of frames display per second (fps)). This process takes an already rendered image, modifies it with the predicted positional information based on the collected positional information obtained from sensors (e.g., sensor(s) housed in a HMD), and then displays the modified image on the VR display. Note: By increasing the frame rate, a second frame rate would be higher than a first frame rate).
Quach does not explicitly disclose the local processing system being external to the head-mounted display unit, and in communication with the head-mounted display unit via a hardware connection, providing, by the local processing system, the rendered frames to the head- mounted display unit via the hardware connection; and warping each rendered frame a threshold number of times, the threshold number of times being two or more, outputting, via a display of the head-mounted display unit, the rendered frames at the first frame rate, and outputting, via the display of the head-mounted display unit, the warped frames at the second frame rate that is higher than the first frame rate and such that a plurality of warped rendered frames are presented to a user between the rendered frames, wherein the display is configured to output the threshold number of warped frames associated with a first rendered frame followed by the threshold number of warped frames associated with a second subsequent rendered frame.
However, Chen teaches the local processing system external to the head-mounted display unit (see para. [0036], para. [0038] and Fig. 1. The computing device 110 may be, for example a personal computer or a laptop with computing capability and includes a processor. The processor is, for example, a central processing unit (CPU), a graphics processing unit (GPU) or any other programmable microprocessor for general or special use, a digital signal processor (DSP), a programmable controller, an application specific integrated circuit (ASIC), a programmable logic device (PLD) or other similar devices, or a combination of these devices. The computing device 110 is disposed separately from the HMD 120) and in communication with the head-mounted display unit via a hardware connection (see Fig. 1, para. [0038], para. [0054]. The computing device 110 is disposed separately from the HMD 120, and coupled to the HMD 120 in a wired or a wireless manner for data transmission), providing, by the local processing system, the rendered frames to the head- mounted display unit via the hardware connection (see para. [0015],para. [0038], para. [0040], para. [0046]. The computing device is configured to generate a plurality of image frames to be displayed on the HMD according to a frame rate. The computing device is configured to transfer the generated image frames to the HMD. The computing device 110 generates a plurality of image frames to be displayed on the HMD 120 according to a frame rate (step S202)) warping, by the head-mounted display unit, each rendered frame (see para. [0015], para. [0026]. The HMD is configured to restore the second frames with the second resolution among the image frames to a regular resolution of the first display and the second display, and to interleaved display the first frames and the restored second frames on the first display and the second display. The application provides a display system for adaptive interleaved image warping), outputting, via a display of the head-mounted display unit, the warped frames (see Fig. 2, para. [0015], para. [0030], para.[0038]-[0039], para. [0049]. The HMD is configured to restore the second frames with the second resolution among the image frames to a regular resolution of the first display and the second display, and to interleaved display the first frames and the restored second frames on the first display and the second display).
Quach and Chen are related to head mounted displays and virtual reality, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the display system disclosed by Quach with Chen’s teachings, since it would have reduced the computation loading of the computing device and the data size for transmission while keeping the quality of the image frames (Chen para. [0026]).
Quach and Chen do not explicitly disclose warping each rendered frame a threshold number of times, the threshold number of times being two or more, outputting, via a display of the head-mounted display unit, the rendered frames at the first frame rate, and outputting, via the display of the head-mounted display unit, the warped frames at the second frame rate that is higher than the first frame rate and such that a plurality of warped rendered frames are presented to a user between the rendered frames, wherein the display is configured to output the threshold number of warped frames associated with a first rendered frame followed by the threshold number of warped frames associated with a second subsequent rendered frame.
However, Hunt teaches outputting, via a display of the head-mounted display unit, the rendered frames at the first frame rate, and outputting, via the display of the head-mounted display unit, the warped frames at the second frame rate that is higher than the first frame rate and such that a plurality of warped rendered frames are presented to a user between the rendered frames (see claim 1, col. 3 lines 3-18, col. 3 lines 44-56, col. 6 lines 48-67, col. 7 lines 1-3 . Generate one or more synthetic frames using one or more of the frames from the processed content and the warping parameters, upsample the content from the first frame rate to a second frame rate by inserting the one or more synthetic frames among the frames received from the console at the first frame rate to generate augmented content, wherein the second frame rate is faster than the first frame rate, and present the augmented content at the second frame rate via an electronic display. A synthetic frame is an image frame created by warping and/or shifting features of an original content frame. The number of synthetic frames to generate between adjacent original frames is (Thigh/Tlow)−1. For example, 100 fps corresponds to a frame every 10 ms, and 500 fps corresponds to a frame every 2 ms. Accordingly, to achieve 500 fps or, in other words, a frame every 2 ms, four (10 ms/2 ms−1) synthetic frames should be generated for every 10 ms time period. And one period of augmented content would have an original frame followed by the four additional synthetic frames).
Quach, Chen and Hunt are related to head mounted displays and virtual reality, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the display system disclosed by Quach and Chen with Hunt’s teachings, since it would have mitigated motion blur that could otherwise occur using a slower frame rate when there is movement of the HMD. In addition, increasing to a fast frame rate allows these intervals to occur more often, and assuming a same display intensity as some slower frame rate, the fast frame rate would result in an increase in brightness of the displayed content. And a brighter display allows for use with HMDs that include optics that attenuate light from the display. Moreover, in some cases, it takes less energy to render a synthetic frame than an original frame (e.g., an I-frame). Accordingly, power requirements are substantially less for rendering augmented content at a given frame rate—than for rendering only full frames at the given frame rate (see Hunt, col. 3, lines 19-41).
Quach, Chen and Hunt do not explicitly disclose warping each rendered frame a threshold number of times, the threshold number being two or more, and wherein the display is configured to output the threshold number of warped frames associated with a first rendered frame followed by the threshold number of warped frames associated with a second subsequent rendered frame.
However, Baran teaches warping each rendered frame a threshold number of times, the threshold number being two or more, wherein the display is configured to output the threshold number of warped frames associated with a first rendered frame followed by the threshold number of warped frames associated with a second subsequent rendered frame (see para. [0090], para. [0129], para. [0133], para. [0273]. The multi-view display apparatus may be part of a wearable (e.g., virtual reality) headset worn by a viewer. A subset of pixels can be updated at a rate that is higher than the equivalent full-refresh frame rate of the display element. The error views may be used to determine how to update the values of the first set of actuation signals to obtain a second set of actuation signals (in order to reduce the error between the display views and the scene views). The second set of actuation signals are then used to determine a second set of display views that would be generated by a multi-view display if it were driven by the second set of actuation signals, and a second set of error views is generated by comparing the second set of display views with the scene views. The second set of error views is then used to determine how to update the values of the second set of actuation signals to obtain a third set of actuation signals in order further reduce the error between the display vies and the scene views. This iterative process may be repeated until the error between the display views and the scene views falls below a predetermined threshold, a threshold number of iterations has been performed, a threshold amount of time has elapsed, or any other suitable stopping criteria has been satisfied).
Quach, Chen, Hunt and Baran are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the method disclosed by Quach, Chen, Hunt with Baran’s teachings, since it would have enhanced the method by allowing performing operations a proper number of times to reduce display errors.
Claims 5-9 and 11-13 are rejected under 35 U.S.C. 103 as being unpatentable over Quach (US 9858637 B1) in view of Chen (US 20190149809 A1), in view of Hunt (US 10785471 B1), further in view of Urey (US 20180003981 A1).
Regarding Claim 5, Quach, Chen and Hunt teach the head-mounted display system of claim 1.
Quach further teaches wherein a processor of the one or more processors head-mounted display unit (see Fig. 1, Fig. 3, GPU 108), col. 4 lines 38-45) is configured to warp the rendered frames based on the orientation information (see abstract and col. 2 lines 46-60. A computer system comprising one or more sensors for tracking translational and rotational motion of a user. A graphic processing unit (GPU) displays rendered images associated with the virtual reality application. A dedicated sensor processing component is electrically coupled to the one or more sensors and the GPU. The dedicated sensor processing component receives sensor data from the one or more sensors and, based on the sensor data, computes an updated position and a speed and acceleration of the user. The dedicated sensor processing component feeds the updated position, the speed, and the acceleration to a warp engine associated with the GPU).
Quach, Chen and Hunt do not explicitly teach the processor of the one or more processors head-mounted display unit is a hardware application-specific integrated circuit (ASIC).
However, Urey teaches the processor of the one or more processors head-mounted display unit is a hardware application-specific integrated circuit (ASIC) (see para. [0392]-[0393]. Processor 13102 may be a microprocessor, a digital signal processor, an application-specific processor, or the like. In some embodiments, processor 13102 is a component within a larger integrated circuit such as a system on chip (SOC) application-specific integrated circuit (ASIC)).
Quach, Chen, Hunt and Urey are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen and Hunt with Urey’s teachings of the processor is a hardware application-specific integrated circuit (ASIC), since it would have been obvious to try from a finite number of processor known in the art that would have yield the same predictable result of processing information. Moreover, it would have allowed the used of integrated circuit such as a system on chip (SOC) architecture (Urey para. [0393]).
Regarding Claim 6, Quach, Chen, Hunt and Urey teach the head-mounted display system of claim 5.
Urey teaches wherein the display comprises a spatial light modulator (see para. [0114]. Near-to-eye display device 100 also includes spatial light modulators (SLM) 110), and wherein the spatial light modulator comprises the hardware ASIC (see para. [0393]-[0395]. Processor 13102 may be a microprocessor, a digital signal processor, an application-specific processor, or the like. In some embodiments, processor 13102 is a component within a larger integrated circuit such as a system on chip (SOC) application-specific integrated circuit (ASIC). The processor 13102 computes the SLM data to be displayed on the SLM).
Quach, Chen, Hunt and Urey are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen, Hunt and Urey with Urey’s teachings of the display comprises a spatial light modulator, since it would have been obvious to try from a finite number of display technologies known in the art that would have yield the same predictable result of display images.
Regarding Claim 7, Quach, Chen, Hunt and Urey teach the head-mounted display system of claim 6.
Urey further teaches wherein the spatial light modulator is configured to adjust pixels of the rendered frames based on the hardware ASIC (see para. [0116]-[0117], para. [0128]-[0129], para. [0139], para. [0215], para. [0236], para. [0245] para. [0393]-[0395]. SLMs are basically dynamically programmable diffractive optical elements. For each frame of the displayed video, point light source 120 generates a coherent light wave of a single wavelength that illuminates a spatial light modulator (SLM 110) that is mounted on the front section of the eyeglass. SLM with a pixelated structure. The pixelated structure of SLMs is intimately linked with sampling and interpolation of light waves. For each video frame, the data on the SLM is a computer-generated holographic image of the virtual scene. S SLMs 13162 are SLMs that impart information to an illumination wave to create the desired light wave distribution in the useful portion of the exit pupil plane. In operation, processor 13102 programs SLMs 13162 using data stored in memory 13110. In some embodiments, processor 13102 computes the SLM data to be displayed on the SLM and stores it in memory 13110. In other embodiments, the SLM data is computed by a separate device, and the SLM data is provided to near-to-eye display device 13100 for later display).
Quach, Chen, Hunt and Urey are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen, Hunt and Urey with Urey’s teachings of the display comprises a spatial light modulator, since it would have been obvious to try from a finite number of display technologies known in the art that would have yield the same predictable result of display images.
Regarding Claim 8, Quach, Chen, Hunt and Urey teach the head-mounted display system of claim 5.
Quach further teaches wherein the processor is configured to provide information corresponding to the warped rendered frames to the display (see abstract, col. 2 lines 46-60, col. 5 lines 28-50, col. 6 lines 7-29. A graphic processing unit (GPU) displays rendered images associated with the virtual reality application. The dedicated sensor processing component feeds the updated position, the speed, and the acceleration to a warp engine associated with the GPU. The warp engine may use the speed data along with eye-tracking information to adjust the rendering resolution. For example, if the user's head (or the eyes) are moving fast enough (e.g., past a certain threshold—100 degrees/second), then rendering may be performed at a lower resolution to minimize DDR bandwidth)
Urey further teaches wherein the hardware ASIC is configured to provide information corresponding to frames to a spatial light modulator associated with the display (see para. [0116]-[0117], para. [0215], para. [0236] para. [0393]-[0395]. For each frame of the displayed video, point light source 120 generates a coherent light wave of a single wavelength that illuminates a spatial light modulator (SLM 110) that is mounted on the front section of the eyeglass. For each video frame, the data on the SLM is a computer-generated holographic image of the virtual scene. SLMs 13162 are SLMs that impart information to an illumination wave to create the desired light wave distribution in the useful portion of the exit pupil plane. In operation, processor 13102 programs SLMs 13162 using data stored in memory 13110. In some embodiments, processor 13102 computes the SLM data to be displayed on the SLM and stores it in memory 13110. In other embodiments, the SLM data is computed by a separate device, and the SLM data is provided to near-to-eye display device 13100 for later display).
Quach, Chen, Hunt and Urey are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen, Hunt and Urey with Urey’s teachings of the display comprises a spatial light modulator, since it would have been obvious to try from a finite number of display technologies known in the art that would have yield the same predictable result of display images.
Regarding Claim 9, Quach, Chen and Hunt teach the head-mounted display system of claim 1.
Quach, Chen and Hunt do not explicitly teach the display comprises an array of micro-LEDs, wherein each pixel of the warped rendered frame is associated with one or more of the micro-LEDs
However, Urey teaches wherein the display comprises an array of micro-LEDs, wherein each pixel of the warped rendered frame is associated with one or more of the micro-LEDs (see Fig. 113, para. [0116]-[0117], para. [0128]-[0129], and para. [0355]. For each video frame, the data on the SLM is a computer-generated holographic image of the virtual scene. The data on the SLM is computed and fed by a computer unit FIG. 113 shows a perspective view of a near-to-eye display device that includes a LED array).
Quach, Chen, Hunt and Urey are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen and Hunt with Urey’s teachings, since it would have been obvious to try from a finite number of display technologies known in the art that would have yield the same predictable result of display images.
Regarding Claim 11, Quach, Chen, Hunt and Urey teach the head-mounted display system of claim 9.
Urey further teaches wherein the display is configured to update the array of micro-LEDs by providing a scanned update (see Figs. 40-42, para. [0187]-[0188], para. [0259]. FIG. 41 shows a moving platform upon which an SLM bar. The entire platform 4010 can move up and down periodically to scan the vertical direction. The display is considered see-through since the moving platform does not continuously block any part of the user's FOV, but does so only for a short duration of time. Both the SLM bar and the LED bar have high refresh rates. Computations for embodiments that include moving SLM bar (e.g., FIGS. 40, 42) are the same with the exception that the SLM is partitioned into a number of slices, and the entire “2D actual digital SLM image” is displayed slice by slice in a time sequential manner depending on the scan location of the SLM bar. The scan is completed in the frame time reserved for the “2D actual digital SLM image”).
Quach, Chen, Hunt and Urey are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen, Hunt and Urey with Urey’s teachings, since it would have been obvious to try from a finite number of display technologies known in the art that would have yield the same predictable result of display images.
Regarding Claim 12, Quach, Chen, Hunt and Urey teach the head-mounted display system of claim 11.
Urey further teaches wherein the scanned update comprises a sequential updating of individual pixels (see para. [0504] and para. [0516]. a spatial light modulator illuminated by the array of point light sources in a time sequential manner).
Quach, Chen, Hunt and Urey are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen, Hunt and Urey with Urey’s teachings, since it would have been obvious to try from a finite number of display technologies known in the art that would have yield the same predictable result of display images.
Regarding Claim 13, Quach, Chen, Hunt and Urey teach the head-mounted display system of claim 11.
Urey further teaches wherein the scanned update comprises sequential updating of groups of pixels at a same time (see para. [0159], para. [0512], para. [0516]. Some embodiments use a second group of point light sources interspersed with the existing group, such that the second group again divides the SLM surface into non-overlapping regions, but this time with boundaries falling into the middle of the regions formed by the first group of light sources. In these embodiments, the first and second groups of light sources are turned on in a time sequential manner).
Quach, Chen, Hunt and Urey are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen, Hunt and Urey with Urey’s teachings, since it would have been obvious to try from a finite number of display technologies known in the art that would have yield the same predictable result of display images.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Quach (US 9858637 B1) in view of Chen (US 20190149809 A1) in view of Hunt (US 10785471 B1) in view of Urey (US 20180003981 A1), further in view of Zalewski (US 20150348327 A1).
Regarding Claim 10, Quach, Chen, Hunt and Urey teach the head-mounted display system of claim 9.
Urey further teaches the “pixels” comprises an array of micro-LEDs (see Figs. 113-114, and para. [0355]-[00359]. The data on the SLM is computed and fed by a computer unit FIG. 113 shows a perspective view of a near-to-eye display device that includes a LED array).
Quach, Chen, Hunt and Urey are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen, Hunt and Urey with Urey’s teachings, since it would have been obvious to try from a finite number of display technologies known in the art that would have yield the same predictable result of display images.
Quach, Chen, Hunt and Urey do not explicitly teach wherein the display is configured to update the “pixels” globally for each warped rendered frame output by the display.
However, Zalewski teaches wherein the display is configured to update the “pixels” globally for each warped rendered frame output by the display (see para. [0143]. A video stream may include different types of video frames. For example, the H.264 standard includes a “P” frame and a “I” frame. I-frames include information to refresh all macro blocks/pixels on a display device, while P-frames include information to refresh a subset thereof. P-frames are typically smaller in data size than are I-frames).
Quach, Chen, Hunt, Urey and Zalewski are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the head mounted display system disclosed by Quach, Chen, Hunt and Urey with Zalewski’s teachings, since it would have allowed the used of different types of video frames.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Quach (US 9858637 B1) in view of Chen (US 20190149809 A1) in view of Hunt (US 10785471 B1), further in view of Zalewski (US 20150348327 A1).
Regarding Claim 15, Quach, Chen and Hunt teach the head-mounted display system of claim 1.
Quach further teaches wherein the orientation sensor is an “one or more accelerometers” and “one or more gyroscopes” (see col. 2 lines 46-60, col. 4 lines 9-38, col. 6 lines 28-43. As illustrated in FIG. 2, the sensor(s) enable the system 100 to track the rotational and translational movement of a user (e.g., the user's head, eyes, and/or other body parts) within three-dimensional space using six degrees of freedom (6DOF). As known in the art, the term 6DOF refers to the freedom of movement of the user in three-dimensional space. One or more accelerometer sensors 126 may sense changes in the magnitude and/or direction of the proper acceleration of the user's head and/or other body parts. One or more gyroscope sensors 128 may be used to track angular motion of the user's head and/or other body parts).
Quach, Chen and Hunt do not explicitly disclose wherein the orientation sensor is an inertial measurement unit.
However, Zalewski teaches wherein the orientation sensor is an inertial measurement unit (see para. [0015]. The system includes an inertial sensor of the housing. The inertial sensor provides movement and orientation data of the housing).
Quach, Chen, Hunt and Zalewski are related to head mounted displays and virtual reality, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness substituting Quach, Chen and Hunt orientation sensor with Zalewski’s inertial sensor, since is a simple substitution of known sensors that would have yield the same predictable result of sensing orientation.
Claims 19 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Quach (US 9858637 B1) in view of Chen (US 20190149809 A1) in view of Hunt (US 10785471 B1), further in view of Baran (US 20170085867 A1), further in view of Morein (US 20200258482 A1).
Regarding Claim 19, Quach, Chen, Hunt and Baran teach the system of claim 16.
Quach, Chen, Hunt and Baran do not explicitly teach wherein the display comprises micro-LEDs.
However, Morein teaches wherein the display comprises micro-LEDs (see Fig. 5 and para. [0066]. The physical display 514 is any type of physical device that includes functionality to show a combined image. For example, the physical display 514 may be a liquid crystal display (LCD), light emitting diode display (LED), micro LED, organic light-emitting diode display (OLED), liquid crystal on silicon (LCos), or other technology).
Quach, Chen, Hunt Baran and Morein are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the display system disclosed by Quach, Chen, Hunt and Baran with Morein’s teachings of the display comprising micro-LEDs, since it would have been obvious to try from a finite number of display technologies known in the art that would have yield the same predictable result of display images.
Regarding Claim 21, Quach, Chen, Hunt and Baran teach the method of claim 20.
Quach, Chen, Hunt and Baran do not explicitly disclose wherein the display comprises micro-LEDs.
However, Morein teaches wherein the display comprises micro-LEDs (see Fig. 5 and para. [0066]. The physical display 514 is any type of physical device that includes functionality to show a combined image. For example, the physical display 514 may be a liquid crystal display (LCD), light emitting diode display (LED), micro LED, organic light-emitting diode display (OLED), liquid crystal on silicon (LCos), or other technology).
Quach, Chen, Hunt, Baran and Morein are related to head mounted displays, thus one of ordinary skill in the art, before the effective filing date of the claimed invention, would have recognized the obviousness of modifying the method disclosed by Quach, Chen, Hunt and Baran with Morein’s teachings of the display comprising micro-LEDs, since it would have been obvious to try from a finite number of display technologies known in the art that would have yield the same predictable result of display images.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IVELISSE MARTINEZ QUILES whose telephone number is (571)270-7618. The examiner can normally be reached Monday thru Friday; 1:00 PM to 5:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Temesghen Ghebretinsae can be reached at 571-272-3017. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IM/Examiner, Art Unit 2626
/TEMESGHEN GHEBRETINSAE/Supervisory Patent Examiner, Art Unit 2626 6/1/26B
Prosecution Insights