DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The objections to the claims and the specifications have been withdrawn in view of the Applicants amendments filed on 07/02/2026
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Marks et al. (U.S. Pub. No. 20190060756) in view of Williams et al. (U.S. Pub. No. 20150029218).
Regarding claim 1, Marks discloses apparatus comprising (para 2, “The present disclosure relates to virtual reality (VR) environment content presented in head mounted displays (HMDs), and methods and systems for integrating access to VR environments by spectating users and methods for enabling users to spectate dynamically changing environments being navigated by HMD users that interact in VR environments, and associated apparatus and methods”): receiving circuitry to receive tracking information indicative of tracked poses of a head mountable display "HMD" by a user wearing the HMD (para 72, “The motion input 318 can be processed from a motion sensor 300 included in the HMD 102, or from image capture device 108 as it captures images of the HMD 102.”; also, para 58, “the HMD 102 may include one or more lights which can be tracked to determine the location and orientation of the HMD 102”; also, para 54, “A user 100 is shown wearing a head-mounted display (HMD) 102.”); rendering circuitry to render images for display having viewpoints corresponding to the tracked poses of the HMD (para 73, “In the illustrated embodiment, a video rendering module 322 is defined to render a video stream for presentation on the HMD 102.”; also, para 72, “The game engine 320 outputs game state data to various rendering modules which process the game state data to define content which will be presented to the user.”); trajectory calculation circuitry to calculate a smoothed trajectory for position of the HMD in dependence on the tracking information (para 87, “a gearing ratio would be applied to the movement of the virtual camera view, such that the virtual camera moves at a rate that trails the movement of the actual view of the HMD player.”; also, para 87, “By using dynamic gearing, the movement of the virtual camera view can be smoothed out, so as to provide a more pleasant viewing experience for the spectator view, even when the HMD moves fast or erratic.”); first image re-projection circuitry to display device different from the HMD (para 47, “Furthermore, the views provided to spectators can also be provided in such a way that prevents fast movements of the content to the spectators, as would be typical if the same exact view generated by the HMD player were to be shown to the spectator.”; also, para 26, “In some embodiments, the device of the spectator is one of an HMD used by the spectator or a television screen used by the spectator, or a computer screen used by the spectator, or hand-held device screen used by the spectator”), in which the first image re-projection circuitry is configured to smoothed trajectory for the position of the HMD (para 47, “In one configuration, the spectator views can follow the movements of the HMD player, but with a smoother delay, so as to prevent disorientation of the spectator.”; also, para 87, “the movement of the virtual camera view can be set to follow the movement of the HMD player's head, with a delay.”); and output circuitry to output the first re-projected images for display by the another display device (para 26, “In some embodiments, the device of the spectator is one of an HMD used by the spectator or a television screen used by the spectator, or a computer screen used by the spectator, or hand-held device screen used by the spectator.”). Marks does not disclose re-project at least some of the rendered images according to a first re-projection operation to obtain first re-projected images, and re-project a given rendered image having a given viewpoint position of the HMD for a different viewpoint position for display the another display device.
However, in a similar field of endeavor, Williams discloses re-project at least some of the rendered images according to a first re-projection operation to obtain first re-projected images (para 23, “In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD (e.g., a predicted pose of the HMD 10 ms or 20 ms in the future), generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image or concurrent with the pre-rendered image being generated, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD.”), and re-project a given rendered image having a given viewpoint position of the HMD for a different viewpoint position for display the another display device (para 23, “The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image.”; also, para 25, “The image reprojection techniques may include per pixel reprojection (e.g., where each pixel of a rendered image is reprojected based on an updated pose), multi-plane homography (e.g., where multiple rendered images associated with multiple planes within a 3D scene are used to generate the composite updated image), single plane homography (e.g., where a single rendered image associated with a single plane within a 3D scene is used to generate the updated image), affine homography, and pixel offset based adjustments.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of the apparatus, the receiving circuitry receiving tracking information indicative of tracked poses of an HMD worn by a user, the rendering circuitry rendering images having viewpoints corresponding to the tracked poses, the trajectory calculation circuitry calculating a smoothed trajectory for position of the HMD, the provision of the smoothed spectator view to another display device different from the HMD corresponding to the smoothed trajectory, and the output circuitry outputting that view, with the features of Williams's invention of re-projecting a given rendered image having a given viewpoint position for a different viewpoint position by a homographic transformation and pixel offset adjustment of the rendered image. A person of ordinary skill would have applied Williams's known image re-projection technique to obtain Marks's smoothed spectator viewpoint from images already rendered for the HMD wearer's tracked poses, because Williams teaches that an image rendered for one viewpoint can be transformed to a different viewpoint by a homographic transformation and pixel offset adjustment, and using that known technique to generate the spectator view yields the predictable result of producing Marks's smoothed trailing viewpoint without a second full rendering pass from a separate virtual camera, reducing rendering workload while achieving the pleasant, non-disorienting spectator presentation Marks seeks.
Regarding claim 2, Marks as modified by Williams discloses the apparatus according to claim 1, wherein Marks further discloses the trajectory calculation circuitry is configured to calculate the smoothed trajectory in dependence on a smoothing calculation (para 87, “a gearing ratio would be applied to the movement of the virtual camera view, such that the virtual camera moves at a rate that trails the movement of the actual view of the HMD player.”; also, para 87, “By using dynamic gearing, the movement of the virtual camera view can be smoothed out, so as to provide a more pleasant viewing experience for the spectator view, even when the HMD moves fast or erratic.”; also, para 87, “The gearing ratio may be modified dynamically by the computer, the game, and/or program executing the views for the spectator into the virtual reality scenes.”) using the tracking information received from the receiving circuitry (para 72, “The motion input 318 can be processed from a motion sensor 300 included in the HMD 102, or from image capture device 108 as it captures images of the HMD 102.”; also, para 87, “the movement of the virtual camera view can be set to follow the movement of the HMD player's head, with a delay.”).
Regarding claim 3, Marks as modified by Williams discloses the apparatus according to claim 2, wherein Marks further discloses the trajectory calculation circuitry is configured to calculate the smoothed trajectory in dependence on a moving average calculation (para 87, “The delay can be similar to a conceptual rubber band that links the virtual camera view to the movements of the HMD. That is, if the user's head moves to the left quickly, the spectator's virtual camera view will move to the left slower, with the delay similar to the way an object would trail when connected by a rubber band to a moving object ahead of it.”) using the tracking information received from the receiving circuitry (para 72, “The motion input 318 can be processed from a motion sensor 300 included in the HMD 102, or from image capture device 108 as it captures images of the HMD 102.”; also, para 87, “the movement of the virtual camera view can be set to follow the movement of the HMD player's head, with a delay.”).
Regarding claim 4, Marks as modified by Williams discloses the apparatus according to claim 2, wherein Marks further disclose the smoothed trajectory comprises a plurality of calculated positions for the HMD for a period of time and the trajectory calculation circuitry is configured to calculate a next position for the smoothed trajectory in dependence on a weighted average of a most recent calculated position for the smoothed trajectory and a current tracked position of the HMD (para 50, “The virtual camera that provides that view can move and follow the view of the HMD player, with a slight delay behind the movements of the HMD player. “; also, para 87, “The delay can be similar to a conceptual rubber band that links the virtual camera view to the movements of the HMD. That is, if the user's head moves to the left quickly, the spectator's virtual camera view will move to the left slower, with the delay similar to the way an object would trail when connected by a rubber band to a moving object ahead of it. “) greater weighting for the most recent calculated position.
However, in a similar field of endeavor, Williams discloses with a greater weighting for the most recent calculated position (para 24, “In some embodiments, the predicted pose may be determined based on a current position and orientation of the HMD and an acceleration and a velocity of the HMD immediately prior to determining the predicted pose (e.g., by extrapolating the predicted pose based on movement of the HMD 5 ms or 10 ms prior to determining the predicted pose).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of calculating a next position for the smoothed trajectory in dependence on a weighted average of a most recent calculated position and a current tracked position of the HMD with the features of Williams's invention of weighting a most recent movement of the HMD when determining a subsequent pose. A person of ordinary skill would have weighted the most recent calculated position more heavily when computing Marks's trailing virtual camera position because Williams teaches that a subsequent pose is determined from the position, velocity, and acceleration of the HMD immediately prior to the determination, and applying that emphasis on the most recent samples yields the predictable result of a spectator trajectory that follows the player's recent motion while still lagging behind the instantaneous head pose, which is the smoothed trailing behavior Marks describes.
Regarding claim 5, Marks as modified by Williams discloses the apparatus according to claim 1, wherein Marks further discloses the trajectory calculation circuitry is configured to on at least a portion of the smoothed trajectory (para 87, “a gearing ratio would be applied to the movement of the virtual camera view, such that the virtual camera moves at a rate that trails the movement of the actual view of the HMD player.”; also, para 87, “By using dynamic gearing, the movement of the virtual camera view can be smoothed out, so as to provide a more pleasant viewing experience for the spectator view, even when the HMD moves fast or erratic.”), calculate an extrapolated trajectory for the position of the HMD, and the extrapolated trajectory comprising one or more predicted positions for the HMD.
However, in a similar field of endeavor, Williams disclose calculate an extrapolated trajectory for the position of the HMD, and the extrapolated trajectory comprising one or more predicted positions for the HMD (para 24, “In some embodiments, the predicted pose may be determined based on a current position and orientation of the HMD and an acceleration and a velocity of the HMD immediately prior to determining the predicted pose (e.g., by extrapolating the predicted pose based on movement of the HMD 5 ms or 10 ms prior to determining the predicted pose).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of calculating a smoothed trajectory for the position of the HMD with the features of Williams's invention of calculating an extrapolated trajectory for the position of the HMD comprising one or more predicted positions for the HMD. A person of ordinary skill would have extrapolated one or more predicted positions of the HMD from at least a portion of Marks's smoothed trajectory because Williams teaches extrapolating a predicted pose associated with a future position of the HMD from the current position, velocity, and acceleration of the HMD, and extrapolating predicted positions from Marks's smoothed trajectory yields the predictable result of anticipating the HMD position so the spectator view can be prepared in advance, reducing perceived latency without altering the smoothed presentation.
Regarding claim 6, Marks as modified by Williams discloses the apparatus according to claim 4, wherein Marks further discloses the first image re-projection circuitry is configured to re-project the given rendered image having the given viewpoint position for the different viewpoint position, in which the different viewpoint position corresponds to one of: a calculated position associated with the smoothed trajectory (para 87, “Still further, the movement of the virtual camera view can be set to follow the movement of the HMD player's head, with a delay.”); predicted position for the HMD.
However, in a similar field of endeavor, Williams discloses a predicted position for the HMD (para 23, “In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD (e.g., a predicted pose of the HMD 10 ms or 20 ms in the future), generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image or concurrent with the pre-rendered image being generated, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of re-projecting the given rendered image for a different viewpoint position corresponding to a calculated position associated with the smoothed trajectory with the features of Williams's invention of a predicted position for the HMD. A person of ordinary skill would have used a predicted HMD position as the re-projection target because Williams teaches re-projecting a rendered image to a predicted future pose of the HMD, and offering the predicted position as an alternative re-projection target in Marks yields the predictable result of a re-projected view aligned to where the HMD is anticipated to be, which is one of the two recited alternatives.
Regarding claim 7, Marks as modified by Williams discloses the apparatus according to claim 1, wherein Marks further disclose the first image re-projection circuitry is configured to re-project the given rendered image having the given viewpoint position for the different viewpoint position corresponding to a calculated position on the smoothed trajectory for a respective time that is between a start time of a rendering operation for rendering the given rendered image (para 87, “Still further, the movement of the virtual camera view can be set to follow the movement of the HMD player's head, with a delay.”) time at which the first image re-projection operation is to be performed for the given rendered image.
However, in a similar field of endeavor, Williams disclose and a time at which the first image re-projection operation is to be performed for the given rendered image (para 23, “The displayed images may include late stage graphical adjustments of pre-rendered scenes (i.e., forward predicted scenes that are rendered at the rendering frame rate) in order to incorporate higher frequency pose estimates.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of re-projecting the given rendered image for a viewpoint position corresponding to a calculated position on the smoothed trajectory with the features of Williams's invention of performing the re-projection as a late stage adjustment between rendering and display. A person of ordinary skill would have selected the re-projection time to fall between the start of rendering and the time the re-projection is performed because Williams teaches late stage graphical adjustment of pre-rendered scenes to incorporate higher frequency pose estimates, and timing the re-projection in that interval yields the predictable result of using the most current trajectory position available when the re-projected spectator image is produced.
Regarding claim 8, Marks as modified by Williams discloses the apparatus according to claim 1, wherein Marks further disclose the trajectory calculation circuitry is configured to calculate a smoothed trajectory for the pose of the HMD in dependence on the tracking information, the smoothed trajectory for the pose of the HMD comprising a plurality of calculated poses for the HMD for a period of time (para 87, “The gearing ratio may be modified dynamically by the computer, the game, and/or program executing the views for the spectator into the virtual reality scenes.”; also, para 87, “a gearing ratio would be applied to the movement of the virtual camera view, such that the virtual camera moves at a rate that trails the movement of the actual view of the HMD player.”; also, para 87, “the movement of the virtual camera view can be set to follow the movement of the HMD player's head, with a delay.”), and the first image re-projection circuitry is configured to smoothed trajectory for the pose of the HMD (para 47, “In one configuration, the spectator views can follow the movements of the HMD player, but with a smoother delay, so as to prevent disorientation of the spectator.”; also, para 87, “By using dynamic gearing, the movement of the virtual camera view can be smoothed out, so as to provide a more pleasant viewing experience for the spectator view, even when the HMD moves fast or erratic.”). Marks does not disclose re-project the given rendered image having a given viewpoint pose for a different viewpoint pose.
However, in a similar field of endeavor, Williams disclose re-project the given rendered image having a given viewpoint pose for a different viewpoint pose (para 23, “In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD (e.g., a predicted pose of the HMD 10 ms or 20 ms in the future), generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image or concurrent with the pre-rendered image being generated, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD.”; also, para 23, “The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of calculating a smoothed trajectory for the pose of the HMD comprising a plurality of calculated poses for the HMD for a period of time and a different viewpoint pose corresponding to the smoothed trajectory for the pose of the HMD with the features of Williams's invention of re-projecting the given rendered image having a given viewpoint pose for a different viewpoint pose by a homographic transformation and pixel offset adjustment of the rendered image. A person of ordinary skill would have applied Williams's known image re-projection technique to obtain Marks's smoothed pose trajectory viewpoint from images already rendered for the HMD wearer's tracked poses, because Williams teaches transforming a rendered image having a given viewpoint pose to a different viewpoint pose by a homographic transformation and pixel offset adjustment, and using that known technique to generate the spectator view yields the predictable result of producing Marks's smoothed pose based viewpoint without a second full rendering pass from a separate virtual camera.
Regarding claim 9, Marks as modified by Williams discloses the apparatus according to claim 1, wherein Marks further discloses comprising projection circuitry configured to output circuitry is configured to output the second re-projected images for display by the HMD (para 68, “In one embodiment, the content being displayed in the HMD 102 is shared to the second screen 107.”). Marks does not disclose second image re-project at least some of the rendered images according to a second re-projection operation to obtain second re-projected images for display by the HMD.
However, in a similar field of endeavor, Williams disclose re-project at least some of the rendered images according to a second re-projection operation to obtain second re-projected images for display by the HMD (para 23, “In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD (e.g., a predicted pose of the HMD 10 ms or 20 ms in the future), generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image or concurrent with the pre-rendered image being generated, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD.”; also, para 23, “The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image.”; also, para 25, “The image reprojection techniques may include per pixel reprojection (e.g., where each pixel of a rendered image is reprojected based on an updated pose), multi-plane homography (e.g., where multiple rendered images associated with multiple planes within a 3D scene are used to generate the composite updated image), single plane homography (e.g., where a single rendered image associated with a single plane within a 3D scene is used to generate the updated image), affine homography, and pixel offset based adjustments.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of displaying content in the HMD and sharing content to a second screen with the features of Williams's invention of re-projecting rendered images according to a re-projection operation to obtain re-projected images for display by the HMD. A person of ordinary skill would have added a second re-projection operation directed to the HMD display because Williams teaches re-projecting a rendered image to an updated pose and displaying the updated image on the HMD, and providing that second re-projection alongside the spectator re-projection yields the predictable result of a world-locked, low-latency view for the HMD wearer while the first re-projection supplies the smoothed spectator view.
Regarding claim 10, discloses the apparatus according to claim 9, wherein the second image re-projection circuitry is configured to re-project the given rendered image according to the second re-projection operation to (para 68, “In one embodiment, the content being displayed in the HMD 102 is shared to the second screen 107.”) given rendered image for a current viewpoint corresponding to a current tracked pose of the HMD.
However, in a similar field of endeavor, Williams disclose re-project the given rendered image for a current viewpoint corresponding to a current tracked pose of the HMD (para 24, ‘The updated pose may be determined based on updated pose information that is provided to a pose tracker at a higher frequency than the rendering frame rate.”; also, para 23, “In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD (e.g., a predicted pose of the HMD 10 ms or 20 ms in the future), generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image or concurrent with the pre-rendered image being generated, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of sharing HMD content for display with the features of Williams's invention of re-projecting the given rendered image for a current viewpoint corresponding to a current tracked pose of the HMD. A person of ordinary skill would have targeted the second re-projection to the current tracked pose because Williams teaches determining an updated pose from pose information provided at a higher frequency than the rendering frame rate and re-projecting to that updated pose, and directing the HMD-facing re-projection to the current pose yields the predictable result of a stable, world-locked image for the HMD wearer.
Regarding claim 11, Marks as modified by Williams discloses the apparatus according to claim 1, wherein the rendering circuitry is configured to render at least some of the images for display having a field of view greater than a display field of view associated with the HMD (para 77, “Implementations of the present disclosure provide for rendering of a game replay with a very wide field of view to allow the spectator to move his head freely using an HMD and view the replay from novel vantage points.”; also, para 78, “It will be appreciated that the extremely wide field of view is in excess of the HMD's field of view, allowing for the spectator wearing the HMD to look around in the replay.”).
Regarding claim 12, Marks as modified by Williams discloses the apparatus according to claim 1, wherein the first image re-projection circuitry is configured to re-project one or more rendered images according to the first re- projection operation to obtain one or more first re-projected images for display by the another display device, in which one or more of the first re-projected images comprise a portion outside a field of view of one or more of the rendered images (para 77, “Implementations of the present disclosure provide for rendering of a game replay with a very wide field of view to allow the spectator to move his head freely using an HMD and view the replay from novel vantage points.”), first image re-projection circuitry is configured to use image data from one or more image sources for the portion outside the field of view.
However, in a similar field of endeavor, Williams disclose and wherein the first image re-projection circuitry is configured to use image data from one or more image sources for the portion outside the field of view (para 25, “The image reprojection techniques may include per pixel reprojection (e.g., where each pixel of a rendered image is reprojected based on an updated pose), multi-plane homography (e.g., where multiple rendered images associated with multiple planes within a 3D scene are used to generate the composite updated image), single plane homography (e.g., where a single rendered image associated with a single plane within a 3D scene is used to generate the updated image), affine homography, and pixel offset based adjustments.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of providing images having a portion outside the field of view of the rendered images with the features of Williams's invention of using image data from one or more image sources when re-projecting. A person of ordinary skill would have drawn on additional image data for the out-of-field portion because Williams teaches re-projection techniques including multi-plane homography that use multiple rendered images associated with multiple planes within a three-dimensional scene to generate the composite updated image, and using such additional image sources yields the predictable result of filling regions of the smoothed spectator view that fall outside the field of view of any single rendered image.
Regarding claim 13, Marks as modified by Williams discloses the apparatus according to claim 1, comprising detection circuitry to detect image-based motion vector information for at least some of the rendered images (para 47, “Furthermore, the views provided to spectators can also be provided in such a way that prevents fast movements of the content to the spectators, as would be typical if the same exact view generated by the HMD player were to be shown to the spectator.”), and first image re-projection circuitry is configured to re-project the given rendered image for the different viewpoint in dependence on the image-based motion vector information.
However, in a similar field of endeavor, Williams disclose wherein the first image re-projection circuitry is configured to re-project the given rendered image for the different viewpoint in dependence on the image-based motion vector information (para 23, “The displayed images may include late stage graphical adjustments of pre-rendered scenes (i.e., forward predicted scenes that are rendered at the rendering frame rate) in order to incorporate higher frequency pose estimates.”; also, para 24, “The updated image may comprise an image rotation, translation, resizing (e.g., stretching or shrinking), shifting, or tilting of at least a portion of the pre-rendered image in order to correct for differences between the predicted pose and the updated pose (e.g., to compensate for an incorrect pose prediction when generating the pre-rendered image).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of providing smoothed spectator views that prevent fast movements of the content with the features of Williams's invention of re-projecting a rendered image by rotation, translation, resizing, shifting, or tilting to adjust for differences between poses. A person of ordinary skill would have re-projected the rendered image in dependence on image-based motion information because Williams teaches adjusting a pre-rendered image by translation and shifting to correct for differences between poses and to incorporate higher frequency estimates, and driving the re-projection with detected motion vector information yields the predictable result of a spectator view that compensates for inter-frame image motion when warping to the smoothed viewpoint.
Regarding claim 14, Marks disclose a computer implemented method comprising (para 9, “A method, system, computer readable media and cloud systems are provided for generating views of a virtual reality environment for a spectator.”): receiving tracking information indicative of tracked poses of a head mountable display "HMD" by a user wearing the HMD (para 54, “A user 100 is shown wearing a head-mounted display (HMD) 102.”; also, para 72, “The motion input 318 can be processed from a motion sensor 300 included in the HMD 102, or from image capture device 108 as it captures images of the HMD 102.”; also, para 58, “the HMD 102 may include one or more lights which can be tracked to determine the location and orientation of the HMD 102”); rendering images for display having viewpoints corresponding to the tracked poses of the HMD (para 73, “In the illustrated embodiment, a video rendering module 322 is defined to render a video stream for presentation on the HMD 102.”; also, para 72, “The game engine 320 outputs game state data to various rendering modules which process the game state data to define content which will be presented to the user.”); calculating a smoothed trajectory for position of the HMD in dependence on the tracking information (para 87, “a gearing ratio would be applied to the movement of the virtual camera view, such that the virtual camera moves at a rate that trails the movement of the actual view of the HMD player.”; also, para 87, “By using dynamic gearing, the movement of the virtual camera view can be smoothed out, so as to provide a more pleasant viewing experience for the spectator view, even when the HMD moves fast or erraticHMD (para 47, “Furthermore, the views provided to spectators can also be provided in such a way that prevents fast movements of the content to the spectators, as would be typical if the same exact view generated by the HMD player were to be shown to the spectator.”; also, para 26, “In some embodiments, the device of the spectator is one of an HMD used by the spectator or a television screen used by the spectator, or a computer screen used by the spectator, or hand-held device screen used by the spectator.”); in which another display device corresponding to the smoothed trajectory for the position of the HMD (para 47, “In one configuration, the spectator views can follow the movements of the HMD player, but with a smoother delay, so as to prevent disorientation of the spectator.”; also, para 87, “the movement of the virtual camera view can be set to follow the movement of the HMD player's head, with a delay.”); and outputting the first re-projected images for display by the another display device (para 26, “In some embodiments, the device of the spectator is one of an HMD used by the spectator or a television screen used by the spectator, or a computer screen used by the spectator, or hand-held device screen used by the spectator.”). Marks does not disclose re-projecting at least some of the rendered images according to a first re-projection operation to obtain first re-projected images, and re-projecting a given rendered image having a given viewpoint position of the HMD according to the first re-projection operation comprises re-projecting the given rendered for a different viewpoint position.
However, in a similar field of endeavor, Williams discloses re-projecting at least some of the rendered images according to a first re-projection operation to obtain first re-projected images (para 23, “In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD (e.g., a predicted pose of the HMD 10 ms or 20 ms in the future), generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image or concurrent with the pre-rendered image being generated, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD.”), and re-projecting a given rendered image having a given viewpoint position of the HMD according to the first re-projection operation comprises re-projecting the given rendered for a different viewpoint position (para 23, “The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image.”; also, para 25, “The image reprojection techniques may include per pixel reprojection (e.g., where each pixel of a rendered image is reprojected based on an updated pose), multi-plane homography (e.g., where multiple rendered images associated with multiple planes within a 3D scene are used to generate the composite updated image), single plane homography (e.g., where a single rendered image associated with a single plane within a 3D scene is used to generate the updated image), affine homography, and pixel offset based adjustments.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of the computer implemented method, receiving tracking information indicative of tracked poses of an HMD worn by a user, rendering images having viewpoints corresponding to the tracked poses, calculating a smoothed trajectory for position of the HMD, providing the smoothed spectator view to another user of a display device different from the HMD corresponding to the smoothed trajectory, and outputting that view, with the features of Williams's invention of re-projecting a given rendered image having a given viewpoint position for a different viewpoint position by a homographic transformation and pixel offset adjustment of the rendered image. A person of ordinary skill would have applied Williams's known image re-projection technique to obtain Marks's smoothed spectator viewpoint from images already rendered for the HMD wearer's tracked poses, because Williams teaches that an image rendered for one viewpoint can be transformed to a different viewpoint by a homographic transformation and pixel offset adjustment, and using that known technique to generate the spectator view shown to another user yields the predictable result of producing Marks's smoothed trailing viewpoint without a second full rendering pass from a separate virtual camera.
Regarding claim 15, Marks discloses a non-transitory computer-readable storage medium storing computer software which, when executed by a computer, causes the computer to a method comprising (para 145, “Video Server System 1420 comprises at least a Video Source 1430, an I/O Device 1445, a Processor 1450, and non-transitory Storage 1455.”; also, para 153, “Storage 1455 includes non-transitory analog and/or digital storage devices.”): receiving tracking information indicative of tracked poses of a head mountable display "HMD" by a user wearing the HMD (para 54, “A user 100 is shown wearing a head-mounted display (HMD) 102.”; also, para 72, “The motion input 318 can be processed from a motion sensor 300 included in the HMD 102, or from image capture device 108 as it captures images of the HMD 102.”; also, para 58, “the HMD 102 may include one or more lights which can be tracked to determine the location and orientation of the HMD 102”); rendering images for display having viewpoints corresponding to the tracked poses of the HMD (para 73, “In the illustrated embodiment, a video rendering module 322 is defined to render a video stream for presentation on the HMD 102.”; also, para 72, “The game engine 320 outputs game state data to various rendering modules which process the game state data to define content which will be presented to the user.”); calculating a smoothed trajectory for position of the HMD in dependence on the tracking information (para 87, “a gearing ratio would be applied to the movement of the virtual camera view, such that the virtual camera moves at a rate that trails the movement of the actual view of the HMD player.”; also, para 87, “By using dynamic gearing, the movement of the virtual camera view can be smoothed out, so as to provide a more pleasant viewing experience for the spectator view, even when the HMD moves fast or erratic.”); the HMD (para 47, “Furthermore, the views provided to spectators can also be provided in such a way that prevents fast movements of the content to the spectators, as would be typical if the same exact view generated by the HMD player were to be shown to the spectator.”; also, para 26, “In some embodiments, the device of the spectator is one of an HMD used by the spectator or a television screen used by the spectator, or a computer screen used by the spectator, or hand-held device screen used by the spectator.”); in which another display device corresponding to the smoothed trajectory for the position of the HMD (para 47, “In one configuration, the spectator views can follow the movements of the HMD player, but with a smoother delay, so as to prevent disorientation of the spectator.”; also, para 87, “the movement of the virtual camera view can be set to follow the movement of the HMD player's head, with a delay.”); and outputting the first re-projected images for display by the another display device (para 26, “In some embodiments, the device of the spectator is one of an HMD used by the spectator or a television screen used by the spectator, or a computer screen used by the spectator, or hand-held device screen used by the spectator.”). Marks does not disclose re-projecting at least some of the rendered images according to a first re-projection operation to obtain first re-projected images, and re-projecting a given rendered image having a given viewpoint position of the HMD according to the first re-projection operation comprises re-projecting the given rendered image for a different viewpoint position.
However, in a similar field of endeavor, Williams discloses re-projecting at least some of the rendered images according to a first re-projection operation to obtain first re-projected images (para 23, “In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD (e.g., a predicted pose of the HMD 10 ms or 20 ms in the future), generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image or concurrent with the pre-rendered image being generated, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD.”), and re-projecting a given rendered image having a given viewpoint position of the HMD according to the first re-projection operation comprises re-projecting the given rendered image for a different viewpoint position (para 23, “The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image.”; also, para 25, “The image reprojection techniques may include per pixel reprojection (e.g., where each pixel of a rendered image is reprojected based on an updated pose), multi-plane homography (e.g., where multiple rendered images associated with multiple planes within a 3D scene are used to generate the composite updated image), single plane homography (e.g., where a single rendered image associated with a single plane within a 3D scene is used to generate the updated image), affine homography, and pixel offset based adjustments.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Marks's invention of the non-transitory computer-readable storage medium storing software that receives tracking information indicative of tracked poses of an HMD worn by a user, renders images having viewpoints corresponding to the tracked poses, calculates a smoothed trajectory for position of the HMD, provides the smoothed spectator view to another user of a display device different from the HMD corresponding to the smoothed trajectory, and outputs that view, with the features of Williams's invention of re-projecting a given rendered image having a given viewpoint position for a different viewpoint position by a homographic transformation and pixel offset adjustment of the rendered image. A person of ordinary skill would have applied Williams's known image re-projection technique to obtain Marks's smoothed spectator viewpoint from images already rendered for the HMD wearer's tracked poses, because Williams teaches that an image rendered for one viewpoint can be transformed to a different viewpoint by a homographic transformation and pixel offset adjustment, and using that known technique to generate the spectator view shown to another user yields the predictable result of producing Marks's smoothed trailing viewpoint without a second full rendering pass from a separate virtual camera.
Response to Arguments
Applicant's arguments filed 07/02/2026 have been fully considered but they are not persuasive.
Applicant argues, at Remarks pages 9-10, that the combination of Marks and Williams teaches away because Williams re-projects to an updated or actual pose to reduce latency and world-lock the scene, whereas the claimed re-projection targets a smoothed trajectory that intentionally lags, and that applying Williams to Marks would correct the smoothed view back toward the player's erratic actual pose, defeating the purpose of the smoothing taught by Marks and producing the motion sickness the present invention avoids.
This argument is not persuasive because it attacks Williams individually rather than the rejection as set out. The rejection does not rely on Williams for the smoothed trajectory, for the spectator, or for the display device different from the HMD. Marks supplies each of those features. Marks teaches that the spectator views can follow the movements of the HMD player, but with a smoother delay, so as to prevent disorientation of the spectator, that a gearing ratio would be applied to the movement of the virtual camera view, such that the virtual camera moves at a rate that trails the movement of the actual view of the HMD player, and that the device of the spectator is one of an HMD used by the spectator or a television screen used by the spectator, or a computer screen used by the spectator, or hand-held device screen used by the spectator. Williams is relied upon only for the re-projection mechanism, namely that an updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image and the per pixel, multi-plane, single plane, and affine homography techniques. That mechanism transforms a rendered image from the viewpoint at which it was rendered to whatever target viewpoint it is supplied, independent of whether that target is a current pose or a smoothed trajectory position. One cannot show nonobviousness by attacking references individually where the rejection is based on a combination of references. The test is what the combined teachings would have suggested to one of ordinary skill in the art, and the rejection need not bodily incorporate Williams's latency-correction purpose into Marks.
Applicant's premise that the Examiner equates the claimed smoothed trajectory with the updated pose of Williams is not the basis of this rejection. As set out above, Marks supplies the smoothed trajectory. The re-projection target fed to Williams's image transformation is Marks's smoothed trailing viewpoint, so the combination produces Marks's intended smoothed spectator view by warping an already-rendered frame rather than by rendering a second virtual camera view. Marks's purpose of a pleasant, non-disorienting spectator view is preserved by the combination, not defeated.
Applicant's argument further relies on features that are described in the specification but are not recited in claim 1. Applicant argues, at Remarks page 10, that the smoothed trajectory intentionally lags behind the actual pose to prevent nausea, that the claimed smoothed trajectory is a filtered version of the tracking information designed to remove the very high-frequency movements and jitters that Williams seeks to capture, and that re-projecting to the smoothed trajectory creates a "comfortable lag" or "rubber band" effect for the spectator. Claim 1 recites only trajectory calculation circuitry to calculate a smoothed trajectory for position of the HMD in dependence on the tracking information, and first image re-projection circuitry to re-project a given rendered image for a different viewpoint position corresponding to the smoothed trajectory. Claim 1 does not recite low-pass filtering, removal of high-frequency movement or jitter, any particular magnitude or duration of lag, the prevention of nausea, or any requirement that the re-projection avoid the current or actual pose of the HMD. Although claims are interpreted in light of the specification, limitations appearing only in the specification are not read into the claims, and arguments must be commensurate in scope with the claims as actually recited. Under the broadest reasonable interpretation, a smoothed trajectory reads on the gearing and dynamic gearing of Marks, which Marks expressly describes as smoothing out the movement of the virtual camera view to provide a more pleasant viewing experience for the spectator view even when the HMD moves fast or erratic, whether or not that smoothing is characterized as filtering or jitter removal.
Applicant's argument also depends on bodily incorporating Williams's latency-correction purpose into Marks. The contention that applying Williams to Marks would correct the smoothed view back toward the erratic actual pose of the user assumes that Williams's re-projection technique necessarily carries with it Williams's goal of aligning the image to the current or actual pose. Obviousness does not require that the secondary reference be bodily incorporated into the primary reference, and the test is not whether the features of Williams may be bodily incorporated into the structure of Marks but what the combined teachings would have suggested to one of ordinary skill in the art.
For these reasons the rejection of claims 1-15 under 35 U.S.C. 103 over Marks in view of Williams is maintained.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jai Li whose telephone number is (571)272-1170. The examiner can normally be reached Mon-Thu between 06:00-16:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAI W LI/Junior Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights