DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 6/3/2025 have been fully considered but they are not persuasive.
Applicant argues that the combination of Nishiyama, Kobayashi, Li, and Young does not explicitly teach the three-dimensional models shown in the generated image represent the objects at a moment corresponding to a peak of the degrees of attention, respectively.
In response, the examiner respectfully disagrees. Young teaches HMD 102 is configurable to display images configured with foveated rendering, wherein portions of images in a foveal region are displayed with high resolution and portions outside the foveal region are displayed with lower resolution. In particular, particle systems are simulated and/or rendered differently depending on whether the effect generated by the particle system is displayed inside or outside a foveal region of an image as displayed. For example, a particle system that is inside the foveal region is simulated and/or rendered using a full complement of particles in the particle system. A particle system that is outside the foveal region (i.e., in the peripheral region) is simulated and/or rendered using a subsystem of particles. As such, instead of computing all the particles in a particle system located in a peripheral region, only a subsystem of particles are simulated and/or rendered, thereby lessening the computation required for rendering the effect produced by the particle system in a corresponding image. Also, each image containing a particle system rendered using a subsystem of particles and displayed outside the foveal region require less defining data than images containing a particle system that is simulated and/or rendered using the full complement of particles of the particle system. In that manner, the total bandwidth for the sequence of video frames being displayed is reduced. [0030].
A video rendering module 183 is defined to render a video stream for presentation on the HMD 102. Foveated view renderer 190 is configured to render foveated images in conjunction with and/or independent of video rendering module 183. Additionally, the functionality provided by the foveated view renderer 190 may be incorporated within the video rendering module 183, in embodiments. In particular, foveated view renderer 190 is configured to perform foveated rendering, wherein portions of images in a foveal region with high resolution and portions outside the foveal region with lower resolution. More particularly, a particle system is simulated and/or rendered differently depending on the displayed location of the corresponding effect in relation to a foveal region of an image. For example, a particle system that is displayed inside the foveal region is simulated and/or rendered using a full complement of particles in the particle system. A particle system that is displayed in the peripheral region is simulated and/or rendered using a subsystem of particles. The foveated particle and/or fragment determiner 192 is configured to determine whether a particle and/or particle system being rendered is displayed inside or outside a foveal region. The particle system and/or subsystem simulator 195 is configured to simulate the effect using the full complement of particles in the particle system, or simulate the effect using the subsystem of particles in various embodiments. For example, for particle systems having limited particles (e.g., less than ten thousand), the full complement of particles in the particle system is simulated regardless of whether the effect is in the foveated or peripheral region. However when the effect generated by the particle system is displayed in the peripheral region the subsystem of particles is then rendered. In this case, the subsystem of particles may be generated by the random particle selector 196. Also, for particle systems having larger amounts of particles (e.g., more than ten thousand), the subsystem of particles is simulated when the effect generated by the particle system is displayed in the peripheral region, and thereafter rendered using the output from the same subsystem of particles. The subsystem of particles in this case may be determined by the particle clusterer and aggregator 198. The particle system and subsystem renderer 194 is configured to render the full complement of particles in a particle system when the effect generated by the particle system is displayed in the foveal region, and to render a subsystem of particles when the effect is displayed in a peripheral region. The particle scaler and color modifier 199 is configured to scale particles in the subsystem of particles, and modify the color of the particles to achieve the same or similar perceivable effect by the user of the of the particle system. For example, each particle in the subsystem is scaled to a larger size (e.g., based on a reduction ratio) and darkened (e.g., modifying alpha values based on the reduction ratio). Components and/or functions of the foveated view renderer 190 may be performed within a CPU or GPU, or combination thereof. [0039].
A gaze tracking camera 165 is included in the HMD 102 to enable tracking of the gaze of the user. Although only one gaze tracking camera 165 is included, it should be noted that more than one gaze tracking camera may be employed to track the gaze of the user. The gaze tracking camera captures images of the user's eyes, which are analyzed to determine the gaze direction of the user. In one embodiment, information about the gaze direction of the user can be utilized to affect the video rendering. For example, if a user's eyes are determined to be looking in a specific direction, then the video rendering for that direction can be prioritized or emphasized, such as by providing greater detail, higher resolution through foveated rendering as provided by foveated view renderer 190, higher resolution of a particle system effect displayed in the foveal region, lower resolution of a particle system effect displayed outside the foveal region, or faster updates in the region where the user is looking. It should be appreciated that the gaze direction of the user can be defined relative to the head mounted display, relative to a real environment in which the user is situated, and/or relative to a virtual environment that is being rendered on the head mounted display. [0042].
FIG. 3A illustrates an image 310 shown on a display 300, wherein the image includes a foveal region 310A of high resolution, wherein the foveal region corresponds to a center of the display, in accordance with one embodiment of the present disclosure. In particular, image 310 includes rows and columns of the letter “E” for simplicity and clarity. The image 310 is partitioned into multiple regions, including a foveal region 310A and a peripheral region 310B.
As shown, the foveal region 310A is static and corresponds to the center of the display 300. The foveal region 310A is assumed to be the region towards which the user mostly directs his or her gaze (e.g., using the fovea of the eye), such as when viewing graphics of a video game. Though the gaze of the user may occasionally be directed off center, the gaze is mostly directed to the center (to view the main content). In some cases, the image is designed to initially bring the gaze of the user off-center (e.g., to view an object of interest), but then to bring the gaze back to the center (e.g., by moving the object towards the foveal region 310A).
In particular, the portion or portions of any image, such as image 310, as displayed and located within the foveal region 310A will be rendered at higher resolution. For example, the graphics pipeline will render portions of the image in the foveal region 310A while minimizing the use of any techniques used to reduce computational complexity. In particular, for embodiments of the present invention, light sources affecting objects displayed using pixels corresponding to the foveal region 310A are individually computed within the graphics pipeline in order to determine each of their effects on the objects (e.g., color, texture, shadowing, etc. on polygons of the objects). Representative of the higher resolution, the letter “E” objects as displayed within the foveal region 310A are shown with clarity, vibrant color, and minimal blurriness. This is consistent with and takes advantage of the gaze of the user being directed towards the foveal region 310A on display 300.
In addition, the portion or portions of an image, such as image 310, as disposed and located in the peripheral region 310B will be rendered at lower resolution (e.g., lower than the resolution of the portions of the image and/or objects located in the foveal region 310A). The gaze of the user is not typically directed to the objects located in and/or displayed in the peripheral region 310B, as the main focus of the gaze is directed to the objects in the foveal region 310A. Consistent with real-life views into a scene, the objects in the peripheral region 310B are rendered with lower resolution, with enough detail so that the user is able to perceive moving objects (e.g., a human walking straight-legged instead of with knees bent) and with sufficient contrast within an object or between objects in the peripheral region 310B, for example. To achieve rendering at lower resolutions, the graphics pipeline may render portions of the image in the peripheral region 310B using computationally efficient techniques that reduce computational complexity. In particular, for embodiments of the present invention, particle systems are simulated and/or rendered differently depending on whether the effect generated by the particle system is displayed inside or outside a foveal region of an image as displayed. For example, a particle system that is inside the foveal region is simulated and/or rendered using a full complement of particles in the particle system. A particle system that is outside the foveal region (i.e., in the peripheral region) is simulated and/or rendered using a subsystem of particles. As such, instead of computing all the particles in a particle system located in a peripheral region, only a subsystem of particles are simulated and/or rendered, thereby lessening the computation required for rendering the effect produced by the particle system in a corresponding image. This reduces computational processing when rendering the overall image (e.g., through simulation and/or rendering of the particle system), and especially for particle systems rendered in the peripheral region 310B. Also, each image containing a particle system rendered using a subsystem of particles and displayed outside the foveal region require less defining data than images containing a particle system that is simulated and/or rendered using the full complement of particles of the particle system. As such, the sequence of video frames can be delivered (e.g., over wired or wireless connections) in real time with minimal or no latency because less data is being transmitted.
FIG. 3B illustrates an image 310′ shown on a display 300 and including a foveal region 310A′ of high resolution, wherein the foveal region corresponds to a location of the display towards which the user is directing his or her gaze, in accordance with one embodiment of the present disclosure. In particular, image 310′ is similar to image 310 shown in FIG. 3A and includes rows and columns of the letter “E” for simplicity and clarity. The image 310 is partitioned into multiple regions, including a foveal region 310A′ and a peripheral region 310B′.
As shown, the foveal region 310A′ is dynamically moving throughout display 300 depending on which direction the gaze of the user is directed towards. As previously described, the gaze may be tracked using gaze tracking camera 165 of HMD 102, for example. As such, the foveal region 310A′ may not necessarily correspond to the center of display 300, but instead correlates to the actual direction and focus of attention of the user within image 310′. That is, the foveal region 310A′ dynamically moves with the movement of the eye and/or eyes of the user.
As previously introduced, the portion or portions of any image, such as image 310′, as displayed and located within the foveal region 310A′ will be rendered at higher resolution by minimizing the use of any rendering techniques used to reduce computational complexity when rendering objects located in the foveal region 310A′. In particular, for embodiments of the present invention, light sources affecting objects displayed using pixels corresponding to the foveal region 310A′ are individually computed within the graphics pipeline in order to determine each of their effects on the objects (e.g., color, texture, shadowing, etc. on polygons of the objects). Representative of the higher resolution, the letter “E” objects as displayed within the foveal region 310A′ are shown with clarity, vibrant color, and minimal blurriness.
In addition, the portion or portions of an image, such as image 310′, as disposed and located in the peripheral region 310B′ will be rendered at lower resolution (e.g., lower than the resolution of the portions of the image and/or objects located in the foveal region 310A). As previously introduced, the gaze of the user is not typically directed to the objects located in and/or displayed in the peripheral region 310B′, as the main focus of the gaze is directed to the objects in the foveal region 310A′. As such, the objects in the peripheral region 310B′ are rendered with lower resolution, with enough detail so that the user is able to perceive moving objects (e.g., a human walking straight-legged instead of with knees bent) and with sufficient contrast within an object or between objects in the peripheral region 310B′, for example. To achieve rendering at lower resolutions, the graphics pipeline may render portions of the image in the peripheral region 310B′ using computationally efficient techniques that reduce computational complexity. In particular, as previous described for embodiments of the present invention, the simulation and/or rendering of a particle system is performed using a subsystem of particles when the effect of the particle system is displayed in a peripheral region of an image. [0060] – [0067].
FIG. 4A illustrates a system 400A implementing a graphics pipeline configured for foveated rendering including particle system simulation of a limited amount of particles (e.g., less than ten thousand) using a central processing unit (CPU) or graphics processing unit (GPU) and the rendering of the particle system in the GPU using the full system of particles or a subsystem of particles generated through sampling (e.g., random, nearest neighbor, similarity, etc.) depending on whether the drawn effect of the particle system is located within a foveal region or peripheral region, in accordance with one embodiment of the present disclosure. The graphics pipeline 400A is illustrative of the general process for rendering images using 3D (three dimensional) polygon rendering processes, but is modified to perform additional programmable elements within the pipeline to perform foveated rendering, such as simulating and/or rendering particle systems using a full complement of particles or a subsystem of particles depending on the location of the effect generated by the particle system with respect to a foveal region of a corresponding image. The graphics pipeline 400A for a rendered image outputs corresponding color information for each of the pixels in a display, wherein the color information may represent texture and shading (e.g., color, shadowing, etc.). Graphics pipeline 400A is implementable within the game console 106 of FIG. 1A, VR content engine 120 of FIG. 1B, client devices 106 of FIGS. 2A and 2B, and/or game title processing engine 211 of FIG. 2B. [0068].
the graphics pipeline of system 400A receives input geometries 405. For example, the input geometries 405 may include vertices within a 3D gaming world, and information corresponding to each of the vertices. A given object within the gaming world can be represented using polygons (e.g., triangles) defined by vertices, wherein the surface of a corresponding polygon is then processed through the graphics pipeline 400A to achieve a final effect (e.g., color, texture, etc.). Vertex attributes may include normal (e.g., which direction is the light in relation to the vertex), color (e.g., RGB—red, green, and blue triple, etc.), and texture coordinate/mapping information. For ease of illustration, the input geometries for the 3D gaming world are shown to be inputted to processor 401, though the geometries may also be partitioned such that geometries for the particle system are input to processor 401, and remaining geometries input to vertex shader 410 of the processor 402. For example, the input geometries may be input into a vertex buffer that can be shared between the processors 401 and 402. [0070].
FIG. 4B illustrates a graphics processor implementing a graphics pipeline configured for foveated rendering including particle system simulation of a large amount of particles (e.g., more than ten thousand), wherein the particle system is both simulated and then rendered using the full system of particles or a subsystem of particles generated through particle clustering and aggregation depending on whether the drawn effect of the particle system is located within a foveal region or peripheral region, in accordance with one embodiment of the present disclosure. The graphics pipeline 400B is illustrative of the general process for rendering images using 3D (three dimensional) polygon rendering processes, but is modified to perform additional programmable elements within the pipeline to perform foveated rendering, such as simulating and/or rendering particle systems using a full complement of particles or a subsystem of particles depending on the location of the effect generated by the particle system with respect to a foveal region of a corresponding image. The graphics pipeline 400B for a rendered image outputs corresponding color information for each of the pixels in a display, wherein the color information may represent texture and shading (e.g., color, shadowing, etc.). Graphics pipeline 400B is implementable within the game console 106 of FIG. 1A, VR content engine 120 of FIG. 1B, client devices 106 of FIGS. 2A and 2B, and/or game title processing engine 211 of FIG. 2B. The graphics pipeline 400B may contain similar components as the pipeline 400A of FIG. 4A, wherein like referenced numerals designate identical or corresponding parts. [0086].
In one embodiment, a method for implementing a graphics pipeline is disclosed. The method includes generating a system of particles creating an effect in a virtual scene, the system of particles comprising a plurality of particle geometries. The method further includes determining a subsystem of particles from the system of particles, the subsystem of particles comprising a subset of particle geometries taken from the plurality of particle geometries. The method includes determining a foveal region when rendering an image of the virtual scene, wherein the foveal region corresponds to where an attention of a user is directed. the method includes determining that at least one portion of the effect is located in the peripheral region for the image. The method includes rendering the subsystem of particles to generate the effect.
In still another embodiment, a computer system is disclosed. The computer system including a processor and memory, wherein the memory is coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method for implementing a graphics pipeline. The method includes generating a system of particles creating an effect in a virtual scene, the system of particles comprising a plurality of particle geometries. The method further includes determining a subsystem of particles from the system of particles, the subsystem of particles comprising a subset of particle geometries taken from the plurality of particle geometries. The method includes determining a foveal region when rendering an image of the virtual scene, wherein the foveal region corresponds to where an attention of a user is directed. the method includes determining that at least one portion of the effect is located in the peripheral region for the image. The method includes rendering the subsystem of particles to generate the effect.
In another embodiment, a non-transitory computer-readable medium storing a computer program for implementing a graphics pipeline is disclosed. The computer-readable medium includes program instructions for generating a system of particles creating an effect in a virtual scene, the system of particles comprising a plurality of particle geometries. The computer-readable medium includes program instructions for determining a subsystem of particles from the system of particles, the subsystem of particles comprising a subset of particle geometries taken from the plurality of particle geometries. The computer-readable medium includes program instructions for determining a foveal region when rendering an image of the virtual scene, wherein the foveal region corresponds to where an attention of a user is directed. The computer-readable medium includes program instructions for determining that at least one portion of the effect is located in the peripheral region for the image. The computer-readable medium includes program instructions for rendering the subsystem of particles to generate the effect. [0008] – [0010].
The foveal region 310A′ is dynamically moving throughout display 300 depending on which direction the gaze of the user is directed towards. As previously described, the gaze may be tracked using gaze tracking camera 165 of HMD 102, for example. As such, the foveal region 310A′ may not necessarily correspond to the center of display 300, but instead correlates to the actual direction and focus of attention of the user within image 310′. That is, the foveal region 310A′ dynamically moves with the movement of the eye and/or eyes of the user. [0065].
The objects in the foveal region corresponds to the objects at a moment corresponding to a peak of the degrees of attention.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-7 and 17-18 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1, 17, and 18 recite “…the three-dimensional models shown in the generated image represent the objects at a moment corresponding to a peak of the degrees of attention.” However, the specification does not support this limitation because the specification only describes three-dimensional models corresponding to the degrees of attention. It is silent about a peak of the degrees of attention.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3-5, and 17-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nishiyama (US 2011/0254973 A1) in view of Kobayashi et al. (US 2019/0266786 A1), Li et al. (US 2012/0081369 A1) and Young et al. (US 2018/0357810 A1).
Consider claim 1, Nishiyama teaches an information processing apparatus comprising: one or more hardware processors ([0104]); and one or more memories storing one or more programs configured to be executed by the one or more hardware processors ([0104]), the one or more programs including instructions for: obtaining viewpoint information regarding virtual viewpoints corresponding to virtual viewpoint images generated based on a plurality of captured images obtained by a plurality of imaging apparatuses performing image capturing from a plurality of directions ([0024] – [0031]); detecting a first object included in at least any of the plurality of captured images and included in a field of view corresponding to a virtual viewpoint identified based on the viewpoint information obtained by the obtaining unit ([0102]).
However, Nishiyama does not explicitly teach based on a detection result of the detection unit associated with a plurality of virtual viewpoints identified based on the viewpoint information obtained by the obtaining unit, generating object information associated with the number of virtual viewpoints of which the fields of view include the first object; and displaying the object information and the first object in such a manner that the object information and the first object are associated with each other.
Kobayashi teaches based on a detection result of the detection unit associated with a plurality of virtual viewpoints identified based on the viewpoint information obtained by the obtaining unit, generating object information associated with the number of virtual viewpoints of which the fields of view include the first object ([0049], [0051], [0072], [0076] – [0077], [0092] – [0095], [0111] – [0113], [0122] – [0127], [0146] – [0147], [0164] – [0166]); and displaying the object information and the first object in such a manner that the object information and the first object are associated with each other ([0155]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the known technique of generating object information because such incorporation would improve the accuracy of a depth image of a 3D model. [0014].
However, the combination of Nishiyama and Kobayashi does not explicitly teach generating, based on the viewpoint information, degrees of attention to objects associated with a number of virtual viewpoints of which fields of view include the objects; and generating an image that shows the objects together with the degree of attention corresponding to the objects.
Li teaches generating, based on the viewpoint information, degrees of attention to objects associated with a number of virtual viewpoints of which fields of view include the objects ([0017], [0021] – [0025], [0029] – [0030], [0037] – [0048], [0053] – [0056]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the known technique of generating degrees of attention to objects because such incorporation would allow determination of hot ranks and generation of recommendations. [0008].
Young teaches generating an image that shows the objects together with the degree of attention corresponding to the objects using three-dimensional models for the object, (display images configured with foveated rendering, wherein portions of images in a foveal region are displayed with high resolution and portions outside the foveal region are displayed with lower resolution. A particle system that is inside the foveal region is simulated and/or rendered using a full complement of particles in the particle system. A particle system that is outside the foveal region (i.e., in the peripheral region) is simulated and/or rendered using a subsystem of particles. [0030], [0039]. If a user's eyes are determined to be looking in a specific direction, then the video rendering for that direction can be prioritized or emphasized, such as by providing greater detail, higher resolution through foveated rendering as provided by foveated view renderer 190, higher resolution of a particle system effect displayed in the foveal region, lower resolution of a particle system effect displayed outside the foveal region, or faster updates in the region where the user is looking [0042]. The foveal region 310A′ is dynamically moving throughout display 300 depending on which direction the gaze of the user is directed towards. As previously described, the gaze may be tracked using gaze tracking camera 165 of HMD 102, for example. As such, the foveal region 310A′ may not necessarily correspond to the center of display 300, but instead correlates to the actual direction and focus of attention of the user within image 310′. That is, the foveal region 310A′ dynamically moves with the movement of the eye and/or eyes of the user. [0060] – [0067]; The graphics pipeline 400A is illustrative of the general process for rendering images using 3D (three dimensional) polygon rendering processes, but is modified to perform additional programmable elements within the pipeline to perform foveated rendering, such as simulating and/or rendering particle systems using a full complement of particles or a subsystem of particles depending on the location of the effect generated by the particle system with respect to a foveal region of a corresponding image. [0068], [0070], [0086]; determining a foveal region when rendering an image of the virtual scene, wherein the foveal region corresponds to where an attention of a user is directed. [0008] – [0010] and [0065]), wherein the three-dimensional models shown in the generated image represent the objects at a moment corresponding to a peak of the degrees of attention, respectively (display images configured with foveated rendering, wherein portions of images in a foveal region are displayed with high resolution and portions outside the foveal region are displayed with lower resolution. A particle system that is inside the foveal region is simulated and/or rendered using a full complement of particles in the particle system. A particle system that is outside the foveal region (i.e., in the peripheral region) is simulated and/or rendered using a subsystem of particles. [0030], [0039]. If a user's eyes are determined to be looking in a specific direction, then the video rendering for that direction can be prioritized or emphasized, such as by providing greater detail, higher resolution through foveated rendering as provided by foveated view renderer 190, higher resolution of a particle system effect displayed in the foveal region, lower resolution of a particle system effect displayed outside the foveal region, or faster updates in the region where the user is looking [0042]. The foveal region 310A′ is dynamically moving throughout display 300 depending on which direction the gaze of the user is directed towards. As previously described, the gaze may be tracked using gaze tracking camera 165 of HMD 102, for example. As such, the foveal region 310A′ may not necessarily correspond to the center of display 300, but instead correlates to the actual direction and focus of attention of the user within image 310′. That is, the foveal region 310A′ dynamically moves with the movement of the eye and/or eyes of the user. [0060] – [0067]; The graphics pipeline 400A is illustrative of the general process for rendering images using 3D (three dimensional) polygon rendering processes, but is modified to perform additional programmable elements within the pipeline to perform foveated rendering, such as simulating and/or rendering particle systems using a full complement of particles or a subsystem of particles depending on the location of the effect generated by the particle system with respect to a foveal region of a corresponding image. [0068], [0070], [0086]; determining a foveal region when rendering an image of the virtual scene, wherein the foveal region corresponds to where an attention of a user is directed. [0008] – [0010] and [0065]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the known technique of generating an image that shows the objects together with the degree of attention corresponding to the objects using three-dimensional models change when the degrees of attention corresponding to the objects fluctuate because such incorporation would allow the sequence of video frames to be delivered in real time with minimal or no latency by reducing the computation complexity. [0007].
Consider claim 3, Nishiyama teaches based on a virtual viewpoint image corresponding to the virtual viewpoints identified based on the obtained viewpoint information, the one or more programs further include instruction for detecting the objects in the fields of view of the virtual viewpoints ([0102]).
Consider claim 4, Nishiyama teaches the objects to be detected are persons or a part of persons ([0025], [0030], [0061] – [0064]).
Consider claim 5, Kobayashi teaches the one or more programs further include instructions for detecting the objects located in a predetermined portion in a range corresponding to the fields of view corresponding to the virtual viewpoints identified based on the viewpoint information ([0074] – [0079], [0091] – [0094], [0114], [0123] – [0127], [0156] – [0157], [0162] – [0169], [0175] – [0178], [0183] – [0184]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the known technique of generating object information because such incorporation would improve the accuracy of a depth image of a 3D model. [0014].
Consider claim 17, claim 17 recites the method implemented by the apparatus recited in claim 1. Thus, it is rejected for the same reasons.
Consider claim 18, claim 18 recites the non-transitory storage medium for causing a computer to execute the method recited in claim 17. Thus, it is rejected for the same reasons.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nishiyama (US 2011/0254973 A1) in view of Kobayashi et al. (US 2019/0266786 A1), Li et al. (US 2012/0081369 A1), Young et al. (US 2018/0357810 A1), and Ogasawara et al. (US 2020/0050833 A1).
Consider claim 2, the combination of Nishiyama, Kobayashi, Li, and Horowitz teaches all the limitations in claim 1 but does not explicitly teach based on position information regarding the objects included in at least any of the plurality of captured images, and the obtained viewpoint information, the one or more programs further include instructions for detecting the objects in the fields of view of the virtual viewpoints.
Ogasawara teaches based on position information regarding the objects included in at least any of the plurality of captured images, and the obtained viewpoint information, the one or more programs further include instructions for detecting the objects in the fields of view of the virtual viewpoints ([0060], [0102], [0108], [0133]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the known technique of detecting the object in the fields of view of the virtual viewpoint based on position information regarding a predetermined object and viewpoint information because such incorporation would allow the state of a target object be acquired efficiently and accurately. [0010].
Claim(s) 6-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nishiyama (US 2011/0254973 A1) in view of Kobayashi et al. (US 2019/0266786 A1), Li et al. (US 2012/0081369 A1), Young et al. (US 2018/0357810 A1), and Duca et al. (US 2019/0259201 A1).
Consider claim 6, the combination of Nishiyama, Kobayashi, Li, and Horowitz teaches all the limitations in claim 1 but does not explicitly teach the one or more programs further include instructions for generating the object information associated with the number of virtual viewpoints of which the fields of view include the same object among a plurality of virtual viewpoints corresponding to a plurality of users and corresponding to a same time.
Duca teaches the one or more programs further include instructions for generating the object information associated with the number of virtual viewpoints of which the fields of view include the same object among a plurality of virtual viewpoints corresponding to a plurality of users and corresponding to a same time ([0077]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the known technique of generating the object information associated with the number of virtual viewpoints of which the fields of view include the same object among a plurality of virtual viewpoints corresponding to a plurality of users and corresponding to a same time because such incorporation would allow unique lighting data to be determined for the same virtual object under different lighting conditions, in different environments, and/or on different user devices at any time. [0077].
Consider claim 7, Duca teaches the one or more programs further include instructions for generating the object information associated with the number of virtual viewpoints of which the fields of view include the same object among a plurality of virtual viewpoints corresponding to a plurality of different times ([0077]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the known technique of generating the object information associated with the number of virtual viewpoints of which the fields of view include the same object among a plurality of virtual viewpoints corresponding to a plurality of users and corresponding to a same time because such incorporation would allow unique lighting data to be determined for the same virtual object under different lighting conditions, in different environments, and/or on different user devices at any time. [0077].
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAT CHI CHIO whose telephone number is (571)272-9563. The examiner can normally be reached Monday-Thursday 10am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JAMIE J ATALA can be reached at 571-272-7384. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TAT C CHIO/ Primary Examiner, Art Unit 2486