DETAILED ACTION
Status of Claims
Claims 1-20 are pending in this application, with claims 1, 11 and 17 being independent.
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Obligation Under 37 CFR 1.56 – Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Drawings
The drawings were received on January 6, 2025. These drawings are acceptable.
Claim Objections
Claim 1 is objected to because of the following informalities:
line 4 of claim 1 recites “to sample incident;” which makes no sense and is apparently a typographical error. It appears that applicant intended to recite “to sample incident light”. Accordingly, to further examine claim 1 at this time, the examiner will interpret claim 1 as such. Appropriate correction is required.
“the one or more exposure values” (line 5 of claim 1, emphasis added) lacks proper antecedent basis. Appropriate correction is required.
Claim 11 is objected to because of the following informalities: “the one or more exposure values” (line 7 of claim 11, emphasis added) lacks proper antecedent basis. Appropriate correction is required.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 11 and 17 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 9 and 14 of U.S. Patent No. 11,308,684. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of instant application are anticipated by and/or are obvious variants of the claims of U.S. Patent No. 11,308,684.
Claims of the Instant Application
Claims of U.S. Patent No. 11,308,684
1. A method comprising:
determining, from application data, one or more locations associated with one or more subjects in a virtual environment;
casting one or more rays at the one or more locations to sample incident [light];
computing the one or more exposure values based at least on the sampled incident light and the one or more subjects; and
rendering one or more images representative of the virtual environment based at least on the one or more exposure values.
1. A method comprising:
casting one or more first rays from a virtual camera into a virtual environment;
determining, based at least in part on one or more intersections of the one or more first rays, one or more subjects in the virtual environment;
instantiating one or more virtual light meters at one or more locations in the virtual environment corresponding to the one or more subjects;
sampling incident light at the one or more locations of the one or more virtual light meters based at least in part on casting one or more second rays from the one or more locations of the one or more virtual light meters using one or more ray tracing algorithms;
determining, based at least in part on the incident light, an exposure value; and
rendering a frame representative of the virtual environment based at least in part on the exposure value.
4. The method of claim 1, further comprising
based at least on the determining of the one or more locations, instantiating one or more virtual light meters at the one or more locations,
wherein the sampling includes casting the one or more rays based at least on the instantiating.
1.
instantiating one or more virtual light meters at one or more locations in the virtual environment corresponding to the one or more subjects;
sampling incident light at the one or more locations of the one or more virtual light meters based at least in part on casting one or more second rays from the one or more locations of the one or more virtual light meters using one or more ray tracing algorithms;
11. A system comprising:
one or more processors to perform operations including:
determining, from application data, one or more locations associated with one or more subjects in a virtual environment;
casting one or more rays to sample incident light at the one or more locations;
computing the one or more exposure values based at least on the sampled incident light and the one or more subjects; and
rendering one or more images representative of the virtual environment based at least on the one or more exposure values.
9. A system comprising:
one or more processors; and one or more memory devices storing instruction thereon that, when executed using the one or more processors, cause the one or more processors to execute operations comprising:
casting one or more rays from a virtual camera into a virtual environment; determining, based at least in part on one or more intersections of the one or more rays, one or more subjects in a virtual environment; instantiating one or more virtual light meters at one or more locations in the virtual environment corresponding to the one or more subjects;
sampling incoming irradiance at the one or more locations of the one or more virtual light meters based at least in part on executing a ray-tracing algorithm at the one or more locations;
determining, based at least in part on the incoming irradiance at the one or more locations, an exposure value; and
rendering, based at least in part on the exposure value, a frame representative of the virtual environment from a perspective of the virtual camera.
Claim 17 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 14 of U.S. Patent No. 11,308,684 in view of BENEDETTO et al. (US 2020/0269143, hereinafter “BENEDETTO”.
Claims of the Instant Application
Claims of U.S. Patent No. 11,308,684
17. At least one processor comprising:
one or more circuits to
render one or more images representative of a virtual environment associated with a game application based at least on one or more exposure values,
the one or more exposure values being computed based at least on:
a determination, from game state information associated with the game application, of one or more locations associated with one or more subjects in the virtual environment; and
casting one or more rays to sample incident light at the one or more locations.
14. A processor comprising:
processing circuitry to:
render, based at least in part on the exposure value, a frame representative of the virtual environment from a perspective of the virtual camera.
cast one or more rays from a virtual camera into a virtual environment;
determine, based at least in part on one or more intersections of the one or more rays, one or more subjects in a virtual environment;
instantiate one or more virtual light meters at one or more locations in the virtual environment corresponding to the one or more subjects;
sample incoming irradiance at the one or more locations of the one or more virtual light meters based at least in part on executing a ray-tracing algorithm at the one or more locations;
determine, based at least in part on the incoming irradiance at the one or more locations, an exposure value; and
Although claim 14 of U.S. Patent No. 11,308,684 does not recite “a virtual environment associated with a game application” and a determination, “from game state information associated with the game application,” of one or more locations associated with one or more subjects in the virtual environment, BENEDETTO teaches at least one processor (¶ [0132]: “graphics subsystem 1114“) comprising:
one or more circuits (e.g., ¶ [0132]: “a graphics processing unit (GPU) 1116 and graphics memory 1118 “) to render one or more images (e.g., ¶ [0132]: “the GPU 1116 includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and/or camera parameters for a scene.” (¶ [0132]: “A graphics subsystem 1114 is further connected with data bus 1122 and the components of the device 1100. The graphics subsystem 1114 includes a graphics processing unit (GPU) 1116 and graphics memory 1118. Graphics memory 1118 includes a display memory (e.g., a frame buffer) used for storing pixel data for each pixel of an output image. Graphics memory 1118 can be integrated in the same device as GPU 1116, connected as a separate device with GPU 1116, and/or implemented within memory 1104. Pixel data can be provided to graphics memory 1118 directly from the CPU 1102. Alternatively, CPU 1102 provides the GPU 1116 with data and/or instructions defining the desired output images, from which the GPU 1116 generates the pixel data of one or more output images. The data and/or instructions defining the desired output images can be stored in memory 1104 and/or graphics memory 1118. In an embodiment, the GPU 1116 includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and/or camera parameters for a scene. The GPU 1116 can further include one or more programmable execution units capable of executing shader programs.” ¶ [0133]: “The graphics subsystem 1114 periodically outputs pixel data for an image from graphics memory 1118 to be displayed on display device 12, or to be projected by projection system 1140. Display device 12 can be any device capable of displaying visual information in response to a signal from the device 1100, including CRT, LCD, plasma, and OLED displays. Device 1100 can provide the display device 12 with an analog or digital signal, for example.”) representative of a virtual environment associated with a game application (e.g., ¶ [0014]: “a gaming environment associated with a particular player playing a gaming application,”) (¶ [0035]: “As shown in FIG. 1A, the gaming application may be executing locally at a client device 100 (e.g., through game title executing engine 111 and game logic 126) of the player 5, or may be executing at a back-end game title executing engine (e.g., game engine) 211 operating at a back-end game server 205 of a cloud game network or game cloud system (GCS) 201.” ¶ [0036]: “In particular, client device 100 may include a game title executing engine 111 (also referred to as game engine) configured for local execution of the gaming application. Game logic 126 of the gaming application runs on top of and/or in association with the game title executing engine 111. For example, the game logic 126 (of the gaming application) that is executing makes function calls down to the game title executing engine 111, such as for physics information, for texture information, etc. More specifically, game title executing engine 111 performs basic processor-based functions for executing the gaming application (e.g., through game logic 126) and services associated with the gaming application. For example, processor-based functions include 2D or 3D rendering, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracking, shadowing, culling, transformation, etc.” ¶ [0037]: “In another embodiment, the gaming application may be executing at a back-end game title executing engine 211 (also referred to as game engine) operating at a back-end game server 205 of GCS 201. Game title executing engine 211 performs similarly as game title executing engine 111, previously described. More particularly, the game title executing engine 211 may be operating within one of many physical and/or virtual game processors of game server 205. Game logic 126 may be executing on top of and/or in association with game title executing engine 211, similar to how game logic 126 executes on top of and/or in association with game title executing engine 111. For example, game logic 126 that is executing makes function calls down to the game title executing engine 211, such as for physics information, for texture information, etc. Game title executing engine 211 and the game logic 126 combined form an instance 175 of the gaming application, as will be further described below. Game server 205 is configured to manage and control a plurality of instances executing one or more gaming applications for one or more players.” ¶ [0038]: “The client device 100 may receive input from various types of input devices, such as game controllers 6, tablet computers 11, keyboards, and gestures captured by video cameras, mice, touch pads, etc. Client device 100 can be any type of computing device having at least a memory and a processor module that is capable of connecting to the game server 205 over network 150. Some examples of client device 100 include a personal computer (PC), a game console, a home theater device, a general purpose computer, mobile computing device, a tablet, a phone, or any other types of computing devices that can interact with the game server 205 to execute an instance of a gaming application.” ¶ [0039]: “Client device 100 is configured for receiving rendered images, and for displaying the rendered images on display 12. For example, through local game processing, the rendered images may be delivered by the local game executing engine 111, in response to input commands that are used to drive game play of player 5. As another example, through cloud based services the rendered images may be delivered by an instance 175 of a gaming application executing on game executing engine 211 of game server 205 in response to input commands that are used to drive game play of player 5. That is, through remote game processing, the rendered images may be delivered by the remote game title executing engine 211. In either case, client device 100 is configured to interact with the executing engine 211 or 111 in association with the game play of player 5, such as through input commands that are used to drive game play.” ¶ [0040]: “Also, client device 100 is configured to interact with the game server 205 to capture and store metadata and/or information of the game play of player 5 when playing a gaming application. The metadata includes information (e.g., game state, etc.) related to the game play. For example, the metadata may include location based information corresponding to a location of a character within a gaming world of the game play of the player 5. In one embodiment, the metadata could include snapshot information that is related to a point in the gaming application at which the snapshot was captured. Relevant information may be stored in one or more databases of database 140.” ¶ [0041]: “In one embodiment, the metadata may include snapshot information, wherein a snapshot provides information that enables execution of an instance of the gaming application beginning from a point in the gaming application associated with the capture of the corresponding snapshot.”), and
a determination, from game state information associated with the game application, of one or more locations associated with one or more subjects in the virtual environment (¶ [0040]: “Also, client device 100 is configured to interact with the game server 205 to capture and store metadata and/or information of the game play of player 5 when playing a gaming application. The metadata includes information (e.g., game state, etc.) related to the game play. For example, the metadata may include location based information corresponding to a location of a character within a gaming world of the game play of the player 5. In one embodiment, the metadata could include snapshot information that is related to a point in the gaming application at which the snapshot was captured. Relevant information may be stored in one or more databases of database 140.” ¶ [0043]: “More particularly, the captured metadata and/or information may include game state data that defines the state of the game play at that point. For example, game state data may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, etc. In that manner, game state data allows for the generation of the gaming environment that existed at the corresponding point in the game play of the gaming application. Game state data may also include the state of every device used for rendering the game play, such as states of CPU, GPU, memory, register values, program counter value, programmable DMA state, buffered data for the DMA, audio chip state, CD-ROM state, etc. Game state data may also identify which parts of the executable code need to be loaded to execute the gaming application from that point. The game state data may be stored locally or in game state database 145 of GCS 201.” ¶ [0045]: “In addition, the captured metadata and/or information may include random seed data that may be generated by an artificial intelligence (AI) module (not shown). The random seed data may not be part of the original game code, but may be added in an overlay to make the gaming environment seem more realistic and/or engaging to the user. That is, random seed data provides additional features for the gaming environment that exists at the corresponding point in the game play of the user. For example, AI characters may be randomly generated and provided in the overlay. The AI characters are not associated with any users playing the game, but are placed into the gaming environment to enhance the user's experience. As an illustration, these AI characters may randomly walk the streets in a city scene. In addition, other objects maybe generated and presented in an overlay. For instance, clouds in the background and birds flying through space may be generated and presented in an overlay. The random seed data may be stored locally or stored in random seed database 143 of GCS 201.” ¶ [0050]: “These various functions performed by a game engine include basic processor based functions for executing the gaming application and services associated with the gaming application. For example, processor based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. In addition, services for the gaming application include streaming, encoding, memory management, multi-thread management, quality of service (QoS), bandwidth testing, social networking, management of social friends, communication with social networks of friends, communication channels, texting, instant messaging, chat support, etc.” ¶ [0053]: “Each of the streaming servers 170 may be configured as having at least a memory and a processor module that is capable of executing the gaming application, such as through a corresponding instance of the gaming application, in support of a corresponding game play. For example, each streaming server 170 may be a server console, gaming console, computer, etc. More particularly, an instance of a gaming application is executed by a corresponding game title execution engine (e.g., game engine) that is running game logic corresponding to the gaming application, as previously described. Each server 205 is configured to stream data 105 (e.g., rendered images and/or frames of a corresponding game play) either directly or through game server 205 back to a corresponding client device through network 150. In that manner, a computationally complex gaming application may be executing at the back-end server in response to controller inputs received and forwarded by a corresponding client device. Each server is able to render images and/or frames that are then encoded (e.g., compressed) and streamed to the corresponding client device for display.” ¶ [0054]: “A plurality of players 115 accesses the GCS 201 via network 150, wherein players (e.g., players 5L, 5M . . . 5Z) access network 150 via corresponding client devices 100A . . . 100N. Each of client device 100A through 100N may be configured similarly as client device 100 of FIG. 1A, or may be configured as a thin client (e.g., client device 100B) providing interfaces with a back end server providing computational functionality. Corresponding client device 100 can be any type of computing device having at least a memory and a processor module that is capable of connecting to the game server 205 over network 150. Also, corresponding client device 100 is configured for generating rendered images executed by game title execution engine 111 executing locally, or executed by the game title execution engine 211 executing remotely, and for displaying the rendered images on a display, including a head mounted display (HMD) (not shown). As previously described, each of client devices 100 may receive input from various types of input devices 11, such as game controllers, tablet computers, keyboards, gestures captured by video cameras, mice touch pads, etc. For example, a client device 100A of a corresponding player 5A is configured for requesting access to gaming applications over a network 150, such as the internet, and for rendering data of a particular gaming application (e.g., video game) executed by the a corresponding steaming server 170a through 170n, as managed by gaming server 205 and delivered to a display device associated with the corresponding player 5A. As such, player 5A may be interacting through client device 100A with an instance of a gaming application executing on a corresponding streaming server as managed by game server 205.” ¶ [0106]: “Game state data allows for the generation of the gaming environment that existed at the corresponding point in the game play of the gaming application. For example, game state may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, the state of every device used for rendering the game play, etc.” ¶ [0107]: “At 620, the method includes tracking a plurality of interaction regions in a gaming world associated with the plurality of game plays, wherein each game play is currently engaging with a corresponding interaction region. The game state for each game play may provide and/or be analyzed to determine a corresponding location of a corresponding game play within the gaming environment of the gaming application. For example, a character in the game play as controlled by player P3 may be within a particular interaction region (e.g., area, region, town, house, map coordinate, etc.). In one embodiment, the interaction region is a map region of a corresponding gaming environment. The map region generally encompasses a large area, and may include one or more towns, or areas (e.g., woods, open area, park, etc.). In other embodiments, the interaction region is smaller than a region, such as an area, or town, or map coordinate. In another embodiment, the interaction region is a quest. Each of the game plays has a corresponding interaction region that can be determined through corresponding game state.” ¶ [0108]: “At 630, the method includes determining a first interaction region in a gaming world associated with a first game play of a first player (e.g., player P3) playing the gaming application. For example, the location of a character being controlled by player P3 may determine the first interaction region. That is, the location of the character (e.g., region, area, town, map coordinate, etc.) determines the interaction region. For purposes of illustration only, the first interaction region may be a map region, such as the region of Velen in the gaming application Witcher 3, wherein the town of Blackbough is located in Velen.” ¶ [0112]: “In particular, player P3 has a corresponding field of view (e.g., FOV P3) taken from a particular point 713 in the gaming environment 700 of the gaming application. As shown, player P3 is able to view object 730 within FOV P3. That is, the depth buffer of player P3 does not limit viewing object 730. In addition, player P2 (Landon) has a corresponding field of view (e.g., FOV′ P2) taken from point 712 in the gaming environment. However, a depth buffer analysis limits the field of view of player P2 because of object 750. That is, FOV′ P2 includes large object 750 (e.g., wall, rock wall, building, etc.). As such, in FOV′ P2 player P2 is unable to view object 730 (e.g., along ray trace or line 720).” NOTE: Clearly, as per ¶ [0112], the location of player characters and objects within the gaming environment are determined by game state information of the gaming application.).
Thus, for one of ordinary skill in the art, in view of BENEDETTO, claim 17 the instant application is an obvious variant of claim 14 of U.S. Patent No. 11,308,684 since, in order to obtain a more versatile processor, it would have been obvious to one of ordinary skill in the art to modified claim 14 so that the claimed virtual environment is associated with a game application and the determination of the location of the one or more locations associated with one or more subjects in the virtual environment are determined from game state information associated with the game application, as taught by BENEDETTO.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 4, 9, 11, 14 and 16 are rejected under 35 U.S.C. 102 (a)(1) and/or 102 (a)(2) as being anticipated by ARMSDEN et al. (US 2013/0002671, hereinafter “ARMSDEN”).
Regarding claim 1, ARMSDEN discloses: a method (¶ [0016]: FIG. 2 illustrates an exemplary process for rendering a scene.”) comprising:
determining, from application data (¶ [0012]: “a computer-animated scene illuminated by indirect light” ), one or more locations (¶ [0024]: “sample location 170.”) associated with one or more subjects (¶ [0024]: “ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170.” ¶ [0012]: “sample locations on a surface element of an object in the scene.”) in a virtual environment (¶ [0023]: “scene 100”; ¶ [0023]: “a virtual world 110.” ¶ [0024]: “the virtual world 110”) (¶ [0023]: “FIG. 1 illustrates an exemplary scene 100 lit by indirect lighting 150. The scene 100 may be a snapshot taken by a virtual camera 130, viewing a virtual world 110. The box 120 highlights the portion of the virtual world 110 that may be visible to the virtual camera 130. Indirect lighting 150 may be light from an out-of-view light source, such as a light bulb, the sun, or the like. The scene 100 may be rendered using a conventional ray tracing engine.” ¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0028]:”FIG. 3 provides one exemplary representation of generating a point cloud representation of a scene. Scene 300 may represent a snapshot taken by a virtual camera 370, viewing a virtual world of teapots, including teapot 310. A point cloud representation 360 of teapot 310 may be computed. To generate the point cloud, the surfaces of different objects in the scene 300, such as teapot 310, may be subdivided into small micropolygons. The energy from each micropolygon may be stored by a point in the point cloud. A point may also store other information, including a position, a surface normal, an effective area, a point-radius, ID, or the like.” ¶ [0029]: “The point cloud representation 350 of the scene 300 may be generated in a pre-computation phase before computing the shading of the pixels in the scene.” NOTE: Using ray-tracing to render a virtual 3D scene requires “application data” defining the 3D scene, including the locations of everything in the scene. Thus, in order to render a scene using ray-tracing, the location of every object in the scene must be determined in order to perform ray tracing. In other words, in order to cast a ray into the virtual scene and determine whether it intersects any of the objects in the virtual scene, the location of each object in the scene, by necessity, must be determined.);
casting one or more rays at the one or more locations to sample incident [light] (¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0025]: “When determining the shading of a sample location 170 on an object 160 in the scene 100 as a result of indirect lighting 150, ray 190 is reflected and/or refracted into rays 191-196. These rays 191-196 are sent out from the sample location 170 using a stochastic sampling method and are randomly distributed in a hemisphere around the sample location 170. These rays 191-196 sample the surrounding geometry of the scene 100 and then are integrated to determine the incoming illumination, or irradiance, of the sample location 170.”);
computing the one or more exposure values (¶ [0025]: “integrated to determine the incoming illumination, or irradiance, of the sample location 170.” ¶ [0013]: “determine the energy incoming (irradiance) to the sample location.”) based at least on the sampled incident light and the one or more subjects (¶ [0025]: “When determining the shading of a sample location 170 on an object 160 in the scene 100 as a result of indirect lighting 150, ray 190 is reflected and/or refracted into rays 191-196. These rays 191-196 are sent out from the sample location 170 using a stochastic sampling method and are randomly distributed in a hemisphere around the sample location 170. These rays 191-196 sample the surrounding geometry of the scene 100 and then are integrated to determine the incoming illumination, or irradiance, of the sample location 170.” ¶ [0026]: “Many of the rays 191-194 are sent away from the indirect light 150 and so contribute no irradiance to the final rendering of the sample location 170. Irradiance is a measure of the amount of light energy incoming or incident to a surface.”); and
rendering one or more images representative of the virtual environment based at least on the one or more exposure values ( ¶ [0023]: “The scene 100 may be rendered using a conventional ray tracing engine.” ¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0026]: “Many of the rays 191-194 are sent away from the indirect light 150 and so contribute no irradiance to the final rendering of the sample location 170. Irradiance is a measure of the amount of light energy incoming or incident to a surface. Color is a specific frequency of this light energy. For example, the color perceived as red has a particular frequency whilst the color perceived as blue has another. In this example, only two of the rays 195-196 are mostly directed toward the indirect light 150 and contribute irradiance to the final rendering of the pixel 180 associated with sample location 170. To achieve the quality necessary for aesthetic images, many orders of magnitude more rays are required to illuminate this scene than to illuminate a well-lit scene, since so many rays are sent away from the indirect light 150 and do not contribute.” ¶ [0002]: “computer systems and processes for efficiently rendering a scene illuminated by indirect light using ray tracing.”).
Regarding claim 4 (depends on claim 1), ARMSDEN discloses:
based at least on the determining of the one or more locations (¶ [0024]: “ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170.”), instantiating one or more virtual light meters at the one or more locations (¶ [0024]: “sample location 170 in the virtual world 110.” ¶ [0025]: “a sample location 170 on an object 160 in the scene 100”; ¶ [0025]: “a hemisphere around the sample location 170.”), wherein the sampling includes casting the one or more rays based at least on the instantiating (¶ [0025]: “ray 190 is reflected and/or refracted into rays 191-196. These rays 191-196 are sent out from the sample location 170 using a stochastic sampling method and are randomly distributed in a hemisphere around the sample location 170.”) (¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0025]: “When determining the shading of a sample location 170 on an object 160 in the scene 100 as a result of indirect lighting 150, ray 190 is reflected and/or refracted into rays 191-196. These rays 191-196 are sent out from the sample location 170 using a stochastic sampling method and are randomly distributed in a hemisphere around the sample location 170. These rays 191-196 sample the surrounding geometry of the scene 100 and then are integrated to determine the incoming illumination, or irradiance, of the sample location 170.”).
Regarding claim 9 (depends on claim 1), ARMSDEN discloses:
wherein the computing the one or more exposure values (¶ [0025]: “integrated to determine the incoming illumination, or irradiance, of the sample location 170.” ¶ [0013]: “determine the energy incoming (irradiance) to the sample location.”) is based at least on filtering one or more incident light values from the computation of the one or more exposure values (¶ [0042]: “instead of randomly distributing rays in a hemisphere around the sample location 600, rays 610-611 are primarily directed toward an area of interest 630.” ¶ [0042]: “In this way, more rays may be directed toward the area of interest 630 without necessarily increasing the number of rays required to adequately render the scene. Fewer rays, or no rays, may be sent to the areas of non-interest in the scene, for example, areas of low energy, low information, low contrast, or the like.” ) based at least on determining the one or more incident light values correspond to one or more of at least one subject of the one or more subjects or one or more environmental features of the virtual environment (e.g., ¶ [0042]: “an area of interest 630. Area 630 may be determined during a pre-computation phase for rendering the scene. For example, a point cloud representation of the scene may be determined during the pre-computation phase. From this point cloud representation, an importance map may be created. The importance map may designate area of interest 630.”) (¶ [0013]: “An importance map may be generated at each sample location by rasterizing a cube or hemi-cube view of the point cloud from the position of the sample location, orientated around the normal to the sample location. This raster image may determine the energy incoming (irradiance) to the sample location. One or more areas of interest may then be designated based on the absolute or relative irradiance. Using this information, a ray tracing engine may then be biased, "sending" rays in the directions of areas of interest. For example, the areas of interest may be the areas of highest or relative energy, radiance, light, or the like in the scene. The scene is shaded using the output from the biased ray tracing engine.” ¶ [0042]: “FIGS. 6A-B illustrate biased results of a ray tracing engine. In FIG. 6A, instead of randomly distributing rays in a hemisphere around the sample location 600, rays 610-611 are primarily directed toward an area of interest 630. Area 630 may be determined during a pre-computation phase for rendering the scene. For example, a point cloud representation of the scene may be determined during the pre-computation phase. From this point cloud representation, an importance map may be created. The importance map may designate area of interest 630. In this way, more rays may be directed toward the area of interest 630 without necessarily increasing the number of rays required to adequately render the scene. Fewer rays, or no rays, may be sent to the areas of non-interest in the scene, for example, areas of low energy, low information, low contrast, or the like.” ¶ [0043]: “FIG. 6A illustrates a single area of interest 630. Alternatively, there may be more than one area of interest. FIG. 6B illustrates another possible scene. This scene contains a sample location 650 and three areas of interest 660, 670, and 680. If there is more than one area of interest, rays may be sent to all or some of the areas of interest. Rays 661-662 are directed toward area of interest 660. Rays 671-673 are directed toward area of interest 670. Ray 681 is directed toward area of interest 680. One of ordinary skill in the art would understand that the number of areas of interest in a scene is not limited to one as exemplified in FIG. 6A or three as exemplified in FIG. 6B. There may be more or fewer areas of interest in any given scene.”).
Regarding claim 11, ARMSDEN discloses: a system (¶ [0050]: “FIG. 7 depicts an exemplary computing system 700 configured to perform any one of the above-described processes.”) comprising:
one or more processors (¶ [0050]: “computing system 700 may include, for example, a processor, memory, storage, and input/output devices (e.g., monitor, keyboard, disk drive, Internet connection, etc.).” ¶ [0051]: “one or more central processing units ("CPU") 708,”) to perform operations including:
determining, from application data (¶ [0012]: “a computer-animated scene illuminated by indirect light”), one or more locations (¶ [0024]: “sample location 170.”) associated with one or more subjects (¶ [0024]: “ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170.” ¶ [0012]: “sample locations on a surface element of an object in the scene.”) in a virtual environment (¶ [0023]: “scene 100”; ¶ [0023]: “a virtual world 110.” ¶ [0024]: “the virtual world 110”) (¶ [0023]: “FIG. 1 illustrates an exemplary scene 100 lit by indirect lighting 150. The scene 100 may be a snapshot taken by a virtual camera 130, viewing a virtual world 110. The box 120 highlights the portion of the virtual world 110 that may be visible to the virtual camera 130. Indirect lighting 150 may be light from an out-of-view light source, such as a light bulb, the sun, or the like. The scene 100 may be rendered using a conventional ray tracing engine.” ¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0028]:”FIG. 3 provides one exemplary representation of generating a point cloud representation of a scene. Scene 300 may represent a snapshot taken by a virtual camera 370, viewing a virtual world of teapots, including teapot 310. A point cloud representation 360 of teapot 310 may be computed. To generate the point cloud, the surfaces of different objects in the scene 300, such as teapot 310, may be subdivided into small micropolygons. The energy from each micropolygon may be stored by a point in the point cloud. A point may also store other information, including a position, a surface normal, an effective area, a point-radius, ID, or the like.” ¶ [0029]: “The point cloud representation 350 of the scene 300 may be generated in a pre-computation phase before computing the shading of the pixels in the scene.” NOTE: Using ray-tracing to render a virtual 3D scene requires “application data” defining the 3D scene, including the locations of everything in the scene. Thus, in order to render a scene using ray-tracing, the location of every object in the scene must be determined in order to perform ray tracing. In other words, in order to cast a ray into the virtual scene and determine whether it intersects any of the objects in the virtual scene, the location of each object in the scene, by necessity, must be determined.);
casting one or more rays to sample incident light at the one or more locations (¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0025]: “When determining the shading of a sample location 170 on an object 160 in the scene 100 as a result of indirect lighting 150, ray 190 is reflected and/or refracted into rays 191-196. These rays 191-196 are sent out from the sample location 170 using a stochastic sampling method and are randomly distributed in a hemisphere around the sample location 170. These rays 191-196 sample the surrounding geometry of the scene 100 and then are integrated to determine the incoming illumination, or irradiance, of the sample location 170.”);
computing the one or more exposure values (¶ [0025]: “determine the incoming illumination, or irradiance, of the sample location 170.” ¶ [0013]: “determine the energy incoming (irradiance) to the sample location.”) based at least on the sampled incident light and the one or more subjects (¶ [0025]: “When determining the shading of a sample location 170 on an object 160 in the scene 100 as a result of indirect lighting 150, ray 190 is reflected and/or refracted into rays 191-196. These rays 191-196 are sent out from the sample location 170 using a stochastic sampling method and are randomly distributed in a hemisphere around the sample location 170. These rays 191-196 sample the surrounding geometry of the scene 100 and then are integrated to determine the incoming illumination, or irradiance, of the sample location 170.” ¶ [0026]: “Many of the rays 191-194 are sent away from the indirect light 150 and so contribute no irradiance to the final rendering of the sample location 170. Irradiance is a measure of the amount of light energy incoming or incident to a surface.”); and
rendering one or more images representative of the virtual environment based at least on the one or more exposure values (¶ [0023]: “The scene 100 may be rendered using a conventional ray tracing engine.” ¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0026]: “Many of the rays 191-194 are sent away from the indirect light 150 and so contribute no irradiance to the final rendering of the sample location 170. Irradiance is a measure of the amount of light energy incoming or incident to a surface. Color is a specific frequency of this light energy. For example, the color perceived as red has a particular frequency whilst the color perceived as blue has another. In this example, only two of the rays 195-196 are mostly directed toward the indirect light 150 and contribute irradiance to the final rendering of the pixel 180 associated with sample location 170. To achieve the quality necessary for aesthetic images, many orders of magnitude more rays are required to illuminate this scene than to illuminate a well-lit scene, since so many rays are sent away from the indirect light 150 and do not contribute.” ¶ [0002]: “computer systems and processes for efficiently rendering a scene illuminated by indirect light using ray tracing.”).
Regarding claim 14 (depends on claim 11), ARMSDEN discloses:
based at least on the determining of the one or more locations (¶ [0024]: “ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170.”), instantiating one or more virtual light meters at the one or more locations (¶ [0024]: “sample location 170 in the virtual world 110.” ¶ [0025]: “a sample location 170 on an object 160 in the scene 100”; ¶ [0025]: “a hemisphere around the sample location 170.”), wherein the sampling includes casting the one or more rays based at least on the instantiating (¶ [0025]: “ray 190 is reflected and/or refracted into rays 191-196. These rays 191-196 are sent out from the sample location 170 using a stochastic sampling method and are randomly distributed in a hemisphere around the sample location 170.”) (¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0025]: “When determining the shading of a sample location 170 on an object 160 in the scene 100 as a result of indirect lighting 150, ray 190 is reflected and/or refracted into rays 191-196. These rays 191-196 are sent out from the sample location 170 using a stochastic sampling method and are randomly distributed in a hemisphere around the sample location 170. These rays 191-196 sample the surrounding geometry of the scene 100 and then are integrated to determine the incoming illumination, or irradiance, of the sample location 170.”).
Regarding claim 16 (depends on claim 11), although ARMSDEN discloses:
the application data corresponds to an application executing at least one of an augmented reality application, a virtual reality application (¶ [0023]: “The scene 100 may be a snapshot taken by a virtual camera 130, viewing a virtual world 110. The box 120 highlights the portion of the virtual world 110 that may be visible to the virtual camera 130. Indirect lighting 150 may be light from an out-of-view light source, such as a light bulb, the sun, or the like. The scene 100 may be rendered using a conventional ray tracing engine.” ¶ [0028]: “Scene 300 may represent a snapshot taken by a virtual camera 370, viewing a virtual world of teapots, including teapot 310.”), or a mixed reality application.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
Determining the scope and contents of the prior art;
Ascertaining the differences between the prior art and the claims at issue;
Resolving the level of ordinary skill in the pertinent art; and
Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2, 5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over ARMSDEN et al. (US 2013/0002671) in view of BLACKMON et al. (US 2017/0169602).
Regarding claim 2 (depends on claim 1), whereas ARMSDEN may not be entirely explicit as to, BLACKMON teaches:
wherein the determining the one or more locations is based at least on the application data indicating that the one or more subjects are a focus of a scene (¶ [0106]: “In other examples, a region of interest may be determined by the content in the image. For example, if an object in a scene is desired to be the focus of attention then a region of the image containing that object may be determined to be a region of interest such that it is rendered using ray tracing. This may be the case irrespective of a user's gaze and irrespective of the position of the object in the image, i.e. the object does not need to be centrally located. As another example, a region of the image containing a moving object may be determined to be a region of interest such that it is processed using a ray tracing technique. As another example, the gaze tracking logic could identify “hot spots” where the user's eyes are determined to frequently be attracted to (e.g. two bits of the image that are both capturing the user's attention, where the gaze is bouncing between them), and these hot spots could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, foveal focus regions from the past (as a subset or simplification of the former) could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, future foveal destinations could be predicted based on a model of the motion of the eye and/or the saccade mechanism, and those predicted future foveal destinations could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, areas of high image complexity could be identified according to a complexity metric (e.g. number of primitives in a region or number of edges in a region, etc.) and those identified areas could be determined to be regions of interest such that they are processed using a ray tracing technique. The complexity metric may be based on known information content and/or augmentations within the image, for example signage may be rendered in high detail because it is likely the user will gaze there.“ ¶ [0001]: “Graphics processing systems are typically configured to receive graphics data, e.g. from an application running on a computer system, and to render the graphics data to provide a rendering output. For example, the graphics data provided to a graphics processing system may describe geometry within a three dimensional (3D) scene to be rendered, and the rendering output may be a rendered image of the scene. The graphics data may include “primitives” representing geometric shapes, which describe surfaces of structures in the scene. A common primitive shape is a triangle, but primitives may be other shapes and may be lines or points. Objects can be composed of one or more (e.g. hundreds, thousands or millions) of such primitives.” ¶ [0002]: “In some systems, sequences of images (or “frames”) are rendered and displayed in real-time. The frame rate of the sequence of images will typically depend on the application for which the images are rendered. To give an example, a gaming application may send images for rendering at a frame rate of 25 frames per second, but other frame rates may be used in other examples. Generally, increasing the frame rate of the sequence of images to be rendered will increase the processing load on the graphics processing system.” ¶ [0006]: “Gaze tracking (or “eye tracking”) may be used to determine where a user is looking to thereby determine where the foveal region 104 is in relation to the image 102. The foveal region 104 can be rendered at a high resolution and high geometric LOD, whereas the rest of the image 102 can be rendered at a lower resolution and lower geometric LOD. The rendered foveal region 104 can then be blended with the rendering of the rest of the image 102 to form a rendered image. Therefore, in the periphery of the image 102, unperceived detail can be omitted such that fewer primitives may be rendered, at a lower pixel density. Since the foveal region 104 is generally a small proportion (e.g. ˜1%) of the area of the image 102, substantial rendering computation savings can be achieved by reducing the resolution and geometric LOD of the periphery compared to the rendering of the foveal region. Furthermore, since human visual acuity decreases rapidly away from the foveal region, high image quality is maintained for the image as perceived by a user who directs their gaze at the centre of the foveal region 104.” ¶ [0008]: “In some examples, foveated rendering is described for rendering an image whereby a ray tracing technique is used to render a region of interest of the image, and a rasterisation technique is used to render other regions of the image. The rendered region of interest of the image is then combined (e.g. blended) with the rendered other regions of the image to form a rendered image. The region of interest may correspond to a foveal region of the image. Ray tracing naturally provides high detail and photo-realistic rendering, which human vision is particularly sensitive to in the foveal region; whereas rasterisation techniques are suited for providing temporal smoothing and anti-aliasing in a simple manner, and is therefore suited for use in the regions of the image that a user will see in the periphery of their vision.”).
Thus, in order to reduce the computational load for rendering each image, it would have been obvious to one of ordinary skill in the art to have modified the image rendering method taught by ARMSDEN so as to include determining the one or more locations based at least on the application data indicating that the one or more subjects are a focus of a scene, as taught by BLACKMON.
Regarding claim 5 (depends on claim 1), whereas ARMSDEN may not be entirely explicit as to, BLACKMON teaches:
wherein the computing the one or more exposure values includes:
selecting, based at least on the application data (¶ [0001]: “graphics data, e.g. from an application running on a computer system,” ¶ [0001]: “the graphics data provided to a graphics processing system may describe geometry within a three dimensional (3D) scene to be rendered,”), a subset of pixels from an image (¶ [0106]: “a region of interest may be determined by the content in the image.”) based at least on the subset of pixels corresponding to the one or more locations (¶ [0106]: “if an object in a scene is desired to be the focus of attention then a region of the image containing that object may be determined to be a region of interest such that it is rendered using ray tracing.”) (¶ [0106]: “In other examples, a region of interest may be determined by the content in the image. For example, if an object in a scene is desired to be the focus of attention then a region of the image containing that object may be determined to be a region of interest such that it is rendered using ray tracing. This may be the case irrespective of a user's gaze and irrespective of the position of the object in the image, i.e. the object does not need to be centrally located. As another example, a region of the image containing a moving object may be determined to be a region of interest such that it is processed using a ray tracing technique. As another example, the gaze tracking logic could identify “hot spots” where the user's eyes are determined to frequently be attracted to (e.g. two bits of the image that are both capturing the user's attention, where the gaze is bouncing between them), and these hot spots could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, foveal focus regions from the past (as a subset or simplification of the former) could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, future foveal destinations could be predicted based on a model of the motion of the eye and/or the saccade mechanism, and those predicted future foveal destinations could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, areas of high image complexity could be identified according to a complexity metric (e.g. number of primitives in a region or number of edges in a region, etc.) and those identified areas could be determined to be regions of interest such that they are processed using a ray tracing technique. The complexity metric may be based on known information content and/or augmentations within the image, for example signage may be rendered in high detail because it is likely the user will gaze there.“ ¶ [0001]: “Graphics processing systems are typically configured to receive graphics data, e.g. from an application running on a computer system, and to render the graphics data to provide a rendering output. For example, the graphics data provided to a graphics processing system may describe geometry within a three dimensional (3D) scene to be rendered, and the rendering output may be a rendered image of the scene. The graphics data may include “primitives” representing geometric shapes, which describe surfaces of structures in the scene. A common primitive shape is a triangle, but primitives may be other shapes and may be lines or points. Objects can be composed of one or more (e.g. hundreds, thousands or millions) of such primitives.” ¶ [0002]: “In some systems, sequences of images (or “frames”) are rendered and displayed in real-time. The frame rate of the sequence of images will typically depend on the application for which the images are rendered. To give an example, a gaming application may send images for rendering at a frame rate of 25 frames per second, but other frame rates may be used in other examples. Generally, increasing the frame rate of the sequence of images to be rendered will increase the processing load on the graphics processing system.” ¶ [0006]: “Gaze tracking (or “eye tracking”) may be used to determine where a user is looking to thereby determine where the foveal region 104 is in relation to the image 102. The foveal region 104 can be rendered at a high resolution and high geometric LOD, whereas the rest of the image 102 can be rendered at a lower resolution and lower geometric LOD. The rendered foveal region 104 can then be blended with the rendering of the rest of the image 102 to form a rendered image. Therefore, in the periphery of the image 102, unperceived detail can be omitted such that fewer primitives may be rendered, at a lower pixel density. Since the foveal region 104 is generally a small proportion (e.g. ˜1%) of the area of the image 102, substantial rendering computation savings can be achieved by reducing the resolution and geometric LOD of the periphery compared to the rendering of the foveal region. Furthermore, since human visual acuity decreases rapidly away from the foveal region, high image quality is maintained for the image as perceived by a user who directs their gaze at the centre of the foveal region 104.” ¶ [0008]: “In some examples, foveated rendering is described for rendering an image whereby a ray tracing technique is used to render a region of interest of the image, and a rasterisation technique is used to render other regions of the image. The rendered region of interest of the image is then combined (e.g. blended) with the rendered other regions of the image to form a rendered image. The region of interest may correspond to a foveal region of the image. Ray tracing naturally provides high detail and photo-realistic rendering, which human vision is particularly sensitive to in the foveal region; whereas rasterisation techniques are suited for providing temporal smoothing and anti-aliasing in a simple manner, and is therefore suited for use in the regions of the image that a user will see in the periphery of their vision.”); and
based at least on the selecting (e.g., ¶ [0106]: “a region of interest may be determined by the content in the image.” ¶ [0106]: “if an object in a scene is desired to be the focus of attention then a region of the image containing that object may be determined to be a region of interest such that it is rendered using ray tracing.”), determining the sampled incident light based at least on the sampled incident light corresponding to the subset of pixels (e.g., ¶ [0106]: “a region of the image containing that object may be determined to be a region of interest such that it is rendered using ray tracing.” NOTE: In other words, in order to further reduce the computational load and delay when rendering dynamic low light scenes using ray tracing as taught by ARMSDEN, it would have been obvious to one of ordinary skill in the art to have modified ARMSDEN’s method so as to incorporate BLACKMON’s technique of only using ray tracing to render a region of interest in an image (i.e., a subset of pixels) corresponding to an object in a scene that is desired to by the focus of attention.) (¶ [0106]: “In other examples, a region of interest may be determined by the content in the image. For example, if an object in a scene is desired to be the focus of attention then a region of the image containing that object may be determined to be a region of interest such that it is rendered using ray tracing. This may be the case irrespective of a user's gaze and irrespective of the position of the object in the image, i.e. the object does not need to be centrally located. As another example, a region of the image containing a moving object may be determined to be a region of interest such that it is processed using a ray tracing technique. As another example, the gaze tracking logic could identify “hot spots” where the user's eyes are determined to frequently be attracted to (e.g. two bits of the image that are both capturing the user's attention, where the gaze is bouncing between them), and these hot spots could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, foveal focus regions from the past (as a subset or simplification of the former) could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, future foveal destinations could be predicted based on a model of the motion of the eye and/or the saccade mechanism, and those predicted future foveal destinations could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, areas of high image complexity could be identified according to a complexity metric (e.g. number of primitives in a region or number of edges in a region, etc.) and those identified areas could be determined to be regions of interest such that they are processed using a ray tracing technique. The complexity metric may be based on known information content and/or augmentations within the image, for example signage may be rendered in high detail because it is likely the user will gaze there.“ ¶ [0001]: “Graphics processing systems are typically configured to receive graphics data, e.g. from an application running on a computer system, and to render the graphics data to provide a rendering output. For example, the graphics data provided to a graphics processing system may describe geometry within a three dimensional (3D) scene to be rendered, and the rendering output may be a rendered image of the scene. The graphics data may include “primitives” representing geometric shapes, which describe surfaces of structures in the scene. A common primitive shape is a triangle, but primitives may be other shapes and may be lines or points. Objects can be composed of one or more (e.g. hundreds, thousands or millions) of such primitives.” ¶ [0002]: “In some systems, sequences of images (or “frames”) are rendered and displayed in real-time. The frame rate of the sequence of images will typically depend on the application for which the images are rendered. To give an example, a gaming application may send images for rendering at a frame rate of 25 frames per second, but other frame rates may be used in other examples. Generally, increasing the frame rate of the sequence of images to be rendered will increase the processing load on the graphics processing system.” ¶ [0006]: “Gaze tracking (or “eye tracking”) may be used to determine where a user is looking to thereby determine where the foveal region 104 is in relation to the image 102. The foveal region 104 can be rendered at a high resolution and high geometric LOD, whereas the rest of the image 102 can be rendered at a lower resolution and lower geometric LOD. The rendered foveal region 104 can then be blended with the rendering of the rest of the image 102 to form a rendered image. Therefore, in the periphery of the image 102, unperceived detail can be omitted such that fewer primitives may be rendered, at a lower pixel density. Since the foveal region 104 is generally a small proportion (e.g. ˜1%) of the area of the image 102, substantial rendering computation savings can be achieved by reducing the resolution and geometric LOD of the periphery compared to the rendering of the foveal region. Furthermore, since human visual acuity decreases rapidly away from the foveal region, high image quality is maintained for the image as perceived by a user who directs their gaze at the centre of the foveal region 104.” ¶ [0008]: “In some examples, foveated rendering is described for rendering an image whereby a ray tracing technique is used to render a region of interest of the image, and a rasterisation technique is used to render other regions of the image. The rendered region of interest of the image is then combined (e.g. blended) with the rendered other regions of the image to form a rendered image. The region of interest may correspond to a foveal region of the image. Ray tracing naturally provides high detail and photo-realistic rendering, which human vision is particularly sensitive to in the foveal region; whereas rasterisation techniques are suited for providing temporal smoothing and anti-aliasing in a simple manner, and is therefore suited for use in the regions of the image that a user will see in the periphery of their vision.”).
Thus, in order to reduce the computational load for rendering each image, it would have been obvious to one of ordinary skill in the art to have modified the image rendering method taught by ARMSDEN so as to compute the one or more exposure values includes by selecting, based at least on the application data, a subset of pixels from an image based at least on the subset of pixels corresponding to the one or more locations, and, based at least on the selecting, determining the sampled incident light based at least on the sampled incident light corresponding to the subset of pixels, as taught by BLACKMON.
Regarding claim 12 (depends on claim 11), whereas ARMSDEN may not be entirely explicit as to, BLACKMON teaches:
wherein the determining the one or more locations is based at least on the application data indicating that the one or more subjects are a focus of a scene (¶ [0106]: “In other examples, a region of interest may be determined by the content in the image. For example, if an object in a scene is desired to be the focus of attention then a region of the image containing that object may be determined to be a region of interest such that it is rendered using ray tracing. This may be the case irrespective of a user's gaze and irrespective of the position of the object in the image, i.e. the object does not need to be centrally located. As another example, a region of the image containing a moving object may be determined to be a region of interest such that it is processed using a ray tracing technique. As another example, the gaze tracking logic could identify “hot spots” where the user's eyes are determined to frequently be attracted to (e.g. two bits of the image that are both capturing the user's attention, where the gaze is bouncing between them), and these hot spots could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, foveal focus regions from the past (as a subset or simplification of the former) could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, future foveal destinations could be predicted based on a model of the motion of the eye and/or the saccade mechanism, and those predicted future foveal destinations could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, areas of high image complexity could be identified according to a complexity metric (e.g. number of primitives in a region or number of edges in a region, etc.) and those identified areas could be determined to be regions of interest such that they are processed using a ray tracing technique. The complexity metric may be based on known information content and/or augmentations within the image, for example signage may be rendered in high detail because it is likely the user will gaze there.“ ¶ [0001]: “Graphics processing systems are typically configured to receive graphics data, e.g. from an application running on a computer system, and to render the graphics data to provide a rendering output. For example, the graphics data provided to a graphics processing system may describe geometry within a three dimensional (3D) scene to be rendered, and the rendering output may be a rendered image of the scene. The graphics data may include “primitives” representing geometric shapes, which describe surfaces of structures in the scene. A common primitive shape is a triangle, but primitives may be other shapes and may be lines or points. Objects can be composed of one or more (e.g. hundreds, thousands or millions) of such primitives.” ¶ [0002]: “In some systems, sequences of images (or “frames”) are rendered and displayed in real-time. The frame rate of the sequence of images will typically depend on the application for which the images are rendered. To give an example, a gaming application may send images for rendering at a frame rate of 25 frames per second, but other frame rates may be used in other examples. Generally, increasing the frame rate of the sequence of images to be rendered will increase the processing load on the graphics processing system.” ¶ [0006]: “Gaze tracking (or “eye tracking”) may be used to determine where a user is looking to thereby determine where the foveal region 104 is in relation to the image 102. The foveal region 104 can be rendered at a high resolution and high geometric LOD, whereas the rest of the image 102 can be rendered at a lower resolution and lower geometric LOD. The rendered foveal region 104 can then be blended with the rendering of the rest of the image 102 to form a rendered image. Therefore, in the periphery of the image 102, unperceived detail can be omitted such that fewer primitives may be rendered, at a lower pixel density. Since the foveal region 104 is generally a small proportion (e.g. ˜1%) of the area of the image 102, substantial rendering computation savings can be achieved by reducing the resolution and geometric LOD of the periphery compared to the rendering of the foveal region. Furthermore, since human visual acuity decreases rapidly away from the foveal region, high image quality is maintained for the image as perceived by a user who directs their gaze at the centre of the foveal region 104.” ¶ [0008]: “In some examples, foveated rendering is described for rendering an image whereby a ray tracing technique is used to render a region of interest of the image, and a rasterisation technique is used to render other regions of the image. The rendered region of interest of the image is then combined (e.g. blended) with the rendered other regions of the image to form a rendered image. The region of interest may correspond to a foveal region of the image. Ray tracing naturally provides high detail and photo-realistic rendering, which human vision is particularly sensitive to in the foveal region; whereas rasterisation techniques are suited for providing temporal smoothing and anti-aliasing in a simple manner, and is therefore suited for use in the regions of the image that a user will see in the periphery of their vision.”).
Thus, in order to reduce the computational load for rendering each image, it would have been obvious to one of ordinary skill in the art to have modified the image rendering system taught by ARMSDEN so as to include determining the one or more locations based at least on the application data indicating that the one or more subjects are a focus of a scene, as taught by BLACKMON.
Claims 3, 13, 17, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over ARMSDEN et al. (US 2013/0002671) in view of BENEDETTO et al. (US 2020/0269143).
Regarding claim 3 (depends on claim 1), whereas ARMSDEN may not be entirely explicit as to, BENEDETTO teaches:
wherein the application data includes application state data (¶ [0040]: “metadata includes information (e.g., game state, etc.) related to the game play. For example, the metadata may include location based information corresponding to a location of a character within a gaming world”) corresponding to a software application (e.g., ¶ [0035]: “gaming application”), and the one or more subjects comprise one or more characters of the application (¶ [0040]: “the metadata may include location based information corresponding to a location of a character within a gaming world”) (¶ [0035]: “As shown in FIG. 1A, the gaming application may be executing locally at a client device 100 (e.g., through game title executing engine 111 and game logic 126) of the player 5, or may be executing at a back-end game title executing engine (e.g., game engine) 211 operating at a back-end game server 205 of a cloud game network or game cloud system (GCS) 201.” ¶ [0036]: “In particular, client device 100 may include a game title executing engine 111 (also referred to as game engine) configured for local execution of the gaming application. Game logic 126 of the gaming application runs on top of and/or in association with the game title executing engine 111. For example, the game logic 126 (of the gaming application) that is executing makes function calls down to the game title executing engine 111, such as for physics information, for texture information, etc. More specifically, game title executing engine 111 performs basic processor-based functions for executing the gaming application (e.g., through game logic 126) and services associated with the gaming application. For example, processor-based functions include 2D or 3D rendering, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracking, shadowing, culling, transformation, etc.” ¶ [0039]: “Client device 100 is configured for receiving rendered images, and for displaying the rendered images on display 12. For example, through local game processing, the rendered images may be delivered by the local game executing engine 111, in response to input commands that are used to drive game play of player 5. As another example, through cloud based services the rendered images may be delivered by an instance 175 of a gaming application executing on game executing engine 211 of game server 205 in response to input commands that are used to drive game play of player 5. That is, through remote game processing, the rendered images may be delivered by the remote game title executing engine 211. In either case, client device 100 is configured to interact with the executing engine 211 or 111 in association with the game play of player 5, such as through input commands that are used to drive game play.” ¶ [0040]: “Also, client device 100 is configured to interact with the game server 205 to capture and store metadata and/or information of the game play of player 5 when playing a gaming application. The metadata includes information (e.g., game state, etc.) related to the game play. For example, the metadata may include location based information corresponding to a location of a character within a gaming world of the game play of the player 5. In one embodiment, the metadata could include snapshot information that is related to a point in the gaming application at which the snapshot was captured. Relevant information may be stored in one or more databases of database 140.” ¶ [0043]: “More particularly, the captured metadata and/or information may include game state data that defines the state of the game play at that point. For example, game state data may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, etc. In that manner, game state data allows for the generation of the gaming environment that existed at the corresponding point in the game play of the gaming application. Game state data may also include the state of every device used for rendering the game play, such as states of CPU, GPU, memory, register values, program counter value, programmable DMA state, buffered data for the DMA, audio chip state, CD-ROM state, etc. Game state data may also identify which parts of the executable code need to be loaded to execute the gaming application from that point. The game state data may be stored locally or in game state database 145 of GCS 201.” ¶ [0050]: “These various functions performed by a game engine include basic processor based functions for executing the gaming application and services associated with the gaming application. For example, processor based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. In addition, services for the gaming application include streaming, encoding, memory management, multi-thread management, quality of service (QoS), bandwidth testing, social networking, management of social friends, communication with social networks of friends, communication channels, texting, instant messaging, chat support, etc.” ¶ [0053]: “Each of the streaming servers 170 may be configured as having at least a memory and a processor module that is capable of executing the gaming application, such as through a corresponding instance of the gaming application, in support of a corresponding game play. For example, each streaming server 170 may be a server console, gaming console, computer, etc. More particularly, an instance of a gaming application is executed by a corresponding game title execution engine (e.g., game engine) that is running game logic corresponding to the gaming application, as previously described. Each server 205 is configured to stream data 105 (e.g., rendered images and/or frames of a corresponding game play) either directly or through game server 205 back to a corresponding client device through network 150. In that manner, a computationally complex gaming application may be executing at the back-end server in response to controller inputs received and forwarded by a corresponding client device. Each server is able to render images and/or frames that are then encoded (e.g., compressed) and streamed to the corresponding client device for display.” ¶ [0106]: “Game state data allows for the generation of the gaming environment that existed at the corresponding point in the game play of the gaming application. For example, game state may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, the state of every device used for rendering the game play, etc.” ¶ [0107]: “At 620, the method includes tracking a plurality of interaction regions in a gaming world associated with the plurality of game plays, wherein each game play is currently engaging with a corresponding interaction region. The game state for each game play may provide and/or be analyzed to determine a corresponding location of a corresponding game play within the gaming environment of the gaming application. For example, a character in the game play as controlled by player P3 may be within a particular interaction region (e.g., area, region, town, house, map coordinate, etc.). In one embodiment, the interaction region is a map region of a corresponding gaming environment. The map region generally encompasses a large area, and may include one or more towns, or areas (e.g., woods, open area, park, etc.). In other embodiments, the interaction region is smaller than a region, such as an area, or town, or map coordinate. In another embodiment, the interaction region is a quest. Each of the game plays has a corresponding interaction region that can be determined through corresponding game state.” ¶ [0108]: “At 630, the method includes determining a first interaction region in a gaming world associated with a first game play of a first player (e.g., player P3) playing the gaming application. For example, the location of a character being controlled by player P3 may determine the first interaction region. That is, the location of the character (e.g., region, area, town, map coordinate, etc.) determines the interaction region. For purposes of illustration only, the first interaction region may be a map region, such as the region of Velen in the gaming application Witcher 3, wherein the town of Blackbough is located in Velen.” ¶ [0112]: “In particular, player P3 has a corresponding field of view (e.g., FOV P3) taken from a particular point 713 in the gaming environment 700 of the gaming application. As shown, player P3 is able to view object 730 within FOV P3. That is, the depth buffer of player P3 does not limit viewing object 730. In addition, player P2 (Landon) has a corresponding field of view (e.g., FOV′ P2) taken from point 712 in the gaming environment. However, a depth buffer analysis limits the field of view of player P2 because of object 750. That is, FOV′ P2 includes large object 750 (e.g., wall, rock wall, building, etc.). As such, in FOV′ P2 player P2 is unable to view object 730 (e.g., along ray trace or line 720).” NOTE: Clearly, as per at least ¶ [0112], the location of player characters and objects within the gaming environment are determined by game state information of the gaming application.).
Thus, in order to obtain a graphic processing system having the cumulative features and/or functionalities taught by ARMSDEN and BENEDETTO, it would have been obvious to one of ordinary skill in the art to have modified the graphic processing system taught by ARMSDEN so that the application data includes application state data corresponding to a software application and the one or more subjects comprise one or more characters of the application, as taught by BENEDETTO.
Regarding claim 13 (depends on claim 11), whereas ARMSDEN may not be entirely explicit as to, BENEDETTO teaches:
wherein the application data includes application state data (¶ [0040]: “metadata includes information (e.g., game state, etc.) related to the game play. For example, the metadata may include location based information corresponding to a location of a character within a gaming world”) corresponding to a software application (e.g., ¶ [0035]: “gaming application”), and the one or more subjects comprise one or more characters of the application (¶ [0040]: “the metadata may include location based information corresponding to a location of a character within a gaming world”) (¶ [0035]: “As shown in FIG. 1A, the gaming application may be executing locally at a client device 100 (e.g., through game title executing engine 111 and game logic 126) of the player 5, or may be executing at a back-end game title executing engine (e.g., game engine) 211 operating at a back-end game server 205 of a cloud game network or game cloud system (GCS) 201.” ¶ [0036]: “In particular, client device 100 may include a game title executing engine 111 (also referred to as game engine) configured for local execution of the gaming application. Game logic 126 of the gaming application runs on top of and/or in association with the game title executing engine 111. For example, the game logic 126 (of the gaming application) that is executing makes function calls down to the game title executing engine 111, such as for physics information, for texture information, etc. More specifically, game title executing engine 111 performs basic processor-based functions for executing the gaming application (e.g., through game logic 126) and services associated with the gaming application. For example, processor-based functions include 2D or 3D rendering, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracking, shadowing, culling, transformation, etc.” ¶ [0039]: “Client device 100 is configured for receiving rendered images, and for displaying the rendered images on display 12. For example, through local game processing, the rendered images may be delivered by the local game executing engine 111, in response to input commands that are used to drive game play of player 5. As another example, through cloud based services the rendered images may be delivered by an instance 175 of a gaming application executing on game executing engine 211 of game server 205 in response to input commands that are used to drive game play of player 5. That is, through remote game processing, the rendered images may be delivered by the remote game title executing engine 211. In either case, client device 100 is configured to interact with the executing engine 211 or 111 in association with the game play of player 5, such as through input commands that are used to drive game play.” ¶ [0040]: “Also, client device 100 is configured to interact with the game server 205 to capture and store metadata and/or information of the game play of player 5 when playing a gaming application. The metadata includes information (e.g., game state, etc.) related to the game play. For example, the metadata may include location based information corresponding to a location of a character within a gaming world of the game play of the player 5. In one embodiment, the metadata could include snapshot information that is related to a point in the gaming application at which the snapshot was captured. Relevant information may be stored in one or more databases of database 140.” ¶ [0043]: “More particularly, the captured metadata and/or information may include game state data that defines the state of the game play at that point. For example, game state data may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, etc. In that manner, game state data allows for the generation of the gaming environment that existed at the corresponding point in the game play of the gaming application. Game state data may also include the state of every device used for rendering the game play, such as states of CPU, GPU, memory, register values, program counter value, programmable DMA state, buffered data for the DMA, audio chip state, CD-ROM state, etc. Game state data may also identify which parts of the executable code need to be loaded to execute the gaming application from that point. The game state data may be stored locally or in game state database 145 of GCS 201.” ¶ [0050]: “These various functions performed by a game engine include basic processor based functions for executing the gaming application and services associated with the gaming application. For example, processor based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. In addition, services for the gaming application include streaming, encoding, memory management, multi-thread management, quality of service (QoS), bandwidth testing, social networking, management of social friends, communication with social networks of friends, communication channels, texting, instant messaging, chat support, etc.” ¶ [0053]: “Each of the streaming servers 170 may be configured as having at least a memory and a processor module that is capable of executing the gaming application, such as through a corresponding instance of the gaming application, in support of a corresponding game play. For example, each streaming server 170 may be a server console, gaming console, computer, etc. More particularly, an instance of a gaming application is executed by a corresponding game title execution engine (e.g., game engine) that is running game logic corresponding to the gaming application, as previously described. Each server 205 is configured to stream data 105 (e.g., rendered images and/or frames of a corresponding game play) either directly or through game server 205 back to a corresponding client device through network 150. In that manner, a computationally complex gaming application may be executing at the back-end server in response to controller inputs received and forwarded by a corresponding client device. Each server is able to render images and/or frames that are then encoded (e.g., compressed) and streamed to the corresponding client device for display.” ¶ [0106]: “Game state data allows for the generation of the gaming environment that existed at the corresponding point in the game play of the gaming application. For example, game state may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, the state of every device used for rendering the game play, etc.” ¶ [0107]: “At 620, the method includes tracking a plurality of interaction regions in a gaming world associated with the plurality of game plays, wherein each game play is currently engaging with a corresponding interaction region. The game state for each game play may provide and/or be analyzed to determine a corresponding location of a corresponding game play within the gaming environment of the gaming application. For example, a character in the game play as controlled by player P3 may be within a particular interaction region (e.g., area, region, town, house, map coordinate, etc.). In one embodiment, the interaction region is a map region of a corresponding gaming environment. The map region generally encompasses a large area, and may include one or more towns, or areas (e.g., woods, open area, park, etc.). In other embodiments, the interaction region is smaller than a region, such as an area, or town, or map coordinate. In another embodiment, the interaction region is a quest. Each of the game plays has a corresponding interaction region that can be determined through corresponding game state.” ¶ [0108]: “At 630, the method includes determining a first interaction region in a gaming world associated with a first game play of a first player (e.g., player P3) playing the gaming application. For example, the location of a character being controlled by player P3 may determine the first interaction region. That is, the location of the character (e.g., region, area, town, map coordinate, etc.) determines the interaction region. For purposes of illustration only, the first interaction region may be a map region, such as the region of Velen in the gaming application Witcher 3, wherein the town of Blackbough is located in Velen.” ¶ [0112]: “In particular, player P3 has a corresponding field of view (e.g., FOV P3) taken from a particular point 713 in the gaming environment 700 of the gaming application. As shown, player P3 is able to view object 730 within FOV P3. That is, the depth buffer of player P3 does not limit viewing object 730. In addition, player P2 (Landon) has a corresponding field of view (e.g., FOV′ P2) taken from point 712 in the gaming environment. However, a depth buffer analysis limits the field of view of player P2 because of object 750. That is, FOV′ P2 includes large object 750 (e.g., wall, rock wall, building, etc.). As such, in FOV′ P2 player P2 is unable to view object 730 (e.g., along ray trace or line 720).” NOTE: Clearly, as per at least ¶ [0112], the location of player characters and objects within the gaming environment are determined by game state information of the gaming application.).
Thus, in order to obtain a graphic processing system having the cumulative features and/or functionalities taught by ARMSDEN and BENEDETTO, it would have been obvious to one of ordinary skill in the art to have modified the graphic processing and rendering system taught by ARMSDEN so that the application data includes application state data corresponding to a software application and the one or more subjects comprise one or more characters of the application, as taught by BENEDETTO.
Regarding claim 17, ARMSDEN discloses at least one processor (¶ [0050]: “computing system 700“) comprising:
one or more circuits (¶ [0050]: “computing system 700 may include, for example, a processor, memory, storage, and input/output devices (e.g., monitor, keyboard, disk drive, Internet connection, etc.). However, computing system 700 may include circuitry or other specialized hardware for carrying out some or all aspects of the processes. In some operational settings, computing system 700 may be configured as a system that includes one or more units, each of which is configured to carry out some aspects of the processes either in software, hardware, or some combination thereof.” ) to render one or more images (¶ [0023]: “A snapshot of a scene 100 using a ray tracing approach may be rendered”) representative of a virtual environment (¶ [0023]: “virtual world 110.”) based at least on one or more exposure values (¶ [0026]: “contribute irradiance to the final rendering of the pixel 180 associated with sample location 170.”) (¶ [0023]: “The scene 100 may be rendered using a conventional ray tracing engine.” ¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0026]: “Many of the rays 191-194 are sent away from the indirect light 150 and so contribute no irradiance to the final rendering of the sample location 170. Irradiance is a measure of the amount of light energy incoming or incident to a surface. Color is a specific frequency of this light energy. For example, the color perceived as red has a particular frequency whilst the color perceived as blue has another. In this example, only two of the rays 195-196 are mostly directed toward the indirect light 150 and contribute irradiance to the final rendering of the pixel 180 associated with sample location 170. To achieve the quality necessary for aesthetic images, many orders of magnitude more rays are required to illuminate this scene than to illuminate a well-lit scene, since so many rays are sent away from the indirect light 150 and do not contribute.” ¶ [0002]: “computer systems and processes for efficiently rendering a scene illuminated by indirect light using ray tracing.”),
the one or more exposure values being computed (¶ [0025]: “determine the incoming illumination, or irradiance, of the sample location 170.” ¶ [0013]: “determine the energy incoming (irradiance) to the sample location.”) based at least on:
a determination, (¶ [0024]: “sample location 170.”) associated with one or more subjects (¶ [0024]: “ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170.” ¶ [0012]: “sample locations on a surface element of an object in the scene.”) in the virtual environment (¶ [0023]: “scene 100”; ¶ [0023]: “a virtual world 110.” ¶ [0024]: “the virtual world 110”) (¶ [0023]: “FIG. 1 illustrates an exemplary scene 100 lit by indirect lighting 150. The scene 100 may be a snapshot taken by a virtual camera 130, viewing a virtual world 110. The box 120 highlights the portion of the virtual world 110 that may be visible to the virtual camera 130. Indirect lighting 150 may be light from an out-of-view light source, such as a light bulb, the sun, or the like. The scene 100 may be rendered using a conventional ray tracing engine.” ¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0028]:”FIG. 3 provides one exemplary representation of generating a point cloud representation of a scene. Scene 300 may represent a snapshot taken by a virtual camera 370, viewing a virtual world of teapots, including teapot 310. A point cloud representation 360 of teapot 310 may be computed. To generate the point cloud, the surfaces of different objects in the scene 300, such as teapot 310, may be subdivided into small micropolygons. The energy from each micropolygon may be stored by a point in the point cloud. A point may also store other information, including a position, a surface normal, an effective area, a point-radius, ID, or the like.” ¶ [0029]: “The point cloud representation 350 of the scene 300 may be generated in a pre-computation phase before computing the shading of the pixels in the scene.” NOTE: Using ray-tracing to render a virtual 3D scene requires “application data” defining the 3D scene, including the locations of everything in the scene. Thus, in order to render a scene using ray-tracing, the location of every object in the scene must be determined in order to perform ray tracing. In other words, in order to cast a ray into the virtual scene and determine whether it intersects any of the objects in the virtual scene, the location of each object in the scene, by necessity, must be determined.); and
casting one or more rays to sample incident light at the one or more locations (¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0025]: “When determining the shading of a sample location 170 on an object 160 in the scene 100 as a result of indirect lighting 150, ray 190 is reflected and/or refracted into rays 191-196. These rays 191-196 are sent out from the sample location 170 using a stochastic sampling method and are randomly distributed in a hemisphere around the sample location 170. These rays 191-196 sample the surrounding geometry of the scene 100 and then are integrated to determine the incoming illumination, or irradiance, of the sample location 170.”).
Whereas ARMSDEN may not be explicit as to, BENEDETTO discloses at least one processor (¶ [0132]: “graphics subsystem 1114“) comprising:
one or more circuits (e.g., ¶ [0132]: “a graphics processing unit (GPU) 1116 and graphics memory 1118 “) to render one or more images (e.g., ¶ [0132]: “the GPU 1116 includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and/or camera parameters for a scene.”) (¶ [0132]: “A graphics subsystem 1114 is further connected with data bus 1122 and the components of the device 1100. The graphics subsystem 1114 includes a graphics processing unit (GPU) 1116 and graphics memory 1118. Graphics memory 1118 includes a display memory (e.g., a frame buffer) used for storing pixel data for each pixel of an output image. Graphics memory 1118 can be integrated in the same device as GPU 1116, connected as a separate device with GPU 1116, and/or implemented within memory 1104. Pixel data can be provided to graphics memory 1118 directly from the CPU 1102. Alternatively, CPU 1102 provides the GPU 1116 with data and/or instructions defining the desired output images, from which the GPU 1116 generates the pixel data of one or more output images. The data and/or instructions defining the desired output images can be stored in memory 1104 and/or graphics memory 1118. In an embodiment, the GPU 1116 includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and/or camera parameters for a scene. The GPU 1116 can further include one or more programmable execution units capable of executing shader programs.” ¶ [0133]: “The graphics subsystem 1114 periodically outputs pixel data for an image from graphics memory 1118 to be displayed on display device 12, or to be projected by projection system 1140. Display device 12 can be any device capable of displaying visual information in response to a signal from the device 1100, including CRT, LCD, plasma, and OLED displays. Device 1100 can provide the display device 12 with an analog or digital signal, for example.”) representative of a virtual environment associated with a game application (e.g., ¶ [0014]: “a gaming environment associated with a particular player playing a gaming application,”) (¶ [0035]: “As shown in FIG. 1A, the gaming application may be executing locally at a client device 100 (e.g., through game title executing engine 111 and game logic 126) of the player 5, or may be executing at a back-end game title executing engine (e.g., game engine) 211 operating at a back-end game server 205 of a cloud game network or game cloud system (GCS) 201.” ¶ [0036]: “In particular, client device 100 may include a game title executing engine 111 (also referred to as game engine) configured for local execution of the gaming application. Game logic 126 of the gaming application runs on top of and/or in association with the game title executing engine 111. For example, the game logic 126 (of the gaming application) that is executing makes function calls down to the game title executing engine 111, such as for physics information, for texture information, etc. More specifically, game title executing engine 111 performs basic processor-based functions for executing the gaming application (e.g., through game logic 126) and services associated with the gaming application. For example, processor-based functions include 2D or 3D rendering, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracking, shadowing, culling, transformation, etc.” ¶ [0037]: “In another embodiment, the gaming application may be executing at a back-end game title executing engine 211 (also referred to as game engine) operating at a back-end game server 205 of GCS 201. Game title executing engine 211 performs similarly as game title executing engine 111, previously described. More particularly, the game title executing engine 211 may be operating within one of many physical and/or virtual game processors of game server 205. Game logic 126 may be executing on top of and/or in association with game title executing engine 211, similar to how game logic 126 executes on top of and/or in association with game title executing engine 111. For example, game logic 126 that is executing makes function calls down to the game title executing engine 211, such as for physics information, for texture information, etc. Game title executing engine 211 and the game logic 126 combined form an instance 175 of the gaming application, as will be further described below. Game server 205 is configured to manage and control a plurality of instances executing one or more gaming applications for one or more players.” ¶ [0038]: “The client device 100 may receive input from various types of input devices, such as game controllers 6, tablet computers 11, keyboards, and gestures captured by video cameras, mice, touch pads, etc. Client device 100 can be any type of computing device having at least a memory and a processor module that is capable of connecting to the game server 205 over network 150. Some examples of client device 100 include a personal computer (PC), a game console, a home theater device, a general purpose computer, mobile computing device, a tablet, a phone, or any other types of computing devices that can interact with the game server 205 to execute an instance of a gaming application.” ¶ [0039]: “Client device 100 is configured for receiving rendered images, and for displaying the rendered images on display 12. For example, through local game processing, the rendered images may be delivered by the local game executing engine 111, in response to input commands that are used to drive game play of player 5. As another example, through cloud based services the rendered images may be delivered by an instance 175 of a gaming application executing on game executing engine 211 of game server 205 in response to input commands that are used to drive game play of player 5. That is, through remote game processing, the rendered images may be delivered by the remote game title executing engine 211. In either case, client device 100 is configured to interact with the executing engine 211 or 111 in association with the game play of player 5, such as through input commands that are used to drive game play.” ¶ [0040]: “Also, client device 100 is configured to interact with the game server 205 to capture and store metadata and/or information of the game play of player 5 when playing a gaming application. The metadata includes information (e.g., game state, etc.) related to the game play. For example, the metadata may include location based information corresponding to a location of a character within a gaming world of the game play of the player 5. In one embodiment, the metadata could include snapshot information that is related to a point in the gaming application at which the snapshot was captured. Relevant information may be stored in one or more databases of database 140.” ¶ [0041]: “In one embodiment, the metadata may include snapshot information, wherein a snapshot provides information that enables execution of an instance of the gaming application beginning from a point in the gaming application associated with the capture of the corresponding snapshot.”), and
a determination, from game state information associated with the game application, of one or more locations associated with one or more subjects in the virtual environment (¶ [0040]: “Also, client device 100 is configured to interact with the game server 205 to capture and store metadata and/or information of the game play of player 5 when playing a gaming application. The metadata includes information (e.g., game state, etc.) related to the game play. For example, the metadata may include location based information corresponding to a location of a character within a gaming world of the game play of the player 5. In one embodiment, the metadata could include snapshot information that is related to a point in the gaming application at which the snapshot was captured. Relevant information may be stored in one or more databases of database 140.” ¶ [0043]: “More particularly, the captured metadata and/or information may include game state data that defines the state of the game play at that point. For example, game state data may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, etc. In that manner, game state data allows for the generation of the gaming environment that existed at the corresponding point in the game play of the gaming application. Game state data may also include the state of every device used for rendering the game play, such as states of CPU, GPU, memory, register values, program counter value, programmable DMA state, buffered data for the DMA, audio chip state, CD-ROM state, etc. Game state data may also identify which parts of the executable code need to be loaded to execute the gaming application from that point. The game state data may be stored locally or in game state database 145 of GCS 201.” ¶ [0045]: “In addition, the captured metadata and/or information may include random seed data that may be generated by an artificial intelligence (AI) module (not shown). The random seed data may not be part of the original game code, but may be added in an overlay to make the gaming environment seem more realistic and/or engaging to the user. That is, random seed data provides additional features for the gaming environment that exists at the corresponding point in the game play of the user. For example, AI characters may be randomly generated and provided in the overlay. The AI characters are not associated with any users playing the game, but are placed into the gaming environment to enhance the user's experience. As an illustration, these AI characters may randomly walk the streets in a city scene. In addition, other objects maybe generated and presented in an overlay. For instance, clouds in the background and birds flying through space may be generated and presented in an overlay. The random seed data may be stored locally or stored in random seed database 143 of GCS 201.” ¶ [0050]: “These various functions performed by a game engine include basic processor based functions for executing the gaming application and services associated with the gaming application. For example, processor based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. In addition, services for the gaming application include streaming, encoding, memory management, multi-thread management, quality of service (QoS), bandwidth testing, social networking, management of social friends, communication with social networks of friends, communication channels, texting, instant messaging, chat support, etc.” ¶ [0053]: “Each of the streaming servers 170 may be configured as having at least a memory and a processor module that is capable of executing the gaming application, such as through a corresponding instance of the gaming application, in support of a corresponding game play. For example, each streaming server 170 may be a server console, gaming console, computer, etc. More particularly, an instance of a gaming application is executed by a corresponding game title execution engine (e.g., game engine) that is running game logic corresponding to the gaming application, as previously described. Each server 205 is configured to stream data 105 (e.g., rendered images and/or frames of a corresponding game play) either directly or through game server 205 back to a corresponding client device through network 150. In that manner, a computationally complex gaming application may be executing at the back-end server in response to controller inputs received and forwarded by a corresponding client device. Each server is able to render images and/or frames that are then encoded (e.g., compressed) and streamed to the corresponding client device for display.” ¶ [0054]: “A plurality of players 115 accesses the GCS 201 via network 150, wherein players (e.g., players 5L, 5M . . . 5Z) access network 150 via corresponding client devices 100A . . . 100N. Each of client device 100A through 100N may be configured similarly as client device 100 of FIG. 1A, or may be configured as a thin client (e.g., client device 100B) providing interfaces with a back end server providing computational functionality. Corresponding client device 100 can be any type of computing device having at least a memory and a processor module that is capable of connecting to the game server 205 over network 150. Also, corresponding client device 100 is configured for generating rendered images executed by game title execution engine 111 executing locally, or executed by the game title execution engine 211 executing remotely, and for displaying the rendered images on a display, including a head mounted display (HMD) (not shown). As previously described, each of client devices 100 may receive input from various types of input devices 11, such as game controllers, tablet computers, keyboards, gestures captured by video cameras, mice touch pads, etc. For example, a client device 100A of a corresponding player 5A is configured for requesting access to gaming applications over a network 150, such as the internet, and for rendering data of a particular gaming application (e.g., video game) executed by the a corresponding steaming server 170a through 170n, as managed by gaming server 205 and delivered to a display device associated with the corresponding player 5A. As such, player 5A may be interacting through client device 100A with an instance of a gaming application executing on a corresponding streaming server as managed by game server 205.” ¶ [0106]: “Game state data allows for the generation of the gaming environment that existed at the corresponding point in the game play of the gaming application. For example, game state may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, the state of every device used for rendering the game play, etc.” ¶ [0107]: “At 620, the method includes tracking a plurality of interaction regions in a gaming world associated with the plurality of game plays, wherein each game play is currently engaging with a corresponding interaction region. The game state for each game play may provide and/or be analyzed to determine a corresponding location of a corresponding game play within the gaming environment of the gaming application. For example, a character in the game play as controlled by player P3 may be within a particular interaction region (e.g., area, region, town, house, map coordinate, etc.). In one embodiment, the interaction region is a map region of a corresponding gaming environment. The map region generally encompasses a large area, and may include one or more towns, or areas (e.g., woods, open area, park, etc.). In other embodiments, the interaction region is smaller than a region, such as an area, or town, or map coordinate. In another embodiment, the interaction region is a quest. Each of the game plays has a corresponding interaction region that can be determined through corresponding game state.” ¶ [0108]: “At 630, the method includes determining a first interaction region in a gaming world associated with a first game play of a first player (e.g., player P3) playing the gaming application. For example, the location of a character being controlled by player P3 may determine the first interaction region. That is, the location of the character (e.g., region, area, town, map coordinate, etc.) determines the interaction region. For purposes of illustration only, the first interaction region may be a map region, such as the region of Velen in the gaming application Witcher 3, wherein the town of Blackbough is located in Velen.” ¶ [0112]: “In particular, player P3 has a corresponding field of view (e.g., FOV P3) taken from a particular point 713 in the gaming environment 700 of the gaming application. As shown, player P3 is able to view object 730 within FOV P3. That is, the depth buffer of player P3 does not limit viewing object 730. In addition, player P2 (Landon) has a corresponding field of view (e.g., FOV′ P2) taken from point 712 in the gaming environment. However, a depth buffer analysis limits the field of view of player P2 because of object 750. That is, FOV′ P2 includes large object 750 (e.g., wall, rock wall, building, etc.). As such, in FOV′ P2 player P2 is unable to view object 730 (e.g., along ray trace or line 720).” NOTE: Clearly, as per ¶ [0112], the location of player characters and objects within the gaming environment are determined by game state information of the gaming application.).
Thus, in order to obtain a graphic gaming processor having the cumulative features and/or functionalities taught by ARMSDEN and BENEDETTO, it would have been obvious to one of ordinary skill in the art to have modified the at least one processor taught by ARMSDEN so as to include the functionality of rendering one or more images representative of a virtual environment associated with a game application and determining, from game state information associated with the game application, of one or more locations associated with one or more subjects in the virtual environment, as taught by BENEDETTO.
Regarding claim 19 (depends on claim 17), whereas ARMSDEN may not be entirely explicit as to, BENEDETTO teaches:
wherein the one or more subjects comprise one or more characters of the game application (¶ [0040]: “the metadata may include location based information corresponding to a location of a character within a gaming world”) (¶ [0035]: “As shown in FIG. 1A, the gaming application may be executing locally at a client device 100 (e.g., through game title executing engine 111 and game logic 126) of the player 5, or may be executing at a back-end game title executing engine (e.g., game engine) 211 operating at a back-end game server 205 of a cloud game network or game cloud system (GCS) 201.” ¶ [0036]: “In particular, client device 100 may include a game title executing engine 111 (also referred to as game engine) configured for local execution of the gaming application. Game logic 126 of the gaming application runs on top of and/or in association with the game title executing engine 111. For example, the game logic 126 (of the gaming application) that is executing makes function calls down to the game title executing engine 111, such as for physics information, for texture information, etc. More specifically, game title executing engine 111 performs basic processor-based functions for executing the gaming application (e.g., through game logic 126) and services associated with the gaming application. For example, processor-based functions include 2D or 3D rendering, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracking, shadowing, culling, transformation, etc.” ¶ [0039]: “Client device 100 is configured for receiving rendered images, and for displaying the rendered images on display 12. For example, through local game processing, the rendered images may be delivered by the local game executing engine 111, in response to input commands that are used to drive game play of player 5. As another example, through cloud based services the rendered images may be delivered by an instance 175 of a gaming application executing on game executing engine 211 of game server 205 in response to input commands that are used to drive game play of player 5. That is, through remote game processing, the rendered images may be delivered by the remote game title executing engine 211. In either case, client device 100 is configured to interact with the executing engine 211 or 111 in association with the game play of player 5, such as through input commands that are used to drive game play.” ¶ [0040]: “Also, client device 100 is configured to interact with the game server 205 to capture and store metadata and/or information of the game play of player 5 when playing a gaming application. The metadata includes information (e.g., game state, etc.) related to the game play. For example, the metadata may include location based information corresponding to a location of a character within a gaming world of the game play of the player 5. In one embodiment, the metadata could include snapshot information that is related to a point in the gaming application at which the snapshot was captured. Relevant information may be stored in one or more databases of database 140.” ¶ [0043]: “More particularly, the captured metadata and/or information may include game state data that defines the state of the game play at that point. For example, game state data may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, etc. In that manner, game state data allows for the generation of the gaming environment that existed at the corresponding point in the game play of the gaming application. Game state data may also include the state of every device used for rendering the game play, such as states of CPU, GPU, memory, register values, program counter value, programmable DMA state, buffered data for the DMA, audio chip state, CD-ROM state, etc. Game state data may also identify which parts of the executable code need to be loaded to execute the gaming application from that point. The game state data may be stored locally or in game state database 145 of GCS 201.” ¶ [0050]: “These various functions performed by a game engine include basic processor based functions for executing the gaming application and services associated with the gaming application. For example, processor based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadowing, culling, transformation, artificial intelligence, etc. In addition, services for the gaming application include streaming, encoding, memory management, multi-thread management, quality of service (QoS), bandwidth testing, social networking, management of social friends, communication with social networks of friends, communication channels, texting, instant messaging, chat support, etc.” ¶ [0053]: “Each of the streaming servers 170 may be configured as having at least a memory and a processor module that is capable of executing the gaming application, such as through a corresponding instance of the gaming application, in support of a corresponding game play. For example, each streaming server 170 may be a server console, gaming console, computer, etc. More particularly, an instance of a gaming application is executed by a corresponding game title execution engine (e.g., game engine) that is running game logic corresponding to the gaming application, as previously described. Each server 205 is configured to stream data 105 (e.g., rendered images and/or frames of a corresponding game play) either directly or through game server 205 back to a corresponding client device through network 150. In that manner, a computationally complex gaming application may be executing at the back-end server in response to controller inputs received and forwarded by a corresponding client device. Each server is able to render images and/or frames that are then encoded (e.g., compressed) and streamed to the corresponding client device for display.” ¶ [0106]: “Game state data allows for the generation of the gaming environment that existed at the corresponding point in the game play of the gaming application. For example, game state may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, the state of every device used for rendering the game play, etc.” ¶ [0107]: “At 620, the method includes tracking a plurality of interaction regions in a gaming world associated with the plurality of game plays, wherein each game play is currently engaging with a corresponding interaction region. The game state for each game play may provide and/or be analyzed to determine a corresponding location of a corresponding game play within the gaming environment of the gaming application. For example, a character in the game play as controlled by player P3 may be within a particular interaction region (e.g., area, region, town, house, map coordinate, etc.). In one embodiment, the interaction region is a map region of a corresponding gaming environment. The map region generally encompasses a large area, and may include one or more towns, or areas (e.g., woods, open area, park, etc.). In other embodiments, the interaction region is smaller than a region, such as an area, or town, or map coordinate. In another embodiment, the interaction region is a quest. Each of the game plays has a corresponding interaction region that can be determined through corresponding game state.” ¶ [0108]: “At 630, the method includes determining a first interaction region in a gaming world associated with a first game play of a first player (e.g., player P3) playing the gaming application. For example, the location of a character being controlled by player P3 may determine the first interaction region. That is, the location of the character (e.g., region, area, town, map coordinate, etc.) determines the interaction region. For purposes of illustration only, the first interaction region may be a map region, such as the region of Velen in the gaming application Witcher 3, wherein the town of Blackbough is located in Velen.” ¶ [0112]: “In particular, player P3 has a corresponding field of view (e.g., FOV P3) taken from a particular point 713 in the gaming environment 700 of the gaming application. As shown, player P3 is able to view object 730 within FOV P3. That is, the depth buffer of player P3 does not limit viewing object 730. In addition, player P2 (Landon) has a corresponding field of view (e.g., FOV′ P2) taken from point 712 in the gaming environment. However, a depth buffer analysis limits the field of view of player P2 because of object 750. That is, FOV′ P2 includes large object 750 (e.g., wall, rock wall, building, etc.). As such, in FOV′ P2 player P2 is unable to view object 730 (e.g., along ray trace or line 720).” NOTE: Clearly, as per at least ¶ [0112], the location of player characters and objects within the gaming environment are determined by game state information of the gaming application.).
Thus, in order to obtain a graphic processing system having the cumulative features and/or functionalities taught by ARMSDEN and BENEDETTO, it would have been obvious to one of ordinary skill in the art to have modified the graphic processing system taught by ARMSDEN so that the one or more subjects comprise one or more characters of the application, as taught by BENEDETTO.
Regarding claim 20 (depends on claim 17), ARMSDEN discloses: the one or more circuits are further to,
based at least on the determination of the one or more locations (¶ [0024]: “ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170.”), instantiate one or more virtual light meters at the one or more locations (¶ [0024]: “sample location 170 in the virtual world 110.” ¶ [0025]: “a sample location 170 on an object 160 in the scene 100”; ¶ [0025]: “a hemisphere around the sample location 170.”), wherein the one or more rays are cast to sample the incident light based at least on the instantiation (¶ [0025]: “ray 190 is reflected and/or refracted into rays 191-196. These rays 191-196 are sent out from the sample location 170 using a stochastic sampling method and are randomly distributed in a hemisphere around the sample location 170.”) (¶ [0024]: “A snapshot of a scene 100 using a ray tracing approach may be rendered by calculating the path of rays from a virtual camera 130 through pixels in an image plane 140 and then into the virtual world 110. For example, consider ray 190. Ray 190 originates at virtual camera 130, passing through the image plane 140 at pixel 180. The ray 190 enters the virtual world 110 and encounters its first object 160 at sample location 170. In this way, the rendered pixel 180 on the image plane 140 may be associated with the shaded sample location 170 in the virtual world 110. In one embodiment, only sample locations within the box 120 may be associated with pixels on the image plane 140 since only these locations are visible to the virtual camera 130. However, sample locations anywhere within the virtual world 110 may affect the shading, or color, of sample locations within the box 120.” ¶ [0025]: “When determining the shading of a sample location 170 on an object 160 in the scene 100 as a result of indirect lighting 150, ray 190 is reflected and/or refracted into rays 191-196. These rays 191-196 are sent out from the sample location 170 using a stochastic sampling method and are randomly distributed in a hemisphere around the sample location 170. These rays 191-196 sample the surrounding geometry of the scene 100 and then are integrated to determine the incoming illumination, or irradiance, of the sample location 170.”).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over ARMSDEN et al. (US 2013/0002671) in view of CARR et al. (US 8223148, hereinafter “CARR”).
Regarding claim 6 (depends on claim 1), whereas ARMSDEN is not explicit as to, CARR teaches:
wherein the sampled incident light (e.g., col. 2, lines 8-9: “irradiance values”; col. 2, lines 16-17: “diffuse indirect lighting computation”; col. 2, lines 39-40: “a new irradiance value for that point on the surface is calculated”) comprises a subset of incident light values (e.g. col. 2, lines 36-37: “enough nearby irradiance points in space whose confidence bounds overlap the point,” i.e., the subset of pixels in the scene 600 (FIG. 9) that are used to generate (col. 14, lines 27-28): “irradiance values at a low resolution”; col. 1, line 43: “a set of carefully chosen sample points.” col. 11, lines 28-29: “an irradiance caching algorithm that sparsely computes irradiance in the environment”; FIG. 11, step 720: “Perform a low resolution pass on some points in the scene to partially populate the irradiance data” col 12, line 28: “a set of sparse locations in the scene”; col. 13, line6-7: “sample locations (sample points)”; col. 19, lines 9-10: “for each of a subset of surface points in a scene,”) and
the computing the one or more exposure values (e.g., col. 2, lines 36-38: “If there are enough nearby irradiance points in space whose confidence bounds overlap the point, then interpolation is performed to determine the irradiance for the point.”) includes retrieving, using a screen-space light buffer (e.g., col. 2, lines 29-34: “for each pixel in the scene, a ray is fired into the pixel. Each time a ray hits a point on a surface in the scene, an irradiance data structure, also referred to as an irradiance cache, that caches or stores determined irradiance values for surface points in the scene, may be checked”), the subset of incident light values (col. 2 lines 7-8: “sparsely computes irradiance in the environment”; col. 2, lines 26-41: “To perform irradiance caching, in one embodiment, rays may be fired, for example starting at the bottom of the scene. The rays may be fired from the perspective of a "viewer" of the scene. In one embodiment, for each pixel in the scene, a ray is fired into the pixel. Each time a ray hits a point on a surface in the scene, an irradiance data structure, also referred to as an irradiance cache, that caches or stores determined irradiance values for surface points in the scene, may be checked to see if there are nearby irradiance points on the surface. If there are enough nearby irradiance points in space whose confidence bounds overlap the point, then interpolation is performed to determine the irradiance for the point. If there are no or not enough nearby irradiance points, a new irradiance value for that point on the surface is calculated and added to the irradiance cache.” col. 4, lines 30-31: “determine the light that reaches each surface point in a scene.”).
Thus, in order to further reduce the computational expense for indirect lighting computation (see CARR, col. 2, lines 2-4), it would have been obvious to one of ordinary skill in the art to have modified computing the one or more exposure values in the graphic rendering system taught by ARMSDEN so as to include retrieving, using a screen-space light buffer, a subset of incident light values comprising the sampled incident light, as taught by CARR.
Claims 7, 8, 10 and 15 is rejected under 35 U.S.C. 103 as being unpatentable over ARMSDEN et al. (US 2013/0002671) in view of MARAIS (US 2020/0184612).
Regarding claim 7 (depends on claim 1), whereas ARMSDEN may not be explicit as to, MARAIS teaches:
wherein the rendering is based at least on adjusting (¶ [0006]: “The luminance value of the bright region is adjusted”), using the one or more exposure values (¶ [0006]: “luminance value”), a relative balance in exposure between a set of pixels (¶ [0006]: “A bright region”) and one or more other pixels of an image associated with the one or more images (¶ [0008]: “images including bright regions having the classifications that may be employed in the HDR content (e.g., sun, moon, fires, explosions, specular highlights etc.).”) (¶ [0006]: “A bright region is identified in an image. The bright region is classified into an assigned classification. A luminance value of the bright region is determined and compared to predefined luminance values corresponding to the classification. The luminance value of the bright region is adjusted to match the predefined luminance values where there is a mismatch.” ¶ [0007]: “Such correction may be applied to the HDR image to produce a corrected HDR image. Corrected luminance values may be generated that match another bright region with the same classification in the same image. In addition to generating corrections to incorrect luminance values, a scale adjustment may be applied to luminance values across the bright region to generate a scaled and adjusted luminance value.” ¶ [0008]: “Identification and classification of the bright regions of the image may be performed in various ways, such as by a suitably trained machine learning model. Such a model may be trained using images including bright regions having the classifications that may be employed in the HDR content (e.g., sun, moon, fires, explosions, specular highlights etc.). The predefined luminance values for each class of bright region may likewise be determined by a suitably trained machine learning model. Such a model would be trained not only with images of the expected classification, but also having luminance values is the desired range for such classification.”).
Thus, in order to obtain a corrected final rendered image having bright regions with appropriate luminance values, it would have been obvious to one of ordinary skill in the art to have modified the rendering system taught by ARMSDEN such that the rendering is based at least on adjusting, using the one or more exposure values, a relative balance in exposure between a set of pixels and one or more other pixels of an image associated with the one or more images, as taught by MARAIS.
Regarding claim 8 (depends on claim 1), whereas ARMSDEN may not be explicit as to, MARAIS teaches:
wherein the one or more images (¶ [0007]: “Such correction may be applied to the HDR image to produce a corrected HDR image.”) are generated based at least on modifying a relative balance in exposure (¶ [0006]: “A bright region is identified in an image. The bright region is classified into an assigned classification. A luminance value of the bright region is determined and compared to predefined luminance values corresponding to the classification. The luminance value of the bright region is adjusted to match the predefined luminance values where there is a mismatch.” ¶ [0007]: “Such correction may be applied to the HDR image to produce a corrected HDR image. Corrected luminance values may be generated that match another bright region with the same classification in the same image. In addition to generating corrections to incorrect luminance values, a scale adjustment may be applied to luminance values across the bright region to generate a scaled and adjusted luminance value.”) depicted in one or more initial rendered images (¶ [0006]: “HDR content that have incorrect and/or inconsistent tones,” ¶ [0007]: “Such correction may be applied to the HDR image to produce a corrected HDR image.”).
Thus, in order to obtain a corrected final rendered image having bright regions with appropriate luminance values, it would have been obvious to one of ordinary skill in the art to have modified the method taught by ARMSDEN so as to incorporate generating the one or more images based at least on modifying a relative balance in exposure depicted in one or more initial rendered images, as taught by MARAIS.
Regarding claim 10 (depends on claim 1), whereas ARMSDEN may not be explicit as to, MARAIS teaches:
wherein the rendering is based at least on increasing or decreasing one or more second exposure values (¶ [0006]: “The luminance value of the bright region is adjusted” ¶ [0007]: “Corrected luminance values may be generated”) based at least on the one or more second exposure values corresponding to one or more different subjects (¶ [0006]: “A bright region”) than the one or more subjects or one or more environmental features of the virtual environment (¶ [0007]: “Corrected luminance values may be generated that match another bright region with the same classification in the same image.”) (¶ [0006]: “A bright region is identified in an image. The bright region is classified into an assigned classification. A luminance value of the bright region is determined and compared to predefined luminance values corresponding to the classification. The luminance value of the bright region is adjusted to match the predefined luminance values where there is a mismatch.” ¶ [0007]: “Such correction may be applied to the HDR image to produce a corrected HDR image. Corrected luminance values may be generated that match another bright region with the same classification in the same image. In addition to generating corrections to incorrect luminance values, a scale adjustment may be applied to luminance values across the bright region to generate a scaled and adjusted luminance value.” ¶ [0008]: “Identification and classification of the bright regions of the image may be performed in various ways, such as by a suitably trained machine learning model. Such a model may be trained using images including bright regions having the classifications that may be employed in the HDR content (e.g., sun, moon, fires, explosions, specular highlights etc.). The predefined luminance values for each class of bright region may likewise be determined by a suitably trained machine learning model. Such a model would be trained not only with images of the expected classification, but also having luminance values is the desired range for such classification.”).
Thus, in order to obtain a corrected rendered image having bright regions with appropriate luminance values, it would have been obvious to one of ordinary skill in the art to have modified the method taught by ARMSDEN so that the rendering is based at least on increasing or decreasing one or more second exposure values based at least on the one or more second exposure values corresponding to one or more different subjects than the one or more subjects or one or more environmental features of the virtual environment, as taught by MARAIS.
Regarding claim 15 (depends on claim 11), whereas ARMSDEN may not be explicit as to, MARAIS teaches:
wherein the rendering is based at least on adjusting (¶ [0006]: “The luminance value of the bright region is adjusted”), using the one or more exposure values (¶ [0006]: “luminance value”), a relative balance in exposure between a set of pixels (¶ [0006]: “A bright region”) and one or more other pixels of an image associated with the one or more images (¶ [0008]: “images including bright regions having the classifications that may be employed in the HDR content (e.g., sun, moon, fires, explosions, specular highlights etc.).”) (¶ [0006]: “A bright region is identified in an image. The bright region is classified into an assigned classification. A luminance value of the bright region is determined and compared to predefined luminance values corresponding to the classification. The luminance value of the bright region is adjusted to match the predefined luminance values where there is a mismatch.” ¶ [0007]: “Such correction may be applied to the HDR image to produce a corrected HDR image. Corrected luminance values may be generated that match another bright region with the same classification in the same image. In addition to generating corrections to incorrect luminance values, a scale adjustment may be applied to luminance values across the bright region to generate a scaled and adjusted luminance value.” ¶ [0008]: “Identification and classification of the bright regions of the image may be performed in various ways, such as by a suitably trained machine learning model. Such a model may be trained using images including bright regions having the classifications that may be employed in the HDR content (e.g., sun, moon, fires, explosions, specular highlights etc.). The predefined luminance values for each class of bright region may likewise be determined by a suitably trained machine learning model. Such a model would be trained not only with images of the expected classification, but also having luminance values is the desired range for such classification.”).
Thus, in order to obtain a corrected rendered image having bright regions with appropriate luminance values, it would have been obvious to one of ordinary skill in the art to have modified the rendering system taught by ARMSDEN such that the rendering is based at least on adjusting, using the one or more exposure values, a relative balance in exposure between a set of pixels and one or more other pixels of an image associated with the one or more images, as taught by MARAIS.
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over ARMSDEN et al. (US 2013/0002671) in view of BENEDETTO et al. (US 2020/0269143), further in view of BLACKMON et al. (US 2017/0169602).
Regarding claim 18 (depends on claim 17), whereas ARMSDEN and BENEDETTO are not explicit as to, BLACKMON teaches:
wherein the determination of the one or more locations is based at least on the game state information indicating that the one or more subjects are a focus of a scene (¶ [0106]: “In other examples, a region of interest may be determined by the content in the image. For example, if an object in a scene is desired to be the focus of attention then a region of the image containing that object may be determined to be a region of interest such that it is rendered using ray tracing. This may be the case irrespective of a user's gaze and irrespective of the position of the object in the image, i.e. the object does not need to be centrally located. As another example, a region of the image containing a moving object may be determined to be a region of interest such that it is processed using a ray tracing technique. As another example, the gaze tracking logic could identify “hot spots” where the user's eyes are determined to frequently be attracted to (e.g. two bits of the image that are both capturing the user's attention, where the gaze is bouncing between them), and these hot spots could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, foveal focus regions from the past (as a subset or simplification of the former) could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, future foveal destinations could be predicted based on a model of the motion of the eye and/or the saccade mechanism, and those predicted future foveal destinations could be determined to be regions of interest such that they are processed using a ray tracing technique. As another example, areas of high image complexity could be identified according to a complexity metric (e.g. number of primitives in a region or number of edges in a region, etc.) and those identified areas could be determined to be regions of interest such that they are processed using a ray tracing technique. The complexity metric may be based on known information content and/or augmentations within the image, for example signage may be rendered in high detail because it is likely the user will gaze there.“ ¶ [0073]: “The region identification logic 708 is configured to identify one or more regions of the initial image. The identified regions are regions for which ray traced data is to be computed (i.e. regions in which more rays are to be processed by the ray tracing logic 710) or regions for which more rays are to be traced (i.e. regions where the ray density is to be increased). In other words, the region identification logic 708 detects where detail is needed or desired in the image.” ¶ [0073]: “As another example, a region corresponding to a particular object in a scene (e.g. a character in a game) may be identified such that ray traced data is added to the particular object even if the particular object is not in the foveal region or in high-detail.” ¶ [0001]: “Graphics processing systems are typically configured to receive graphics data, e.g. from an application running on a computer system, and to render the graphics data to provide a rendering output. For example, the graphics data provided to a graphics processing system may describe geometry within a three dimensional (3D) scene to be rendered, and the rendering output may be a rendered image of the scene. The graphics data may include “primitives” representing geometric shapes, which describe surfaces of structures in the scene. A common primitive shape is a triangle, but primitives may be other shapes and may be lines or points. Objects can be composed of one or more (e.g. hundreds, thousands or millions) of such primitives.” ¶ [0002]: “In some systems, sequences of images (or “frames”) are rendered and displayed in real-time. The frame rate of the sequence of images will typically depend on the application for which the images are rendered. To give an example, a gaming application may send images for rendering at a frame rate of 25 frames per second, but other frame rates may be used in other examples. Generally, increasing the frame rate of the sequence of images to be rendered will increase the processing load on the graphics processing system.” ¶ [0006]: “Gaze tracking (or “eye tracking”) may be used to determine where a user is looking to thereby determine where the foveal region 104 is in relation to the image 102. The foveal region 104 can be rendered at a high resolution and high geometric LOD, whereas the rest of the image 102 can be rendered at a lower resolution and lower geometric LOD. The rendered foveal region 104 can then be blended with the rendering of the rest of the image 102 to form a rendered image. Therefore, in the periphery of the image 102, unperceived detail can be omitted such that fewer primitives may be rendered, at a lower pixel density. Since the foveal region 104 is generally a small proportion (e.g. ˜1%) of the area of the image 102, substantial rendering computation savings can be achieved by reducing the resolution and geometric LOD of the periphery compared to the rendering of the foveal region. Furthermore, since human visual acuity decreases rapidly away from the foveal region, high image quality is maintained for the image as perceived by a user who directs their gaze at the centre of the foveal region 104.” ¶ [0008]: “In some examples, foveated rendering is described for rendering an image whereby a ray tracing technique is used to render a region of interest of the image, and a rasterisation technique is used to render other regions of the image. The rendered region of interest of the image is then combined (e.g. blended) with the rendered other regions of the image to form a rendered image. The region of interest may correspond to a foveal region of the image. Ray tracing naturally provides high detail and photo-realistic rendering, which human vision is particularly sensitive to in the foveal region; whereas rasterisation techniques are suited for providing temporal smoothing and anti-aliasing in a simple manner, and is therefore suited for use in the regions of the image that a user will see in the periphery of their vision.”).
Thus, in order to reduce the computational load for rendering each image, it would have been obvious to one of ordinary skill in the art to have modified the image rendering method taught by ARMSDEN so as to include determining the one or more locations based at least on the game state data indicating that the one or more subjects are a focus of a scene, as taught by BLACKMON.
Conclusion
At present, it is not apparent to the examiner which part of the application could serve as a basis for new and allowable claims. However, should the applicant nevertheless regard some particular matter as patentable, the examiner encourages applicant to appropriately amend the claims to include such matter and to indicate in the REMARKS the difference(s) between the prior art and the claimed invention as well as the significance thereof.
Furthermore, should applicant decide to amend the claims, examiner respectfully requests that the applicant please indicate in the REMARKS from which page(s), line(s) or claim(s) of the originally filed application that any amendments are derived. See MPEP § 2163(II)(A) (There is a strong presumption that an adequate written description of the claimed invention is present in the specification as filed, Wertheim, 541 F.2d at 262, 191 USPQ at 96; however, with respect to newly added or amended claims, applicant should show support in the original disclosure for the new or amended claims.).
A shortened statutory period for reply to this action is set to expire THREE MONTHS from the mailing date of this action. Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action. Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 USC § 133).
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT PEREN who can be reached by telephone at (571) 270-7781, or via email at vincent.peren@uspto.gov. The examiner can normally be reached on Monday-Friday from 10:00 A.M. to 6:00 P.M.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KING POON, can be reached at telephone number (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/VINCENT PEREN/
Examiner, Art Unit 2617
/KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights