DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
2. The information disclosure statement (IDS) submitted on 12/20/2024. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
3. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
4. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
6. Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Vorba (US 2022/0036641 A1) in view of Muller et al. (US 2022/0284657 A1).
7. With reference to claim 1, Vorba teaches A computer-implemented method for performing path guiding, the computer-implemented method comprising: receiving a representation of a three-dimensional (3D) scene and a virtual camera location; (“A path guiding method as described herein can provide realistic rendering of complex lighting situations for use in computer graphics and animation, as described herein. ... One general aspect includes a computer-implemented method for generating a mask for a light source used in rendering virtual scenes under control of one or more computer systems configured with executable instructions, perform steps of: determining a bounding volume (e.g., a bounding box) for a scene based on a frustum or spherical aperture of a virtual camera having a virtual camera position in the scene; generating a path-traced first image of a portion of the scene within the bounding box by projecting a plurality of first light paths from the virtual camera position, where each first light path of the plurality of first light paths includes a plurality of vertices;” [0008]) Vorba also teaches generating a lightpath that originates at the virtual camera location and reaches a point included in the 3D scene; (“generating a path-traced first image of a portion of the scene within the bounding box by projecting a plurality of first light paths from the virtual camera position, where each first light path of the plurality of first light paths includes a plurality of vertices; storing a light paths subset of the plurality of first light paths, where a stored light path in the light paths subset is a light path that exits at the light source; removing objects from the scene that are sampled less than a predetermined threshold by the first light paths to form a modified virtual scene;” [0008]) Vorba further teaches selecting, from a set of candidate directions, a direction in which to extend the generated lightpath from the point, extending the generated lightpath in the selected direction; (“As the rendering system incrementally constructs paths vertex by vertex, the path guiding method biases random decisions taken in the process to guide the paths towards important regions in the scene. In these regions, paths are disproportionately likely to make significant contributions to the image. The method can bias or change the probability of multiple decisions along each path, such as choosing direction after each scattering event, free path sampling in volumes, absorption, or choosing of a light source for connection.” [0040] “path guiding can be used to learn the optimal sampling distribution for sampling the position of emitted light paths, based on the methods described herein. Such methods may for example be based on keeping a number of 2D guiding distributions for position sampling. Each distribution is relevant for a compact set of directions, while the union of sets forms the whole sphere of directions. In other words, after sampling the initial direction of the emitted light path, the method finds the corresponding guiding distribution for sampling the starting position.” [0082]) Vorba teaches generating a two-dimensional (2D) rendering of the 3D scene based at least on the generated lightpath. (“The system may then generate a path-traced image by using the masked light source to generate a plurality of light paths and storing a subset of the paths that exit at the virtual camera. The system may then use quad trees to identify a further subset of paths that are poorly sampled (e.g., sampled less than a pre-determined threshold). For at least some light paths of this further subset, the system may compute a distribution (e.g., a 2D Gaussian distribution) for each vertex, sample the vertex positions from the computed distributions, and then construct a guide path from the sampled vertices. Next, the system may modify the guide path by iteratively resampling the positions of the vertices, until the guide path exits at the virtual camera. The system can then construct a path-guided image using only the modified guide paths and combine the path-traced image with the path-guided image to form a fully rendered scene image storable in computer memory.” [0057] “FIG. 9 illustrates an exemplary placement for guided path samples 920 in a two-dimensional (2D) image space 900.” [0116])
PNG
media_image1.png
708
523
media_image1.png
Greyscale
Vorba does not explicitly teach the selecting is based at least on one or estimates of incident light characteristics associated with the 3D scene and predicted by a machine learning model; This is what Muller teaches (“a neural network radiance cache model processes a 3D position associated with a light transport path through a scene to produce a radiance prediction (estimated radiance or reflected light) at the 3D position. In an embodiment, the light transport path starts from a camera. In an embodiment, the light transport path starts at a position between a left and right eye. In an embodiment, the light transport path starts at an intermediate vertex of a rendering path. In an embodiment, the neural network radiance cache is the neural radiance cache 135. In an embodiment, the 3D position is a vertex. In an embodiment, a second 3D position that is associated with the rendered path and at which a second radiance prediction is computed, is importance sampled based on the 3D position. The 3D position and the second 3D position define a segment of the rendered path and a direction originating at the 3D position towards a next 3D position (e.g., the second 3D position) is chosen with probability proportional to the predicted reflected light at the second 3D position.” [0062]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Muller into Vorba, in order to shorten the path and reduce noise.
8. With reference to claim 2, Vorba teaches the representation of the 3D scene includes surface or lighting characteristics associated with one or more objects, surfaces, or light sources included in the 3D scene. (“One general aspect includes a computer-implemented method for generating a mask for a light source used in rendering virtual scenes under control of one or more computer systems configured with executable instructions, perform steps of: determining a bounding volume (e.g., a bounding box) for a scene based on a frustum or spherical aperture of a virtual camera having a virtual camera position in the scene; generating a path-traced first image of a portion of the scene within the bounding box by projecting a plurality of first light paths from the virtual camera position, where each first light path of the plurality of first light paths includes a plurality of vertices;” [0008] “FIG. 8 is a schematic overview of the guiding method employing full paths. Visible are light paths 800 emitted from a virtual eye, camera, or sensor 810, reflecting off of a surface 820 to a light source 830. Some paths also refract through a transparent object 840.” [0109])
9. With reference to claim 3, Vorba teaches generating the set of candidate directions based on a distribution function, wherein the distribution function includes a uniform distribution, a bidirectional scattering distribution function (BxDF) distribution, a cosine distribution, or a Next Event Estimation (NEE) distribution. (“Path sampling can benefit from directional guiding as long as the path guiding method can efficiently decide whether it is worthwhile to continue tracing the path or to terminate it and start tracing a new path from the camera. It may be important for the method to guide this decision, as it may be important to achieving high performance in simple scenes where most of the energy is transported over short paths, as well as in more complex scenes where light undergoes many scattering events before reaching the camera. This problem is addressed by guided Russian roulette and splitting (also known as adjoint-driven Russian roulette and splitting). Methods for directional guiding may be classified for example according to their capability of handling surfaces with low roughness or volumes with high mean cosine. Ideally, to generate high quality samples, choosing direction at a scattering event should consider the product of incident illumination and BSDF (or phase function in volumes).” [0052] “Volume sampling and Russian roulette can be used to guide the full paths in path space. Guiding new samples along full guide paths (instead of marginalized distributions which only guide low dimensional parts at a time) transparently includes all aspects of the high dimensional path space, including but not limited to path length, BSDF, incident illumination, and free distances in volumes.” [0101])
10. With reference to claim 4, Vorba does not explicitly teach the machine learning model includes a neural network that estimates an amount of incident light arriving at the point included in the 3D scene from a given direction. This is what Muller teaches (“the path tracer 130 receives scene and camera data and traces short rendering paths, inputting intersection data to the neural (network) radiance cache 135 to generate a reflected light approximation. The intersection data may include one or more of a 3D position of a vertex, a direction of incidence (view direction), and attributes such as material properties, surface normal vector (if the query vertex is not in a volume), and the like.” [0047] “a neural network radiance cache model processes a 3D position associated with a light transport path through a scene to produce a radiance prediction (estimated radiance or reflected light) at the 3D position. In an embodiment, the light transport path starts from a camera. In an embodiment, the light transport path starts at a position between a left and right eye. In an embodiment, the light transport path starts at an intermediate vertex of a rendering path. In an embodiment, the neural network radiance cache is the neural radiance cache 135. In an embodiment, the 3D position is a vertex. In an embodiment, a second 3D position that is associated with the rendered path and at which a second radiance prediction is computed, is importance sampled based on the 3D position. The 3D position and the second 3D position define a segment of the rendered path and a direction originating at the 3D position towards a next 3D position (e.g., the second 3D position) is chosen with probability proportional to the predicted reflected light at the second 3D position.” [0062] “at step 285 the path is terminated, the first vertex is a terminal 3D position and the method 270 proceeds to step 290. Otherwise, at step 280, the path is extended by another segment before returning to step 285. At step 290, the neural radiance cache 135 processes the terminal 3D position to produce a terminal radiance prediction (estimated radiance or reflected light) at the terminal 3D position.” [0082]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Muller into Vorba, in order to shorten the path and reduce noise.
11. With reference to claim 5, Vorba teaches selecting the direction in which to extend the generated lightpath is further based on a bidirectional scattering distribution function (BxDF). (“As the rendering system incrementally constructs paths vertex by vertex, the path guiding method biases random decisions taken in the process to guide the paths towards important regions in the scene. In these regions, paths are disproportionately likely to make significant contributions to the image. The method can bias or change the probability of multiple decisions along each path, such as choosing direction after each scattering event, free path sampling in volumes, absorption, or choosing of a light source for connection.” [0040] “Path sampling can benefit from directional guiding as long as the path guiding method can efficiently decide whether it is worthwhile to continue tracing the path or to terminate it and start tracing a new path from the camera. It may be important for the method to guide this decision, as it may be important to achieving high performance in simple scenes where most of the energy is transported over short paths, as well as in more complex scenes where light undergoes many scattering events before reaching the camera. This problem is addressed by guided Russian roulette and splitting (also known as adjoint-driven Russian roulette and splitting). Methods for directional guiding may be classified for example according to their capability of handling surfaces with low roughness or volumes with high mean cosine. Ideally, to generate high quality samples, choosing direction at a scattering event should consider the product of incident illumination and BSDF (or phase function in volumes).” [0052] “path guiding can be used to learn the optimal sampling distribution for sampling the position of emitted light paths, based on the methods described herein. Such methods may for example be based on keeping a number of 2D guiding distributions for position sampling. Each distribution is relevant for a compact set of directions, while the union of sets forms the whole sphere of directions. In other words, after sampling the initial direction of the emitted light path, the method finds the corresponding guiding distribution for sampling the starting position.” [0082] “Volume sampling and Russian roulette can be used to guide the full paths in path space. Guiding new samples along full guide paths (instead of marginalized distributions which only guide low dimensional parts at a time) transparently includes all aspects of the high dimensional path space, including but not limited to path length, BSDF, incident illumination, and free distances in volumes.” [0101])
12. With reference to claim 6, Vorba does not explicitly teach approximating, via a neural radiance cache, an integrated amount of reflected radiance from the point included in the 3D scene into a direction from which the lightpath reached the point. This is what Muller teaches (“the path tracer 130 receives scene and camera data and traces short rendering paths, inputting intersection data to the neural (network) radiance cache 135 to generate a reflected light approximation. The intersection data may include one or more of a 3D position of a vertex, a direction of incidence (view direction), and attributes such as material properties, surface normal vector (if the query vertex is not in a volume), and the like.” [0047] “a neural network radiance cache model processes a 3D position associated with a light transport path through a scene to produce a radiance prediction (estimated radiance or reflected light) at the 3D position. In an embodiment, the light transport path starts from a camera. In an embodiment, the light transport path starts at a position between a left and right eye. In an embodiment, the light transport path starts at an intermediate vertex of a rendering path. In an embodiment, the neural network radiance cache is the neural radiance cache 135. In an embodiment, the 3D position is a vertex. In an embodiment, a second 3D position that is associated with the rendered path and at which a second radiance prediction is computed, is importance sampled based on the 3D position. The 3D position and the second 3D position define a segment of the rendered path and a direction originating at the 3D position towards a next 3D position (e.g., the second 3D position) is chosen with probability proportional to the predicted reflected light at the second 3D position.” [0062] “at step 285 the path is terminated, the first vertex is a terminal 3D position and the method 270 proceeds to step 290. Otherwise, at step 280, the path is extended by another segment before returning to step 285. At step 290, the neural radiance cache 135 processes the terminal 3D position to produce a terminal radiance prediction (estimated radiance or reflected light) at the terminal 3D position.” [0082]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Muller into Vorba, in order to shorten the path and reduce noise.
13. With reference to claim 7, Vorba does not explicitly teach the machine learning model includes a first neural network that estimates an amount of incident direct illumination at the point included in the 3D scene and a second neural network that estimates an amount of incident indirect illumination at the point included in the 3D scene. This is what Muller teaches (“the path tracer 130 receives scene and camera data and traces short rendering paths, inputting intersection data to the neural (network) radiance cache 135 to generate a reflected light approximation. The intersection data may include one or more of a 3D position of a vertex, a direction of incidence (view direction), and attributes such as material properties, surface normal vector (if the query vertex is not in a volume), and the like.” [0047] “a neural network radiance cache model processes a 3D position associated with a light transport path through a scene to produce a radiance prediction (estimated radiance or reflected light) at the 3D position. In an embodiment, the light transport path starts from a camera. In an embodiment, the light transport path starts at a position between a left and right eye. In an embodiment, the light transport path starts at an intermediate vertex of a rendering path. In an embodiment, the neural network radiance cache is the neural radiance cache 135. In an embodiment, the 3D position is a vertex. In an embodiment, a second 3D position that is associated with the rendered path and at which a second radiance prediction is computed, is importance sampled based on the 3D position. The 3D position and the second 3D position define a segment of the rendered path and a direction originating at the 3D position towards a next 3D position (e.g., the second 3D position) is chosen with probability proportional to the predicted reflected light at the second 3D position.” [0062] “at step 285 the path is terminated, the first vertex is a terminal 3D position and the method 270 proceeds to step 290. Otherwise, at step 280, the path is extended by another segment before returning to step 285. At step 290, the neural radiance cache 135 processes the terminal 3D position to produce a terminal radiance prediction (estimated radiance or reflected light) at the terminal 3D position.” [0082] “The neural network cache 135 may be implemented as a fully-connected neural network in a GPU programming language to take full advantage of the GPU's memory hierarchy. In the context of the following description, a fully-connected neural network configured for execution by a processor by limiting slow global memory accesses to reading and writing inputs to and outputs from the fully-connected neural network is referred to as a “fully-fused” neural network.” [0086] “the set of training data may be used to train one or more neural networks within the control variate neural network system 100.” [0183]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Muller into Vorba, in order to shorten the path and reduce noise.
14. With reference to claim 8, Vorba teaches discarding a generated lightpath that exits the 3D scene without reaching any of one or more light sources included in the 3D scene. (“determining a bounding volume (e.g., a bounding box) for a scene based on a frustum or spherical aperture of a virtual camera having a virtual camera position in the scene; generating a path-traced first image of a portion of the scene within the bounding box by projecting a plurality of first light paths from the virtual camera position, where each first light path of the plurality of first light paths includes a plurality of vertices; storing a light paths subset of the plurality of first light paths, where a stored light path in the light paths subset is a light path that exits at the light source; removing objects from the scene that are sampled less than a predetermined threshold by the first light paths to form a modified virtual scene; generating an initial mask for the light source from the light paths subset such that at least some positions on the light source is assigned an emission probability in the initial mask based on a density of first light paths exiting at that position on the light source, thereby generating a masked light source data structure; refining the initial mask, to form a refined mask, by generating successive second images of the scene by repeatedly: (a) generating a path-traced second image using the masked light source data structure and the modified virtual scene to generate a plurality of second light paths; (b) storing the second light paths that exit at the virtual camera position; and (c) modifying an emission probability of each position on the light source based on a cumulative average across a plurality of the successive second images of a density of second light paths emitted from that position that exit at the virtual camera.” [0008] “the system may determine a bounding box for a scene based on a frustum of a virtual camera, and then generate a path-traced “oracle” image of the scene (or, more properly, a portion of the scene within the bounding box) by projecting a plurality of oracle light paths from the virtual camera, and storing a subset of the oracle paths that exit at a light source. The system may then remove objects from the scene that are poorly sampled by the oracle paths (e.g., sampled less than a predetermined threshold). … The system may then generate a path-traced image by using the masked light source to generate a plurality of light paths and storing a subset of the paths that exit at the virtual camera. The system may then use quad trees to identify a further subset of paths that are poorly sampled (e.g., sampled less than a pre-determined threshold). For at least some light paths of this further subset, the system may compute a distribution (e.g., a 2D Gaussian distribution) for each vertex, sample the vertex positions from the computed distributions, and then construct a guide path from the sampled vertices. Next, the system may modify the guide path by iteratively resampling the positions of the vertices, until the guide path exits at the virtual camera. The system can then construct a path-guided image using only the modified guide paths and combine the path-traced image with the path-guided image to form a fully rendered scene image storable in computer memory.” [0055-0057])
15. With reference to claim 9, Vorba teaches generating, for each candidate direction included in the set of candidate directions, a resampling weight associated with the candidate direction. (“As the rendering system incrementally constructs paths vertex by vertex, the path guiding method biases random decisions taken in the process to guide the paths towards important regions in the scene. In these regions, paths are disproportionately likely to make significant contributions to the image. The method can bias or change the probability of multiple decisions along each path, such as choosing direction after each scattering event, free path sampling in volumes, absorption, or choosing of a light source for connection.” [0040] “the system may place a virtual camera within the scene, and generate a mask for a light source that assigns an emission probability to each position on the light source (e.g., based on a density of light paths emitted by the virtual camera exiting at that position on the light source or, alternatively, based on a density of light paths emitted from that position on the light source that exit at the virtual camera). The system may then generate a path-traced image by using the masked light source to generate a plurality of light paths and storing a subset of the paths that exit at the virtual camera. The system may then use quad trees to identify a further subset of paths that are poorly sampled (e.g., sampled less than a pre-determined threshold). For at least some light paths of this further subset, the system may compute a distribution (e.g., a 2D Gaussian distribution) for each vertex, sample the vertex positions from the computed distributions, and then construct a guide path from the sampled vertices. Next, the system may modify the guide path by iteratively resampling the positions of the vertices, until the guide path exits at the virtual camera. The system can then construct a path-guided image using only the modified guide paths and combine the path-traced image with the path-guided image to form a fully rendered scene image storable in computer memory.” [0057] “path guiding can be used to learn the optimal sampling distribution for sampling the position of emitted light paths, based on the methods described herein. Such methods may for example be based on keeping a number of 2D guiding distributions for position sampling. Each distribution is relevant for a compact set of directions, while the union of sets forms the whole sphere of directions. In other words, after sampling the initial direction of the emitted light path, the method finds the corresponding guiding distribution for sampling the starting position.” [0082])
16. Claim 10 is similar in scope to the combination of claim 1, and thus is rejected under similar rationale. Vorba additionally teaches One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps (“Some implementations include a non-transitory computer-readable storage medium storing instructions, which when executed by at least one processor of a computer system, causes the computer system to carry out the method.” [0009])
17. Claims 11-18 are similar in scope to the combination of claims 2-9, and they are rejected under similar rationale.
18. Claim 19 is similar in scope to the combination of claim 1, and thus is rejected under similar rationale. Vorba additionally teaches A system comprising: one or more memories storing instructions; and one or more processors for executing the instruction (“Computer system 1500 also includes a main memory 1506, such as a random-access memory (RAM) or other dynamic storage device, coupled to bus 1502 for storing information and instructions to be executed by processor 1504.” [0165])
19. Claim 20 is similar in scope to the combination of claim 4, and thus is rejected under similar rationale.
Conclusion
20. Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michelle Chin whose telephone number is (571)270-3697. The examiner can normally be reached on Monday-Friday 8:00 AM-4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http:/Awww.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Kent Chang can be reached on (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https:/Awww.uspto.gov/patents/apply/patent- center for more information about Patent Center and https:/Awww.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE CHIN/
Primary Examiner, Art Unit 2614
Prosecution Insights