DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
No new amendments were added. Claims 1-20 remain pending in the application.
Response to Arguments
Applicant’s arguments, see Pages 7-11 of Remarks, filed April 14, 2026, with respect to the rejection of claims 1-20 under 35 USC 103, have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Sambongi et al. (US 20170046819 A1) and Prunier (Introduction to Shading).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-9, 11, 14-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20240020897 A1) in view of Zimmer et al. (US 10832375 B2), Sambongi et al. (US 20170046819 A1), and Prunier (Introduction to Shading), hereinafter Wang, Zimmer, Sambongi, and Prunier respectively.
Regarding claim 1, Wang teaches a method (Paragraph 0058 – “a portrait relighting process”; Note: a process is equivalent to a method) comprising:
obtaining a plurality of lighting components for an image frame in a current time instance (Paragraph 0056, 0086 – “an input image 102 can be provided as input to a geometry network 104 that can infer geometric information about one or more objects in image 102, as may include surface normals and one or more light maps…these temporal residual networks 212, 214 take outputs from a past frame and predictions from an image-only relighting part of this framework at a current frame as inputs”; Note: the light maps are equivalent to the lighting components), the plurality of lighting components for the image frame being associated with a plurality of lighting coefficients (Paragraph 0077 – “a framework can compute an initial rendering from predicted albedo, diffuse, and specular light maps…coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficients are associated with each light map, as shown in the equation
PNG
media_image1.png
30
142
media_image1.png
Greyscale
, where C is a coefficient and L is a light map);
warping lighting components of a previous image frame to provide a warped image frame referenced to the current time instance (Paragraph 0089 – “a flow can be applied to warp a previous relit frame and compute its different to a current relit frame”; Note: the lighting of a previous frame is warped, which produces a warped image that corresponds to a current frame and can be used to compare to the actual current frame);
applying first lighting coefficients to combine pixel values of a first lighting component to provide pixels values of the first lighting component in an output image frame (Paragraph 0075-0076 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… a specular network can take foreground image I, albedo map A, and these four specular light maps, L.sub.S.sup.1, L.sub.S.sup.2, L.sub.S.sup.3, L.sub.S.sup.4 as input, and can predict a four channel weight map W.sub.S.sup.1, L.sub.S.sup.16, L.sub.S.sup.32, W.sub.S.sup.64. In at least one embodiment, a weighted specular light map L.sub.S can be computed by:
PNG
media_image2.png
79
211
media_image2.png
Greyscale
where i=1, 16, 32, 64 and ⊙ indicates pixel-wise multiplication. In at least one embodiment, this rendering network takes albedo map A, diffuse light map L.sub.D, and specular light map L.sub.S as input and predicts relit image R”; Note: Weights Wis, which is equivalent to the first lighting coefficients, are used to combine the specular light maps, which is the first lighting component. The combination operation, shown in the equation, corresponds to pixel values in the pixel space. The combined specular light map is then used to provide the specular lighting for a relit image, which is equivalent to the relit image R);
and applying second lighting coefficients to combine pixel values of a second lighting component to provide pixels values of the second lighting component in the output image frame (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… this rendering network takes albedo map A, diffuse light map L.sub.D, and specular light map L.sub.S as input and predicts relit image R…coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficient of the diffuse light map, which is the second lighting coefficient, is used in a combination operation related to the diffuse component, which corresponds to pixel values in the pixel space. The combination provides pixel values of the diffuse component in the output/relit image);
wherein the first and second lighting coefficients are derived from an output tensor of a neural network (Paragraph 0078, 0512 – “coefficients and a residual map are predicted by f.sub.R, which takes input foreground image I, albedo map A, diffuse light map L.sub.d, and specular light map L.sub.S as input…tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inferencing”; Note: f.sub.R is a neural network that outputs lighting coefficients. Additionally, tensors are used in the neural networks, implying there is an output tensor).
Wang does not teach the rendered image frame in the limitation: “obtaining a plurality of lighting components for a rendered image frame in a current time instance”. However, Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the image of Wang could have been substituted for the rendered image of Zimmer because both the image and rendered image serve the purpose of representing the details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of retrieving light components from the image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the image of Wang for the rendered image of Zimmer according to known methods to yield the predictable result of retrieving light components from the image.
Wang modified by Zimmer still does not teach that the combination of pixel values occurs for pixel values of two different image frames, as described in the limitation: “applying first lighting coefficients to combine pixel values of a first lighting component in the rendered image frame with pixel values of the first lighting component in the warped image frame to provide pixels values of the first lighting component in an output image frame”; nor combining pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame. However, Sambongi teaches combining pixel values of a first lighting component in an image frame with pixel values of the first lighting component in the warped image frame to provide pixels values of the first lighting component in an output image frame (Paragraph 0055, 0065 – “as shown in FIG. 12, a plurality of specular-reflection components that overlap each other so as to neighbor the specified specular-reflection component in the up, down, left, and right directions are combined…pixels at which the pixel value is high are assumed to represent the specular-reflection components, and therefore, by using the threshold-setting unit, it is possible to set a threshold value on the basis of the difference images, and by using the reflection-component separating unit, it is possible to extract specular-reflection components from the plurality of transformed images on the basis of the threshold value”; Note: pixel values of specular/first lighting component from multiple images are combined to provide specular lighting for an output image; the specular lighting for the output is shown on the right side of the screenshot of Fig. 12 below, and the output image (combined image) is shown in screenshot of Fig. 13 below. The transformed image is equivalent to the warped image frame. The same process can be applied to the rendered image (previously taught by Zimmer) and transformed/warped image);
PNG
media_image3.png
343
539
media_image3.png
Greyscale
Screenshot of Fig. 12 (taken from Sambongi)
PNG
media_image4.png
204
534
media_image4.png
Greyscale
Screenshot of Fig. 13 (taken from Sambongi)
and combining pixel values of a second lighting component in the image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in an output image frame (Paragraph 0049, 0065 – “The interpolation unit 27 interpolates a region from which the specular-reflection component among the other components of the reference image sent thereto has been removed, as shown in FIG. 9, to create an interpolated image…pixels at which the pixel value is high are assumed to represent the specular-reflection components, and therefore, by using the threshold-setting unit, it is possible to set a threshold value on the basis of the difference images, and by using the reflection-component separating unit, it is possible to extract specular-reflection components from the plurality of transformed images on the basis of the threshold value”; Note: pixel values in a region of removed specular components are interpolated/combined. The leftover lighting components after removal of specular components are equivalent to the second lighting component. The combination occurs separate from the combination of the specular components since the specular components are already removed. The interpolated image provides the pixel values of the second lighting component for an output/combined image, as shown in Fig. 13 above. The same process can be applied to the rendered image (previously taught by Zimmer) and the transformed/warped image). Wang teaches a warped image (Paragraph 0089 – “a flow can be applied to warp a previous relit frame and compute its different to a current relit frame”), and Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). Wang can be modified to use the process of combining pixel values of lighting components, taught by Sambongi, on the warped image and rendered image, instead of the images of Sambongi. A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the images of Sambongi could have been substituted for the warped and rendered images of Wang and Zimmer respectively because all of them serve the purpose of representing details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. The substitution achieves the predictable result of providing an output image with combined pixel values from two images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the images of Sambongi with the warped and rendered images of Wang and Zimmer respectively. Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Sambogni to combine the pixel values of specular lighting in the warped image and rendered image because “an advantage is afforded in that, even though the position of a light source S is not actually moved, it is possible to obtain a combined image in which the position of the light source S is virtually moved… it is possible to create an image in which the focal position of the specular-reflection component is virtually shifted. By simultaneously combining a plurality of specular-reflection components whose parallaxes are shifted, it is possible to achieve the same effect as if the focus is shifted” (Sambogni: Paragraph 0053, 0055). In other words, the combination operation allows for creating an image with new lighting based on previous lighting. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Sambogni to combine the pixel values of a second lighting component in the warped image and rendered image because the areas where the specular light is removed can be filled with the leftover lighting component to create a natural effect of the specular light moving.
Finally, Wang modified by Zimmer and Sambongi still does not teach applying second lighting coefficients to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame. However, Prunier teaches applying second lighting coefficients to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame (Paragraph 1-2 on Page 2 – “this principle simply means that in the renderer, the contribution of each light just needs to be summed up. In other words, the total amount of light arriving at a point is just the linear sum of the amount of light that each light is contributing to. In mathematical terms, for a diffuse surface, this concept can be written using the following formula:
PNG
media_image5.png
66
371
media_image5.png
Greyscale
Where Sp stands for the shading point. The symbol ∑ in mathematics means "sum". In other words, for each light in the scene (there is n lights in total), we compute the diffuse equation by replacing in the equation the term LiN and LN with the current light intensity and direction. This is the same as:
PNG
media_image6.png
38
1
media_image6.png
Greyscale
”; Note: the coefficient
PNG
media_image6.png
38
1
media_image6.png
Greyscale
is equivalent to the second/diffuse lighting coefficient, and LiN and LN are equivalent to the pixel values of the second/diffuse lighting component, which are being combined. The rendered image and warped image were previously taught by Zimmer and Wang earlier in the rejection respectively. The application of diffuse/second coefficients occurs separately from the specular/first coefficients since the equation only applies for diffuse light. The output shading point is equivalent to the provided pixel values of diffuse lighting for an output image). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Prunier to apply diffuse lighting coefficients to combine pixel values of diffuse lighting of different images because the different images may have different light contributions, and thus, combining the diffuse lighting with a diffuse lighting coefficient allows for accurate calculation of the final lighting: “the contribution of each light adds up linearly. This is an important observation for two reasons. First, if you wish to create photorealistic images then your renderer needs to follow the same principle. It is also important for artists to recompose CG renders from individual layers where each layer represents the contribution of one particular light from the scene” (Prunier: Paragraph 1 on Page 2). To incorporate Prunier’s features of the second lighting component and coefficient into Wang’s configuration:
PNG
media_image1.png
30
142
media_image1.png
Greyscale
, the coefficient Cd would be
PNG
media_image6.png
38
1
media_image6.png
Greyscale
and the diffuse light map Ld would be the summation in the equation from Prunier:
PNG
media_image6.png
38
1
media_image6.png
Greyscale
.
Regarding claim 4, Wang in view of Zimmer, Sambogni, and Prunier teaches the method of claim 1. Wang further teaches wherein the plurality of lighting components comprise at least a specular lighting component and a diffuse lighting component (Paragraph 0074 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals”; Note: the specular light map is equivalent to a specular lighting component, and the diffuse light map is equivalent to a diffuse lighting component).
Regarding claim 5, Wang in view of Zimmer, Sambogni, and Prunier teaches the method of claim 1. Wang further teaches wherein the plurality of lighting components are rendered (Paragraph 0074 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals. In at least one embodiment, these components are concatenated as input to rendering residual network 112 to produce a final, relit output image”; Note: diffuse and specular light maps, which are lighting components, are rendered in the rendering residual network). Wang separately teaches ray tracing (Paragraph 0321 – “one or more slices 1801A-1801N includes one or more ray tracing units to compute ray tracing operations”). While Wang does not directly teach wherein the plurality of lighting components are rendered using ray tracing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine ray tracing with rendering lighting components because ray tracing is a common rendering technique in the state of the art and is useful for simulating realistic lighting effects.
Regarding claim 6, Wang in view of Zimmer, Sambogni, and Prunier teaches the method of claim 1. Wang does not teach applying one or more first motion vectors to pixel values for the first lighting component to provide pixel values in the warped image frame for the first lighting component; and applying one or more second motion vectors to pixel values for the second lighting component to provide pixel values in the warped image frame for the second lighting component. However, Zimmer teaches applying one or more first motion vectors to pixel values for the first lighting component to provide pixel values in the warped image frame for the first lighting component (Col. 4 lines 6-7 and 57-60, Col. 11 lines 52-55 – “each component is characterized by a single motion vector and a single spatial structure per image pixel…embodiments may decompose a scene into more or fewer components, e.g., just two components: diffuse (ED.*) and specular (sum of ET.* and ER.*)…embodiments warp every component of adjacent frames, as well as the corresponding feature buffers, using the computed per-component motion vectors, which aligns them to the current frame”; Note: motion vectors are used for specular components, which is equivalent to the first lighting component, to warp an image frame. Each motion vector corresponds to an image pixel); and applying one or more second motion vectors to pixel values for the second lighting component to provide pixel values in the warped image frame for the second lighting component (Col. 4 lines 6-7 and 57-60, Col. 11 lines 52-55 – “each component is characterized by a single motion vector and a single spatial structure per image pixel…embodiments may decompose a scene into more or fewer components, e.g., just two components: diffuse (ED.*) and specular (sum of ET.* and ER.*)…embodiments warp every component of adjacent frames, as well as the corresponding feature buffers, using the computed per-component motion vectors, which aligns them to the current frame”; Note: motion vectors are used for diffuse components, which is equivalent to the second lighting component, to warp an image frame. Each motion vector corresponds to an image pixel). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Zimmer to apply motion vectors to pixel values for warping because “Image-based methods such as frame interpolation and temporally stable denoising require accurate motion vectors for each component of the decomposition” and also because “it is straightforward to extract motion vectors of visible surface positions on the scene geometry” (Zimmer: Col. 6 lines 2-9). In other words, obtaining and using motion vectors is common and pertinent in the art.
Regarding claim 7, Wang in view of Zimmer, Sambogni, and Prunier teaches the method of claim 1. Wang further teaches combining the first and second lighting components to generate an output image frame (Paragraph 0074 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals. In at least one embodiment, these components are concatenated as input to rendering residual network 112 to produce a final, relit output image”; Note: the specular light map is equivalent to the first lighting component, and the diffuse light map is equivalent to the second lighting component. They are combined to generate an output image).
Regarding claim 8, Wang in view of Zimmer, Sambogni, and Prunier teaches the method of claim 7. Wang further teaches using an albedo to determine a proportion of first and second lighting components used to generate the output image frame (Paragraph 0076-0077 – “rendering network takes albedo map A, diffuse light map L.sub.D, and specular light map L.sub.S as input and predicts relit image R… coefficients can be predicted to linearly combine diffuse and specular light maps, then multiply this combined map by this albedo map to obtain an initial result”; Note: an output image is obtained by using an albedo map to determine a proportion of diffuse and specular light maps. The proportion is determined using multiplication).
Regarding claim 9, Wang in view of Zimmer, Sambogni, and Prunier teaches the method of claim 7. Wang does not teach filtering the combined first and second lighting components to reduce noise in generating the output image frame. However, Zimmer teaches filtering the first and second lighting components to reduce noise in generating the output image frame (Fig. 10, Col. 4 lines 57-60, Col. 12 lines 47-63 – “embodiments may decompose a scene into more or fewer components, e.g., just two components: diffuse (ED.*) and specular (sum of ET.* and ER.*)… Denoising may be significantly more robust when leveraging embodiments described herein…joint NL-Means filtering of each component guided by auxiliary features (‘decomposition and features’) robustly recovers fine details in the scene and yields a result close to the ground truth. (FIG. 10 also shows an example of the relative mean-square error (“MSE”) of each image for the full frame (first value) and the crop shown (second value))”; Note: the joint NL-Means filter is used to filter the diffuse and specular components for the purpose of denoising, and the generated output images are shown in Fig. 10; see screenshot of Fig. 10 below). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Zimmer to filter the lighting components for the benefit of achieving “denoising results that are visually very close to a high sample-count ground truth rendering with a low relative mean-square error” (Zimmer: Col. 12 lines 48-52). In other words, doing so would result in a final image that accurately represents the original scene.
PNG
media_image7.png
346
928
media_image7.png
Greyscale
Screenshot of Fig. 10 (taken from Zimmer)
Regarding claim 11, Wang teaches a computing device (Paragraph 0413 – “system 2600 is a mobile phone, a smart phone, a tablet computing device or a mobile Internet device”), comprising:
a memory comprising one more storage devices (Paragraph 0417 – “a memory device 2620 can be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, flash memory device, phase-change memory device, or some other memory device having suitable performance to serve as process memory. In at least one embodiment, memory device 2620 can operate as system memory for system 2600, to store data 2622 and instructions 2621”);
and one or more processors coupled to the memory (Fig. 26, Paragraph 0416 – “one or more processor(s) 2602 are coupled with one or more interface bus(es) 2610 to transmit communication signals such as address, data, or control signals between processor 2602 and other components in system 2600”; Note: The processor is coupled to the memory, as shown in Fig. 26; see modified screenshot of Fig. 26 below), the one or more processors operable to:
PNG
media_image8.png
822
680
media_image8.png
Greyscale
Modified screenshot of Fig. 26 (taken from Wang)
obtain from the memory a plurality of lighting components for an image frame in a current time instance (Paragraph 0056, 0086, 0102 – “an input image 102 can be provided as input to a geometry network 104 that can infer geometric information about one or more objects in image 102, as may include surface normals and one or more light maps…these temporal residual networks 212, 214 take outputs from a past frame and predictions from an image-only relighting part of this framework at a current frame as inputs…inference and/or training logic 715 may include, without limitation, one or more arithmetic logic unit(s) (“ALU(s)”) 710, including integer and/or floating point units, to perform logical and/or mathematical operations based, at least in part on, or indicated by, training and/or inference code (e.g., graph code), a result of which may produce activations (e.g., output values from layers or neurons within a neural network) stored in an activation storage 720 that are functions of input/output and/or weight parameter data stored in code and/or data storage 701 and/or code and/or data storage 705”; Note: the light maps are equivalent to the lighting components. They are obtained from activation storage of the geometry network, which is equivalent to memory in this case), the plurality of lighting components for the image frame being associated with a plurality of lighting coefficients (Paragraph 0077 – “a framework can compute an initial rendering from predicted albedo, diffuse, and specular light maps…coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficients are associated with each light map, as shown in the equation
PNG
media_image1.png
30
142
media_image1.png
Greyscale
, where C is a coefficient and L is a light map);
warp lighting components of a previous image frame to provide a warped image frame referenced to the current time instance (Paragraph 0089 – “a flow can be applied to warp a previous relit frame and compute its different to a current relit frame”; Note: the lighting of a previous frame is warped, which produces a warped image that corresponds to a current frame and can be used to compare to the actual current frame);
apply first lighting coefficients to combine pixel values of a first lighting component to provide pixels values of the first lighting component in an output image frame (Paragraph 0075-0076 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… a specular network can take foreground image I, albedo map A, and these four specular light maps, L.sub.S.sup.1, L.sub.S.sup.2, L.sub.S.sup.3, L.sub.S.sup.4 as input, and can predict a four channel weight map W.sub.S.sup.1, L.sub.S.sup.16, L.sub.S.sup.32, W.sub.S.sup.64. In at least one embodiment, a weighted specular light map L.sub.S can be computed by:
PNG
media_image2.png
79
211
media_image2.png
Greyscale
where i=1, 16, 32, 64 and ⊙ indicates pixel-wise multiplication. In at least one embodiment, this rendering network takes albedo map A, diffuse light map L.sub.D, and specular light map L.sub.S as input and predicts relit image R”; Note: Weights Wis, which is equivalent to the first lighting coefficients, are used to combine the specular light maps, which is the first lighting component. The combination operation, shown in the equation, corresponds to pixel values in the pixel space. The combined specular light map is then used to provide the specular lighting for a relit image, which is equivalent to the relit image R);
and apply second lighting coefficients to combine pixel values of a second lighting component to provide pixels values of the second lighting component in the output image frame (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… this rendering network takes albedo map A, diffuse light map L.sub.D, and specular light map L.sub.S as input and predicts relit image R…coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficient of the diffuse light map, which is the second lighting coefficient, is used in a combination operation related to the diffuse component, which corresponds to pixel values in the pixel space. The combination provides pixel values of the diffuse component in the output/relit image);
wherein the first and second lighting coefficients are derived from an output tensor of a neural network (Paragraph 0078, 0512 – “coefficients and a residual map are predicted by f.sub.R, which takes input foreground image I, albedo map A, diffuse light map L.sub.d, and specular light map L.sub.S as input…tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inferencing”; Note: f.sub.R is a neural network that outputs lighting coefficients. Additionally, tensors are used in the neural networks, implying there is an output tensor).
Wang does not teach the rendered image frame in the limitation: “obtain a plurality of lighting components for a rendered image frame in a current time instance”. However, Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the image of Wang could have been substituted for the rendered image of Zimmer because both the image and rendered image serve the purpose of representing the details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of retrieving light components from the image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the image of Wang for the rendered image of Zimmer according to known methods to yield the predictable result of retrieving light components from the image.
Wang modified by Zimmer still does not teach that the combination of pixel values occurs for pixel values of two different image frames, as described in the limitation: “apply first lighting coefficients to combine pixel values of a first lighting component in the rendered image frame with pixel values of the first lighting component in the warped image frame to provide pixels values of the first lighting component in an output image frame”; nor combining pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame. However, Sambongi teaches combining pixel values of a first lighting component in an image frame with pixel values of the first lighting component in the warped image frame to provide pixels values of the first lighting component in an output image frame (Paragraph 0055, 0065 – “as shown in FIG. 12, a plurality of specular-reflection components that overlap each other so as to neighbor the specified specular-reflection component in the up, down, left, and right directions are combined…pixels at which the pixel value is high are assumed to represent the specular-reflection components, and therefore, by using the threshold-setting unit, it is possible to set a threshold value on the basis of the difference images, and by using the reflection-component separating unit, it is possible to extract specular-reflection components from the plurality of transformed images on the basis of the threshold value”; Note: pixel values of specular/first lighting component from multiple images are combined to provide specular lighting for an output image; the specular lighting for the output is shown on the right side of the screenshot of Fig. 12 above, and the output image (combined image) is shown in screenshot of Fig. 13 above. The transformed image is equivalent to the warped image frame. The same process can be applied to the rendered image (previously taught by Zimmer) and transformed/warped image); and combining pixel values of a second lighting component in the image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in an output image frame (Paragraph 0049, 0065 – “The interpolation unit 27 interpolates a region from which the specular-reflection component among the other components of the reference image sent thereto has been removed, as shown in FIG. 9, to create an interpolated image…pixels at which the pixel value is high are assumed to represent the specular-reflection components, and therefore, by using the threshold-setting unit, it is possible to set a threshold value on the basis of the difference images, and by using the reflection-component separating unit, it is possible to extract specular-reflection components from the plurality of transformed images on the basis of the threshold value”; Note: pixel values in a region of removed specular components are interpolated/combined. The leftover lighting components after removal of specular components are equivalent to the second lighting component. The combination occurs separate from the combination of the specular components since the specular components are already removed. The interpolated image provides the pixel values of the second lighting component for an output/combined image, as shown in Fig. 13 above. The same process can be applied to the rendered image (previously taught by Zimmer) and the transformed/warped image). Wang teaches a warped image (Paragraph 0089 – “a flow can be applied to warp a previous relit frame and compute its different to a current relit frame”), and Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). Wang can be modified to use the process of combining pixel values of lighting components, taught by Sambongi, on the warped image and rendered image, instead of the images of Sambongi. A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the images of Sambongi could have been substituted for the warped and rendered images of Wang and Zimmer respectively because all of them serve the purpose of representing details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. The substitution achieves the predictable result of providing an output image with combined pixel values from two images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the images of Sambongi with the warped and rendered images of Wang and Zimmer respectively. Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Sambogni to combine the pixel values of specular lighting in the warped image and rendered image because “an advantage is afforded in that, even though the position of a light source S is not actually moved, it is possible to obtain a combined image in which the position of the light source S is virtually moved… it is possible to create an image in which the focal position of the specular-reflection component is virtually shifted. By simultaneously combining a plurality of specular-reflection components whose parallaxes are shifted, it is possible to achieve the same effect as if the focus is shifted” (Sambogni: Paragraph 0053, 0055). In other words, the combination operation allows for creating an image with new lighting based on previous lighting. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Sambogni to combine the pixel values of a second lighting component in the warped image and rendered image because the areas where the specular light is removed can be filled with the leftover lighting component to create a natural effect of the specular light moving.
Finally, Wang modified by Zimmer and Sambongi still does not teach applying second lighting coefficients to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame. However, Prunier teaches applying second lighting coefficients to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame (Paragraph 1-2 on Page 2 – “this principle simply means that in the renderer, the contribution of each light just needs to be summed up. In other words, the total amount of light arriving at a point is just the linear sum of the amount of light that each light is contributing to. In mathematical terms, for a diffuse surface, this concept can be written using the following formula:
PNG
media_image5.png
66
371
media_image5.png
Greyscale
Where Sp stands for the shading point. The symbol ∑ in mathematics means "sum". In other words, for each light in the scene (there is n lights in total), we compute the diffuse equation by replacing in the equation the term LiN and LN with the current light intensity and direction. This is the same as:
PNG
media_image6.png
38
1
media_image6.png
Greyscale
”; Note: the coefficient
PNG
media_image6.png
38
1
media_image6.png
Greyscale
is equivalent to the second/diffuse lighting coefficient, and LiN and LN are equivalent to the pixel values of the second/diffuse lighting component, which are being combined. The rendered image and warped image were previously taught by Zimmer and Wang earlier in the rejection respectively. The application of diffuse/second coefficients occurs separately from the specular/first coefficients since the equation only applies for diffuse light. The output shading point is equivalent to the provided pixel values of diffuse lighting for an output image). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Prunier to apply diffuse lighting coefficients to combine pixel values of diffuse lighting of different images because the different images may have different light contributions, and thus, combining the diffuse lighting with a diffuse lighting coefficient allows for accurate calculation of the final lighting: “the contribution of each light adds up linearly. This is an important observation for two reasons. First, if you wish to create photorealistic images then your renderer needs to follow the same principle. It is also important for artists to recompose CG renders from individual layers where each layer represents the contribution of one particular light from the scene” (Prunier: Paragraph 1 on Page 2). To incorporate Prunier’s features of the second lighting component and coefficient into Wang’s configuration:
PNG
media_image1.png
30
142
media_image1.png
Greyscale
, the coefficient Cd would be
PNG
media_image6.png
38
1
media_image6.png
Greyscale
and the diffuse light map Ld would be the summation in the equation from Prunier:
PNG
media_image6.png
38
1
media_image6.png
Greyscale
.
Regarding claim 14, Wang in view of Zimmer, Sambogni, and Prunier teaches the computing device of claim 11. Wang further teaches wherein the plurality of lighting components comprise at least a specular lighting component and a diffuse lighting component (Paragraph 0074 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals”; Note: the specular light map is equivalent to a specular lighting component, and the diffuse light map is equivalent to a diffuse lighting component).
Regarding claim 15, Wang in view of Zimmer, Sambogni, and Prunier teaches the computing device of claim 11. Wang further teaches wherein the plurality of lighting components are rendered (Paragraph 0074 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals. In at least one embodiment, these components are concatenated as input to rendering residual network 112 to produce a final, relit output image”; Note: diffuse and specular light maps, which are lighting components, are rendered in the rendering residual network). Wang separately teaches ray tracing (Paragraph 0321 – “one or more slices 1801A-1801N includes one or more ray tracing units to compute ray tracing operations”). While Wang does not directly teach wherein the plurality of lighting components are rendered using ray tracing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine ray tracing with rendering lighting components because ray tracing is a common rendering technique in the state of the art and is useful for simulating realistic lighting effects.
Regarding claim 16, Wang in view of Zimmer, Sambogni, and Prunier teaches the computing device of claim 11. Wang does not teach wherein application of one or more first motion vectors to pixel values for the first lighting component to provide pixel values in the warped image frame for the first lighting component; and application of one or more second motion vectors to pixel values for the second lighting component to provide pixel values in the warped image frame for the second lighting component. However, Zimmer teaches application of one or more first motion vectors to pixel values for the first lighting component to provide pixel values in the warped image frame for the first lighting component (Col. 4 lines 6-7 and 57-60, Col. 11 lines 52-55 – “each component is characterized by a single motion vector and a single spatial structure per image pixel…embodiments may decompose a scene into more or fewer components, e.g., just two components: diffuse (ED.*) and specular (sum of ET.* and ER.*)…embodiments warp every component of adjacent frames, as well as the corresponding feature buffers, using the computed per-component motion vectors, which aligns them to the current frame”; Note: motion vectors are used for specular components, which is equivalent to the first lighting component, to warp an image frame. Each motion vector corresponds to an image pixel); and application of one or more second motion vectors to pixel values for the second lighting component to provide pixel values in the warped image frame for the second lighting component (Col. 4 lines 6-7 and 57-60, Col. 11 lines 52-55 – “each component is characterized by a single motion vector and a single spatial structure per image pixel…embodiments may decompose a scene into more or fewer components, e.g., just two components: diffuse (ED.*) and specular (sum of ET.* and ER.*)…embodiments warp every component of adjacent frames, as well as the corresponding feature buffers, using the computed per-component motion vectors, which aligns them to the current frame”; Note: motion vectors are used for diffuse components, which is equivalent to the second lighting component, to warp an image frame. Each motion vector corresponds to an image pixel). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Zimmer to apply motion vectors to pixel values for warping because “Image-based methods such as frame interpolation and temporally stable denoising require accurate motion vectors for each component of the decomposition” and also because “it is straightforward to extract motion vectors of visible surface positions on the scene geometry” (Zimmer: Col. 6 lines 2-9). In other words, obtaining and using motion vectors is common and pertinent in the art.
Regarding claim 17, Wang in view of Zimmer, Sambogni, and Prunier teaches the computing device of claim 11. Wang further teaches the one or more processors are further operable to combine the first and second lighting components to generate an output image frame (Paragraph 0074, 0414 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals. In at least one embodiment, these components are concatenated as input to rendering residual network 112 to produce a final, relit output image…one or more processors 2602 each include one or more processor cores 2607 to process instructions which, when executed, perform operations for system and user software”; Note: the specular light map is equivalent to the first lighting component, and the diffuse light map is equivalent to the second lighting component. They are combined to generate an output image).
Regarding claim 18, Wang in view of Zimmer, Sambogni, and Prunier teaches the computing device of claim 17. Wang further teaches the one or more processors are further operable to apply an albedo to determine a proportion of first and second lighting components used to generate the output image frame (Paragraph 0076-0077, 0414 – “rendering network takes albedo map A, diffuse light map L.sub.D, and specular light map L.sub.S as input and predicts relit image R… coefficients can be predicted to linearly combine diffuse and specular light maps, then multiply this combined map by this albedo map to obtain an initial result…one or more processors 2602 each include one or more processor cores 2607 to process instructions which, when executed, perform operations for system and user software”; Note: an output image is obtained by using an albedo map to determine a proportion of diffuse and specular light maps. The proportion is determined using multiplication).
Regarding claim 20, Wang teaches a method of training a neural network (Paragraph 0092 – “a process 400 for training a network to generate an image with one or more different lighting aspects can be performed as illustrated in FIG. 4”; Note: the network refers to a neural network), comprising:
receiving an input tensor in an input layer of a neural network, the input tensor representing one or more characteristics of an image frame (Paragraph 0073, 0512 – “a foreground image is obtained using an off-the-shelf matting network given an input image 102, such as an input portrait image. In at least one embodiment, this image and an environment map can serve as inputs to a relighting network to predict a relit image…tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inferencing”; Note: input image is received by a neural network. Additionally, tensors are used in the neural networks, so it is implied that there would be an input tensor because tensor operations could not be performed otherwise);
providing an output tensor to an output layer of the neural network, the output tensor representing first and second lighting coefficients (Paragraph 0078, 0512 – “coefficients and a residual map are predicted by f.sub.R, which takes input foreground image I, albedo map A, diffuse light map L.sub.d, and specular light map L.sub.S as input…tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inferencing”; Note: f.sub.R is a neural network that outputs lighting coefficients. Additionally, tensors are used in the neural networks, implying there is an output tensor),
the first lighting coefficients used to combine pixel values of a first lighting component to provide pixels values of the first lighting component in an output image frame (Paragraph 0075-0076 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… a specular network can take foreground image I, albedo map A, and these four specular light maps, L.sub.S.sup.1, L.sub.S.sup.2, L.sub.S.sup.3, L.sub.S.sup.4 as input, and can predict a four channel weight map W.sub.S.sup.1, L.sub.S.sup.16, L.sub.S.sup.32, W.sub.S.sup.64. In at least one embodiment, a weighted specular light map L.sub.S can be computed by:
PNG
media_image2.png
79
211
media_image2.png
Greyscale
where i=1, 16, 32, 64 and ⊙ indicates pixel-wise multiplication. In at least one embodiment, this rendering network takes albedo map A, diffuse light map L.sub.D, and specular light map L.sub.S as input and predicts relit image R”; Note: Weights Wis, which is equivalent to the first lighting coefficients, are used to combine the specular light maps, which is the first lighting component. The combination operation, shown in the equation, corresponds to pixel values in the pixel space. The combined specular light map is then used to provide the specular lighting for a relit image, which is equivalent to the relit image R);
and the second lighting coefficients used to combine pixel values of a second lighting component to provide pixels values of the second lighting component in the output image frame (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… this rendering network takes albedo map A, diffuse light map L.sub.D, and specular light map L.sub.S as input and predicts relit image R…coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficient of the diffuse light map, which is the second lighting coefficient, is used in a combination operation related to the diffuse component, which corresponds to pixel values in the pixel space. The combination provides pixel values of the diffuse component in the output/relit image);
the output layer of the neural network connected by one or more intermediate layers of the neural network (Fig. 8 – the output layer is connected to an intermediate layer; see modified screenshot of Fig. 8 below);
PNG
media_image9.png
448
681
media_image9.png
Greyscale
Modified screenshot of Fig. 8 (taken from Wang)
and training the neural network to predict the provided output tensor when provided with the received input tensor by using backpropagation to adjust a weight of one or more activation functions linking one or more nodes of one or more layers of the neural network (Paragraph 0109 – “untrained neural network 806 is trained in a supervised manner and processes inputs from training dataset 802 and compares resulting outputs against a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through untrained neural network 806. In at least one embodiment, training framework 804 adjusts weights that control untrained neural network 806”; Note: a neural network is trained to predict output based on received input and uses back propagation to adjust weights throughout the neural network).
Wang does not teach the rendered image frame in the limitation: “first lighting coefficients used to combine pixel values of a first lighting component in a rendered image frame…”. However, Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the image of Wang could have been substituted for the rendered image of Zimmer because both the image and rendered image serve the purpose of representing the details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of retrieving light components from the image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the image of Wang for the rendered image of Zimmer according to known methods to yield the predictable result of retrieving light components from the image.
Wang modified by Zimmer still does not teach that the combination of pixel values occurs for pixel values of two different image frames, as described in the limitation: “first lighting coefficients used to combine pixel values of a first lighting component in a rendered image frame with pixel values of the first lighting component in a warped image frame to provide pixels values of the first lighting component in an output image frame”; nor combining pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame. However, Sambongi teaches combining pixel values of a first lighting component in an image frame with pixel values of the first lighting component in a warped image frame to provide pixels values of the first lighting component in an output image frame (Paragraph 0055, 0065 – “as shown in FIG. 12, a plurality of specular-reflection components that overlap each other so as to neighbor the specified specular-reflection component in the up, down, left, and right directions are combined…pixels at which the pixel value is high are assumed to represent the specular-reflection components, and therefore, by using the threshold-setting unit, it is possible to set a threshold value on the basis of the difference images, and by using the reflection-component separating unit, it is possible to extract specular-reflection components from the plurality of transformed images on the basis of the threshold value”; Note: pixel values of specular/first lighting component from multiple images are combined to provide specular lighting for an output image; the specular lighting for the output is shown on the right side of the screenshot of Fig. 12 above, and the output image (combined image) is shown in screenshot of Fig. 13 above. The transformed image is equivalent to the warped image frame. The same process can be applied to the rendered image (previously taught by Zimmer) and transformed/warped image); and combining pixel values of a second lighting component in the image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in an output image frame (Paragraph 0049, 0065 – “The interpolation unit 27 interpolates a region from which the specular-reflection component among the other components of the reference image sent thereto has been removed, as shown in FIG. 9, to create an interpolated image…pixels at which the pixel value is high are assumed to represent the specular-reflection components, and therefore, by using the threshold-setting unit, it is possible to set a threshold value on the basis of the difference images, and by using the reflection-component separating unit, it is possible to extract specular-reflection components from the plurality of transformed images on the basis of the threshold value”; Note: pixel values in a region of removed specular components are interpolated/combined. The leftover lighting components after removal of specular components are equivalent to the second lighting component. The combination occurs separate from the combination of the specular components since the specular components are already removed. The interpolated image provides the pixel values of the second lighting component for an output/combined image, as shown in Fig. 13 above. The same process can be applied to the rendered image (previously taught by Zimmer) and the transformed/warped image). Wang teaches a warped image (Paragraph 0089 – “a flow can be applied to warp a previous relit frame and compute its different to a current relit frame”), and Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). Wang can be modified to use the process of combining pixel values of lighting components, taught by Sambongi, on the warped image and rendered image, instead of the images of Sambongi. A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the images of Sambongi could have been substituted for the warped and rendered images of Wang and Zimmer respectively because all of them serve the purpose of representing details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. The substitution achieves the predictable result of providing an output image with combined pixel values from two images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the images of Sambongi with the warped and rendered images of Wang and Zimmer respectively. Additionally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Sambogni to combine the pixel values of specular lighting in the warped image and rendered image because “an advantage is afforded in that, even though the position of a light source S is not actually moved, it is possible to obtain a combined image in which the position of the light source S is virtually moved… it is possible to create an image in which the focal position of the specular-reflection component is virtually shifted. By simultaneously combining a plurality of specular-reflection components whose parallaxes are shifted, it is possible to achieve the same effect as if the focus is shifted” (Sambogni: Paragraph 0053, 0055). In other words, the combination operation allows for creating an image with new lighting based on previous lighting. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Sambogni to combine the pixel values of a second lighting component in the warped image and rendered image because the areas where the specular light is removed can be filled with the leftover lighting component to create a natural effect of the specular light moving.
Finally, Wang modified by Zimmer and Sambongi still does not teach second lighting coefficients used to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame. However, Prunier teaches second lighting coefficients used to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame (Paragraph 1-2 on Page 2 – “this principle simply means that in the renderer, the contribution of each light just needs to be summed up. In other words, the total amount of light arriving at a point is just the linear sum of the amount of light that each light is contributing to. In mathematical terms, for a diffuse surface, this concept can be written using the following formula:
PNG
media_image5.png
66
371
media_image5.png
Greyscale
Where Sp stands for the shading point. The symbol ∑ in mathematics means "sum". In other words, for each light in the scene (there is n lights in total), we compute the diffuse equation by replacing in the equation the term LiN and LN with the current light intensity and direction. This is the same as:
PNG
media_image6.png
38
1
media_image6.png
Greyscale
”; Note: the coefficient
PNG
media_image6.png
38
1
media_image6.png
Greyscale
is equivalent to the second/diffuse lighting coefficient, and LiN and LN are equivalent to the pixel values of the second/diffuse lighting component, which are being combined. The rendered image and warped image were previously taught by Zimmer and Wang earlier in the rejection respectively. The application of diffuse/second coefficients occurs separately from the specular/first coefficients since the equation only applies for diffuse light. The output shading point is equivalent to the provided pixel values of diffuse lighting for an output image). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Prunier to apply diffuse lighting coefficients to combine pixel values of diffuse lighting of different images because the different images may have different light contributions, and thus, combining the diffuse lighting with a diffuse lighting coefficient allows for accurate calculation of the final lighting: “the contribution of each light adds up linearly. This is an important observation for two reasons. First, if you wish to create photorealistic images then your renderer needs to follow the same principle. It is also important for artists to recompose CG renders from individual layers where each layer represents the contribution of one particular light from the scene” (Prunier: Paragraph 1 on Page 2). To incorporate Prunier’s features of the second lighting component and coefficient into Wang’s configuration:
PNG
media_image1.png
30
142
media_image1.png
Greyscale
, the coefficient Cd would be
PNG
media_image6.png
38
1
media_image6.png
Greyscale
and the diffuse light map Ld would be the summation in the equation from Prunier:
PNG
media_image6.png
38
1
media_image6.png
Greyscale
.
Claims 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Zimmer, Sambogni, Prunier, and Rossin et al. (US 6707453 B1), hereinafter Rossin.
Regarding claim 2, Wang in view of Zimmer, Sambogni, and Prunier teaches the method of claim 1. Wang does not teach interpolating pixel values of the first lighting component; nor interpolating pixel values of the second lighting component. However, Rossin teaches interpolating pixel values of the first lighting component (Col. 14 lines 1-7 – “Each shap stepper or interpolator 310-314 determines the value of a component at each pixel along each span of the primitive. Each span is generally a row of interior pixels bounded by edge pixels. As noted, a consolidated interpolator to interpolate specular and diffuse lighting components can be implemented for each type of interpolator in scan converter 204”; Note: pixel values of specular lighting components are interpolated); and interpolating pixel values of the second lighting component (Col. 14 lines 1-7 – “Each shap stepper or interpolator 310-314 determines the value of a component at each pixel along each span of the primitive. Each span is generally a row of interior pixels bounded by edge pixels. As noted, a consolidated interpolator to interpolate specular and diffuse lighting components can be implemented for each type of interpolator in scan converter 204”; Note: pixel values of diffuse lighting components are interpolated). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Rossin to interpolate the pixel values of the lighting components for the benefit of “insuring that all object surfaces, including those that have low color intensity, properly exhibit the specular lighting contribution” (Rossin: Col. 4 lines 10-13) and ensuring that the lighting is accurately represented in the final image.
Regarding claim 12, Wang in view of Zimmer, Sambogni, and Prunier teaches the computing device of claim 11. Wang does not teach wherein: application of the first lighting coefficients to combine pixel values of the first lighting component in the warped image frame is based, at least in part, on an interpolation pixel values of the first lighting component; application of the second lighting coefficients to combine pixel values of the second lighting component in the warped image frame is based, at least in part, on an interpolation of pixel values of the second lighting component. However, Rossin teaches interpolating pixel values of the first lighting component (Col. 14 lines 1-7 – “Each shap stepper or interpolator 310-314 determines the value of a component at each pixel along each span of the primitive. Each span is generally a row of interior pixels bounded by edge pixels. As noted, a consolidated interpolator to interpolate specular and diffuse lighting components can be implemented for each type of interpolator in scan converter 204”; Note: pixel values of specular lighting components are interpolated); and interpolating pixel values of the second lighting component (Col. 14 lines 1-7 – “Each shap stepper or interpolator 310-314 determines the value of a component at each pixel along each span of the primitive. Each span is generally a row of interior pixels bounded by edge pixels. As noted, a consolidated interpolator to interpolate specular and diffuse lighting components can be implemented for each type of interpolator in scan converter 204”; Note: pixel values of diffuse lighting components are interpolated). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Rossin to interpolate the pixel values of the lighting components for the benefit of “insuring that all object surfaces, including those that have low color intensity, properly exhibit the specular lighting contribution” (Rossin: Col. 4 lines 10-13) and ensuring that the lighting is accurately represented in the final image. Interpolation is a common and reliable way for combining pixel values between images.
Claims 3, 10, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Zimmer, Sambogni, Prunier, and Wizadwongsa et al. (NeX: Real-time View Synthesis with Neural Basis Expansion), hereinafter Wizadwongsa.
Regarding claim 3, Wang in view of Zimmer, Sambogni, and Prunier teaches the method of claim 1. Wang does not teach wherein first and second lighting coefficients are per-pixel coefficients. However, Wizadwongsa teaches wherein first and second lighting coefficients are per-pixel coefficients (Fig. 1 Caption on page 1 – “Each pixel in NeX multiplane image consists of an alpha transparency value, base color k0, and view-dependent reflectance coefficients k1…kn. A linear combination of these coefficients and basis functions learned from a neural network produces the final color value”; Note: the reflectance coefficients correspond to lighting coefficients, and each pixel has them). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Wizadwongsa to have the lighting coefficients for each pixel for the benefit of improving “fine detail” and producing “sharper results” (Wizadwongsa: Col. 1 Paragraph 1 on page 2). Having a lighting coefficient for each pixel would help make the final output accurately represent the lighting of the original image.
Regarding claim 10, Wang in view of Zimmer, Sambogni, and Prunier teaches the method of claim 1. Wang does not teach wherein the first and second lighting coefficients are derived from disocclusion of one or more pixels in the rendered image frame in the current time instance or a change in view-dependent lighting for one or more pixels in the rendered image frame in the current time instance, or a combination thereof. However, Wizadwongsa teaches wherein the first and second lighting coefficients are derived from a change in view-dependent lighting for one or more pixels in the image frame (Fig. 1 Caption on page 1, Col. 2 Paragraphs 1 and 3 on Page 6 – “Each pixel in NeX multiplane image consists of an alpha transparency value, base color k0, and view-dependent reflectance coefficients k1…kn. A linear combination of these coefficients and basis functions learned from a neural network produces the final color value…We train our model on 12 input views, then evaluate on 4 held-out views… We vary the number of basis coefficients from zero, which represents no view-dependent modeling, to 20”; Note: the reflectance coefficients correspond to lighting coefficients, and they derive from view-dependent lighting, where each coefficient corresponds to a different view). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Wizadwongsa to derive lighting coefficients from changes in view-dependent lighting for the benefit of accounting for lighting differences, which will make the final result more realistic. Considering view-dependent lighting helps with capturing “fine detail” and reproducing “complex view-dependent effects…thus allowing real-time rendering” (Wizadwongsa: Col. 1 Paragraph 1 on page 2, Col. 2 Paragraph 2 on Page 8).
Regarding claim 13, Wang in view of Zimmer, Sambogni, and Prunier teaches the computing device of claim 11. Wang does not teach wherein first and second lighting coefficients are per-pixel coefficients. However, Wizadwongsa teaches wherein first and second lighting coefficients are per-pixel coefficients (Fig. 1 Caption on page 1 – “Each pixel in NeX multiplane image consists of an alpha transparency value, base color k0, and view-dependent reflectance coefficients k1…kn. A linear combination of these coefficients and basis functions learned from a neural network produces the final color value”; Note: the reflectance coefficients correspond to lighting coefficients, and each pixel has them). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Wizadwongsa to have the lighting coefficients for each pixel for the benefit of improving “fine detail” and producing “sharper results” (Wizadwongsa: Col. 1 Paragraph 1 on page 2). Having a lighting coefficient for each pixel would help make the final output accurately represent the lighting of the original image.
Regarding claim 19, Wang in view of Zimmer, Sambogni, and Prunier teaches the computing device of claim 11. Wang does not teach wherein the first and second lighting coefficients are derived from disocclusion of one or more pixels in the rendered image frame in the current time instance or a change in view-dependent lighting for one or more pixels in the rendered image frame in the current time instance, or a combination thereof. However, Wizadwongsa teaches wherein the first and second lighting coefficients are derived from a change in view-dependent lighting for one or more pixels in the image frame (Fig. 1 Caption on page 1, Col. 2 Paragraphs 1 and 3 on Page 6 – “Each pixel in NeX multiplane image consists of an alpha transparency value, base color k0, and view-dependent reflectance coefficients k1…kn. A linear combination of these coefficients and basis functions learned from a neural network produces the final color value…We train our model on 12 input views, then evaluate on 4 held-out views… We vary the number of basis coefficients from zero, which represents no view-dependent modeling, to 20”; Note: the reflectance coefficients correspond to lighting coefficients, and they derive from view-dependent lighting, where each coefficient corresponds to a different view). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Wizadwongsa to derive lighting coefficients from changes in view-dependent lighting for the benefit of accounting for lighting differences, which will make the final result more realistic. Considering view-dependent lighting helps with capturing “fine detail” and reproducing “complex view-dependent effects…thus allowing real-time rendering” (Wizadwongsa: Col. 1 Paragraph 1 on page 2, Col. 2 Paragraph 2 on Page 8).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Watanabe et al. (US 20140093158 A1) teaches a method of separating a target image into a diffuse reflection image and non-diffuse reflection image based on pixel values in the image in order to generate a multi-view image. Mitsumine et al. (JP 2017068764 A) teaches a method of computing illumination information for a diffuse reflection component and specular reflection component from acquired images.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE HAU MA whose telephone number is (571)272-2187. The examiner can normally be reached M-Th 7-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE HAU MA/Examiner, Art Unit 2617
/KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights