DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on December 5, 2025 has been entered.
Response to Amendment
The amendment filed December 5, 2025 has been entered. Claims 1-20 remain pending in the application.
Response to Arguments
Applicant's arguments filed December 5, 2025 have been fully considered, and the examiner agrees that Wang, Zimmer, and Imagawa fail to directly teach applying coefficients separately for different lighting components. Wang teaches that the application of first and second lighting coefficients occur together (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… coefficients can be predicted to linearly combine diffuse and specular light maps”). However, it still would have been obvious to try separately applying two different lighting coefficients, as there are a finite number of potential solutions known to persons of ordinary skill in the art. The solutions being: applying the two different lighting coefficients together (as done in Wang) or separately applying two different lighting coefficients, such as using them in two different equations. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to apply the first and second lighting coefficients separately for the benefit of calculating and knowing the individual values of each lighting component for lighting a new output image or scene. For instance, in Wang, the first and second lighting components are equivalent to specular and diffuse light respectively. Specular and diffuse light define different properties of light; specular light defines reflectiveness and diffuse light defines a base color. Wang combines these specular and diffuse light values using their respective coefficients to determine how much each component affects the output scene. But in cases like in Imagawa, where the output scene results from a synthesis of multiple images, then combining the individual light components separately first is important to determine how much of a lighting component in each image will affect the output scene. Wang already performs a similar process using a specular network, where weights (coefficients) are used to determine an output specular light map based on a combination of four specular light maps (Paragraph 0076 – “a specular network can take foreground image I, albedo map A, and these four specular light maps L.sub.d, L.sub.S.sup.1, L.sub.S.sup.2, L.sub.S.sup.3, L.sub.S.sup.4 as input, and can predict a four channel weight map W.sub.S.sup.1, L.sub.S.sup.16, L.sub.S.sup.32, W.sub.S.sup.64. In at least one embodiment, a weighted specular light map L.sub.S can be computed”; Note specular coefficients are used to combine specular light). Thus, when there are multiple images affecting how the light will appear in an output image, it would be beneficial to separately use coefficients for each lighting component, in order to know how much each individual lighting component in each image will contribute to the final output and in order to generate realistic lighting for the final image. Furthermore, when there are two types of coefficients, there is a finite number of ways to apply them; either they can be applied together in the same operation, or separately. One of ordinary skill in the art could have applied the first and second lighting coefficients separately with a reasonable expectation of success and would have done so for the benefit of being able to know the individual specular and diffuse values for the sake of providing a quality output rendering. Therefore, it would have been obvious to try the solution of applying the first and second lighting coefficients separately.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4-9, 11, 14-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20240020897 A1) in view of Zimmer et al. (US 10832375 B2) and Imagawa (US 9489728 B2), hereinafter Wang, Zimmer, and Imagawa respectively.
Regarding claim 1, Wang teaches a method (Paragraph 0058 – “a portrait relighting process”; Note: a process is equivalent to a method) comprising:
obtaining a plurality of lighting components for an image frame in a current time instance (Paragraph 0056, 0086 – “an input image 102 can be provided as input to a geometry network 104 that can infer geometric information about one or more objects in image 102, as may include surface normals and one or more light maps…these temporal residual networks 212, 214 take outputs from a past frame and predictions from an image-only relighting part of this framework at a current frame as inputs”; Note: the light maps are equivalent to the lighting components), the plurality of lighting components for the image frame being associated with a plurality of lighting coefficients (Paragraph 0077 – “a framework can compute an initial rendering from predicted albedo, diffuse, and specular light maps…coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficients are associated with each light map, as shown in the equation
PNG
media_image1.png
30
142
media_image1.png
Greyscale
, where C is a coefficient and L is a light map);
warping lighting components of a previous image frame to provide a warped image frame referenced to the current time instance (Paragraph 0089 – “a flow can be applied to warp a previous relit frame and compute its different to a current relit frame”; Note: the lighting of a previous frame is warped, which produces a warped image that corresponds to a current frame and can be used to compare to the actual current frame);
applying first lighting coefficients to combine pixel values of a first lighting component (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficient of the specular light map, which is the first lighting coefficient, is used in a combination operation related to the specular component, which corresponds to pixel values in the pixel space);
and applying second lighting coefficients to combine pixel values of a second lighting component (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficient of the diffuse light map, which is the second lighting coefficient, is used in a combination operation related to the diffuse component, which corresponds to pixel values in the pixel space);
wherein the first and second lighting coefficients are derived from an output tensor of a neural network (Paragraph 0078, 0512 – “coefficients and a residual map are predicted by f.sub.R, which takes input foreground image I, albedo map A, diffuse light map L.sub.d, and specular light map L.sub.S as input…tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inferencing”; Note: f.sub.R is a neural network that outputs lighting coefficients. Additionally, tensors are used in the neural networks, implying there is an output tensor).
Wang does not teach the rendered image frame in the limitation: “obtaining a plurality of lighting components for a rendered image frame in a current time instance”. Wang also does not teach applying first lighting coefficients to combine pixel values of a first lighting component in the rendered image frame with pixel values of the first lighting component in the warped image frame to provide pixels values of the first lighting component in an output image frame; nor applying second lighting coefficients to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame. However, Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the image of Wang could have been substituted for the rendered image of Zimmer because both the image and rendered image serve the purpose of representing the details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of retrieving light components from the image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the image of Wang for the rendered image of Zimmer according to known methods to yield the predictable result of retrieving light components from the image. Wang modified by Zimmer still does not teach applying first lighting coefficients to combine pixel values of a first lighting component in the rendered image frame with pixel values of the first lighting component in the warped image frame to provide pixels values of the first lighting component in an output image frame; nor applying second lighting coefficients to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame. However, Imagawa teaches applying first lighting coefficients to combine pixel values of a first lighting component in a normal image frame with pixel values of the first lighting component in a polarization image frame to provide pixels values of the first lighting component in an output image frame (Col. 3 lines 8-12, Col. 12 lines 18-35 – “synthesizing the normal image and the polarization image using a pixel value obtained by multiplying the coefficient by the at least one of the pixel value of the normal image and the pixel value of the polarization image to generate a synthesized image…the coefficient calculating unit 105 sets the synthetic coefficient α(x,y) to each pixel based on Equation 15 below (S803).
PNG
media_image2.png
42
168
media_image2.png
Greyscale
In the calculating of a synthetic coefficient described with reference to FIG. 12, a diffuse reflection coefficient outside of the reflection region in the normal image that is a color image is calculated, and the synthetic coefficient is calculated based on the color properties of the specular reflection light and the diffuse reflection light”; Note: a coefficient based on specular light, which is equivalent to the first lighting component, is applied to pixel values to combine the pixels of a normal image and a polarization image, providing pixels of a combined output image. The same process can be applied to the rendered image and warped image previously taught by Zimmer and Wang respectively); and applying second lighting coefficients to combine pixel values of a second lighting component in a normal image frame with pixel values of the second lighting component in a polarization image frame to provide pixels values of the second lighting component in an output image frame (Col. 3 lines 8-12, Col. 12 lines 18-35 – “synthesizing the normal image and the polarization image using a pixel value obtained by multiplying the coefficient by the at least one of the pixel value of the normal image and the pixel value of the polarization image to generate a synthesized image…the coefficient calculating unit 105 sets the synthetic coefficient α(x,y) to each pixel based on Equation 15 below (S803).
PNG
media_image2.png
42
168
media_image2.png
Greyscale
In the calculating of a synthetic coefficient described with reference to FIG. 12, a diffuse reflection coefficient outside of the reflection region in the normal image that is a color image is calculated, and the synthetic coefficient is calculated based on the color properties of the specular reflection light and the diffuse reflection light”; Note: a coefficient based on diffuse light, which is equivalent to the second lighting component, is applied to pixel values to combine the pixels of a normal image and a polarization image, providing pixels of a combined output image. The same process can be applied to the rendered image and warped image previously taught by Zimmer and Wang respectively). Wang teaches a warped image (Paragraph 0089 – “a flow can be applied to warp a previous relit frame and compute its different to a current relit frame”), and Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). Wang can be modified to use the process of applying lighting coefficients to combine pixel values of lighting components, taught by Imagawa, on the warped image and rendered image, instead of the normal image and polarization image. A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the normal and polarization images of Imagawa could have been substituted for the warped and rendered images of Wang and Zimmer respectively because all of them serve the purpose of representing details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. The substitution achieves the predictable result of providing an output image with combined pixel values from two images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the normal and polarization images of Imagawa with the warped and rendered images of Wang and Zimmer respectively. Finally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Imagawa to combine the pixel values of the warped image and rendered image using a lighting coefficient for the benefit of accommodating for lack of lighting and achieving a final image with accurate lighting (Imagawa: Col. 1 lines 51-62).
Additionally, Wang modified by Zimmer and Imagawa does not directly teach that application of the second lighting coefficients occurs separate from application of the first lighting coefficients to the pixel values of the first lighting component. Instead, Wang teaches that the application of first and second lighting coefficients occur together (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… coefficients can be predicted to linearly combine diffuse and specular light maps”). However, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to apply the first and second lighting coefficients separately in order to know the individual values of each lighting component. For instance, in Wang, the first and second lighting components are equivalent to specular and diffuse light respectively. Specular and diffuse light define different properties of light. Wang combines these specular and diffuse light values using their respective coefficients to determine how much each component affects the output scene. But in cases like in Imagawa, where the output scene results from a synthesis of multiple images, then combining the individual light components separately first is important to determine how much of a lighting component in each image will affect the output scene. Wang already performs a similar process using a specular network, where weights (coefficients) are used to determine an output specular light map based on a combination of four specular light maps (Paragraph 0076 – “a specular network can take foreground image I, albedo map A, and these four specular light maps L.sub.d, L.sub.S.sup.1, L.sub.S.sup.2, L.sub.S.sup.3, L.sub.S.sup.4 as input, and can predict a four channel weight map W.sub.S.sup.1, L.sub.S.sup.16, L.sub.S.sup.32, W.sub.S.sup.64. In at least one embodiment, a weighted specular light map L.sub.S can be computed”; Note specular coefficients are used to combine specular light). Thus, when there are multiple images affecting how the light will appear in an output image, it would be beneficial to separately use coefficients for each lighting component, in order to know how much each individual lighting component in each image will contribute to the final output and in order to generate realistic lighting for the final image. Furthermore, when there are two types of coefficients, there is a finite number of ways to apply them; either they can be applied together in the same operation, or separately. One of ordinary skill in the art could have applied the first and second lighting coefficients separately with a reasonable expectation of success and would have done so for the benefit of being able to know the individual specular and diffuse values for the sake of providing a quality output rendering. Therefore, it would have been obvious to try the solution of applying the first and second lighting coefficients separately.
Regarding claim 4, Wang in view of Zimmer and Imagawa teaches the method of claim 1. Wang further teaches wherein the plurality of lighting components comprise at least a specular lighting component and a diffuse lighting component (Paragraph 0074 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals”; Note: the specular light map is equivalent to a specular lighting component, and the diffuse light map is equivalent to a diffuse lighting component).
Regarding claim 5, Wang in view of Zimmer and Imagawa teaches the method of claim 1. Wang further teaches wherein the plurality of lighting components are rendered (Paragraph 0074 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals. In at least one embodiment, these components are concatenated as input to rendering residual network 112 to produce a final, relit output image”; Note: diffuse and specular light maps, which are lighting components, are rendered in the rendering residual network). Wang separately teaches ray tracing (Paragraph 0321 – “one or more slices 1801A-1801N includes one or more ray tracing units to compute ray tracing operations”). While Wang does not directly teach wherein the plurality of lighting components are rendered using ray tracing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine ray tracing with rendering lighting components because ray tracing is a common rendering technique in the state of the art and is useful for simulating realistic lighting effects.
Regarding claim 6, Wang in view of Zimmer and Imagawa teaches the method of claim 1. Wang does not teach applying one or more first motion vectors to pixel values for the first lighting component to provide pixel values in the warped image frame for the first lighting component; and applying one or more second motion vectors to pixel values for the second lighting component to provide pixel values in the warped image frame for the second lighting component. However, Zimmer teaches applying one or more first motion vectors to pixel values for the first lighting component to provide pixel values in the warped image frame for the first lighting component (Col. 4 lines 6-7 and 57-60, Col. 11 lines 52-55 – “each component is characterized by a single motion vector and a single spatial structure per image pixel…embodiments may decompose a scene into more or fewer components, e.g., just two components: diffuse (ED.*) and specular (sum of ET.* and ER.*)…embodiments warp every component of adjacent frames, as well as the corresponding feature buffers, using the computed per-component motion vectors, which aligns them to the current frame”; Note: motion vectors are used for specular components, which is equivalent to the first lighting component, to warp an image frame. Each motion vector corresponds to an image pixel); and applying one or more second motion vectors to pixel values for the second lighting component to provide pixel values in the warped image frame for the second lighting component (Col. 4 lines 6-7 and 57-60, Col. 11 lines 52-55 – “each component is characterized by a single motion vector and a single spatial structure per image pixel…embodiments may decompose a scene into more or fewer components, e.g., just two components: diffuse (ED.*) and specular (sum of ET.* and ER.*)…embodiments warp every component of adjacent frames, as well as the corresponding feature buffers, using the computed per-component motion vectors, which aligns them to the current frame”; Note: motion vectors are used for diffuse components, which is equivalent to the second lighting component, to warp an image frame. Each motion vector corresponds to an image pixel). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Zimmer to apply motion vectors to pixel values for warping because “Image-based methods such as frame interpolation and temporally stable denoising require accurate motion vectors for each component of the decomposition” and also because “it is straightforward to extract motion vectors of visible surface positions on the scene geometry” (Zimmer: Col. 6 lines 2-9). In other words, obtaining and using motion vectors is common and pertinent in the art.
Regarding claim 7, Wang in view of Zimmer and Imagawa teaches the method of claim 1. Wang further teaches combining the first and second lighting components to generate an output image frame (Paragraph 0074 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals. In at least one embodiment, these components are concatenated as input to rendering residual network 112 to produce a final, relit output image”; Note: the specular light map is equivalent to the first lighting component, and the diffuse light map is equivalent to the second lighting component. They are combined to generate an output image).
Regarding claim 8, Wang in view of Zimmer and Imagawa teaches the method of claim 7. Wang further teaches using an albedo to determine a proportion of first and second lighting components used to generate the output image frame (Paragraph 0076-0077 – “rendering network takes albedo map A, diffuse light map L.sub.D, and specular light map L.sub.S as input and predicts relit image R… coefficients can be predicted to linearly combine diffuse and specular light maps, then multiply this combined map by this albedo map to obtain an initial result”; Note: an output image is obtained by using an albedo map to determine a proportion of diffuse and specular light maps. The proportion is determined using multiplication).
Regarding claim 9, Wang in view of Zimmer and Imagawa teaches the method of claim 7. Wang does not teach filtering the combined first and second lighting components to reduce noise in generating the output image frame. However, Zimmer teaches filtering the first and second lighting components to reduce noise in generating the output image frame (Fig. 10, Col. 4 lines 57-60, Col. 12 lines 47-63 – “embodiments may decompose a scene into more or fewer components, e.g., just two components: diffuse (ED.*) and specular (sum of ET.* and ER.*)… Denoising may be significantly more robust when leveraging embodiments described herein…joint NL-Means filtering of each component guided by auxiliary features (‘decomposition and features’) robustly recovers fine details in the scene and yields a result close to the ground truth. (FIG. 10 also shows an example of the relative mean-square error (“MSE”) of each image for the full frame (first value) and the crop shown (second value))”; Note: the joint NL-Means filter is used to filter the diffuse and specular components for the purpose of denoising, and the generated output images are shown in Fig. 10; see screenshot of Fig. 10 below). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Zimmer to filter the lighting components for the benefit of achieving “denoising results that are visually very close to a high sample-count ground truth rendering with a low relative mean-square error” (Zimmer: Col. 12 lines 48-52). In other words, doing so would result in a final image that accurately represents the original scene.
PNG
media_image3.png
346
928
media_image3.png
Greyscale
Screenshot of Fig. 10 (taken from Zimmer)
Regarding claim 11, Wang teaches a computing device (Paragraph 0413 – “system 2600 is a mobile phone, a smart phone, a tablet computing device or a mobile Internet device”), comprising:
a memory comprising one more storage devices (Paragraph 0417 – “a memory device 2620 can be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, flash memory device, phase-change memory device, or some other memory device having suitable performance to serve as process memory. In at least one embodiment, memory device 2620 can operate as system memory for system 2600, to store data 2622 and instructions 2621”);
and one or more processors coupled to the memory (Fig. 26, Paragraph 0416 – “one or more processor(s) 2602 are coupled with one or more interface bus(es) 2610 to transmit communication signals such as address, data, or control signals between processor 2602 and other components in system 2600”; Note: The processor is coupled to the memory, as shown in Fig. 26; see modified screenshot of Fig. 26 below), the one or more processors operable to:
PNG
media_image4.png
822
680
media_image4.png
Greyscale
Modified screenshot of Fig. 26 (taken from Wang)
obtain from the memory a plurality of lighting components for an image frame in a current time instance (Paragraph 0056, 0086, 0102 – “an input image 102 can be provided as input to a geometry network 104 that can infer geometric information about one or more objects in image 102, as may include surface normals and one or more light maps…these temporal residual networks 212, 214 take outputs from a past frame and predictions from an image-only relighting part of this framework at a current frame as inputs…inference and/or training logic 715 may include, without limitation, one or more arithmetic logic unit(s) (“ALU(s)”) 710, including integer and/or floating point units, to perform logical and/or mathematical operations based, at least in part on, or indicated by, training and/or inference code (e.g., graph code), a result of which may produce activations (e.g., output values from layers or neurons within a neural network) stored in an activation storage 720 that are functions of input/output and/or weight parameter data stored in code and/or data storage 701 and/or code and/or data storage 705”; Note: the light maps are equivalent to the lighting components. They are obtained from activation storage of the geometry network, which is equivalent to memory in this case), the plurality of lighting components for the image frame being associated with a plurality of lighting coefficients (Paragraph 0077 – “a framework can compute an initial rendering from predicted albedo, diffuse, and specular light maps…coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficients are associated with each light map, as shown in the equation
PNG
media_image1.png
30
142
media_image1.png
Greyscale
, where C is a coefficient and L is a light map);
warp lighting components of a previous image frame to provide a warped image frame referenced to the current time instance (Paragraph 0089 – “a flow can be applied to warp a previous relit frame and compute its different to a current relit frame”; Note: the lighting of a previous frame is warped, which produces a warped image that corresponds to a current frame and can be used to compare to the actual current frame);
apply first lighting coefficients to combine pixel values of a first lighting component (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficient of the specular light map, which is the first lighting coefficient, is used in a combination operation related to the specular component, which corresponds to pixel values in the pixel space);
and apply second lighting coefficients to combine pixel values of a second lighting component (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficient of the diffuse light map, which is the second lighting coefficient, is used in a combination operation related to the diffuse component, which corresponds to pixel values in the pixel space);
wherein the first and second lighting coefficients are derived from an output tensor of a neural network (Paragraph 0078, 0512 – “coefficients and a residual map are predicted by f.sub.R, which takes input foreground image I, albedo map A, diffuse light map L.sub.d, and specular light map L.sub.S as input…tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inferencing”; Note: f.sub.R is a neural network that outputs lighting coefficients. Additionally, tensors are used in the neural networks, implying there is an output tensor).
Wang does not teach the rendered image frame in the limitation: “obtaining a plurality of lighting components for a rendered image frame in a current time instance”. Wang also does not teach applying first lighting coefficients to combine pixel values of a first lighting component in the rendered image frame with pixel values of the first lighting component in the warped image frame to provide pixels values of the first lighting component in an output image frame; nor applying second lighting coefficients to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame. However, Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the image of Wang could have been substituted for the rendered image of Zimmer because both the image and rendered image serve the purpose of representing the details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of retrieving light components from the image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the image of Wang for the rendered image of Zimmer according to known methods to yield the predictable result of retrieving light components from the image. Wang modified by Zimmer still does not teach applying first lighting coefficients to combine pixel values of a first lighting component in the rendered image frame with pixel values of the first lighting component in the warped image frame to provide pixels values of the first lighting component in an output image frame; nor applying second lighting coefficients to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame separate from application of the first lighting coefficients to the pixel values of the first lighting component to provide pixels values of the second lighting component in the output image frame. However, Imagawa teaches applying first lighting coefficients to combine pixel values of a first lighting component in a normal image frame with pixel values of the first lighting component in a polarization image frame to provide pixels values of the first lighting component in an output image frame (Col. 3 lines 8-12, Col. 12 lines 18-35 – “synthesizing the normal image and the polarization image using a pixel value obtained by multiplying the coefficient by the at least one of the pixel value of the normal image and the pixel value of the polarization image to generate a synthesized image…the coefficient calculating unit 105 sets the synthetic coefficient α(x,y) to each pixel based on Equation 15 below (S803).
PNG
media_image2.png
42
168
media_image2.png
Greyscale
In the calculating of a synthetic coefficient described with reference to FIG. 12, a diffuse reflection coefficient outside of the reflection region in the normal image that is a color image is calculated, and the synthetic coefficient is calculated based on the color properties of the specular reflection light and the diffuse reflection light”; Note: a coefficient based on specular light, which is equivalent to the first lighting component, is applied to pixel values to combine the pixels of a normal image and a polarization image, providing pixels of a combined output image. The same process can be applied to the rendered image and warped image previously taught by Zimmer and Wang respectively); and applying second lighting coefficients to combine pixel values of a second lighting component in a normal image frame with pixel values of the second lighting component in a polarization image frame to provide pixels values of the second lighting component in an output image frame (Col. 3 lines 8-12, Col. 12 lines 18-35 – “synthesizing the normal image and the polarization image using a pixel value obtained by multiplying the coefficient by the at least one of the pixel value of the normal image and the pixel value of the polarization image to generate a synthesized image…the coefficient calculating unit 105 sets the synthetic coefficient α(x,y) to each pixel based on Equation 15 below (S803).
PNG
media_image2.png
42
168
media_image2.png
Greyscale
In the calculating of a synthetic coefficient described with reference to FIG. 12, a diffuse reflection coefficient outside of the reflection region in the normal image that is a color image is calculated, and the synthetic coefficient is calculated based on the color properties of the specular reflection light and the diffuse reflection light”; Note: a coefficient based on diffuse light, which is equivalent to the second lighting component, is applied to pixel values to combine the pixels of a normal image and a polarization image, providing pixels of a combined output image. The same process can be applied to the rendered image and warped image previously taught by Zimmer and Wang respectively). Wang teaches a warped image (Paragraph 0089 – “a flow can be applied to warp a previous relit frame and compute its different to a current relit frame”), and Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). Wang can be modified to use the process of applying lighting coefficients to combine pixel values of lighting components, taught by Imagawa, on the warped image and rendered image, instead of the normal image and polarization image. A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the normal and polarization images of Imagawa could have been substituted for the warped and rendered images of Wang and Zimmer respectively because all of them serve the purpose of representing details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. The substitution achieves the predictable result of providing an output image with combined pixel values from two images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the normal and polarization images of Imagawa with the warped and rendered images of Wang and Zimmer respectively. Finally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Imagawa to combine the pixel values of the warped image and rendered image using a lighting coefficient for the benefit of accommodating for lack of lighting and achieving a final image with accurate lighting (Imagawa: Col. 1 lines 51-62).
Additionally, Wang modified by Zimmer and Imagawa does not directly teach that application of the second lighting coefficients occurs separate from application of the first lighting coefficients to the pixel values of the first lighting component. Instead, Wang teaches that the application of first and second lighting coefficients occur together (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… coefficients can be predicted to linearly combine diffuse and specular light maps”). However, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to apply the first and second lighting coefficients separately in order to know the individual values of each lighting component. For instance, in Wang, the first and second lighting components are equivalent to specular and diffuse light respectively. Specular and diffuse light define different properties of light. Wang combines these specular and diffuse light values using their respective coefficients to determine how much each component affects the output scene. But in cases like in Imagawa, where the output scene results from a synthesis of multiple images, then combining the individual light components separately first is important to determine how much of a lighting component in each image will affect the output scene. Wang already performs a similar process using a specular network, where weights (coefficients) are used to determine an output specular light map based on a combination of four specular light maps (Paragraph 0076 – “a specular network can take foreground image I, albedo map A, and these four specular light maps L.sub.d, L.sub.S.sup.1, L.sub.S.sup.2, L.sub.S.sup.3, L.sub.S.sup.4 as input, and can predict a four channel weight map W.sub.S.sup.1, L.sub.S.sup.16, L.sub.S.sup.32, W.sub.S.sup.64. In at least one embodiment, a weighted specular light map L.sub.S can be computed”; Note specular coefficients are used to combine specular light). Thus, when there are multiple images affecting how the light will appear in an output image, it would be beneficial to separately use coefficients for each lighting component, in order to know how much each individual lighting component in each image will contribute to the final output and in order to generate realistic lighting for the final image. Furthermore, when there are two types of coefficients, there is a finite number of ways to apply them; either they can be applied together in the same operation, or separately. One of ordinary skill in the art could have applied the first and second lighting coefficients separately with a reasonable expectation of success and would have done so for the benefit of being able to know the individual specular and diffuse values for the sake of providing a quality output rendering. Therefore, it would have been obvious to try the solution of applying the first and second lighting coefficients separately.
Regarding claim 14, Wang in view of Zimmer and Imagawa teaches the computing device of claim 11. Wang further teaches wherein the plurality of lighting components comprise at least a specular lighting component and a diffuse lighting component (Paragraph 0074 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals”; Note: the specular light map is equivalent to a specular lighting component, and the diffuse light map is equivalent to a diffuse lighting component).
Regarding claim 15, Wang in view of Zimmer and Imagawa teaches the computing device of claim 11. Wang further teaches wherein the plurality of lighting components are rendered (Paragraph 0074 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals. In at least one embodiment, these components are concatenated as input to rendering residual network 112 to produce a final, relit output image”; Note: diffuse and specular light maps, which are lighting components, are rendered in the rendering residual network). Wang separately teaches ray tracing (Paragraph 0321 – “one or more slices 1801A-1801N includes one or more ray tracing units to compute ray tracing operations”). While Wang does not directly teach wherein the plurality of lighting components are rendered using ray tracing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine ray tracing with rendering lighting components because ray tracing is a common rendering technique in the state of the art and is useful for simulating realistic lighting effects.
Regarding claim 16, Wang in view of Zimmer and Imagawa teaches the computing device of claim 11. Wang does not teach wherein application of one or more first motion vectors to pixel values for the first lighting component to provide pixel values in the warped image frame for the first lighting component; and application of one or more second motion vectors to pixel values for the second lighting component to provide pixel values in the warped image frame for the second lighting component. However, Zimmer teaches application of one or more first motion vectors to pixel values for the first lighting component to provide pixel values in the warped image frame for the first lighting component (Col. 4 lines 6-7 and 57-60, Col. 11 lines 52-55 – “each component is characterized by a single motion vector and a single spatial structure per image pixel…embodiments may decompose a scene into more or fewer components, e.g., just two components: diffuse (ED.*) and specular (sum of ET.* and ER.*)…embodiments warp every component of adjacent frames, as well as the corresponding feature buffers, using the computed per-component motion vectors, which aligns them to the current frame”; Note: motion vectors are used for specular components, which is equivalent to the first lighting component, to warp an image frame. Each motion vector corresponds to an image pixel); and application of one or more second motion vectors to pixel values for the second lighting component to provide pixel values in the warped image frame for the second lighting component (Col. 4 lines 6-7 and 57-60, Col. 11 lines 52-55 – “each component is characterized by a single motion vector and a single spatial structure per image pixel…embodiments may decompose a scene into more or fewer components, e.g., just two components: diffuse (ED.*) and specular (sum of ET.* and ER.*)…embodiments warp every component of adjacent frames, as well as the corresponding feature buffers, using the computed per-component motion vectors, which aligns them to the current frame”; Note: motion vectors are used for diffuse components, which is equivalent to the second lighting component, to warp an image frame. Each motion vector corresponds to an image pixel). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Zimmer to apply motion vectors to pixel values for warping because “Image-based methods such as frame interpolation and temporally stable denoising require accurate motion vectors for each component of the decomposition” and also because “it is straightforward to extract motion vectors of visible surface positions on the scene geometry” (Zimmer: Col. 6 lines 2-9). In other words, obtaining and using motion vectors is common and pertinent in the art.
Regarding claim 17, Wang in view of Zimmer and Imagawa teaches the computing device of claim 11. Wang further teaches the one or more processors are further operable to combine the first and second lighting components to generate an output image frame (Paragraph 0074, 0414 – “diffuse and specular light maps 116, 118 are predicted, such as by using specular network 110, with respect to a target environment map illumination using predicted normals. In at least one embodiment, these components are concatenated as input to rendering residual network 112 to produce a final, relit output image…one or more processors 2602 each include one or more processor cores 2607 to process instructions which, when executed, perform operations for system and user software”; Note: the specular light map is equivalent to the first lighting component, and the diffuse light map is equivalent to the second lighting component. They are combined to generate an output image).
Regarding claim 18, Wang in view of Zimmer and Imagawa teaches the computing device of claim 17. Wang further teaches the one or more processors are further operable to apply an albedo to determine a proportion of first and second lighting components used to generate the output image frame (Paragraph 0076-0077, 0414 – “rendering network takes albedo map A, diffuse light map L.sub.D, and specular light map L.sub.S as input and predicts relit image R… coefficients can be predicted to linearly combine diffuse and specular light maps, then multiply this combined map by this albedo map to obtain an initial result…one or more processors 2602 each include one or more processor cores 2607 to process instructions which, when executed, perform operations for system and user software”; Note: an output image is obtained by using an albedo map to determine a proportion of diffuse and specular light maps. The proportion is determined using multiplication).
Regarding claim 20, Wang teaches a method of training a neural network (Paragraph 0092 – “a process 400 for training a network to generate an image with one or more different lighting aspects can be performed as illustrated in FIG. 4”; Note: the network refers to a neural network), comprising:
receiving an input tensor in an input layer of a neural network, the input tensor representing one or more characteristics of an image frame (Paragraph 0073, 0512 – “a foreground image is obtained using an off-the-shelf matting network given an input image 102, such as an input portrait image. In at least one embodiment, this image and an environment map can serve as inputs to a relighting network to predict a relit image…tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inferencing”; Note: input image is received by a neural network. Additionally, tensors are used in the neural networks, so it is implied that there would be an input tensor because tensor operations could not be performed otherwise);
providing an output tensor to an output layer of the neural network, the output tensor representing first and second lighting coefficients (Paragraph 0078, 0512 – “coefficients and a residual map are predicted by f.sub.R, which takes input foreground image I, albedo map A, diffuse light map L.sub.d, and specular light map L.sub.S as input…tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations for neural network training and inferencing”; Note: f.sub.R is a neural network that outputs lighting coefficients. Additionally, tensors are used in the neural networks, implying there is an output tensor),
the first lighting coefficients used to combine pixel values of a first lighting component (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficient of the specular light map, which is the first lighting coefficient, is used in a combination operation related to the specular component, which corresponds to pixel values in the pixel space);
and the second lighting coefficients used to combine pixel values of a second lighting component (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… coefficients can be predicted to linearly combine diffuse and specular light maps”; Note: the coefficient of the diffuse light map, which is the second lighting coefficient, is used in a combination operation related to the diffuse component, which corresponds to pixel values in the pixel space);
the output layer of the neural network connected by one or more intermediate layers of the neural network (Fig. 8 – the output layer is connected to an intermediate layer; see modified screenshot of Fig. 8 below);
PNG
media_image5.png
448
681
media_image5.png
Greyscale
Modified screenshot of Fig. 8 (taken from Wang)
and training the neural network to predict the provided output tensor when provided with the received input tensor by using backpropagation to adjust a weight of one or more activation functions linking one or more nodes of one or more layers of the neural network (Paragraph 0109 – “untrained neural network 806 is trained in a supervised manner and processes inputs from training dataset 802 and compares resulting outputs against a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through untrained neural network 806. In at least one embodiment, training framework 804 adjusts weights that control untrained neural network 806”; Note: a neural network is trained to predict output based on received input and uses back propagation to adjust weights throughout the neural network).
Wang does not teach that the first lighting coefficients are used to combine pixel values of a first lighting component in a rendered image frame with pixel values of the first lighting component in a warped image frame to provide pixels values of the first lighting component in an output image frame, nor the second lighting coefficients are used to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame to provide pixels values of the second lighting component in the output image frame. However, Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the image of Wang could have been substituted for the rendered image of Zimmer because both the image and rendered image serve the purpose of representing the details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of retrieving light components from the image. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the image of Wang for the rendered image of Zimmer according to known methods to yield the predictable result of retrieving light components from the image. Wang modified by Zimmer still does not teach that the first lighting coefficients are used to combine pixel values of a first lighting component in the rendered image frame with pixel values of the first lighting component in the warped image frame to provide pixels values of the first lighting component in an output image frame; nor the second lighting coefficients are used to combine pixel values of a second lighting component in the rendered image frame with pixel values of the second lighting component in the warped image frame to provide pixels values of the second lighting component in the output image frame. However, Imagawa teaches that the first lighting coefficients are used to combine pixel values of a first lighting component in a normal image frame with pixel values of the first lighting component in a polarization image frame to provide pixels values of the first lighting component in an output image frame (Col. 3 lines 8-12, Col. 12 lines 18-35 – “synthesizing the normal image and the polarization image using a pixel value obtained by multiplying the coefficient by the at least one of the pixel value of the normal image and the pixel value of the polarization image to generate a synthesized image…the coefficient calculating unit 105 sets the synthetic coefficient α(x,y) to each pixel based on Equation 15 below (S803).
PNG
media_image2.png
42
168
media_image2.png
Greyscale
In the calculating of a synthetic coefficient described with reference to FIG. 12, a diffuse reflection coefficient outside of the reflection region in the normal image that is a color image is calculated, and the synthetic coefficient is calculated based on the color properties of the specular reflection light and the diffuse reflection light”; Note: a coefficient based on specular light, which is equivalent to the first lighting component, is applied to pixel values to combine the pixels of a normal image and a polarization image, providing pixels of a combined output image); and the second lighting coefficients are used to combine pixel values of a second lighting component in a normal image frame with pixel values of the second lighting component in a polarization image frame to provide pixels values of the second lighting component in an output image frame (Col. 3 lines 8-12, Col. 12 lines 18-35 – “synthesizing the normal image and the polarization image using a pixel value obtained by multiplying the coefficient by the at least one of the pixel value of the normal image and the pixel value of the polarization image to generate a synthesized image…the coefficient calculating unit 105 sets the synthetic coefficient α(x,y) to each pixel based on Equation 15 below (S803).
PNG
media_image2.png
42
168
media_image2.png
Greyscale
In the calculating of a synthetic coefficient described with reference to FIG. 12, a diffuse reflection coefficient outside of the reflection region in the normal image that is a color image is calculated, and the synthetic coefficient is calculated based on the color properties of the specular reflection light and the diffuse reflection light”; Note: a coefficient based on diffuse light, which is equivalent to the second lighting component, is applied to pixel values to combine the pixels of a normal image and a polarization image, providing pixels of a combined output image). Wang teaches a warped image (Paragraph 0089 – “a flow can be applied to warp a previous relit frame and compute its different to a current relit frame”), and Zimmer teaches a rendered image (Col. 3 line 41 – “the rendered image”). Wang can be modified to use the process of applying lighting coefficients to combine pixel values of lighting components, taught by Imagawa, on the warped image and rendered image, instead of the normal image and polarization image. A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the normal and polarization images of Imagawa could have been substituted for the warped and rendered images of Wang and Zimmer respectively because all of them serve the purpose of representing details and light in a scene. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. The substitution achieves the predictable result of providing an output image with combined pixel values from two images. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the normal and polarization images of Imagawa with the warped and rendered images of Wang and Zimmer respectively. Finally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Imagawa to combine the pixel values of the warped image and rendered image using a lighting coefficient for the benefit of accommodating for lack of lighting and achieving a final image with accurate lighting (Imagawa: Col. 1 lines 51-62).
Additionally, Wang modified by Zimmer and Imagawa does not directly teach that application of the second lighting coefficients occurs separate from application of the first lighting coefficients to the pixel values of the first lighting component. Instead, Wang teaches that the application of first and second lighting coefficients occur together (Paragraph 0075, 0077 – “these diffuse and specular light maps can be more efficient representations of lighting than an original environment map as they embed diffuse and specular components of illumination in pixel space… coefficients can be predicted to linearly combine diffuse and specular light maps”). However, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to apply the first and second lighting coefficients separately in order to know the individual values of each lighting component. For instance, in Wang, the first and second lighting components are equivalent to specular and diffuse light respectively. Specular and diffuse light define different properties of light. Wang combines these specular and diffuse light values using their respective coefficients to determine how much each component affects the output scene. But in cases like in Imagawa, where the output scene results from a synthesis of multiple images, then combining the individual light components separately first is important to determine how much of a lighting component in each image will affect the output scene. Wang already performs a similar process using a specular network, where weights (coefficients) are used to determine an output specular light map based on a combination of four specular light maps (Paragraph 0076 – “a specular network can take foreground image I, albedo map A, and these four specular light maps L.sub.d, L.sub.S.sup.1, L.sub.S.sup.2, L.sub.S.sup.3, L.sub.S.sup.4 as input, and can predict a four channel weight map W.sub.S.sup.1, L.sub.S.sup.16, L.sub.S.sup.32, W.sub.S.sup.64. In at least one embodiment, a weighted specular light map L.sub.S can be computed”; Note specular coefficients are used to combine specular light). Thus, when there are multiple images affecting how the light will appear in an output image, it would be beneficial to separately use coefficients for each lighting component, in order to know how much each individual lighting component in each image will contribute to the final output and in order to generate realistic lighting for the final image. Furthermore, when there are two types of coefficients, there is a finite number of ways to apply them; either they can be applied together in the same operation, or separately. One of ordinary skill in the art could have applied the first and second lighting coefficients separately with a reasonable expectation of success and would have done so for the benefit of being able to know the individual specular and diffuse values for the sake of providing a quality output rendering. Therefore, it would have been obvious to try the solution of applying the first and second lighting coefficients separately.
Claims 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Zimmer, Imagawa, and Rossin et al. (US 6707453 B1), hereinafter Rossin.
Regarding claim 2, Wang in view of Zimmer and Imagawa teaches the method of claim 1. Wang does not teach interpolating pixel values of the first lighting component; nor interpolating pixel values of the second lighting component. However, Rossin teaches interpolating pixel values of the first lighting component (Col. 14 lines 1-7 – “Each shap stepper or interpolator 310-314 determines the value of a component at each pixel along each span of the primitive. Each span is generally a row of interior pixels bounded by edge pixels. As noted, a consolidated interpolator to interpolate specular and diffuse lighting components can be implemented for each type of interpolator in scan converter 204”; Note: pixel values of specular lighting components are interpolated); and interpolating pixel values of the second lighting component (Col. 14 lines 1-7 – “Each shap stepper or interpolator 310-314 determines the value of a component at each pixel along each span of the primitive. Each span is generally a row of interior pixels bounded by edge pixels. As noted, a consolidated interpolator to interpolate specular and diffuse lighting components can be implemented for each type of interpolator in scan converter 204”; Note: pixel values of diffuse lighting components are interpolated). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Rossin to interpolate the pixel values of the lighting components for the benefit of “insuring that all object surfaces, including those that have low color intensity, properly exhibit the specular lighting contribution” (Rossin: Col. 4 lines 10-13) and ensuring that the lighting is accurately represented in the final image.
Regarding claim 12, Wang in view of Zimmer and Imagawa teaches the computing device of claim 11. Wang does not teach wherein: application of the first lighting coefficients to combine pixel values of the first lighting component in the warped image frame is based, at least in part, on an interpolation pixel values of the first lighting component; application of the second lighting coefficients to combine pixel values of the second lighting component in the warped image frame is based, at least in part, on an interpolation of pixel values of the second lighting component. However, Rossin teaches interpolating pixel values of the first lighting component (Col. 14 lines 1-7 – “Each shap stepper or interpolator 310-314 determines the value of a component at each pixel along each span of the primitive. Each span is generally a row of interior pixels bounded by edge pixels. As noted, a consolidated interpolator to interpolate specular and diffuse lighting components can be implemented for each type of interpolator in scan converter 204”; Note: pixel values of specular lighting components are interpolated); and interpolating pixel values of the second lighting component (Col. 14 lines 1-7 – “Each shap stepper or interpolator 310-314 determines the value of a component at each pixel along each span of the primitive. Each span is generally a row of interior pixels bounded by edge pixels. As noted, a consolidated interpolator to interpolate specular and diffuse lighting components can be implemented for each type of interpolator in scan converter 204”; Note: pixel values of diffuse lighting components are interpolated). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Rossin to interpolate the pixel values of the lighting components for the benefit of “insuring that all object surfaces, including those that have low color intensity, properly exhibit the specular lighting contribution” (Rossin: Col. 4 lines 10-13) and ensuring that the lighting is accurately represented in the final image. Interpolation is a common and reliable way for combining pixel values between images.
Claims 3, 10, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Zimmer, Imagawa, and Wizadwongsa et al. (NeX: Real-time View Synthesis with Neural Basis Expansion), hereinafter Wizadwongsa.
Regarding claim 3, Wang in view of Zimmer and Imagawa teaches the method of claim 1. Wang does not teach wherein first and second lighting coefficients are per-pixel coefficients. However, Wizadwongsa teaches wherein first and second lighting coefficients are per-pixel coefficients (Fig. 1 Caption on page 1 – “Each pixel in NeX multiplane image consists of an alpha transparency value, base color k0, and view-dependent reflectance coefficients k1…kn. A linear combination of these coefficients and basis functions learned from a neural network produces the final color value”; Note: the reflectance coefficients correspond to lighting coefficients, and each pixel has them). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Wizadwongsa to have the lighting coefficients for each pixel for the benefit of improving “fine detail” and producing “sharper results” (Wizadwongsa: Col. 1 Paragraph 1 on page 2). Having a lighting coefficient for each pixel would help make the final output accurately represent the lighting of the original image.
Regarding claim 10, Wang in view of Zimmer and Imagawa teaches the method of claim 1. Wang does not teach wherein the first and second lighting coefficients are derived from disocclusion of one or more pixels in the rendered image frame in the current time instance or a change in view-dependent lighting for one or more pixels in the rendered image frame in the current time instance, or a combination thereof. However, Wizadwongsa teaches wherein the first and second lighting coefficients are derived from a change in view-dependent lighting for one or more pixels in the image frame (Fig. 1 Caption on page 1, Col. 2 Paragraphs 1 and 3 on Page 6 – “Each pixel in NeX multiplane image consists of an alpha transparency value, base color k0, and view-dependent reflectance coefficients k1…kn. A linear combination of these coefficients and basis functions learned from a neural network produces the final color value…We train our model on 12 input views, then evaluate on 4 held-out views… We vary the number of basis coefficients from zero, which represents no view-dependent modeling, to 20”; Note: the reflectance coefficients correspond to lighting coefficients, and they derive from view-dependent lighting, where each coefficient corresponds to a different view). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Wizadwongsa to derive lighting coefficients from changes in view-dependent lighting for the benefit of accounting for lighting differences, which will make the final result more realistic. Considering view-dependent lighting helps with capturing “fine detail” and reproducing “complex view-dependent effects…thus allowing real-time rendering” (Wizadwongsa: Col. 1 Paragraph 1 on page 2, Col. 2 Paragraph 2 on Page 8).
Regarding claim 13, Wang in view of Zimmer and Imagawa teaches the computing device of claim 11. Wang does not teach wherein first and second lighting coefficients are per-pixel coefficients. However, Wizadwongsa teaches wherein first and second lighting coefficients are per-pixel coefficients (Fig. 1 Caption on page 1 – “Each pixel in NeX multiplane image consists of an alpha transparency value, base color k0, and view-dependent reflectance coefficients k1…kn. A linear combination of these coefficients and basis functions learned from a neural network produces the final color value”; Note: the reflectance coefficients correspond to lighting coefficients, and each pixel has them). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Wizadwongsa to have the lighting coefficients for each pixel for the benefit of improving “fine detail” and producing “sharper results” (Wizadwongsa: Col. 1 Paragraph 1 on page 2). Having a lighting coefficient for each pixel would help make the final output accurately represent the lighting of the original image.
Regarding claim 19, Wang in view of Zimmer and Imagawa teaches the computing device of claim 11. Wang does not teach wherein the first and second lighting coefficients are derived from disocclusion of one or more pixels in the rendered image frame in the current time instance or a change in view-dependent lighting for one or more pixels in the rendered image frame in the current time instance, or a combination thereof. However, Wizadwongsa teaches wherein the first and second lighting coefficients are derived from a change in view-dependent lighting for one or more pixels in the image frame (Fig. 1 Caption on page 1, Col. 2 Paragraphs 1 and 3 on Page 6 – “Each pixel in NeX multiplane image consists of an alpha transparency value, base color k0, and view-dependent reflectance coefficients k1…kn. A linear combination of these coefficients and basis functions learned from a neural network produces the final color value…We train our model on 12 input views, then evaluate on 4 held-out views… We vary the number of basis coefficients from zero, which represents no view-dependent modeling, to 20”; Note: the reflectance coefficients correspond to lighting coefficients, and they derive from view-dependent lighting, where each coefficient corresponds to a different view). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Wang to incorporate the teachings of Wizadwongsa to derive lighting coefficients from changes in view-dependent lighting for the benefit of accounting for lighting differences, which will make the final result more realistic. Considering view-dependent lighting helps with capturing “fine detail” and reproducing “complex view-dependent effects…thus allowing real-time rendering” (Wizadwongsa: Col. 1 Paragraph 1 on page 2, Col. 2 Paragraph 2 on Page 8).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Silverbrook et al. (US 6636216 B1) teaches a method for generating a warped image that takes into account image light sources. Cabral et al. (US 20050099418 A1) teaches a method of rendering an object using radiance environment maps associated with viewing vectors that are warped.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE HAU MA whose telephone number is (571)272-2187. The examiner can normally be reached M-Th 7-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE HAU MA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617