Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
TITLE
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed to.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 6-7, 13-14 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 6, 13 and 20 recite the limitation "the candidate" in lines 5-6. There is insufficient antecedent basis for this limitation in the claim.
Claims 7 and 14 are rejected based on the rejection of claim 6 and claim 13.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2,6, 8-9, 13, 15-16 and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by BAO LINCHAO; LU HUCHUAN; ZHANG YING; KANG DI; CHEN JIANCHUAN, IDS, CN113822977(English translated) (”LINCHAO”)
Regarding independent claim 1, LINCHAO teaches an image processing method performed by an electronic device (n0004] This application provides an image rendering method, apparatus, device, and storage medium that can improve rendering effects”) , the method comprising:
obtaining an initial value of a target material parameter of a target object ([n0006] Based on sample video frames, obtain the shape parameters and pose parameters of the sample object, wherein the sample video frames include the sample object”; [n0012] In one possible implementation, the first rendering parameters include color parameters and density parameters, and determining the color and opacity of the pixel based on the virtual ray between the pixel and the virtual camera, and the first rendering parameters corresponding to the pixel, includes:[n0013] The color is obtained by integrating the first relational data on the virtual ray, wherein the first relational data is associated with the color parameter and the density parameter. [n0014] The opacity is obtained by integrating the second relational data on the virtual ray, and the second relational data is associated with the density parameter. [n0098] The camera parameters describe the relevant attributes of the virtual camera, such as the focal length, the size of the captured image, and the position of the camera. The focal length describes the virtual camera's focal length; the size of the captured image describes the height and width of the image captured by the virtual camera; and the position parameters describe the position of the virtual camera. In some embodiments, the intersection parameters of the virtual camera and the size parameters of the captured images are also referred to as the intrinsic parameters of the virtual camera, and the position parameters of the virtual camera are also referred to as the extrinsic parameters of the virtual camera. The first rendering parameter is the parameter used to render the image.”);
determining a photographing parameter of a photographed image of the target object ([n0030] “The rendering module is used to determine multiple first rendering parameters based on the camera parameters of the virtual camera, the shape parameters, and the pose parameters through an image rendering model. The camera parameters of the virtual camera are the same as the camera parameters of the real camera that captured the sample video frame” );
rendering the target object according to the initial value and the photographing parameter, to obtain a rendered image of the target object ([n0052] The rendering image display module is used to display a first rendered image in response to a rendering operation on the first target image. The first rendered image is obtained by rendering the first target image based on a plurality of first rendering parameters through a trained image rendering model. The plurality of first rendering parameters are determined by the image rendering model based on the camera parameters, shape parameters, and pose parameters of the virtual camera. The image rendering model is used to render the image captured by the virtual camera.”; ([n0098] The camera parameters describe the relevant attributes of the virtual camera, such as the focal length, the size of the captured image, and the position of the camera. The focal length describes the virtual camera's focal length; the size of the captured image describes the height and width of the image captured by the virtual camera; and the position parameters describe the position of the virtual camera. In some embodiments, the intersection parameters of the virtual camera and the size parameters of the captured images are also referred to as the intrinsic parameters of the virtual camera, and the position parameters of the virtual camera are also referred to as the extrinsic parameters of the virtual camera. The first rendering parameter is the parameter used to render the image. [n0099] 204. The server renders the first target image based on multiple first rendering parameters using an image rendering model, and outputs the first rendered image. The first target image is an image obtained by a virtual camera capturing a 3D model. [n0100]The first target image is an image obtained by a virtual camera capturing a 3D model. This means that the first target image is an image obtained by using a virtual camera to capture a 3D model of a sample object at a specific position and angle. The specific position and angle are determined by the camera parameters of the virtual camera.”;);
determining difference information between the rendered image and the photographed image (n0191] 308. The server trains the image rendering model based on the difference information between the sample video frames and the first rendered image ([n0192] In one possible implementation, the server constructs a first loss function based on the color difference information between the sample video frames and the first rendered image, and uses the first loss function to train the image rendering model. [n0193] For example, the server constructs a first loss function based on the color difference information between the colors determined by the coarse network and the colors of the sample video frames, and the color difference information between the colors determined by the fine network and the colors of the sample video frames”);
adjusting the initial value according to the difference information, to obtain a target value of the target material parameter ([n0196] In one possible implementation, the server applies regularization constraints to the pose parameters of adjacent sample video frames to ensure that the pose parameters between adjacent sample video frames are as close as possible, and to ensure that the optimized pose parameters are not significantly different from the original pose parameters. In other words, in order to obtain stable and smooth pose parameters, a second loss function is added during the image rendering process. The optimized pose parameters and the initial parameters should not have too large aberrations, and the pose parameters between adjacent frames should be as similar as possible. Here, the optimized pose parameters refer to the average pose parameters of multiple sample video frames in the sample video.”; [0200] For example, the server constructs a third loss function based on the opacity difference information between the opacity determined by the coarse network and the opacity of the sample video frame, and the opacity difference information between the opacity determined by the fine network and the opacity of the sample video frame.”); and
updating the rendered image of the target object by rendering the target object according to the target value (see at least [n0009]The image rendering model is trained based on the difference information between the sample video frames and the first rendered image. The image rendering model is used to render the image captured by the virtual camera [n0192]-[n0203] In one possible implementation, the server constructs a first loss function based on the color difference information between the sample video frames and the first rendered image, and uses the first loss function to train the image rendering model. [n0193] For example, the server constructs a first loss function based on the color difference information between the colors determined by the coarse network and the colors of the sample video frames, and the color difference information between the colors determined by the fine network and the colors of the sample video frames.[n0196] In one possible implementation, the server applies regularization constraints to the pose parameters of adjacent sample video frames to ensure that the pose parameters between adjacent sample video frames are as close as possible, and to ensure that the optimized pose parameters are not significantly different from the original pose parameters. In other words, in order to obtain stable and smooth pose parameters, a second loss function is added during the image rendering process. The optimized pose parameters and the initial parameters should not have too large aberrations, and the pose parameters between adjacent frames should be as similar as possible. Here, the optimized pose parameters refer to the average pose parameters of multiple sample video frames in the sample video…[n0203] In one possible implementation, the server can use at least two of the above three loss functions to train the image rendering model”)
Regarding claim 2, LINCHAO teaches the method according to claim 1, wherein the determining difference information between the rendered image and the photographed image comprises: determining a rendering value of the target material parameter corresponding to the rendered image; determining a real value of the target material parameter corresponding to the photographed image ; and determining the difference information according to the rendering value and the real value (see at least [n0191] 308. The server trains the image rendering model based on the difference information between the sample video frames and the first rendered image. [n0192] In one possible implementation, the server constructs a first loss function based on the color difference information between the sample video frames and the first rendered image, and uses the first loss function to train the image rendering model.[n0193] For example, the server constructs a first loss function based on the color difference information between the colors determined by the coarse network and the colors of the sample video frames, and the color difference information between the colors determined by the fine network and the colors of the sample video frames.”)
Regarding claim 6, LINCHAO teaches the method according to claim 1, wherein the obtaining an initial value of a target material parameter of a target object comprises: obtaining at least two material parameters of the target object; and selecting the target material parameter from the at least two material parameters; and rendering the target object according to the initial value, a set value corresponding to the candidate material parameter, and the photographing parameter, to obtain the rendered image of the target object, , wherein the candidate material parameter is a parameter in the at least two material parameters except the target material parameter (see at least [n0006] Based on sample video frames, obtain the shape parameters and pose parameters of the sample object, wherein the sample video frames include the sample object”; [n0012] In one possible implementation, the first rendering parameters include color parameters and density parameters, and determining the color and opacity of the pixel based on the virtual ray between the pixel and the virtual camera, and the first rendering parameters corresponding to the pixel, includes:[n0013] The color is obtained by integrating the first relational data on the virtual ray, wherein the first relational data is associated with the color parameter and the density parameter. [n0192]-[n0203] In one possible implementation, the server constructs a first loss function based on the color difference information between the sample video frames and the first rendered image, and uses the first loss function to train the image rendering model. [n0193] For example, the server constructs a first loss function based on the color difference information between the colors determined by the coarse network and the colors of the sample video frames, and the color difference information between the colors determined by the fine network and the colors of the sample video frames.[n0196] In one possible implementation, the server applies regularization constraints to the pose parameters of adjacent sample video frames to ensure that the pose parameters between adjacent sample video frames are as close as possible, and to ensure that the optimized pose parameters are not significantly different from the original pose parameters. In other words, in order to obtain stable and smooth pose parameters, a second loss function is added during the image rendering process. The optimized pose parameters and the initial parameters should not have too large aberrations, and the pose parameters between adjacent frames should be as similar as possible. Here, the optimized pose parameters refer to the average pose parameters of multiple sample video frames in the sample video…[n0203] In one possible implementation, the server can use at least two of the above three loss functions to train the image rendering model”).
Regarding independent claim 8, LINCHAO teaches an electronic device, comprising a processor and a memory, the memory having a computer program stored therein, and the computer program, when executed by the processor, causing the electronic device to implement an image processing method (n0053] On one hand, a computer device is provided, the computer device including one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the image rendering method”)including: Remaining limitations of claim 8 is similar scope to claim 1 and therefore rejected under the same rationale.
Regarding claim 9, LINCHAO teaches the electronic device according to claim 8, Remaining limitations of claim 9 is similar scope to claim 2 and therefore rejected under the same rationale.
Regarding claim 13, LINCHAO teaches the electronic device according to claim 8, Remaining limitations of claim 13 is similar scope to claim 6 and therefore rejected under the same rationale.
Regarding independent claim 15, LINCHAO teaches a non-transitory computer-readable storage medium having a computer program stored therein( [n0313] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a computer program that can be executed by a processor to perform the image rendering method in the above embodiments.), and the computer program, when executed by a processor of an electronic device, causing the electronic device to implement an image processing method ([n0314] In an exemplary embodiment, a computer program product or computer program is also provided, which includes program code stored in a computer-readable storage medium. A processor of a computer device reads the program code from the computer-readable storage medium and executes the program code, causing the computer device to perform the image rendering method described above.”) including: Remaining limitations of claim 15 is similar scope to claim 1 and therefore rejected under the same rationale.
Regarding claim 16, LINCHAO teaches the non-transitory computer-readable storage medium according to claim 15, Remaining limitations of claim 16 is similar scope to claim 2 and therefore rejected under the same rationale.
Regarding claim 20, LINCHAO teaches the non-transitory computer-readable storage medium according to claim 15, Remaining limitations of claim 20 is similar scope to claim 6 and therefore rejected under the same rationale.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
1. Claims 3-4, 10-11 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over BAO LINCHAO; LU HUCHUAN; ZHANG YING; KANG DI; CHEN JIANCHUAN, IDS, CN113822977(English translated) (”LINCHAO) in view of EN DAXUAN; LI JIATONGl ;MEI HAIYI, IDS, CN115272548 -English translated (“Daxuan”)
Regarding claim 3, LINCHAO teaches the method according to claim 1, wherein the adjusting the initial value according to the difference information, to obtain a target value of the target material parameter comprises: adjusting the initial value according to the difference information, to obtain a candidate value of the target material parameter; rendering the target object according to the candidate value and the photographing parameter; determining the difference information between the rendered image and the photographed image; repeating the adjusting and rendering operations until the difference information satisfies; and using the candidate value as the target value (see at least [n0192]-[n0203] In one possible implementation, the server constructs a first loss function based on the color difference information between the sample video frames and the first rendered image, and uses the first loss function to train the image rendering model. [n0193] For example, the server constructs a first loss function based on the color difference information between the colors determined by the coarse network and the colors of the sample video frames, and the color difference information between the colors determined by the fine network and the colors of the sample video frames.[n0196] In one possible implementation, the server applies regularization constraints to the pose parameters of adjacent sample video frames to ensure that the pose parameters between adjacent sample video frames are as close as possible, and to ensure that the optimized pose parameters are not significantly different from the original pose parameters. In other words, in order to obtain stable and smooth pose parameters, a second loss function is added during the image rendering process. The optimized pose parameters and the initial parameters should not have too large aberrations, and the pose parameters between adjacent frames should be as similar as possible. Here, the optimized pose parameters refer to the average pose parameters of multiple sample video frames in the sample video…[n0203] In one possible implementation, the server can use at least two of the above three loss functions to train the image rendering model”) LINCHAO is understood to be silent on the remaining limitations of claim 3.
In the same field of endeavor, Daxuan teaches adjusting the initial value according to the difference information, to obtain a candidate value of the target material parameter; rendering the target object according to the candidate value; determining the difference information between the rendered image and the photographed image; repeating the adjusting and rendering operations until the difference information satisfies a preset condition; and using the candidate value as the target value [n0117] of Daxuan “Specifically, based on the difference results, the rendering parameters are adjusted using a Bayesian optimization algorithm. Among them, the Bayesian optimization algorithm is a method that uses Bayes' theorem to guide the search to find the minimum or maximum value of the objective function. That is, in each process of adjusting the rendering parameters, it is necessary to determine the current rendering parameters based on the confidence level of the previous rendering parameter distribution until the optimal rendering parameters are found. In this way, the efficiency of adjusting the rendering parameters can be improved.”; [n0119] Specifically, after adjusting the rendering parameters, the initial rendering parameters (adjusted rendering parameters) are redefined for the renderer, and the renderer is used to render the target 3D model based on the adjusted rendering parameters to obtain the adjusted rendered image. Then, the difference between the adjusted rendered image and the real image is redefined, and it is determined whether the rendering parameters need to be adjusted again based on the difference. If the above adjustment process meets the preset conditions, the adjusted target rendering parameters can be obtained; if the above adjustment process does not meet the preset conditions, the rendering parameters need to be adjusted again until the adjustment process meets the preset conditions.”; [n0120]
In one possible implementation, the adjustment process satisfies at least one of the following preset conditions: the rate of change of the difference result meets a preset requirement, the number of times the rendering parameters are adjusted reaches a preset number, and the adjustment time of the rendering parameters reaches a preset time.
Specifically, if the rate of change of the difference between the sample image and the target image is less than a preset threshold (for example, if the difference results obtained after multiple adjustments to the rendering parameters do not show significant changes), the difference result is determined to meet the preset requirements. [n0121] In this embodiment of the disclosure, the rendered image obtained by the renderer is compared with the real image to obtain the difference result. Then, the rendering parameters of the renderer are automatically adjusted based on the difference result. This process is repeated until the adjustment process meets the preset conditions and the adjusted target rendering parameters are obtained. This avoids the process of manually adjusting the parameters, improves the efficiency of parameter adjustment, and reduces labor costs. Furthermore, by automatically iterating and adjusting the rendering parameters, the accuracy of the rendering parameter adjustment can be improved, making the rendered image closer to the real image, which in turn can improve the accuracy of subsequent model training. [n0157] S206, using the renderer, based on the adjusted target rendering parameters, the target 3D model is rendered to obtain a rendered target sample image.[n0158] After obtaining the adjusted target rendering parameters, the renderer can render the target 3D model according to the adjusted target rendering parameters to obtain the rendered target sample image. It can be understood that the adjusted target rendering parameters are the optimal rendering parameters. Therefore, the target sample image obtained by rendering with the adjusted target rendering parameters is the image that is closest to the real image.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of rendering the target image based on a plurality of rendering parameters of LINCHAO with adjusting rendering parameters based on the difference between rendered image and real image as seen in Daxuan because this modification would obtain the target sample image by rendering with the adjusted target rendering parameters is the image that is closest to the real image ([n0158] of Daxuan).
Thus, the combination of LINCHAO and Daxuan teaches wherein the adjusting the initial value according to the difference information, to obtain a target value of the target material parameter comprises: adjusting the initial value according to the difference information, to obtain a candidate value of the target material parameter; rendering the target object according to the candidate value and the photographing parameter; determining the difference information between the rendered image and the photographed image; repeating the adjusting and rendering operations until the difference information satisfies a preset condition; and using the candidate value as the target value.
Regarding claim 4, LINCHAO teaches the method according to claim 1, wherein the adjusting the initial value according to the difference information, to obtain a target value of the target material parameter comprises: determining change of the target material parameter according to the difference information; obtaining an optimization step corresponding to the target material parameter; and adjusting the initial value according to the optimization step and the change, to obtain the target value (see at least [n0192]-[n0203] In one possible implementation, the server constructs a first loss function based on the color difference information between the sample video frames and the first rendered image, and uses the first loss function to train the image rendering model. [n0193] For example, the server constructs a first loss function based on the color difference information between the colors determined by the coarse network and the colors of the sample video frames, and the color difference information between the colors determined by the fine network and the colors of the sample video frames.[n0196] In one possible implementation, the server applies regularization constraints to the pose parameters of adjacent sample video frames to ensure that the pose parameters between adjacent sample video frames are as close as possible, and to ensure that the optimized pose parameters are not significantly different from the original pose parameters. In other words, in order to obtain stable and smooth pose parameters, a second loss function is added during the image rendering process. The optimized pose parameters and the initial parameters should not have too large aberrations, and the pose parameters between adjacent frames should be as similar as possible. Here, the optimized pose parameters refer to the average pose parameters of multiple sample video frames in the sample video…[n0203] In one possible implementation, the server can use at least two of the above three loss functions to train the image rendering model”) LINCHAO is understood to be silent on the remaining limitations of claim 4.
In the same field of endeavor, Daxuan teaches wherein the adjusting the initial value according to the difference information, to obtain a target value of the target material parameter comprises: determining an amount of change of the target material parameter according to the difference information; obtaining an optimization step corresponding to the target material parameter; and adjusting the initial value according to the optimization step and the amount of change, to obtain the target value ( [n0117] “Specifically, based on the difference results, the rendering parameters are adjusted using a Bayesian optimization algorithm. Among them, the Bayesian optimization algorithm is a method that uses Bayes' theorem to guide the search to find the minimum or maximum value of the objective function. That is, in each process of adjusting the rendering parameters, it is necessary to determine the current rendering parameters based on the confidence level of the previous rendering parameter distribution until the optimal rendering parameters are found. In this way, the efficiency of adjusting the rendering parameters can be improved.”; [n0119] Specifically, after adjusting the rendering parameters, the initial rendering parameters (adjusted rendering parameters) are redefined for the renderer, and the renderer is used to render the target 3D model based on the adjusted rendering parameters to obtain the adjusted rendered image. Then, the difference between the adjusted rendered image and the real image is redefined, and it is determined whether the rendering parameters need to be adjusted again based on the difference. If the above adjustment process meets the preset conditions, the adjusted target rendering parameters can be obtained; if the above adjustment process does not meet the preset conditions, the rendering parameters need to be adjusted again until the adjustment process meets the preset conditions.”; [n0120] In one possible implementation, the adjustment process satisfies at least one of the following preset conditions: the rate of change of the difference result meets a preset requirement, the number of times the rendering parameters are adjusted reaches a preset number, and the adjustment time of the rendering parameters reaches a preset time. Specifically, if the rate of change of the difference between the sample image and the target image is less than a preset threshold (for example, if the difference results obtained after multiple adjustments to the rendering parameters do not show significant changes), the difference result is determined to meet the preset requirements. [n0121] In this embodiment of the disclosure, the rendered image obtained by the renderer is compared with the real image to obtain the difference result. Then, the rendering parameters of the renderer are automatically adjusted based on the difference result. This process is repeated until the adjustment process meets the preset conditions and the adjusted target rendering parameters are obtained. This avoids the process of manually adjusting the parameters, improves the efficiency of parameter adjustment, and reduces labor costs. Furthermore, by automatically iterating and adjusting the rendering parameters, the accuracy of the rendering parameter adjustment can be improved, making the rendered image closer to the real image, which in turn can improve the accuracy of subsequent model training. [n0157] S206, using the renderer, based on the adjusted target rendering parameters, the target 3D model is rendered to obtain a rendered target sample image.[n0158] After obtaining the adjusted target rendering parameters, the renderer can render the target 3D model according to the adjusted target rendering parameters to obtain the rendered target sample image. It can be understood that the adjusted target rendering parameters are the optimal rendering parameters. Therefore, the target sample image obtained by rendering with the adjusted target rendering parameters is the image that is closest to the real image.”) In addition, the same motivation is used as the rejection for claim 3.
Thus, the combination of LINCHAO and Daxuan teaches wherein the adjusting the initial value according to the difference information, to obtain a target value of the target material parameter comprises: determining an amount of change of the target material parameter according to the difference information; obtaining an optimization step corresponding to the target material parameter; and adjusting the initial value according to the optimization step and the amount of change, to obtain the target value.
Regarding claim 10, LINCHAO teaches the electronic device according to claim 8, Remaining limitations of claim 10 is similar scope to claim 3 and therefore rejected under the same rationale.
Regarding claim 11, LINCHAO teaches the electronic device according to claim 8, Remaining limitations of claim 11 is similar scope to claim 4 and therefore rejected under the same rationale.
Regarding claim 17, LINCHAO teaches the non-transitory computer-readable storage medium according to claim 15, Remaining limitations of claim 17 is similar scope to claim 3 and therefore rejected under the same rationale.
Regarding claim 18, LINCHAO teaches the non-transitory computer-readable storage medium according to claim 15, Remaining limitations of claim 18 is similar scope to claim 4 and therefore rejected under the same rationale.
2. Claims 5, 12 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over BAO LINCHAO; LU HUCHUAN; ZHANG YING; KANG DI; CHEN JIANCHUAN, IDS, CN113822977(English translated) (”LINCHAO”) in view of Chekh et al., U.S Patent Application Publication No.20190350680 (“Chekh”) further in view of KOUNOSU, U.S Patent Application Publication No.2023/0392985 (“KOUNOSU”)
Regarding claim 5, LINCHAO teaches the method according to claim 1, wherein the determining difference information between the rendered image and the photographed image comprises: obtain a rendered image; obtain a photographed image; and determining the difference information according to the rendered image and the photographed image (n0191] 308. The server trains the image rendering model based on the difference information between the sample video frames and the first rendered image ([n0192] In one possible implementation, the server constructs a first loss function based on the color difference information between the sample video frames and the first rendered image, and uses the first loss function to train the image rendering model. [n0193] For example, the server constructs a first loss function based on the color difference information between the colors determined by the coarse network and the colors of the sample video frames, and the color difference information between the colors determined by the fine network and the colors of the sample video frames”) LINCHAO is understood to be silent on the remaining limitations of claim 5.
In the same field of endeavor, Chekh teaches performing color space conversion on the rendered image, to obtain a converted rendered image ([0364] The method may start at block 4704 instead of block 4702. At block 4704, the color space of the rendered image of the teeth is converted to the L*a*b* color space. The L*a*b* color space, also referred to as the CIELAB color space, uses three numerical values, L*, a*, and b* to define the color for each pixel in a rendered image. The L* is the luminance or lightness channel, and may range from 0-100 with 0 representing the darkest luminance and 100 representing the white point, or brightest value, for example. a* is the green-red color channel and may range from −128 to +127 with all green and no red being represented by −128 and all red and no green being represented by +127. Similarly, b* is the blue-yellow color channel and may range from −128 to +127 with all blue and no yellow being represented by −128 and all yellow and no blue being represented by +127.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of training the image rendering model based on the difference information between the sample video frames and the first rendered image of LINCHAO with converting rendered image to CIELAB color space of rendered image as seen in Chekh because this modification would use three numerical values, L*, a*, and b* to define the color for each pixel in a rendered image ([0364] of Chekh). Kaplanyan, Daxuan and Chekh are understood to be silent on the remaining limitations of claim 5.
In the same field of endeavor, KOUNOSU teaches performing color space conversion on the image, to obtain a converted image ([0033] The learning unit 13 inputs a label image converted to CIELAB by the color space conversion unit 12”); performing color space conversion on the photographed image, to obtain a converted photographed image ([0028] The input image storage unit 10 stores images to be learned. The images to be learned may be captured with a sensor (to be described later) that is part of the image processing device 1, or images captured by an external sensor may be input. Alternatively, images stored in a fog computer (to be described later) or a cloud server (to be described later) may be acquired.’; [0032] The learning unit 13 includes a neural network 14. The neural network 14 creates inferred images from the input images converted to CIELAB by the color space conversion unit 12. The color space of the created inferred image is CIELAB.”); and determining the difference information according to the converted image and the converted photographed image ([0033] The learning unit 13 inputs a label image converted to CIELAB by the color space conversion unit 12 and compares the converted label image with the inferred image created by the neural network 14. An error between the two images is obtained as a result of comparison between the label image and the inferred image. This error is a color difference. A color difference is an index defined as a distance in the color space to represent the difference between two colors. The metrics that define a color difference in the CIELAB color space include the Euclidean distance, CIE76, CIE94, and CIEDE2000.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of training the image rendering model based on the difference information between the sample video frames and the first rendered image of LINCHAO and converting rendered image to CIELAB color space of rendered image of Chekh with using color space conversion to convert two different source images to same format image as seen in KOUNOSU because this modification would compare the converted images and defined color difference between the two images ([0033])
Thus, the combination LINCHAO, Chekh and KOUNOSU teaches wherein the determining difference information between the rendered image and the photographed image comprises: performing color space conversion on the rendered image, to obtain a converted rendered image; performing color space conversion on the photographed image, to obtain a converted photographed image; and determining the difference information according to the converted rendered image and the converted photographed image.
Regarding claim 12, LINCHAO teaches the electronic device according to claim 8, Remaining limitations of claim 12 is similar scope to claim 5 and therefore rejected under the same rationale.
Regarding claim 19, LINCHAO teaches the non-transitory computer-readable storage medium according to claim 15, Remaining limitations of claim 19 is similar scope to claim 5 and therefore rejected under the same rationale.
3. Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over BAO LINCHAO; LU HUCHUAN; ZHANG YING; KANG DI; CHEN JIANCHUAN, IDS, CN113822977(English translated) (”LINCHAO”) in view of Chen et al., U.S Patent Application Publication No.20170184714 (“Chen”)
Regarding claim 7, LINCHAO and Kaplanyan teach the method according to claim 6, wherein the selecting the target material parameter from the at least two material parameters comprises: the at least two material parameters; and selecting the target material parameter from the at least two material parameters (see at least see at least [n0006] Based on sample video frames, obtain the shape parameters and pose parameters of the sample object, wherein the sample video frames include the sample object”; [n0012] In one possible implementation, the first rendering parameters include color parameters and density parameters, and determining the color and opacity of the pixel based on the virtual ray between the pixel and the virtual camera, and the first rendering parameters corresponding to the pixel, includes:[n0013] The color is obtained by integrating the first relational data on the virtual ray, wherein the first relational data is associated with the color parameter and the density parameter. [n0192]-[n0203] In one possible implementation, the server constructs a first loss function based on the color difference information between the sample video frames and the first rendered image, and uses the first loss function to train the image rendering model. [n0193] For example, the server constructs a first loss function based on the color difference information between the colors determined by the coarse network and the colors of the sample video frames, and the color difference information between the colors determined by the fine network and the colors of the sample video frames.[n0196] In one possible implementation, the server applies regularization constraints to the pose parameters of adjacent sample video frames to ensure that the pose parameters between adjacent sample video frames are as close as possible, and to ensure that the optimized pose parameters are not significantly different from the original pose parameters. In other words, in order to obtain stable and smooth pose parameters, a second loss function is added during the image rendering process. The optimized pose parameters and the initial parameters should not have too large aberrations, and the pose parameters between adjacent frames should be as similar as possible. Here, the optimized pose parameters refer to the average pose parameters of multiple sample video frames in the sample video…[n0203] In one possible implementation, the server can use at least two of the above three loss functions to train the image rendering model”) LINCHAO is understood to be silent on the remaining limitations of claim 7.
In the same field of endeavor, Chen teaches determining a correlation between the at least two material parameters; and selecting the target material parameter from the at least two material parameters according to the correlation (see at least [0043] In another embodiment, the data analysis processor 124 performs the hierarchical correlation method to obtain the reference sequence. The hierarchical correlation method comprises the following steps of: (a) gathering the color value sequences into an i.sup.th reference sequence set, wherein an initial value of i is 1; (b) obtaining a plurality of correlation coefficient values of the i.sup.th reference sequence set by calculating each pair of the color value sequences in the i.sup.th reference sequence set through a correlation function; (c) selecting the pair of the color value sequences in the i.sup.th reference sequence set, which have a largest correlation coefficient value; (d) averaging the selected pair of the color value sequences in the i.sup.th reference sequence set to generate a new color value sequence; (e) gathering the new color value sequence and the color value sequences in the i.sup.th reference sequence set except for the selected pair of the color value sequences into a (i+1).sup.th reference sequence set; (f) obtaining a plurality of correlation coefficient values of the (i+1).sup.th reference sequence set by calculating each pair of the color value sequences in the (i+1).sup.th reference sequence set through the correlation function; (g) determining whether all of the correlation coefficient values of the (i+1).sup.th reference sequence set are less than a predetermined correlation threshold; and (h) when one of the correlation coefficient values of the (i+1).sup.th reference sequence set is not less than the predetermined correlation threshold, setting i=i+1 and repeating steps (c), (d), (e), (f) and (g), and when the all of the correlation coefficient values of the (i+1).sup.th reference sequence set are less than the predetermined correlation threshold, outputting the color value sequence in the (i+1).sup.th reference sequence set which is generated by averaging a most number of the color value sequences in the first reference sequence set as a reference sequence.” Where color values are considered as material parameters)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of rendering image based on material parameters of LINCHAO with selecting the pair of the color value sequences in the reference sequence set, which have a largest correlation coefficient value as seen in Chen because this modification would achieve the expected benefit of identifying the strongest linear relationship between color channels.
Thus, the combination of LINCHAO and Chen teaches wherein the selecting the target material parameter from the at least two material parameters comprises: determining a correlation between the at least two material parameters; and selecting the target material parameter from the at least two material parameters according to the correlation.
Regarding claim 14, LINCHAO teaches the electronic device according to claim 13, Remaining limitations of claim 14 is similar scope to claim 7 and therefore rejected under the same rationale.
===========================================
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
1. Claims 1-4,6, 8-11,13, 15-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kaplanyan et al., IDS, U.S Patent Application Publication No.2021/0241519 (“Kaplanyan”) in view of REN DAXUAN; LI JIATONGl ;MEI HAIYI, IDS, CN115272548 -English translated (“Daxuan”)
Regarding independent claim 1, Kaplanyan teaches an image processing method performed by an electronic device (Fig.2 and Fig.5), the method comprising:
obtaining an initial value of a target material parameter of a target object (see at least [0020] FIG. 2 illustrates an example process 200 of estimating material properties of objects within a 3D scene and positions of light sources within the 3D scene. In particular embodiments, the process 200 may include a plurality of different phases 202, 204, 206. The phases 202, 204, 206 may include an input phase 202, a processing phase 204, and a generation phase 206. In particular embodiments, the virtual reality system may retrieve, access, or the like to obtain data during the input phase 202. As an example and not by way of limitation, inputs may comprise the captured imagery, scene geometry, object segmentation of the scene, and an arbitrary initial guess of the illumination and material parameters. Material and emission properties may then be estimated by optimizing for rendered imagery to match the captured images. In particular embodiments, the input phase 202 may include an input photos step 208 where the virtual reality system accesses one or more input images 210 and a geometry scan and object segmentation step 212 where the virtual reality system determines the geometry of the 3D environment 214 with respect to the input photos 210. In particular embodiments, the virtual reality system may access a 3D model or generate a 3D model 214 based on the input photos 210. In particular embodiments, the processing phase 204 may include a path tracing step 216 and a backpropagate step 226. In particular embodiments, during the path tracing step 216, the virtual reality system may perform path tracing from a point of view 218 to determine the position of the light sources 222 through the light paths 220. In particular embodiments, the light paths 220 may bounce off the objects 224 within the 3D scene 214 and back to the light sources 222. In particular embodiments, during the backpropagate step 226, the virtual reality system may use light paths 228 to update the material properties of the objects 224 and update the positions of the light sources 222. By updating the material parameters of the material properties of objects 224 and the light source parameters of the lights sources 222, the virtual reality system may accurately render and re-render a virtual representation of the 3D scene 214. In particular embodiments, the generation phase 206 may generate images indicative of the material properties 232a, 232b and light source positions 232c among other images 232. These images 232 may be used to determine the material parameters and the light source parameters. After the parameters are determined, the virtual reality system may render an image based on the parameters and compare the rendered image to the input images 210 to determine whether an update to the parameters is required as described herein. In particular embodiments, one or more of the phases 202, 204, 206 may be repeated to accurately estimate the material parameters and the light source parameters. As an example and not by way of limitation, the backpropagate step 226 may be repeated to update the material parameters of the objects 224 within the 3D scene 214 and used to generate further images 232. These images 232 may be used to re-render an image to be compared to the input images 210.”);
determining a photographing parameter of a photographed image of the target object (see at least [0020] FIG. 2 illustrates an example process 200 of estimating material properties of objects within a 3D scene and positions of light sources within the 3D scene. In particular embodiments, the process 200 may include a plurality of different phases 202, 204, 206. The phases 202, 204, 206 may include an input phase 202, a processing phase 204, and a generation phase 206. In particular embodiments, the virtual reality system may retrieve, access, or the like to obtain data during the input phase 202. As an example and not by way of limitation, inputs may comprise the captured imagery, scene geometry, object segmentation of the scene, and an arbitrary initial guess of the illumination and material parameters. Material and emission properties may then be estimated by optimizing for rendered imagery to match the captured images. In particular embodiments, the input phase 202 may include an input photos step 208 where the virtual reality system accesses one or more input images 210 and a geometry scan and object segmentation step 212 where the virtual reality system determines the geometry of the 3D environment 214 with respect to the input photos 210. In particular embodiments, the virtual reality system may access a 3D model or generate a 3D model 214 based on the input photos 210. In particular embodiments, the processing phase 204 may include a path tracing step 216 and a backpropagate step 226. In particular embodiments, during the path tracing step 216, the virtual reality system may perform path tracing from a point of view 218 to determine the position of the light sources 222 through the light paths 220. In particular embodiments, the light paths 220 may bounce off the objects 224 within the 3D scene 214 and back to the light sources 222. In particular embodiments, during the backpropagate step 226, the virtual reality system may use light paths 228 to update the material properties of the objects 224 and update the positions of the light sources 222. By updating the material parameters of the material properties of objects 224 and the light source parameters of the lights sources 222, the virtual reality system may accurately render and re-render a virtual representation of the 3D scene 214. In particular embodiments, the generation phase 206 may generate images indicative of the material properties 232a, 232b and light source positions 232c among other images 232. These images 232 may be used to determine the material parameters and the light source parameters. After the parameters are determined, the virtual reality system may render an image based on the parameters and compare the rendered image to the input images 210 to determine whether an update to the parameters is required as described herein. In particular embodiments, one or more of the phases 202, 204, 206 may be repeated to accurately estimate the material parameters and the light source parameters. As an example and not by way of limitation, the backpropagate step 226 may be repeated to update the material parameters of the objects 224 within the 3D scene 214 and used to generate further images 232. These images 232 may be used to re-render an image to be compared to the input images 210.”);
rendering the target object according to the initial value and the photographing parameter, to obtain a rendered image of the target object (see at least [0020] FIG. 2 illustrates an example process 200 of estimating material properties of objects within a 3D scene and positions of light sources within the 3D scene. In particular embodiments, the process 200 may include a plurality of different phases 202, 204, 206. The phases 202, 204, 206 may include an input phase 202, a processing phase 204, and a generation phase 206. In particular embodiments, the virtual reality system may retrieve, access, or the like to obtain data during the input phase 202. As an example and not by way of limitation, inputs may comprise the captured imagery, scene geometry, object segmentation of the scene, and an arbitrary initial guess of the illumination and material parameters. Material and emission properties may then be estimated by optimizing for rendered imagery to match the captured images. In particular embodiments, the input phase 202 may include an input photos step 208 where the virtual reality system accesses one or more input images 210 and a geometry scan and object segmentation step 212 where the virtual reality system determines the geometry of the 3D environment 214 with respect to the input photos 210. In particular embodiments, the virtual reality system may access a 3D model or generate a 3D model 214 based on the input photos 210. In particular embodiments, the processing phase 204 may include a path tracing step 216 and a backpropagate step 226. In particular embodiments, during the path tracing step 216, the virtual reality system may perform path tracing from a point of view 218 to determine the position of the light sources 222 through the light paths 220. In particular embodiments, the light paths 220 may bounce off the objects 224 within the 3D scene 214 and back to the light sources 222. In particular embodiments, during the backpropagate step 226, the virtual reality system may use light paths 228 to update the material properties of the objects 224 and update the positions of the light sources 222. By updating the material parameters of the material properties of objects 224 and the light source parameters of the lights sources 222, the virtual reality system may accurately render and re-render a virtual representation of the 3D scene 214. In particular embodiments, the generation phase 206 may generate images indicative of the material properties 232a, 232b and light source positions 232c among other images 232. These images 232 may be used to determine the material parameters and the light source parameters. After the parameters are determined, the virtual reality system may render an image based on the parameters and compare the rendered image to the input images 210 to determine whether an update to the parameters is required as described herein. In particular embodiments, one or more of the phases 202, 204, 206 may be repeated to accurately estimate the material parameters and the light source parameters. As an example and not by way of limitation, the backpropagate step 226 may be repeated to update the material parameters of the objects 224 within the 3D scene 214 and used to generate further images 232. These images 232 may be used to re-render an image to be compared to the input images 210.”);
determining difference information between the rendered image and the photographed image (see at least [0020] “…. In particular embodiments, the virtual reality system may access a 3D model or generate a 3D model 214 based on the input photos 210. In particular embodiments, the processing phase 204 may include a path tracing step 216 and a backpropagate step 226. In particular embodiments, during the path tracing step 216, the virtual reality system may perform path tracing from a point of view 218 to determine the position of the light sources 222 through the light paths 220. In particular embodiments, the light paths 220 may bounce off the objects 224 within the 3D scene 214 and back to the light sources 222. In particular embodiments, during the backpropagate step 226, the virtual reality system may use light paths 228 to update the material properties of the objects 224 and update the positions of the light sources 222. By updating the material parameters of the material properties of objects 224 and the light source parameters of the lights sources 222, the virtual reality system may accurately render and re-render a virtual representation of the 3D scene 214. In particular embodiments, the generation phase 206 may generate images indicative of the material properties 232a, 232b and light source positions 232c among other images 232. These images 232 may be used to determine the material parameters and the light source parameters. After the parameters are determined, the virtual reality system may render an image based on the parameters and compare the rendered image to the input images 210 to determine whether an update to the parameters is required as described herein. In particular embodiments, one or more of the phases 202, 204, 206 may be repeated to accurately estimate the material parameters and the light source parameters. As an example and not by way of limitation, the backpropagate step 226 may be repeated to update the material parameters of the objects 224 within the 3D scene 214 and used to generate further images 232. These images 232 may be used to re-render an image to be compared to the input images 210.”);
the initial value according to the difference information, to obtain a target value of the target material parameter (see at least [0020] “…. In particular embodiments, the virtual reality system may access a 3D model or generate a 3D model 214 based on the input photos 210. In particular embodiments, the processing phase 204 may include a path tracing step 216 and a backpropagate step 226. In particular embodiments, during the path tracing step 216, the virtual reality system may perform path tracing from a point of view 218 to determine the position of the light sources 222 through the light paths 220. In particular embodiments, the light paths 220 may bounce off the objects 224 within the 3D scene 214 and back to the light sources 222. In particular embodiments, during the backpropagate step 226, the virtual reality system may use light paths 228 to update the material properties of the objects 224 and update the positions of the light sources 222. By updating the material parameters of the material properties of objects 224 and the light source parameters of the lights sources 222, the virtual reality system may accurately render and re-render a virtual representation of the 3D scene 214. In particular embodiments, the generation phase 206 may generate images indicative of the material properties 232a, 232b and light source positions 232c among other images 232. These images 232 may be used to determine the material parameters and the light source parameters. After the parameters are determined, the virtual reality system may render an image based on the parameters and compare the rendered image to the input images 210 to determine whether an update to the parameters is required as described herein. In particular embodiments, one or more of the phases 202, 204, 206 may be repeated to accurately estimate the material parameters and the light source parameters. As an example and not by way of limitation, the backpropagate step 226 may be repeated to update the material parameters of the objects 224 within the 3D scene 214 and used to generate further images 232. These images 232 may be used to re-render an image to be compared to the input images 210.”); and
updating the rendered image of the target object by rendering the target object according to the target value (see at least [0020] “….. In particular embodiments, the virtual reality system may access a 3D model or generate a 3D model 214 based on the input photos 210. In particular embodiments, the processing phase 204 may include a path tracing step 216 and a backpropagate step 226. In particular embodiments, during the path tracing step 216, the virtual reality system may perform path tracing from a point of view 218 to determine the position of the light sources 222 through the light paths 220. In particular embodiments, the light paths 220 may bounce off the objects 224 within the 3D scene 214 and back to the light sources 222. In particular embodiments, during the backpropagate step 226, the virtual reality system may use light paths 228 to update the material properties of the objects 224 and update the positions of the light sources 222. By updating the material parameters of the material properties of objects 224 and the light source parameters of the lights sources 222, the virtual reality system may accurately render and re-render a virtual representation of the 3D scene 214. In particular embodiments, the generation phase 206 may generate images indicative of the material properties 232a, 232b and light source positions 232c among other images 232. These images 232 may be used to determine the material parameters and the light source parameters. After the parameters are determined, the virtual reality system may render an image based on the parameters and compare the rendered image to the input images 210 to determine whether an update to the parameters is required as described herein. In particular embodiments, one or more of the phases 202, 204, 206 may be repeated to accurately estimate the material parameters and the light source parameters. As an example and not by way of limitation, the backpropagate step 226 may be repeated to update the material parameters of the objects 224 within the 3D scene 214 and used to generate further images 232. These images 232 may be used to re-render an image to be compared to the input images 210.”)
In the same field of endeavor, Daxuan teaches obtaining an initial value of a target material parameter of a target object ([n0006] The initial rendering parameters are determined for the renderer, and the renderer is used to render the 3D model based on the rendering parameters to obtain a rendered image”; [n0102] Specifically, rendering parameters are the parameters called by the renderer during the rendering process of the target 3D model. Rendering parameters can include lighting parameters, virtual camera lens parameters, point light source parameters, color temperature parameters, etc. For example, lighting parameters may include brightness, saturation, etc., and lens parameters of the virtual camera may include the focal length parameter of the virtual camera, the position parameter of the virtual camera, the size parameter of the captured image, etc. A point light source is a light source that emits light uniformly from a single point into the surrounding space. Color temperature is a parameter used to represent the color components contained in light. The lower the color temperature, the warmer the hue (more reddish); the higher the color temperature, the cooler the hue (more bluish).”);
determining difference information between the rendered image and the photographed image ([n0007] The difference between the rendered image and the real image is determined; wherein the real image is an image of the target environment captured by a real camera; and the environment reflected in the rendered image is the same as the target environment [n0110] S102, determine the difference between the rendered image and the real image; wherein, the real image is an image of the target environment captured by a real camera; the environment reflected by the rendered image is the same as the target environment.”);
adjusting the initial value according to the difference information, to obtain a target value of the target material parameter ([n0102] Specifically, rendering parameters are the parameters called by the renderer during the rendering process of the target 3D model. Rendering parameters can include lighting parameters, virtual camera lens parameters, point light source parameters, color temperature parameters, etc. For example, lighting parameters may include brightness, saturation, etc., and lens parameters of the virtual camera may include the focal length parameter of the virtual camera, the position parameter of the virtual camera, the size parameter of the captured image, etc. A point light source is a light source that emits light uniformly from a single point into the surrounding space. Color temperature is a parameter used to represent the color components contained in light. The lower the color temperature, the warmer the hue (more reddish); the higher the color temperature, the cooler the hue (more bluish). The virtual camera's position parameters are used to describe the height of the virtual camera.”; [n0115] S103, Based on the difference results, adjust the rendering parameters. [n0116] After determining the difference between the rendered image and the real image, the rendering parameters can be adjusted based on the difference. [n0117] Specifically, based on the difference results, the rendering parameters are adjusted using a Bayesian optimization algorithm. Among them, the Bayesian optimization algorithm is a method that uses Bayes' theorem to guide the search to find the minimum or maximum value of the objective function. That is, in each process of adjusting the rendering parameters, it is necessary to determine the current rendering parameters based on the confidence level of the previous rendering parameter distribution until the optimal rendering parameters are found. In this way, the efficiency of adjusting the rendering parameters can be improved.”); and
updating the rendered image of the target object by rendering the target object according to the target value ([n0117] Specifically, based on the difference results, the rendering parameters are adjusted using a Bayesian optimization algorithm. Among them, the Bayesian optimization algorithm is a method that uses Bayes' theorem to guide the search to find the minimum or maximum value of the objective function. That is, in each process of adjusting the rendering parameters, it is necessary to determine the current rendering parameters based on the confidence level of the previous rendering parameter distribution until the optimal rendering parameters are found. In this way, the efficiency of adjusting the rendering parameters can be improved.”; [n0118] S104. Repeat the above steps until the adjustment process meets the preset conditions and the adjusted target rendering parameters are obtained [n0119] Specifically, after adjusting the rendering parameters, the initial rendering parameters (adjusted rendering parameters) are redefined for the renderer, and the renderer is used to render the target 3D model based on the adjusted rendering parameters to obtain the adjusted rendered image. Then, the difference between the adjusted rendered image and the real image is redefined, and it is determined whether the rendering parameters need to be adjusted again based on the difference. If the above adjustment process meets the preset conditions, the adjusted target rendering parameters can be obtained; if the above adjustment process does not meet the preset conditions, the rendering parameters need to be adjusted again until the adjustment process meets the preset conditions.”; [n0157] S206, using the renderer, based on the adjusted target rendering parameters, the target 3D model is rendered to obtain a rendered target sample image.[n0158] After obtaining the adjusted target rendering parameters, the renderer can render the target 3D model according to the adjusted target rendering parameters to obtain the rendered target sample image. It can be understood that the adjusted target rendering parameters are the optimal rendering parameters. Therefore, the target sample image obtained by rendering with the adjusted target rendering parameters is the image that is closest to the real image.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of rendering image based on material parameters of Kaplanyan with adjusting rendering parameters based on the difference between rendered image and real image as seen in Daxuan because this modification would obtain the target sample image by rendering with the adjusted target rendering parameters is the image that is closest to the real image ([n0158] of Daxuan).
Thus, the combination of Kaplanyan and Daxuan teaches an image processing method performed by an electronic device, the method comprising: obtaining an initial value of a target material parameter of a target object; determining a photographing parameter of a photographed image of the target object; rendering the target object according to the initial value and the photographing parameter, to obtain a rendered image of the target object; determining difference information between the rendered image and the photographed image; adjusting the initial value according to the difference information, to obtain a target value of the target material parameter; and updating the rendered image of the target object by rendering the target object according to the target value.
Regarding claim 2, Kaplanyan and Daxuan teach the method according to claim 1, wherein the determining difference information between the rendered image and the photographed image comprises: determining a rendering value of the target material parameter corresponding to the rendered image; determining a real value of the target material parameter corresponding to the photographed image; and determining the difference information according to the rendering value and the real value (see at least [0020] of Kaplanyan “.. In particular embodiments, the virtual reality system may access a 3D model or generate a 3D model 214 based on the input photos 210. In particular embodiments, the processing phase 204 may include a path tracing step 216 and a backpropagate step 226. In particular embodiments, during the path tracing step 216, the virtual reality system may perform path tracing from a point of view 218 to determine the position of the light sources 222 through the light paths 220. In particular embodiments, the light paths 220 may bounce off the objects 224 within the 3D scene 214 and back to the light sources 222. In particular embodiments, during the backpropagate step 226, the virtual reality system may use light paths 228 to update the material properties of the objects 224 and update the positions of the light sources 222. By updating the material parameters of the material properties of objects 224 and the light source parameters of the lights sources 222, the virtual reality system may accurately render and re-render a virtual representation of the 3D scene 214. In particular embodiments, the generation phase 206 may generate images indicative of the material properties 232a, 232b and light source positions 232c among other images 232. These images 232 may be used to determine the material parameters and the light source parameters. After the parameters are determined, the virtual reality system may render an image based on the parameters and compare the rendered image to the input images 210 to determine whether an update to the parameters is required as described herein. In particular embodiments, one or more of the phases 202, 204, 206 may be repeated to accurately estimate the material parameters and the light source parameters. As an example and not by way of limitation, the backpropagate step 226 may be repeated to update the material parameters of the objects 224 within the 3D scene 214 and used to generate further images 232. These images 232 may be used to re-render an image to be compared to the input images 210.” Where compare the rendered image to the input images to determine whether an update to the parameters which is considered as determining the rendering value and the real value; [n0117] of Daxuan “Specifically, based on the difference results, the rendering parameters are adjusted using a Bayesian optimization algorithm. Among them, the Bayesian optimization algorithm is a method that uses Bayes' theorem to guide the search to find the minimum or maximum value of the objective function. That is, in each process of adjusting the rendering parameters, it is necessary to determine the current rendering parameters based on the confidence level of the previous rendering parameter distribution until the optimal rendering parameters are found. In this way, the efficiency of adjusting the rendering parameters can be improved.”; [n0119] Specifically, after adjusting the rendering parameters, the initial rendering parameters (adjusted rendering parameters) are redefined for the renderer, and the renderer is used to render the target 3D model based on the adjusted rendering parameters to obtain the adjusted rendered image. Then, the difference between the adjusted rendered image and the real image is redefined, and it is determined whether the rendering parameters need to be adjusted again based on the difference. If the above adjustment process meets the preset conditions, the adjusted target rendering parameters can be obtained; if the above adjustment process does not meet the preset conditions, the rendering parameters need to be adjusted again until the adjustment process meets the preset conditions.”; [n0157] S206, using the renderer, based on the adjusted target rendering parameters, the target 3D model is rendered to obtain a rendered target sample image.[n0158] After obtaining the adjusted target rendering parameters, the renderer can render the target 3D model according to the adjusted target rendering parameters to obtain the rendered target sample image. It can be understood that the adjusted target rendering parameters are the optimal rendering parameters. Therefore, the target sample image obtained by rendering with the adjusted target rendering parameters is the image that is closest to the real image.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 3, Kaplanyan and Daxuan teach the method according to claim 1, wherein the adjusting the initial value according to the difference information, to obtain a target value of the target material parameter comprises:
adjusting the initial value according to the difference information, to obtain a candidate value of the target material parameter; rendering the target object according to the candidate value and the photographing parameter; determining the difference information between the rendered image and the photographed image; repeating the adjusting and rendering operations until the difference information satisfies a preset condition; and using the candidate value as the target value (see at least [0020] of Kaplanyan “.. In particular embodiments, the virtual reality system may access a 3D model or generate a 3D model 214 based on the input photos 210. In particular embodiments, the processing phase 204 may include a path tracing step 216 and a backpropagate step 226. In particular embodiments, during the path tracing step 216, the virtual reality system may perform path tracing from a point of view 218 to determine the position of the light sources 222 through the light paths 220. In particular embodiments, the light paths 220 may bounce off the objects 224 within the 3D scene 214 and back to the light sources 222. In particular embodiments, during the backpropagate step 226, the virtual reality system may use light paths 228 to update the material properties of the objects 224 and update the positions of the light sources 222. By updating the material parameters of the material properties of objects 224 and the light source parameters of the lights sources 222, the virtual reality system may accurately render and re-render a virtual representation of the 3D scene 214. In particular embodiments, the generation phase 206 may generate images indicative of the material properties 232a, 232b and light source positions 232c among other images 232. These images 232 may be used to determine the material parameters and the light source parameters. After the parameters are determined, the virtual reality system may render an image based on the parameters and compare the rendered image to the input images 210 to determine whether an update to the parameters is required as described herein. In particular embodiments, one or more of the phases 202, 204, 206 may be repeated to accurately estimate the material parameters and the light source parameters. As an example and not by way of limitation, the backpropagate step 226 may be repeated to update the material parameters of the objects 224 within the 3D scene 214 and used to generate further images 232. These images 232 may be used to re-render an image to be compared to the input images 210.” [n0117] of Daxuan “Specifically, based on the difference results, the rendering parameters are adjusted using a Bayesian optimization algorithm. Among them, the Bayesian optimization algorithm is a method that uses Bayes' theorem to guide the search to find the minimum or maximum value of the objective function. That is, in each process of adjusting the rendering parameters, it is necessary to determine the current rendering parameters based on the confidence level of the previous rendering parameter distribution until the optimal rendering parameters are found. In this way, the efficiency of adjusting the rendering parameters can be improved.”; [n0119] Specifically, after adjusting the rendering parameters, the initial rendering parameters (adjusted rendering parameters) are redefined for the renderer, and the renderer is used to render the target 3D model based on the adjusted rendering parameters to obtain the adjusted rendered image. Then, the difference between the adjusted rendered image and the real image is redefined, and it is determined whether the rendering parameters need to be adjusted again based on the difference. If the above adjustment process meets the preset conditions, the adjusted target rendering parameters can be obtained; if the above adjustment process does not meet the preset conditions, the rendering parameters need to be adjusted again until the adjustment process meets the preset conditions.”; [n0120]
In one possible implementation, the adjustment process satisfies at least one of the following preset conditions: the rate of change of the difference result meets a preset requirement, the number of times the rendering parameters are adjusted reaches a preset number, and the adjustment time of the rendering parameters reaches a preset time.
Specifically, if the rate of change of the difference between the sample image and the target image is less than a preset threshold (for example, if the difference results obtained after multiple adjustments to the rendering parameters do not show significant changes), the difference result is determined to meet the preset requirements. [n0121] In this embodiment of the disclosure, the rendered image obtained by the renderer is compared with the real image to obtain the difference result. Then, the rendering parameters of the renderer are automatically adjusted based on the difference result. This process is repeated until the adjustment process meets the preset conditions and the adjusted target rendering parameters are obtained. This avoids the process of manually adjusting the parameters, improves the efficiency of parameter adjustment, and reduces labor costs. Furthermore, by automatically iterating and adjusting the rendering parameters, the accuracy of the rendering parameter adjustment can be improved, making the rendered image closer to the real image, which in turn can improve the accuracy of subsequent model training. [n0157] S206, using the renderer, based on the adjusted target rendering parameters, the target 3D model is rendered to obtain a rendered target sample image.[n0158] After obtaining the adjusted target rendering parameters, the renderer can render the target 3D model according to the adjusted target rendering parameters to obtain the rendered target sample image. It can be understood that the adjusted target rendering parameters are the optimal rendering parameters. Therefore, the target sample image obtained by rendering with the adjusted target rendering parameters is the image that is closest to the real image.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 4, Kaplanyan and Daxuan teach the method according to claim 1, wherein the adjusting the initial value according to the difference information, to obtain a target value of the target material parameter comprises:
determining an amount of change of the target material parameter according to the difference information; obtaining an optimization step corresponding to the target material parameter; and adjusting the initial value according to the optimization step and the amount of change, to obtain the target value (see at least [0030] of Kaplanyan “In particular embodiments, the material model may satisfy several properties. In particular embodiments, the material model may cover as much variability in appearance as possible, including such common effects as specular highlights, multilayered materials, and spatially varying textures. In particular embodiments, since each parameter adds another unknown to the optimization, the number of parameters may be kept to a minimal. Since a goal may be directed to re-rendering and related tasks, the material model may have interpretable parameters, so the users can adjust the parameters to achieve the desired appearance. In particular embodiments, the material properties may be optimized using first-order gradient-based optimization, and the range of the material parameters may be similar. [0034] In particular embodiments, the observed color of an object in a scene may be most easily explained by assigning emission to the triangle. This may be avoided by differences in shading of the different parts of the object. However, it can happen that there are no observable differences in the shading of an object, especially if the object covers only a few pixels in the input image. This may be a source of error during optimization. Another source of error may be Monte Carlo and SGD noise. These errors may lead to incorrect emission parameters for many objects after the optimization. The objects usually have a small estimated emission value when they should have none. In particular embodiments, to address the issue of a small estimated emission value an L1-regularizer may be used for the emission. The vast majority of objects in the scene is not an emitter and having such a regularizer may suppress the small errors for the emission parameters after optimization. [0035] In particular embodiments, ADAM may be used as an optimizer with batch size B=8 estimated pixels and learning rate 5.Math.10.sup.−3. To form a batch, B pixels may be sampled uniformly from the set of all pixels of all images. In particular embodiments, a higher batch size may be used to reduce the variance of each iteration. In particular embodiments, a smaller batch size may be used to have faster iterations.”; [n0110] of Daxuan “ S102, determine the difference between the rendered image and the real image; wherein, the real image is an image of the target environment captured by a real camera; the environment reflected by the rendered image is the same as the target environment.[n0115] “S103, Based on the difference results, adjust the rendering parameters.[n0116] After determining the difference between the rendered image and the real image, the rendering parameters can be adjusted based on the difference. [n0117] Specifically, based on the difference results, the rendering parameters are adjusted using a Bayesian optimization algorithm. Among them, the Bayesian optimization algorithm is a method that uses Bayes' theorem to guide the search to find the minimum or maximum value of the objective function. That is, in each process of adjusting the rendering parameters, it is necessary to determine the current rendering parameters based on the confidence level of the previous rendering parameter distribution until the optimal rendering parameters are found. In this way, the efficiency of adjusting the rendering parameters can be improved.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding claim 6, Kaplanyan and Daxuan teach the method according to claim 1, wherein the obtaining an initial value of a target material parameter of a target object comprises:
obtaining at least two material parameters of the target object; and selecting the target material parameter from the at least two material parameters (see at least [0030] of Kaplanyan “In particular embodiments, the material model may satisfy several properties. In particular embodiments, the material model may cover as much variability in appearance as possible, including such common effects as specular highlights, multilayered materials, and spatially varying textures. In particular embodiments, since each parameter adds another unknown to the optimization, the number of parameters may be kept to a minimal. Since a goal may be directed to re-rendering and related tasks, the material model may have interpretable parameters, so the users can adjust the parameters to achieve the desired appearance. In particular embodiments, the material properties may be optimized using first-order gradient-based optimization, and the range of the material parameters may be similar.; [0031] As an example and not by way of limitation, the materials may be represented using the Disney material model, the state-of-the-art physically based material model used in movie and game rendering. In particular embodiments, the material model may have a “base color” parameter which is used by both diffuse and specular reflectance, as well as 10 other parameters describing the roughness, anisotropy, and specularity of the material. In particular embodiments, all these parameters may be perceptually mapped to [0, 1], which may be both interpretable and suitable for optimization; [0032] In particular embodiments, triangle meshes may be used to represent the scene geometry. Surface normals may be defined per-vertex and interpolated within each triangle using barycentric coordinates. The optimization may be performed on a per-object basis, e.g., every object has a single unknown emission and a set of material parameters that are assumed constant across the whole object. This may be enough to obtain accurate lighting and an average constant value for the albedo of an object”; [n0102] of Daxuan “Specifically, rendering parameters are the parameters called by the renderer during the rendering process of the target 3D model. Rendering parameters can include lighting parameters, virtual camera lens parameters, point light source parameters, color temperature parameters, etc. For example, lighting parameters may include brightness, saturation, etc., and lens parameters of the virtual camera may include the focal length parameter of the virtual camera, the position parameter of the virtual camera, the size parameter of the captured image, etc. A point light source is a light source that emits light uniformly from a single point into the surrounding space. Color temperature is a parameter used to represent the color components contained in light. The lower the color temperature, the warmer the hue (more reddish); the higher the color temperature, the cooler the hue (more bluish). The virtual camera's position parameters are used to describe the height of the virtual camera.”).; and
rendering the target object according to the initial value, a set value corresponding to the candidate material parameter, and the photographing parameter, to obtain the rendered image of the target object, wherein the candidate material parameter is a parameter in the at least two material parameters except the target material parameter (see at least [0030] of Kaplanyan “In particular embodiments, the material model may satisfy several properties. In particular embodiments, the material model may cover as much variability in appearance as possible, including such common effects as specular highlights, multilayered materials, and spatially varying textures. In particular embodiments, since each parameter adds another unknown to the optimization, the number of parameters may be kept to a minimal. Since a goal may be directed to re-rendering and related tasks, the material model may have interpretable parameters, so the users can adjust the parameters to achieve the desired appearance. In particular embodiments, the material properties may be optimized using first-order gradient-based optimization, and the range of the material parameters may be similar. [0031] As an example and not by way of limitation, the materials may be represented using the Disney material model, the state-of-the-art physically based material model used in movie and game rendering. In particular embodiments, the material model may have a “base color” parameter which is used by both diffuse and specular reflectance, as well as 10 other parameters describing the roughness, anisotropy, and specularity of the material. In particular embodiments, all these parameters may be perceptually mapped to [0, 1], which may be both interpretable and suitable for optimization. ([0038] In particular embodiments, most diffuse global illumination effects may be approximated by as few as two bounces of light. As such, an image may be rendered with 10 bounces and it may be used as ground truth for the optimization. In particular embodiments, approximations of the ground truth may be done by renderings with one, two, and three bounces, respectively. One bounce may correspond to direct illumination; adding more bounces may allow the system to take into account indirect illumination as well. Optimization with only a single bounce may be the fastest, but the error remains high even after convergence. Having more than two bounces may lead to high variance and takes a lot of time to converge. As such, in particular embodiments, two bounces may be used to obtain a balance between convergence speed and accuracy. [0039] In particular embodiments, both real and synthetic scenes with textured objects may be considered in order to evaluate surfaces with high-frequency surface signal. As such, the light sources and material parameters on the coarse per-object resolution following the non-textured resolution may be optimized first. Once converged, the light sources may be kept fixed, and all other regions may be subdivided based on the surface texture where the re-rendering error is high; i.e., every triangle may be subdivided based on its average l.sub.2 error and continue until convergence. This coarse-to-fine strategy may allow the system to first separate out material and lighting in the more well-conditioned setting; in the second step, high-resolution material information may be obtained.”; [n0006] of Daxuan “Determine the initial rendering parameters for the renderer, and use the renderer to render the 3D model based on the rendering parameters to obtain a rendered image”; [n0117] of Daxuan “Specifically, based on the difference results, the rendering parameters are adjusted using a Bayesian optimization algorithm. Among them, the Bayesian optimization algorithm is a method that uses Bayes' theorem to guide the search to find the minimum or maximum value of the objective function. That is, in each process of adjusting the rendering parameters, it is necessary to determine the current rendering parameters based on the confidence level of the previous rendering parameter distribution until the optimal rendering parameters are found. In this way, the efficiency of adjusting the rendering parameters can be improved.”; [n0119] Specifically, after adjusting the rendering parameters, the initial rendering parameters (adjusted rendering parameters) are redefined for the renderer, and the renderer is used to render the target 3D model based on the adjusted rendering parameters to obtain the adjusted rendered image. Then, the difference between the adjusted rendered image and the real image is redefined, and it is determined whether the rendering parameters need to be adjusted again based on the difference. If the above adjustment process meets the preset conditions, the adjusted target rendering parameters can be obtained; if the above adjustment process does not meet the preset conditions, the rendering parameters need to be adjusted again until the adjustment process meets the preset conditions.”; [n0157] S206, using the renderer, based on the adjusted target rendering parameters, the target 3D model is rendered to obtain a rendered target sample image.[n0158] After obtaining the adjusted target rendering parameters, the renderer can render the target 3D model according to the adjusted target rendering parameters to obtain the rendered target sample image. It can be understood that the adjusted target rendering parameters are the optimal rendering parameters. Therefore, the target sample image obtained by rendering with the adjusted target rendering parameters is the image that is closest to the real image.”) In addition, the same motivation is used as the rejection for claim 1.
Regarding independent claim 8, Kaplanyan teaches an electronic device (Fig.6), comprising a processor and a memory ([0061] In particular embodiments, computer system 600 includes a processor 602, memory 604…”), the memory having a computer program stored therein, and the computer program, when executed by the processor ( see at least [0062] In particular embodiments, processor 602 includes hardware for executing instructions, such as those making up a computer program. As an example and not by way of limitation, to execute instructions, processor 602 may retrieve (or fetch) the instructions from an internal register, an internal cache, memory 604, or storage 606; decode and execute them; and then write one or more results to an internal register, an internal cache, memory 604, or storage 606”)., causing the electronic device to implement an image processing method including: Remaining limitations of claim 8 is similar scope to claim 1 and therefore rejected under the same rationale.
Regarding claim 9, Kaplanyan and Daxuan teach the electronic device according to claim 8, Remaining limitations of claim 9 is similar scope to claim 2 and therefore rejected under the same rationale.
Regarding claim 10, Kaplanyan and Daxuan teach the electronic device according to claim 8, Remaining limitations of claim 10 is similar scope to claim 3 and therefore rejected under the same rationale.
Regarding claim 11, Kaplanyan and Daxuan teach the electronic device according to claim 8, Remaining limitations of claim 11 is similar scope to claim 4 and therefore rejected under the same rationale.
Regarding claim 13, Kaplanyan and Daxuan teach the electronic device according to claim 8, Remaining limitations of claim 13 is similar scope to claim 6 and therefore rejected under the same rationale.
Regarding independent claim 15, Kaplanyan teaches a non-transitory computer-readable storage medium having a computer program stored therein, and the computer program, when executed by a processor of an electronic device (see at least [0063] In particular embodiments, memory 604 includes main memory for storing instructions for processor 602 to execute or data for processor 602 to operate on. As an example and not by way of limitation, computer system 600 may load instructions from storage 606 or another source (such as, for example, another computer system 600) to memory 604. Processor 602 may then load the instructions from memory 604 to an internal register or internal cache. To execute the instructions, processor 602 may retrieve the instructions from the internal register or internal cache and decode them. During or after execution of the instructions, processor 602 may write one or more results (which may be intermediate or final results) to the internal register or internal cache.”), causing the electronic device to implement an image processing method including: Remaining limitations of claim 15 is similar scope to claim 1 and therefore rejected under the same rationale.
Regarding claim 16, Kaplanyan and Daxuan teach the non-transitory computer-readable storage medium according to claim 15, Remaining limitations of claim 16 is similar scope to claim 2 and therefore rejected under the same rationale.
Regarding claim 17, Kaplanyan and Daxuan teach the non-transitory computer-readable storage medium according to claim 15, Remaining limitations of claim 17 is similar scope to claim 3 and therefore rejected under the same rationale.
Regarding claim 18, Kaplanyan and Daxuan teach the non-transitory computer-readable storage medium according to claim 15, Remaining limitations of claim 18 is similar scope to claim 4 and therefore rejected under the same rationale.
Regarding claim 20, Kaplanyan and Daxuan teach the non-transitory computer-readable storage medium according to claim 15, Remaining limitations of claim 20 is similar scope to claim 6 and therefore rejected under the same rationale.
2. Claims 5, 12 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Kaplanyan et al., IDS, U.S Patent Application Publication No.2021/0241519 (“Kaplanyan”) in view of REN DAXUAN; LI JIATONGl ;MEI HAIYI, IDS, CN115272548 -English translated (“Daxuan”) further in view of Chekh et al., U.S Patent Application Publication No.20190350680 (“Chekh”) further in view of KOUNOSU, U.S Patent Application Publication No.2023/0392985 (“KOUNOSU”)
Regarding claim 5, Kaplanyan and Daxuan teach the method according to claim 1, wherein the determining difference information between the rendered image and the photographed image comprises:
determining the difference information according to the rendered image and the photographed image (see at least [0020] of Kaplanyan “....In particular embodiments, the virtual reality system may access a 3D model or generate a 3D model 214 based on the input photos 210. In particular embodiments, the processing phase 204 may include a path tracing step 216 and a backpropagate step 226. In particular embodiments, during the path tracing step 216, the virtual reality system may perform path tracing from a point of view 218 to determine the position of the light sources 222 through the light paths 220. In particular embodiments, the light paths 220 may bounce off the objects 224 within the 3D scene 214 and back to the light sources 222. In particular embodiments, during the backpropagate step 226, the virtual reality system may use light paths 228 to update the material properties of the objects 224 and update the positions of the light sources 222. By updating the material parameters of the material properties of objects 224 and the light source parameters of the lights sources 222, the virtual reality system may accurately render and re-render a virtual representation of the 3D scene 214. In particular embodiments, the generation phase 206 may generate images indicative of the material properties 232a, 232b and light source positions 232c among other images 232. These images 232 may be used to determine the material parameters and the light source parameters. After the parameters are determined, the virtual reality system may render an image based on the parameters and compare the rendered image to the input images 210 to determine whether an update to the parameters is required as described herein. In particular embodiments, one or more of the phases 202, 204, 206 may be repeated to accurately estimate the material parameters and the light source parameters. As an example and not by way of limitation, the backpropagate step 226 may be repeated to update the material parameters of the objects 224 within the 3D scene 214 and used to generate further images 232. These images 232 may be used to re-render an image to be compared to the input images 210.”; [n0110] of Daxuan “ S102, determine the difference between the rendered image and the real image; wherein, the real image is an image of the target environment captured by a real camera; the environment reflected by the rendered image is the same as the target environment”) In addition, the same motivation is used as the rejection for claim 1. Both Kaplanyan and Daxuan are understood to be silent on the remaining limitations of claim 5.
In the same field of endeavor, Chekh teaches performing color space conversion on the rendered image, to obtain a converted rendered image ([0364] The method may start at block 4704 instead of block 4702. At block 4704, the color space of the rendered image of the teeth is converted to the L*a*b* color space. The L*a*b* color space, also referred to as the CIELAB color space, uses three numerical values, L*, a*, and b* to define the color for each pixel in a rendered image. The L* is the luminance or lightness channel, and may range from 0-100 with 0 representing the darkest luminance and 100 representing the white point, or brightest value, for example. a* is the green-red color channel and may range from −128 to +127 with all green and no red being represented by −128 and all red and no green being represented by +127. Similarly, b* is the blue-yellow color channel and may range from −128 to +127 with all blue and no yellow being represented by −128 and all yellow and no blue being represented by +127.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of rendering image based on material parameters of Kaplanyan and Daxuan with converting rendered image to CIELAB color space of rendered image as seen in Chekh because this modification would use three numerical values, L*, a*, and b* to define the color for each pixel in a rendered image ([0364] of Chekh). Kaplanyan, Daxuan and Chekh are understood to be silent on the remaining limitations of claim 5.
In the same field of endeavor, KOUNOSU teaches performing color space conversion on the image, to obtain a converted image ([0033] The learning unit 13 inputs a label image converted to CIELAB by the color space conversion unit 12”); performing color space conversion on the photographed image, to obtain a converted photographed image ([0028] The input image storage unit 10 stores images to be learned. The images to be learned may be captured with a sensor (to be described later) that is part of the image processing device 1, or images captured by an external sensor may be input. Alternatively, images stored in a fog computer (to be described later) or a cloud server (to be described later) may be acquired.’; [0032] The learning unit 13 includes a neural network 14. The neural network 14 creates inferred images from the input images converted to CIELAB by the color space conversion unit 12. The color space of the created inferred image is CIELAB.”); and determining the difference information according to the converted image and the converted photographed image ([0033] The learning unit 13 inputs a label image converted to CIELAB by the color space conversion unit 12 and compares the converted label image with the inferred image created by the neural network 14. An error between the two images is obtained as a result of comparison between the label image and the inferred image. This error is a color difference. A color difference is an index defined as a distance in the color space to represent the difference between two colors. The metrics that define a color difference in the CIELAB color space include the Euclidean distance, CIE76, CIE94, and CIEDE2000.”)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of rendering image based on material parameters of Kaplanyan, Daxuan and converting rendered image to CIELAB color space of rendered image of Chekh with using color space conversion to convert two different source images to same format image as seen in KOUNOSU because this modification would compare the converted images and defined color difference between the two images ([0033])
Thus, the combination Kaplanyan, Daxuan, Chekh and KOUNOSU teaches wherein the determining difference information between the rendered image and the photographed image comprises: performing color space conversion on the rendered image, to obtain a converted rendered image; performing color space conversion on the photographed image, to obtain a converted photographed image; and determining the difference information according to the converted rendered image and the converted photographed image.
Regarding claim 12, Kaplanyan and Daxuan teach the electronic device according to claim 8, Remaining limitations of claim 12 is similar scope to claim 5 and therefore rejected under the same rationale.
Regarding claim 19, Kaplanyan and Daxuan teach the non-transitory computer-readable storage medium according to claim 15, Remaining limitations of claim 19 is similar scope to claim 5 and therefore rejected under the same rationale.
3. Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Kaplanyan et al., IDS, U.S Patent Application Publication No.2021/0241519 (“Kaplanyan”) in view of REN DAXUAN; LI JIATONGl ;MEI HAIYI, IDS, CN115272548 -English translated (“Daxuan”) further in view of Chen et al., U.S Patent Application Publication No. 20170184714 (“Chen”)
Regarding claim 7, Kaplanyan and Daxuan teach the method according to claim 6, wherein the selecting the target material parameter from the at least two material parameters comprises: the at least two material parameters; and selecting the target material parameter from the at least two material parameters (see at least [0030] of Kaplanyan “In particular embodiments, the material model may satisfy several properties. In particular embodiments, the material model may cover as much variability in appearance as possible, including such common effects as specular highlights, multilayered materials, and spatially varying textures. In particular embodiments, since each parameter adds another unknown to the optimization, the number of parameters may be kept to a minimal. Since a goal may be directed to re-rendering and related tasks, the material model may have interpretable parameters, so the users can adjust the parameters to achieve the desired appearance. In particular embodiments, the material properties may be optimized using first-order gradient-based optimization, and the range of the material parameters may be similar. [0031] As an example and not by way of limitation, the materials may be represented using the Disney material model, the state-of-the-art physically based material model used in movie and game rendering. In particular embodiments, the material model may have a “base color” parameter which is used by both diffuse and specular reflectance, as well as 10 other parameters describing the roughness, anisotropy, and specularity of the material. In particular embodiments, all these parameters may be perceptually mapped to [0, 1], which may be both interpretable and suitable for optimization) Kaplanyan and Daxuan are understood to be silent on the remaining limitations of claim 7.
In the same field of endeavor, Chen teaches determining a correlation between the at least two material parameters; and selecting the target material parameter from the at least two material parameters according to the correlation (see at least [0043] In another embodiment, the data analysis processor 124 performs the hierarchical correlation method to obtain the reference sequence. The hierarchical correlation method comprises the following steps of: (a) gathering the color value sequences into an i.sup.th reference sequence set, wherein an initial value of i is 1; (b) obtaining a plurality of correlation coefficient values of the i.sup.th reference sequence set by calculating each pair of the color value sequences in the i.sup.th reference sequence set through a correlation function; (c) selecting the pair of the color value sequences in the i.sup.th reference sequence set, which have a largest correlation coefficient value; (d) averaging the selected pair of the color value sequences in the i.sup.th reference sequence set to generate a new color value sequence; (e) gathering the new color value sequence and the color value sequences in the i.sup.th reference sequence set except for the selected pair of the color value sequences into a (i+1).sup.th reference sequence set; (f) obtaining a plurality of correlation coefficient values of the (i+1).sup.th reference sequence set by calculating each pair of the color value sequences in the (i+1).sup.th reference sequence set through the correlation function; (g) determining whether all of the correlation coefficient values of the (i+1).sup.th reference sequence set are less than a predetermined correlation threshold; and (h) when one of the correlation coefficient values of the (i+1).sup.th reference sequence set is not less than the predetermined correlation threshold, setting i=i+1 and repeating steps (c), (d), (e), (f) and (g), and when the all of the correlation coefficient values of the (i+1).sup.th reference sequence set are less than the predetermined correlation threshold, outputting the color value sequence in the (i+1).sup.th reference sequence set which is generated by averaging a most number of the color value sequences in the first reference sequence set as a reference sequence.” Where color values are considered as material parameters)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the method of rendering image based on material parameters of Kaplanyan and Daxuan with selecting the pair of the color value sequences in the reference sequence set, which have a largest correlation coefficient value as seen in Chen because this modification would achieve the expected benefit of identifying the strongest linear relationship between color channels.
Thus, the combination of Kaplanyan, Daxuan and Chen teaches wherein the selecting the target material parameter from the at least two material parameters comprises: determining a correlation between the at least two material parameters; and selecting the target material parameter from the at least two material parameters according to the correlation.
Regarding claim 14, Kaplanyan and Daxuan teach the electronic device according to claim 13, Remaining limitations of claim 14 is similar scope to claim 7 and therefore rejected under the same rationale.
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARAH LE whose telephone number is (571)270-7842. The examiner can normally be reached Monday: 8AM-4:30PM EST, Tuesday: 8 AM-3:30PM EST, Wednesday: 8AM-2:30PM EST, Thursday and Friday off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SARAH LE/Primary Examiner, Art Unit 2614
Prosecution Insights