DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-13, 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over PURUSHWALKAM (US 20240386653 A1), and in view of Litwiller (US 11783451 B2).
Re Claim 1, PURUSHWALKAM discloses a computer-implemented method, comprising:
producing a first noised image by adding first noise at a first timestep to a reference image generated using a three-dimensional (3D) generative model (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and,
-- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]);
computing a first gradient between the reference image and a first denoised image (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]);
the first denoised image being obtained by removing the first noise from the first noised image using a diffusion model and based on a received text prompt (see PURUSHWALKAM: e.g., -- the expectation in computing gradient 135 may be estimated by using stratified sample parameters 136, such as (for example and without limitation), any rendering parameters 121, noise term 123, timestep for diffusion used in diffusion model 130, prompt, and/or the like.
[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.
[0039] In one embodiment, in addition to the SDS gradient, 3D model 110 may be trained using supervisory signals from the input image 102. For example, a depth map D may be estimated based on the input image 102. T; --, in [0036]-[0039]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]);
producing a second noised image by adding noise at a second timestep to the reference image (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and,
-- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]);
PURUSHWALKAM however does not explicitly disclose that producing a second noised image by adding second noise at a second timestep to the reference image;
LITWILLER discloses producing a second noised image by adding second noise at a second timestep to the reference image (see LITWILLER: e.g., Fig. 6, and, -- synthesized colored noise in the spatial frequency domain, and an example of the synthesized colored noise in the spatial domain--, and, -- The acquired medical image, which may include colored noise, is then processed using a trained deep neural network, which may optionally be given one or more noise parameters for the image. A medical image including colored noise is herein defined as a noisy image. Noise parameters may correlate to a type and distribution of colored noise in a medical image, and may include various imaging system settings used to acquire the medical image, such as a k-space sampling pattern. Noise parameters may further include a noise power in the spatial frequency domain, and other information about the noisy image, for example. In other words, the noise parameters may correspond to a source of colored noise in the medical image. The deep neural network may reduce colored noise in the image and output an image with substantially less colored noise (e.g., a de-noised image).--, in lines 1-18, col. 4; and, --Noise may be synthesized in the frequency domain as a function of noise power, and may be distributed differently in the frequency domain based on characteristics of the noise. Thus, de-noising comprises a process of at least partially reducing the presence of at least one type of colored noise in an image in order to improve image resolution and diagnostic quality. De-noised images may be produced by de-noising a noisy image (e.g., a medical image with colored noise), and conversely, noisy images may be produced by introducing one or more sources of colored noise into a pristine image (e.g., a medical image without colored noise). In particular, a noisy image may be produced from a corresponding pristine image based on a k-space sampling pattern and/or expected noise characteristics, and the resulting image pair (e.g., the noisy image and the corresponding pristine image) may be used as training data for a deep neural network.--, in lines 26-45, col. 4; -- (63) The deep neural network(s) may be trained by using a plurality of pairs of noisy medical images and corresponding pristine images (e.g., image pairs), referred to herein as training data. The noisy image in each image pair is used as input to the deep neural network and the pristine image in each image pair is used as the ground truth for reference. In some examples, an image pair may be produced by applying colored noise to a pristine image, thus generating a corresponding noisy image. Colored noise may be synthesized by weighting a k-space representation of a white-noise image by the noise power derived from a sampling density, and then transforming noise back to the image space. For example, an image space representation of colored noise (e.g., a representation of colored noise in the spatial domain) may be produced by scaling a field of white noise based on a k-space representation of noise power for a k-space sampling pattern, and transforming the resulting noise from the k-space (frequency domain) back to the image space. In this way, a noisy image, with colored noise based on a k-space sampling pattern, may be generated for a pristine image.
(64) Thus, FIG. 5 shows a flow chart of a method 500 for synthesizing colored noise resulting from nonuniform sampling based on noise power for a k-space sampling pattern. Method 500 may be implemented by the image processing system 100, an edge device connected to the imaging device, a cloud in communication with the imaging device, a computing device in wired communication with the imaging device, or any appropriate combination thereof. Elements of method 500 may be performed sequentially or simultaneously.
(65) At operation 502, method 500 includes synthesizing white noise in image space (e.g., the spatial domain). For example, the synthesized white noise may be a synthesized white noise image, such that noise in the synthesized white noise is equally distributed across all frequencies. In some examples, white noise may be directly synthesized in k-space, rather than in image space.--, in lines 9-46, col. 14;
and, see various noise patterns could be added in the following reproduced Fig. 6:
PNG
media_image1.png
962
788
media_image1.png
Greyscale
);
PURUSHWALKAM and LITWILLER are combinable as they are in the same field of endeavor: image processing and denoising using neural network. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify PURUSHWALKAM’s method using LITWILLER’s teachings by including producing a second noised image by adding second noise at a second timestep to the reference image to PURUSHWALKAM’s synthesizing noisy images for neural network denoising model training in order to train the deep neural network(s) by using a plurality of pairs of noisy medical images and corresponding pristine images (e.g., image pairs), referred to herein as training data including the noisy image in each image pair is used as input to the deep neural network and the pristine image in each image pair is used as the ground truth for reference (see LITWILLER: e.g. in lines 1-18, and lines 26-45, col. 4; and in lines 9-46, col. 14);
PURUSHWALKAM as modified by LITWILLER further disclose computing a second gradient between the reference image and a second denoised image (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]; and, -- the expectation in computing gradient 135 may be estimated by using stratified sample parameters 136, such as (for example and without limitation), any rendering parameters 121, noise term 123, timestep for diffusion used in diffusion model 130, prompt, and/or the like.
[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.
[0039] In one embodiment, in addition to the SDS gradient, 3D model 110 may be trained using supervisory signals from the input image 102. For example, a depth map D may be estimated based on the input image 102. T; --, in [0036]-[0039]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]),
the second denoised image being obtained by removing the second noise from the second noised image using the diffusion model and based on the received text prompt (see PURUSHWALKAM: e.g., -- the expectation in computing gradient 135 may be estimated by using stratified sample parameters 136, such as (for example and without limitation), any rendering parameters 121, noise term 123, timestep for diffusion used in diffusion model 130, prompt, and/or the like.
[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.
[0039] In one embodiment, in addition to the SDS gradient, 3D model 110 may be trained using supervisory signals from the input image 102. For example, a depth map D may be estimated based on the input image 102. T; --, in [0036]-[0039]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]); and
generating a combined gradient based at least on the first gradient and the second gradient (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]); and
updating one or more parameters for the 3D generative model based at least on the combined gradient (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]; and, -- the expectation in computing gradient 135 may be estimated by using stratified sample parameters 136, such as (for example and without limitation), any rendering parameters 121, noise term 123, timestep for diffusion used in diffusion model 130, prompt, and/or the like.
[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.
[0039] In one embodiment, in addition to the SDS gradient, 3D model 110 may be trained using supervisory signals from the input image 102. For example, a depth map D may be estimated based on the input image 102. T; --, in [0036]-[0039]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]).
Re Claim 2, PURUSHWALKAM as modified by LITWILLER further disclose wherein the reference image comprises a set of images at different camera views (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and, ., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]).
Re Claim 3, PURUSHWALKAM as modified by LITWILLER further disclose wherein the combined gradient is an average of the first gradient and the second gradient (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]).
Re Claim 4, PURUSHWALKAM as modified by LITWILLER further disclose wherein one or both of the first timestep and the second timestep are randomly selected (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]).
Re Claim 5, PURUSHWALKAM as modified by LITWILLER further disclose wherein the first noise is selected from a normal distribution (see LITWILLER: e.g., Fig. 6, and, -- synthesized colored noise in the spatial frequency domain, and an example of the synthesized colored noise in the spatial domain--, and, -- The acquired medical image, which may include colored noise, is then processed using a trained deep neural network, which may optionally be given one or more noise parameters for the image. A medical image including colored noise is herein defined as a noisy image. Noise parameters may correlate to a type and distribution of colored noise in a medical image, and may include various imaging system settings used to acquire the medical image, such as a k-space sampling pattern. Noise parameters may further include a noise power in the spatial frequency domain, and other information about the noisy image, for example. In other words, the noise parameters may correspond to a source of colored noise in the medical image. The deep neural network may reduce colored noise in the image and output an image with substantially less colored noise (e.g., a de-noised image).--, in lines 1-18, col. 4; and, --Noise may be synthesized in the frequency domain as a function of noise power, and may be distributed differently in the frequency domain based on characteristics of the noise. Thus, de-noising comprises a process of at least partially reducing the presence of at least one type of colored noise in an image in order to improve image resolution and diagnostic quality. De-noised images may be produced by de-noising a noisy image (e.g., a medical image with colored noise), and conversely, noisy images may be produced by introducing one or more sources of colored noise into a pristine image (e.g., a medical image without colored noise). In particular, a noisy image may be produced from a corresponding pristine image based on a k-space sampling pattern and/or expected noise characteristics, and the resulting image pair (e.g., the noisy image and the corresponding pristine image) may be used as training data for a deep neural network.--, in lines 26-45, col. 4; -- (63) The deep neural network(s) may be trained by using a plurality of pairs of noisy medical images and corresponding pristine images (e.g., image pairs), referred to herein as training data. The noisy image in each image pair is used as input to the deep neural network and the pristine image in each image pair is used as the ground truth for reference. In some examples, an image pair may be produced by applying colored noise to a pristine image, thus generating a corresponding noisy image. Colored noise may be synthesized by weighting a k-space representation of a white-noise image by the noise power derived from a sampling density, and then transforming noise back to the image space. For example, an image space representation of colored noise (e.g., a representation of colored noise in the spatial domain) may be produced by scaling a field of white noise based on a k-space representation of noise power for a k-space sampling pattern, and transforming the resulting noise from the k-space (frequency domain) back to the image space. In this way, a noisy image, with colored noise based on a k-space sampling pattern, may be generated for a pristine image.
(64) Thus, FIG. 5 shows a flow chart of a method 500 for synthesizing colored noise resulting from nonuniform sampling based on noise power for a k-space sampling pattern. Method 500 may be implemented by the image processing system 100, an edge device connected to the imaging device, a cloud in communication with the imaging device, a computing device in wired communication with the imaging device, or any appropriate combination thereof. Elements of method 500 may be performed sequentially or simultaneously.
(65) At operation 502, method 500 includes synthesizing white noise in image space (e.g., the spatial domain). For example, the synthesized white noise may be a synthesized white noise image, such that noise in the synthesized white noise is equally distributed across all frequencies. In some examples, white noise may be directly synthesized in k-space, rather than in image space.--, in lines 9-46, col. 14;
and, see various noise patterns could be added in the following reproduced Fig. 6:
PNG
media_image1.png
962
788
media_image1.png
Greyscale
).
Re Claim 6, PURUSHWALKAM as modified by LITWILLER further disclose determining one or more stop conditions for gradient generation are not satisfied (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]);
iteratively producing gradients based on iteratively generated noised images and
iteratively generated denoised images until the one or more stop conditions are satisfied (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]; and, -- the expectation in computing gradient 135 may be estimated by using stratified sample parameters 136, such as (for example and without limitation), any rendering parameters 121, noise term 123, timestep for diffusion used in diffusion model 130, prompt, and/or the like.
[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.
[0039] In one embodiment, in addition to the SDS gradient, 3D model 110 may be trained using supervisory signals from the input image 102. For example, a depth map D may be estimated based on the input image 102. T; --, in [0036]-[0039]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]).
Re Claim 7, PURUSHWALKAM as modified by LITWILLER further disclose wherein the text prompt is used to generate the reference image (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and,
-- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]).
Re Claim 8, PURUSHWALKAM as modified by LITWILLER further disclose updating the one or more parameters for the 3D generative model (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]; and, -- the expectation in computing gradient 135 may be estimated by using stratified sample parameters 136, such as (for example and without limitation), any rendering parameters 121, noise term 123, timestep for diffusion used in diffusion model 130, prompt, and/or the like.
[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.
[0039] In one embodiment, in addition to the SDS gradient, 3D model 110 may be trained using supervisory signals from the input image 102. For example, a depth map D may be estimated based on the input image 102. T; --, in [0036]-[0039]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]);
generating the reference image (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]; and, -- the expectation in computing gradient 135 may be estimated by using stratified sample parameters 136, such as (for example and without limitation), any rendering parameters 121, noise term 123, timestep for diffusion used in diffusion model 130, prompt, and/or the like.
[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.
[0039] In one embodiment, in addition to the SDS gradient, 3D model 110 may be trained using supervisory signals from the input image 102. For example, a depth map D may be estimated based on the input image 102. T; --, in [0036]-[0039]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]; also see LITWILLER : e.g., -- (81) Following operation 808, method 800 may end. It will be noted that 808 may be repeated until the weights and biases of the deep neural network converge, a threshold difference metric is obtained (for the training data or on a separate validation dataset), or the rate of change of the weights and/or biases of the deep neural network for each iteration are under a threshold. In this way, method 800 enables a deep neural network to be trained to identify colored noise in medical images, and to produce de-noised images with substantially less colored noise, thus increasing image resolution and diagnostic quality.
(82) As an example, a method for training a deep neural network to reduce colored noise in medical images comprises: selecting a first medical image devoid of colored noise, synthesizing a colored noise image based on a noise parameter, generating a second medical image by adding the synthesized colored noise to the first medical image, mapping the second medical image to a predicted de-noised medical image via the deep neural network, determining a loss based on a difference between the predicted de-noised image and the first medical image, and updating parameters of the deep neural network based on the loss. In the previous example, additionally or optionally, mapping the second medical image to the predicted de-noised medical image via the deep neural network further includes indexing the noise parameter to the second medical image, and selecting pre-trained weights and biases for the deep neural net based on the noise parameter.--, in line 47, col. 18 through line 7, col. 19);
determining a quality metric for the reference image is below a threshold (see LITWILLER : e.g., -- (81) Following operation 808, method 800 may end. It will be noted that 808 may be repeated until the weights and biases of the deep neural network converge, a threshold difference metric is obtained (for the training data or on a separate validation dataset), or the rate of change of the weights and/or biases of the deep neural network for each iteration are under a threshold. In this way, method 800 enables a deep neural network to be trained to identify colored noise in medical images, and to produce de-noised images with substantially less colored noise, thus increasing image resolution and diagnostic quality.
(82) As an example, a method for training a deep neural network to reduce colored noise in medical images comprises: selecting a first medical image devoid of colored noise, synthesizing a colored noise image based on a noise parameter, generating a second medical image by adding the synthesized colored noise to the first medical image, mapping the second medical image to a predicted de-noised medical image via the deep neural network, determining a loss based on a difference between the predicted de-noised image and the first medical image, and updating parameters of the deep neural network based on the loss. In the previous example, additionally or optionally, mapping the second medical image to the predicted de-noised medical image via the deep neural network further includes indexing the noise parameter to the second medical image, and selecting pre-trained weights and biases for the deep neural net based on the noise parameter.--, in line 47, col. 18 through line 7, col. 19) ; and
retraining the 3D generative model (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]; and, -- the expectation in computing gradient 135 may be estimated by using stratified sample parameters 136, such as (for example and without limitation), any rendering parameters 121, noise term 123, timestep for diffusion used in diffusion model 130, prompt, and/or the like.
[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.
[0039] In one embodiment, in addition to the SDS gradient, 3D model 110 may be trained using supervisory signals from the input image 102. For example, a depth map D may be estimated based on the input image 102. T; --, in [0036]-[0039]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]).
Re Claim 9, claim 9 is the corresponding processor claim to claim 1, respectively. Claim 9 thus is rejected for the similar reasons for claim 1. See above discussions with regard to claim 1 respectively. PURUSHWALKAM as modified by LITWILLER further disclose a processor, comprising: one or more circuits to perform the method (see PURUSHWALKAM: e.g., -- [0047] FIG. 4 is a simplified diagram illustrating a computing device implementing the 3D reconstruction system described in FIG. 1, according to one embodiment described herein. As shown in FIG. 4A, computing device 400 includes a processor 410 coupled to memory 420. Operation of computing device 400 is controlled by processor 410. And although computing device 400 is shown with only one processor 410, it is understood that processor 410 may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) and/or the like in computing device 400. Computing device 400 may be implemented as a stand-alone subsystem, as a board added to a computing device, and/or as a virtual machine.--, in [0047]-[0050]); and further disclose:
generate a first denoised image from a first input noisy image based, at least, on a text prompt (see PURUSHWALKAM: e.g., -- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and,
-- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]; and, -- the expectation in computing gradient 135 may be estimated by using stratified sample parameters 136, such as (for example and without limitation), any rendering parameters 121, noise term 123, timestep for diffusion used in diffusion model 130, prompt, and/or the like.
[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.
[0039] In one embodiment, in addition to the SDS gradient, 3D model 110 may be trained using supervisory signals from the input image 102. For example, a depth map D may be estimated based on the input image 102. T; --, in [0036]-[0039]);
determine a first difference between the first denoised image and a reference
image used to generate the first input noisy image (see PURUSHWALKAM: e.g., Eqs. 7-10, and, --[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.--, in [0037]-[0040]);
generate a second denoised image from a second input noisy image based, at
least, on the text prompt (see PURUSHWALKAM: e.g., -- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and,
-- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]; and, -- the expectation in computing gradient 135 may be estimated by using stratified sample parameters 136, such as (for example and without limitation), any rendering parameters 121, noise term 123, timestep for diffusion used in diffusion model 130, prompt, and/or the like.
[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.
[0039] In one embodiment, in addition to the SDS gradient, 3D model 110 may be trained using supervisory signals from the input image 102. For example, a depth map D may be estimated based on the input image 102. T; --, in [0036]-[0039]);
determine a second difference between the second denoised image and the
reference image (see PURUSHWALKAM: e.g., Eqs. 7-10, and, --[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.--, in [0037]-[0040]); and,
update one or more parameters for a machine learning model used to generate the reference image using the first difference and the second difference (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]; and, -- the expectation in computing gradient 135 may be estimated by using stratified sample parameters 136, such as (for example and without limitation), any rendering parameters 121, noise term 123, timestep for diffusion used in diffusion model 130, prompt, and/or the like.
[0038] This SDS gradient may be used to update 3D model 110 via backpropagation 137. In one implementation, each update is computed using a randomly sampled timestep t and noise sample ϵ. This may be equivalent to first perturbing the rendered image 121 using ϵ, t and then updating the radiance field of 3D model 110 using the difference between the diffusion model estimated noise 133 and injected noise ϵ 123.
[0039] In one embodiment, in addition to the SDS gradient, 3D model 110 may be trained using supervisory signals from the input image 102. For example, a depth map D may be estimated based on the input image 102. T; --, in [0036]-[0039]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]).
Re Claim 10, PURUSHWALKAM as modified by LITWILLER further disclose wherein the first input noisy image includes the reference image and first noise associated with a first timestep (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]).
Re Claim 11, PURUSHWALKAM as modified by LITWILLER further disclose wherein the second input noisy image includes the reference image and second noise associated with a second timestep that is different from the first timestep. (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]).
Re Claim 12, PURUSHWALKAM as modified by LITWILLER further disclose generate a combined gradient including an average of the first difference and the second difference (see PURUSHWALKAM: e.g., -- This data is used to a train a denoising model {circumflex over (ϵ)}.sub.ϕ(I.sub.t, y, t) which estimates the noise ϵ using the noisy image I.sub.t, the timestep t and an optional embedding y of the caption associated with the image. The resulting pretrained diffusion model 130 is then used to synthesize images by following the inverse process. First, a noise sample I.sub.T is drawn from custom-character(0,I). Then {circumflex over (ϵ)}.sub.ϕ is iteratively used to estimate the sequences of images {I.sub.T-1, I.sub.T-2, . . . , I.sub.0} where I.sub.0 is the finally synthesized image. In this way, the diffusion model 130 “denoises” the input noisy image of the rendered 2D image 121 with injected random noise 123 into a predicted noise 133. [0037] Therefore, given the randomly sampled camera pose p used to render 2D image 121, and images from the radiance field for these viewpoints I.sub.θ.sup.p, the gradient module 135 may compute a gradient of the Score Distillation Sampling (SDS) objective--, in [0036]-[0039]; and, --[0101] At step 822, a gradient may be computed based on an injected random noise and the predicted injected noise, e.g., Eq. (7).--, in [0101]-[0106]).
Re Claim 13, PURUSHWALKAM as modified by LITWILLER further disclose randomly select noise for the first input noisy image and the second input noisy image (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and,
-- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]).
Re Claim 15, claim 15 is the corresponding system claim to claim 1 and claim 9, respectively. Claim 15 thus is rejected for the similar reasons for claim 1 and claim 9. See above discussions with regard to claim 1 and claim 9 respectively. PURUSHWALKAM as modified by LITWILLER further disclose a system, comprising: one or more processing units to generate a combined gradient for updating a model based on differences between a reference image and at least two denoised images, wherein the at least two denoised images are generated by removing noise added to the reference image at different timesteps based on an input text prompt (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and,
-- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]; also see LITWILLER : e.g., -- (81) Following operation 808, method 800 may end. It will be noted that 808 may be repeated until the weights and biases of the deep neural network converge, a threshold difference metric is obtained (for the training data or on a separate validation dataset), or the rate of change of the weights and/or biases of the deep neural network for each iteration are under a threshold. In this way, method 800 enables a deep neural network to be trained to identify colored noise in medical images, and to produce de-noised images with substantially less colored noise, thus increasing image resolution and diagnostic quality. (82) As an example, a method for training a deep neural network to reduce colored noise in medical images comprises: selecting a first medical image devoid of colored noise, synthesizing a colored noise image based on a noise parameter, generating a second medical image by adding the synthesized colored noise to the first medical image, mapping the second medical image to a predicted de-noised medical image via the deep neural network, determining a loss based on a difference between the predicted de-noised image and the first medical image, and updating parameters of the deep neural network based on the loss. In the previous example, additionally or optionally, mapping the second medical image to the predicted de-noised medical image via the deep neural network further includes indexing the noise parameter to the second medical image, and selecting pre-trained weights and biases for the deep neural net based on the noise parameter.--, in line 47, col. 18 through line 7, col. 19).
Re Claim 16, PURUSHWALKAM as modified by LITWILLER further disclose wherein the different timesteps are randomly selected from a uniform distribution (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and,
-- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]).
Re Claim 17, PURUSHWALKAM as modified by LITWILLER further disclose wherein the respective noise added to the reference image is randomly selected from a normal distribution (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and,
-- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]).
Re Claim 18, PURUSHWALKAM as modified by LITWILLER further disclose wherein the at least two denoised images are generated by a trained diffusion model conditioned on a text prompt (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and,
-- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]).
Re Claim 19, PURUSHWALKAM as modified by LITWILLER further disclose wherein the reference image is a rendered image generated by a three-dimensional (3D) content generation model (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and,
-- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]).
Claims 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over PURUSHWALKAM as modified by LITWILLER, and further in view of CHRISTOPH (WO 2024233961 A2).
Re Claim 14, PURUSHWALKAM as modified by LITWILLER further disclose wherein the processor is comprised in at least one of:
a system for performing simulation operations (see PURUSHWALKAM: e.g., --[0020] In one embodiment, the 3D generation system may generate a 3D representation comprising colors and densities corresponding to 3D spatial location coordinates by the radiance field method. For example, a neural network may encode a 3D scene as a volumetric function, and then predict a volume density, and view-dependent emitted radiance given the 3D spatial location coordinates and viewing direction in Euler angles of a camera pose. The 3D representation may be constrained by the one viewpoint of the target 3D object shown in the input image, based on which other viewpoints of the 3D object can be inferred. Based on the visible surface depicted by the input image, points between the reference camera and the surface are assigned with zero density, and points on the surface are assigned with high density and color equal to the corresponding pixel in the input image. In addition, the radiance field is used to obtain an estimate of the depth map, and the observations are in turn incorporated into the color of the radiance field so as to use the input image as a constraint for inferring 3D space around the target object.--, in [002]);
a system for performing simulation operations to test or validate autonomous machine applications (see PURUSHWALKAM: e.g., -- [0064] Parameters of the neural network are updated backwardly from the last layer to the input layer (backpropagating) based on the computed negative gradient using an optimization algorithm to minimize the loss. The backpropagation from the last layer 443 to the input layer 441 may be conducted for a number of training samples in a number of iterative training epochs. In this way, parameters of the neural network may be gradually updated in a direction to result in a lesser or minimized loss, indicating the neural network has been trained to generate a predicted output value closer to the target output value with improved prediction accuracy. Training may continue until a stopping criterion is met, such as reaching a maximum number of epochs or achieving satisfactory performance on the validation data. At this point, the trained network can be used to make predictions on new, unseen data.
[0065] Therefore, the training process transforms the neural network into an “updated” trained neural network with updated parameters such as weights, activation functions, and biases. The trained neural network thus improves neural network technology in 3D object reconstruction.--, in [0064]-[0065]);
PURUSHWALKAM as modified by LITWILLER however still do not explicitly disclose a system for performing digital twin operations;
CHRISTOPH discloses a system for performing digital twin operations {in denoising} (see CHRISTOPH: e.g., -- applying the model for treatment of a subject. 106. The method of any one of the previous aspects, wherein the model comprises a digital twin of a subject. 107. The method of any one of the previous aspects, wherein the method further comprises generating a virtual simulation of the tissue using the model. 108. The method of any one of the previous aspects, wherein the configuring comprises simulating a tissue function as the model learns the contractile motion and electrophysiological wave pattern associated with the tissue. --, in claims 105, 106, and 107; see pages 72-73; also see: -- Example 2: Dreaming of Electrical Waves: Generative Modeling of Cardiac Excitation Waves using Diffusion Models 2.0. Overview Electrical waves in the heart form rotating spiral or scroll waves during life- threatening arrhythmias such as atrial or ventricular fibrillation. The wave dynamics are typically modeled using coupled partial differential equations, which describe reaction- diffusion dynamics in excitable media. More recently, data-driven generative modeling has emerged as an alternative to generate spatio-temporal patterns in physical and biological systems. Here, denoising diffusion probabilistic models were explore for the generative modeling of electrical wave patterns in cardiac tissue. Diffusion models were trained with simulated electrical wave patterns to be able to generate such wave patterns in unconditional and conditional generation tasks. For instance, inpainting tasks, such as reconstructing three-dimensional wave dynamics from superficial two- dimensional measurements, and evolving and generating parameter-specific dynamics were explored. The diffusion-generated solutions were characterized and compared to solutions obtained with biophysical models and it was found that diffusion models learn to replicate spiral and scroll waves dynamics so well that they could serve as an alternative data-driven approach for the modeling of excitation waves in cardiac tissue. For instance, it was found that it is possible to initiate ventricular fibrillation (VF) dynamics instantaneously without having to apply pacing protocols in order to induce wavebreak… During the training procedure, noise is iteratively added to the training data and the neural networks, termed denoising diffusion probabilistic models (DDPMs) [18] or diffusion models, learn to reverse this process, ultimately enabling them to create data from noise, see FIG.37. Diffusion models are very successful in generating data such as images [19–21], videos [22], and audio [23], and they are increasingly also used for technical applications in physics, engineering, medicine, and biology [24–27]. Diffusion models likely also have many useful applications in cardiology that have yet to be explored. For example, they could be used in electrophysiological studies to generate synthetic action potential wave patterns and arrhythmia morphologies, either to fill in or reconstruct missing measurement data, or to simulate cardiac dynamics in a data-driven fashion. --, in pages 119-120);
PURUSHWALKAM (as modified by LITWILLER) and CHRISTOPH are combinable as they are in the same field of endeavor: image processing and denoising diffusion. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify PURUSHWALKAM (as modified by LITWILLER)’s processor and system using CHRISTOPH’s teachings by including a system for performing digital twin operations to PURUSHWALKAM’(as modified by LITWILLER)’s processor and system of denoising diffusion model in order to perform training procedure, in which noise is iteratively added to the training data and the neural networks, termed denoising diffusion probabilistic models (DDPMs) or diffusion models (see CHRISTOPH: e.g. in claims 105, 106, and 107, and in pages 119-120);
PURUSHWALKAM as modified by LITWILLER and CHRISTOPH further disclose at least one of a system for performing light transport simulation; a system for rendering graphical output; a system for performing deep learning operations; a system implemented using an edge device; a system for generating or presenting virtual reality (VR) content; a system for generating or presenting augmented reality (AR) content;
a system for generating or presenting mixed reality (MR) content; a system incorporating one or more Virtual Machines (VMs); a system for performing operations for a conversational AI application; a system for performing operations for a generative AI application; a system for performing operations using a language model; a system for performing one or more generative operations using a large language model (LLM);
a system for performing one or more generative operations using a vision language model (VLM); a system implemented at least partially in a data center; a system for performing hardware testing using simulation; a system for performing one or more generative content operations using a language model; a system for synthetic data generation; a collaborative content creation platform for 3D assets; or a system implemented at least partially using cloud computing resources (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and,
-- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]; also see LITWILLER : e.g., -- (81) Following operation 808, method 800 may end. It will be noted that 808 may be repeated until the weights and biases of the deep neural network converge, a threshold difference metric is obtained (for the training data or on a separate validation dataset), or the rate of change of the weights and/or biases of the deep neural network for each iteration are under a threshold. In this way, method 800 enables a deep neural network to be trained to identify colored noise in medical images, and to produce de-noised images with substantially less colored noise, thus increasing image resolution and diagnostic quality.
(82) As an example, a method for training a deep neural network to reduce colored noise in medical images comprises: selecting a first medical image devoid of colored noise, synthesizing a colored noise image based on a noise parameter, generating a second medical image by adding the synthesized colored noise to the first medical image, mapping the second medical image to a predicted de-noised medical image via the deep neural network, determining a loss based on a difference between the predicted de-noised image and the first medical image, and updating parameters of the deep neural network based on the loss. In the previous example, additionally or optionally, mapping the second medical image to the predicted de-noised medical image via the deep neural network further includes indexing the noise parameter to the second medical image, and selecting pre-trained weights and biases for the deep neural net based on the noise parameter.--, in line 47, col. 18 through line 7, col. 19).
Re Claim 20, claim 20 is the corresponding system claim to claim 14, respectively. Claim 20 thus is rejected for the similar reasons for claim 14. See above discussions with regard to claim 14 respectively. PURUSHWALKAM as modified by LITWILLER and CHRISTOPH further disclose a system, comprising: one or more processing units to generate a combined gradient for updating a model based
on differences between a reference image and at least two denoised images, wherein the at least two denoised images are generated by removing noise added to the reference image at different timesteps based on an input text prompt (see PURUSHWALKAM: e.g., --[0034] In one embodiment, rendering module 120 may render the constrained 3D representation 116 into one or more two-dimensional (2D) images 121, e.g., using a randomly sampled camera pose such as a camera azimuth, a camera elevation, a camera field-of-view, a lighting direction, and/or the like. Additional rendering parameters may include binary choices between a black and white or colored background, shade, albedo, and/or the like. The rendered 2D image 121 may be injected with a noise term 123 to result in a noisy image, which may be fed to a pretrained diffusion model 130.--, in [0034]-[0036]; and,
-- the diffusion model 130 synthesizes images by iteratively denoising the input noisy image from a Gaussian distribution to generate a predicted noise 133 according to a text prompt 122, e.g., “a photo of chair” corresponding to the target 3D object “chair.” For example, diffusion model 130 may be a text conditioned diffusion model. A text prompt 122 such as a caption “a photo of a <token>” where the embedding of the special token is inferred using Textual Inversion. In order to learn an embedding that accurately captures the input image, multiple synthetically generated augmentations of the input image 102 may be passed by randomly cropping, resizing and blurring the input image 102.the diffusion model 130 has been pretrained by added noise to a real training image I iteratively for T timesteps to synthesize training data {I.sub.0, I.sub.1, . . . , I.sub.T}. Specifically, the noised image for timestep t can be computed as I.sub.t=√{square root over (α.sub.tI)}+√{square root over (−α.sub.tϵ)} where ϵ∈custom-character(0,I) and α.sub.t is predefined noising schedule.--, in [0035]-[0036]; and, -- the 3D generation and training framework described in FIGS. 1-2 may also be used to generate 3D model from one or more text prompts. For example, a pretrained text-to-image (e.g., Stable Diffusion, etc.) model may be used to first generate a 2D image from the text prompt describing an object. The generated 2D image may then serve as the input image 102 to generate a 3D reconstruction of this object using the 3D generation and training framework described in FIGS. 1-2.--, in [0044]-[0046]; also see LITWILLER : e.g., -- (81) Following operation 808, method 800 may end. It will be noted that 808 may be repeated until the weights and biases of the deep neural network converge, a threshold difference metric is obtained (for the training data or on a separate validation dataset), or the rate of change of the weights and/or biases of the deep neural network for each iteration are under a threshold. In this way, method 800 enables a deep neural network to be trained to identify colored noise in medical images, and to produce de-noised images with substantially less colored noise, thus increasing image resolution and diagnostic quality.
(82) As an example, a method for training a deep neural network to reduce colored noise in medical images comprises: selecting a first medical image devoid of colored noise, synthesizing a colored noise image based on a noise parameter, generating a second medical image by adding the synthesized colored noise to the first medical image, mapping the second medical image to a predicted de-noised medical image via the deep neural network, determining a loss based on a difference between the predicted de-noised image and the first medical image, and updating parameters of the deep neural network based on the loss. In the previous example, additionally or optionally, mapping the second medical image to the predicted de-noised medical image via the deep neural network further includes indexing the noise parameter to the second medical image, and selecting pre-trained weights and biases for the deep neural net based on the noise parameter.--, in line 47, col. 18 through line 7, col. 19).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEIWEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on Monday-Friday 8:30am-4:30pm east.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
