DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-12, and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Girardot (WO 2022238640 A1) in view of Feldman (KR 20250025432 A).
As per claim 1, Girardot teaches the claimed:
1. A system, comprising:
a processor that executes computer-executable components stored in a non-transitory computer-readable memory, wherein the computer-executable components comprise:
(Girardot (page 12, line 12): “According to a fourth aspect, a device is proposed comprising one or more processors and at least one storage medium readable by the processor(s). … The storage medium includes a set of program code instructions which, when the program is executed by the processor(s)”.
(Girardot (page 20, line 18): “This generation 104 of artificial segmentation masks 23 can be implemented by an electronic device, such as a computer for example. The device comprises for example one or more processors and at least one storage medium readable by the processor(s).”)
an access component that accesses a scanned medical image depicting an anatomical structure of a medical patient; and
(Girardot (Abstract): “The method (100) comprises generating (101) majority segmentation masks associated with real medical images without anomaly”.
(Girardot (page 7, line 31): “The term "real medical image" means a medical image acquired on a patient using a medical imaging device”.
Girardot teaches the generation of a segmentation mask associated with the real medical images, and in order to generate these masks the user must have accessed these medical images at one point, which corresponds to the access component that accesses the medical images.)
Girardot alone does not explicitly teach the remaining claim limitations.
However, Girardot in combination with Feldman teaches the claimed:
a synthesis component that generates, via a diffusion neural network executed in a truncated reverse-diffusion process beginning at an intermediate level of noise rather than full noise, a synthetic version of the scanned medical image,
(Feldman (page 17, line 31): “The diffusion module (208) can use the object masks to spatially blend the noisy versions of the complete object with the corresponding noisy versions of the inpainted image. For example, the diffusion module (208) can use the object masks to blend each noisy version of the complete object with each corresponding noisy version of the inpainted image, where the object masks delineate the boundaries of the complete object such that the object masks delineate areas where the object masks are modified during the blending process. In some embodiments, the diffusion process can include local complete object-induced diffusion where an image generation loss determined during the training process is used under the object masks during the local object generation diffusion.”
Feldman (page 18, line 5): “The diffusion module (208) restores consistency by performing a denoising diffusion step after each blend and projecting it onto the next manifold. Once the spatial blending is complete, the diffusion module (208) preserves the background by replacing the area outside the object mask with the corresponding area from the inpainted image.”
Feldman teaches a process, described in the passages above, called a blended diffusion process where an image transitions through the noisier manifolds, but never to the point of full noise as it maintains an anchor to real data, which is the inpainted image. Therefore, since this forward diffusion process does not fully destroy and noise the image, the starting point of the reverse diffusion process would start at an intermediate noise level.)
wherein the synthetic version of the scanned medical image depicts the anatomical structure exhibiting a foreign object.
(Girardot (Abstract): “The method (100) … generating (102) minority segmentation masks associated with real medical images with an anomaly, training (103) a neural network to generate a synthetic medical image on the basis of a segmentation mask, generating (104) artificial segmentation masks on the basis of the majority and minority segmentation masks by combining a segmentation of the anatomy of interest by a majority segmentation mask with a segmentation of the anomaly by a minority segmentation mask, and generating (105) synthetic medical images on the basis of the artificial segmentation masks and using the previously trained neural network.”
Girardot teaches generating segmentation masks (synthetic version) associated with the real medical images (scanned medical image) with an anomaly (exhibits a foreign object).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the diffusion module as taught by Feldman with the system of Girardot in order to blend an object seamlessly without any sharp edges or mismatched textures and output a modified image where the object is merged in with the surrounding areas of the background image to create a more realistic appearance that is based on an actual real image.
As per claim 10, this claim is similar in scope to limitations recited in claim 1, and thus is rejected under the same rationale.
As per claim 2, Girardot and Feldman teaches the claimed:
2. The system of claim 1, wherein the synthesis component:
pastes or blends the foreign object into the scanned medical image, thereby yielding a post-paste or post-blend image;
(Feldman (page 5, line 13): “The diffusion model blends one or more versions of the inpainted image with one or more versions of the complete object using the object mask to output a modified image in which the complete object is seamlessly merged with the inpainted image.”
Feldman teaches blending an object onto an inpainted image to output a modified image with the object merged within the inpainted image, which corresponds to the blending of the foreign object into the scanned medical image resulting in a post-paste/blend image.)
iteratively inserts, via a truncated forward-diffusion process, noise into the post-paste or post-blend image, thereby yielding a sequence of progressively-noisier versions of the post-paste or post-blend image, wherein a noisiest version of the post-paste or post-blend image in the sequence of progressively-noisier versions of the post-paste or post-blend image is not full noise; and
(Feldman (page 17, line 31): “The diffusion module (208) can use the object masks to spatially blend the noisy versions of the complete object with the corresponding noisy versions of the inpainted image. For example, the diffusion module (208) can use the object masks to blend each noisy version of the complete object with each corresponding noisy version of the inpainted image, where the object masks delineate the boundaries of the complete object such that the object masks delineate areas where the object masks are modified during the blending process. In some embodiments, the diffusion process can include local complete object-induced diffusion where an image generation loss determined during the training process is used under the object masks during the local object generation diffusion. … In some embodiments, the diffusion module (208) generates progressively noisier versions of the complete object compared to previous versions, and progressively noisier versions of the inpainted image compared to previous versions.”
Feldman teaches the forward diffusion process that generates progressively noisier versions of the object and background. In addition to that, the workflow Feldman teaches noises the inpainted image (background) and the object only partially since the blending is restricted. If the entire image becomes full noise, then there would be no distinct boundary or background. Feldman also states the local complete object induced diffusion that forces the generation only within the object masks, while outside the masks the background is preserved in its noisy state.)
iteratively executes the diffusion neural network in the truncated reverse-diffusion process, wherein the truncated reverse-diffusion process begins with the noisiest version of the post-paste or post-blend image, and wherein a final time-step output of the truncated reverse-diffusion process is the synthetic version of the scanned medical image.
(Feldman (page 18, line 5): “The diffusion module (208) generates an incrementally denoised version of the complete object compared to the previous version and an incrementally denoised version of the inpainted image compared to the previous version. For example, a backward Markovian process transforms Gaussian noise samples by iteratively denoising the inpainted image using a learned posterior. Each step of the denoising diffusion process projects the noisy image onto the next less noisy manifold. … The diffusion module (208) restores consistency by performing a denoising diffusion step after each blend and projecting it onto the next manifold. Once the spatial blending is complete, the diffusion module (208) preserves the background by replacing the area outside the object mask with the corresponding area from the inpainted image.”
Feldman teaches incrementally executing the denoising diffusion step, and since Feldman states that the diffusion module generates a denoised version of a previous version, therefore this denoising diffusion process must begin at the noisiest version. Feldman also shows that during this process the object and background are blended together so that they share the same texture and noise at the end. After that, the diffusion module uses the object mask to identify and preserve it and then pastes it onto the inpainted image, creating a synthetic image of the preserved background with an object blended realistically onto it.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the diffusion module as taught by Feldman with the system of Girardot in order to blend an object seamlessly without any sharp edges or mismatched textures and output a modified image where the object is merged in with the surrounding areas of the background image to create a more realistic appearance.
As per claim 11, this claim is similar in scope to limitations recited in claim 2, and thus is rejected under the same rationale.
As per claim 3, Girardot and Feldman teaches the claimed:
3. The system of claim 2, wherein the post-paste or post-blend image depicts one or more pasting or blending artifacts, wherein the one or more pasting or blending artifacts are not visibly discernible in the noisiest version of the post-paste or post-blend image, and wherein the anatomical structure and the foreign object are nevertheless visibly discernible in the noisiest version of the post-paste or post-blend image.
(Girardot (page 18, line 31): “Several minority segmentation masks 21 with different semantic values (i.e. with different types of anomaly: tumor, cyst, ablation region, artefact, ...) can be combined with the majority segmentation masks 22 and thus increase the number of potential combinations.”
Girardot teaches the multiple minority segmentations masks (foreign objects) that can be combined onto a majority segmentation mask (background), which then can be used to create a post paste or blend image, corresponding to the post paste or blend image that depicts more than one pasting or blending artifacts.
Feldman (page 17, line 31): “The diffusion module (208) can use the object masks to spatially blend the noisy versions of the complete object with the corresponding noisy versions of the inpainted image. For example, the diffusion module (208) can use the object masks to blend each noisy version of the complete object with each corresponding noisy version of the inpainted image, where the object masks delineate the boundaries of the complete object such that the object masks delineate areas where the object masks are modified during the blending process. In some embodiments, the diffusion process can include local complete object-induced diffusion where an image generation loss determined during the training process is used under the object masks during the local object generation diffusion.”
Similar to the claim above, Feldman teaches the pasting or blending of artifacts that are not visibly discernable as stated “where the object masks delineate the boundaries of the complete object”, therefore showing that when pasting or blending the objects onto the background, the sharp and rigid edges will not be seen. Feldman also teaches that the background (anatomical structure) and objects are still visible, even in the noisiest version of the image. Feldman states that the diffusion module can be used to “blend each noisy version of the complete object with each corresponding noisy version of the impainted image.” This states that at every step of the process, even the noisiest version, there is a corresponding version of the background and object that are not allowed to merge, since if the background and objects did, there would be no corresponding version of those parts that are discernable.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the diffusion module as taught by Feldman with the system of Girardot in order to blend an object seamlessly without any sharp edges or mismatched textures and output a modified image where the object is merged in with the surrounding areas of the background image to create a more realistic appearance.
As per claim 12, this claim is similar in scope to limitations recited in claim 3, and thus is rejected under the same rationale.
As per claim 5, Girardot and Feldman teaches the claimed:
5. The system of claim 2, wherein, at a current time-step of the truncated reverse-diffusion process, the synthesis component:
accesses a first reverse-diffused image produced during a previous time-step of the truncated reverse-diffusion process; and
executes the diffusion neural network on the first reverse-diffused image, thereby producing a second reverse-diffused image that contains incrementally less noise than the first reverse-diffused image, wherein the second reverse-diffused image is treated as input for the diffusion neural network during a succeeding time-step of the truncated reverse-diffusion process.
(Feldman (page 18, line 5): “The diffusion module (208) generates an incrementally denoised version of the complete object compared to the previous version and an incrementally denoised version of the inpainted image compared to the previous version. For example, a backward Markovian process transforms Gaussian noise samples by iteratively denoising the inpainted image using a learned posterior. Each step of the denoising diffusion process projects the noisy image onto the next less noisy manifold.”
Feldman teaches each step of the denoising diffusion process that projects the noisy image onto the next less noisy manifold, which corresponds to the first reverse diffused image produced from a previous timestep. Feldman also teaches the diffusion module that executes by denoising the image incrementally compared to the previous version, which implies that the previous version must be the input to produce an output that has less noise in the succeeding timestep.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the diffusion module as taught by Feldman with the system of Girardot in order to blend an object seamlessly without any sharp edges or mismatched textures and output a modified image where the object is merged in with the surrounding areas of the background image to create a more realistic appearance.
As per claim 14, this claim is similar in scope to limitations recited in claim 5, and thus is rejected under the same rationale.
As per claim 6, Girardot and Feldman teaches the claimed:
6. The system of claim 2, wherein the synthesis component:
overlays a mask onto the post-paste or post-blend image, such that the mask circumscribes the foreign object but does not cover an entirety of the post-paste or post-blend image; and
(already taught in claim 5)
replaces an unmasked portion of the second reverse-diffused image with an unmasked portion of whichever one of the sequence of progressively-noisier versions of the post-paste or post-blend image corresponds to a succeeding time-step of the truncated reverse-diffusion process, thereby yielding a third reverse-diffused image that is treated as input for the diffusion neural network during the succeeding time-step.
(Feldman (page 17, line 31): “The diffusion module (208) can use the object masks to spatially blend the noisy versions of the complete object with the corresponding noisy versions of the inpainted image. For example, the diffusion module (208) can use the object masks to blend each noisy version of the complete object with each corresponding noisy version of the inpainted image, where the object masks delineate the boundaries of the complete object such that the object masks delineate areas where the object masks are modified during the blending process.”
Feldman teaches the object mask, which only covers the portion of the object, not the whole background image. The object mask is then used to blend a corresponding noisy version of the inpainted image with the object. This corresponding noisy version of the inpainted image corresponds to the succeeding timestep of the reverse diffusion process as Feldman () states “The diffusion module (208) generates … an incrementally denoised version of the inpainted image compared to the previous version.” And similarly in claim 5, as Feldman () states “Each step of the denoising diffusion process projects the noisy image onto the next less noisy manifold.”, which corresponds to this new reverse diffused image that will be used as input for the succeeding step in the process.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the diffusion module with the object mask as taught by Feldman with the system of Girardot in order to blend an object seamlessly without any sharp edges or mismatched textures and output a modified image where the object is merged in with the surrounding areas of the background image to create a more realistic appearance. In addition to that, be more selective of which parts of the diffused image to keep more preserved of the original image.
As per claim 15, this claim is similar in scope to limitations recited in claim 6, and thus is rejected under the same rationale.
As per claim 7, Girardot teaches the claimed:
7. The system of claim 1, wherein the computer-executable components comprise:
an object component that:
selects, based on execution of a large language model, the foreign object from a foreign object library; or
(Girardot (page 18, line 19): “In the example illustrated in FIG. 6, the generation step 104 of an artificial segmentation mask 23 comprises: a selection of a majority segmentation mask 21, a selection of a minority segmentation mask 22, … The majority segmentation mask 21 and the minority segmentation mask 22 can be selected randomly. The number of combinations of artificial segmentation masks can thus amount to the product of the number of majority segmentation masks 21 by the number of minority segmentation masks 22.”
Girardot, in FIG. 6, shows different minority segmentation masks 22 that represent different anomalies, where one is chosen to create a new segmentation mask that incorporates the foreign object on a medical image.)
augments, based on execution of the large language model, the foreign object via a geometric or intensity-based transformation.
(Girardot (page 19, line 18): “As illustrated in FIG. 7, the generation step 104 of artificial segmentation masks 23 can comprise a transformation of the segmentation of the anomaly 25 of the minority segmentation mask 22 selected. Such arrangements make it possible to generate a greater number of different artificial segmentation masks. The transformation of the segmentation of the anomaly 25 corresponds for example to a rotation (as illustrated for the artificial segmentation mask 23-1), a displacement (as illustrated for the artificial segmentation mask 23-2), enlargement (as shown for artificial segmentation mask 23-3), reduction (as shown for artificial segmentation mask 23-4), distortion (as shown for artificial segmentation mask 23 -5) or a combination of these different possible transformations (as illustrated for the artificial segmentation mask 23-6 for which the segmentation of the anomaly 25 has been simultaneously moved, reduced, and rotated).”
Girardot, in FIG. 7, shows different types of ways the anomaly 25 can be incorporated onto the medical image, which includes rotations, displacements, enlargements, reductions, etc. that correspond to the geometric transformations.)
As per claim 16, this claim is similar in scope to limitations recited in claim 7, and thus is rejected under the same rationale.
As per claim 8, Girardot teaches the claimed:
8. The system of claim 1, wherein the computer-executable components further comprise:
an action component that trains, on the synthetic version of the scanned medical image, another neural network to perform an inferencing task.
(Girardot (page 23, line 28): “FIG. 13 schematically represents the training step 202, from a synthetic medical image 13, of the automatic learning algorithm 40. The automatic learning algorithm 40 aims to detect or characterize an anomaly in the anatomy of interest of a patient on a medical image. In the example considered, the machine learning algorithm 40 is a deep neural network. … As illustrated in Figure 13, during the training phase, the machine learning algorithm 40 takes as input a medical image (this is a synthetic medical image 13 in the example illustrated in Figure 13). … The information obtained and the information expected are compared and, depending on the result of the comparison, the parameters of the neural network are updated by a backpropagation loop 41. The training is continued until the machine learning algorithm 40 is able to provide the expected information with a satisfactory success rate.”
Girardot teaches the training step (action component that trains), from a synthetic medical image, of the automatic learning algorithm, which is a deep neural network. This neural network works to detect anomalies of interest of a patient, which corresponds to the inference task.)
As per claim 17, this claim is similar in scope to limitations recited in claim 8, and thus is rejected under the same rationale.
As per claim 9, Girardot teaches the claimed:
9. The system of claim 1, wherein the foreign object is a cyst, a lesion, a surgical implant, or an imaging artifact.
(Girardot (page 7, line 27): “An anomaly present within the anatomy of interest generally corresponds to a lesion, such as a tumor, a cyst, an ablation zone, an aneurysm, etc.”)
As per claim 18, this claim is similar in scope to limitations recited in claim 9, and thus is rejected under the same rationale.
As per claim 19, the reasons and rationale for the rejection of claim 1 and 8 is incorporated herein. In particular, only additional features unique to claim 19 that were not present in claim 1 and 8 will be explicitly addressed here.
Girardot teaches the claimed: (strikethrough is taught in claims 1 and 8)
19. A computer program product for facilitating aberrant image synthesis via truncated reverse-diffusion,
(Girardot [Abstract]: “The method (100) … generating (102) minority segmentation masks associated with real medical images with an anomaly, training (103) a neural network to generate a synthetic medical image on the basis of a segmentation mask, generating (104) artificial segmentation masks on the basis of the majority and minority segmentation masks by combining a segmentation of the anatomy of interest by a majority segmentation mask with a segmentation of the anomaly by a minority segmentation mask, and generating (105) synthetic medical images on the basis of the artificial segmentation masks and using the previously trained neural network.”
Girardot teaches generating segmentation masks (synthetic version) associated with the real medical images (scanned medical image) with an anomaly (exhibits a foreign object). This also corresponds to a pathological version of a scanned medical image because a pathological image is one that specifically shows evidence of any defect, anomaly, or disease, whether it be real or synthetic. Since the synthetic image being generated here have anomalies and foreign objects, this can be considered a pathological image of a scanned medical image.)
As per claim 20, the reasons and rationale for the rejection of claim 2 is incorporated herein. In particular, only additional features unique to claim 20 that were not present in claim 2 will be explicitly addressed here.
Girardot teaches the claimed: (strikethrough taught in claim 2)
20. The computer program product of claim 19, wherein the processor generates the pathological version of the scanned medical image by:
(Girardot [Abstract]: “The method (100) … generating (102) minority segmentation masks associated with real medical images with an anomaly, training (103) a neural network to generate a synthetic medical image on the basis of a segmentation mask, generating (104) artificial segmentation masks on the basis of the majority and minority segmentation masks by combining a segmentation of the anatomy of interest by a majority segmentation mask with a segmentation of the anomaly by a minority segmentation mask, and generating (105) synthetic medical images on the basis of the artificial segmentation masks and using the previously trained neural network.”
Similar to claim 19, Girardot teaches generating segmentation masks (synthetic version) associated with the real medical images (scanned medical image) with an anomaly (exhibits a foreign object). This also corresponds to a pathological version of a scanned medical image because a pathological image is one that specifically shows evidence of any defect, anomaly, or disease, whether it be real or synthetic. Since the synthetic image being generated here have anomalies and foreign objects, this can be considered a pathological image of a scanned medical image.)
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Girardot (WO 2022238640 A1) in view of Feldman (KR 20250025432 A) and in further view of Aggarwal (US 20240355018 A1).
As per claim 4, Girardot and Feldman alone does not explicitly teach the claimed limitations.
However, Girardot and Feldman in combination with Aggarwal teaches the claimed:
4. The system of claim 2, wherein the truncated forward-diffusion process comprises a fraction of a total number of time-steps of a forward-diffusion process on which the diffusion neural network was trained.
(Aggarwal [0070]: “In particular, the mask aware image editing system 102 can gradually add noise to an existing real image until some timestep ‘T’ according to the forward diffusion process of the diffusion model. Then, the model is used to run the regular reverse denoising process starting from the noised image at timestep ‘T’ until timestep zero. … Because the denoising process starts from an intermediate timestep, the generated image (based on the intermediate point) will have style information from the guide image while maintaining structure of the original image that was noised. The amount of structure preservation from the original image decreases with increase in the number of noising steps, i.e. higher the T, the lower the structure preservation.”
Aggarwal teaches the forward diffusion process that gradually adds noise until a certain timestep, and a reverse denoising process that denoises starting from that same timestep. Later in the same paragraph Aggarwal mentions that the denoising process starts at an intermediate step, which implies that the forward diffusion process also stopped at that intermediate timestep instead of the full length.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the mask aware image editing system as taught by Aggarwal with the system of Girardot as modified by Feldman in order to decide how much of the original image is to be preserved during the diffusion process by choosing the number of timesteps for the whole process.
As per claim 13, this claim is similar in scope to limitations recited in claim 4, and thus is rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA SUO whose telephone number is (571) 272-8387. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSHUA SUO/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616