DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The reply filed on 4/28/2026 has been entered. Applicant’s arguments with respect to claims 1-24 have been considered but are moot in view of new ground(s) of rejection caused by the amendments. Claims 1-11 and 16-24 are pending in this application and have been considered below. Claims 12-15 are canceled by the applicant. The previous 112 rejections have been withdrawn.
Priority
Receipt is acknowledged that application is a National Stage application of PCT/EP2022/072887. Priority to EP21192182.0 with a priority date of 8/19/2021 is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Information Disclosure Statement
The IDS dated 6/03/2024 that has been previously considered remain placed in the application file.
1st Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obliga+9tion under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 4, 6, 10, and 18 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2019 0030371 A1, (Han) in view of US Patent Publication 2016 0035093 A1, (Kateb et al.) and US Patent Publication 2022 0237785 A1, (Mitchell).
Claim 1
Regarding Claim 1, Han teaches a method for training an artificial neural network using at least one processor, the method comprising: ("The image processing device 212 may include a memory device 216, a processor 214 and a communication interface," par. 28) wherein each image of the plurality of images comprises at least one object of interest, ("The image acquisition device 232 may be configured to acquire one or more images of the patient's anatomy for a region of interest," par. 44) and wherein the at least one object of interest comprises a surgical cavity, a tissue region exhibiting post-operative anatomical distortion, or both the surgical cavity and the tissue region ("Medical images 246 may include information such as imaging data associated with a patient anatomical region, organ, or volume of interest segmentation data. Patient data 245 may include information such as (1) functional organ modeling data (e.g., serial versus parallel organs, appropriate dose response models, etc.); (2) radiation dosage data (e.g., dose-volume histogram (DVH) information; or (3) other clinical information about the patient and course of treatment," par. 31); for each image of the plurality of images, b) receiving a ground-truth pixel-based annotation of the image, wherein the ground-truth pixel-based annotation comprises a ground-truth segmentation mask for the at least one object of interest, ("The 3D ground truth label map may be divided to sequential 2D ground truth label maps, respectively corresponding to the sequential stacks of adjacent 2D images, and pixels of the 2D ground truth label maps are associated with known anatomical structures," par. 59) c) obtaining a predicted segmentation mask ("The segmentation unit 403 may use at least one trained CNN model received from CNN model training unit 402 to predict the anatomical structure each voxel of a 4D image represents," par. 63) by feeding the image to the artificial neural network implementing a prediction function, ("The encoding portion 524 of the CNN model 510 may include one or more convolutional layers 528. Each convolutional layer 528 may have a plurality of parameters, such as the width (“W”) and height (“H”) determined by the upper input layer (e.g., the size of the input of convolutional layer 528), and a count of filters or kernels (“N”) in the layer and their sizes," par. 72) d) calculating a loss using a training function, when the predicted segmentation mask and the ground-truth segmentation mask are given as input to the training function; ("During the training of CNN model 510, the loss layer may determine how the network training penalizes the deviation between the predicted 2D label map and the 2D ground truth
PNG
media_image1.png
288
396
media_image1.png
Greyscale
label map," par. 81) e) optimizing the model parameters of the prediction function by minimizing the loss with respect to the model parameters for the plurality of the images based on the loss for each image; ("The model 700A parameters may be established from training data, such as by minimizing a loss function," [AltContent: textbox (Figure 7 shows the prediction and training arms workflow of the DCNN. )]par. 88) f) replacing the model parameters of the prediction function with the optimized model parameters; ("The model 700A parameters may be established from training data, such as by minimizing a loss function," par. 88) and outputting the artificial neural network ("A loss function may be used to establish model 700A parameters as mentioned above in the training arm 772, or other techniques may be used to establish the model 700A parameters. The model may be provided for use in a prediction arm," par. 89).
Han does not explicitly teach all of a) receiving a plurality of images from Magnetic Resonance Imaging (MRI) brain scan sequences of post-surgical Glioblastoma patients, and wherein model parameters of the prediction function are randomly initialized.
However, Kateb et al. teach a) receiving a plurality of images from Magnetic Resonance Imaging (MRI) ("first images obtained from biopsy, Infrared Imaging, Ultraviolet Imaging, Diffusion Tensor Imaging (DTI), Computed Tomography (CT), Magnetic Resonance Imaging (MRI)," par. 12) brain scan sequences of post-surgical ("One or more first images can comprise a pre-operative image and/or an intra-operative image and/or a post operative image of one or more patients," par. 13) Glioblastoma patients ("images of the tissue 108 (e.g., the brain, e.g., the same tissue as imaged in FIG. 1A, or different tissue as compared to the tissue imaged in FIG. 1A) including a diseased/abnormal region 110 (e.g., a cancerous tumor)," par. 33) ("patients with glioblastoma," par. 8).
Therefore, taking the teachings of Han and Kateb et al. as a whole, it would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the segmentation and prediction architecture as taught by Han to use post-surgical glioblastoma patient images as taught by Kateb et al. The suggestion/motivation for doing so would have been that, “Since real time intra-operative mapping of brain cancer and epileptic areas is critical in removing the abnormal tissue and leaving the healthy tissue intact, there is a great need for the multimodality intra-operative optical imaging technology according to one or more embodiments of the invention” as noted by the Kateb et al. disclosure in paragraph [0016], which also motivates combination because the combination would predictably have additional utility as there is a reasonable expectation that that using the post-surgical patient images would successfully train or operate the prediction architecture to yield highly accurate real-time tissue mapping; and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Additionally, Mitchell teach wherein model parameters of the prediction function are randomly initialized ("The segmenting computer 150 initializes each of the multiple instances of the neural network with respectively randomized weight parameters," par. 90).
Therefore, taking the teachings of Han, Kateb et al., and Mitchell as a whole, it would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the segmentation and prediction architecture as taught by Han to use post-surgical glioblastoma patient images as taught by Kateb et al. and random parameter initialization as taught by Mitchell. At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify Han’s segmentation and prediction architecture to include Mitchell’s random parameter initialization because such a modification is based on the use of known techniques to improve similar devices in the same way. More specifically, Mitchell’s teaching of initializing instances of a neural network with randomized weights is comparable to Han’s method of establishing model parameters. Therefore, it is within the capabilities of one of ordinary skill in the art to modify Han’s segmentation and prediction architecture to include Mitchell’s random parameter initialization with the predictable result of improving the robustness and accuracy of the segmentation results by reducing overfitting and avoiding local minima.
Claim 4
Regarding Claim 4, Han, Kateb et al., and Mitchell the method of claim 1 as noted above.
[AltContent: textbox (Figure 4 shows the image segmentation system using multiple CNN models. )]
PNG
media_image2.png
382
448
media_image2.png
Greyscale
Han teaches wherein the prediction function is a single ensemble of multiple base models, and wherein the method is performed for each base model ("FIG. 4 illustrates an example including an image segmentation system 400 for segmenting 3D images, such as using first and second CNN models, as mentioned in relation to other examples described in this document," par. 57).
Han, Kateb et al., and Mitchell are combined as per claim 1.
Claim 6
Regarding Claim 6, Han, Kateb et al., and Mitchell the method of claim 1 as noted above.
Han teaches wherein the training is performed on four separate sets of received images, wherein the sets are defined based on ranges of volume distributions of a contrast-enhancing tumor and regions of edema ("a computer-implemented method for segmentation of anatomical features from 3D medical imaging information may include receiving the three-dimensional (3D) medical imaging information defining a first volume, applying a first trained convolutional neural network (CNN) to the three-dimensional medical imaging information, using an output from the first trained CNN determine a region-of-interest within the first volume, the region-of-interest defining a lesser, second volume, applying a different, second trained CNN to the region-of-interest … at least one of the first and second loss functions comprises a cross-entropy loss function, and the second CNN provides enhanced segmentation detail as compared to the first CNN when the first and second CNNs are applied serially to the 3D medical imaging information," par. 6 and 7, wherein additional CNNs processing additional sets of images is an obvious addition to increase segmentation accuracy/detail).
Han, Kateb et al., and Mitchell are combined as per claim 1.
Claim 10
Regarding Claim 10, Han, Kateb et al., and Mitchell the method of claim 1 as noted above.
Han teaches wherein optimizing the model parameters is performed using a stochastic gradient descent ("Each of the first and second DCNN models (e.g., the 2.5D model and the 3D model) were trained from scratch using a stochastic gradient descent with momentum optimization," par. 95).
Han, Kateb et al., and Mitchell are combined as per claim 1.
Claim 18
Regarding Claim 18, Han teaches a computer-program product tangibly embodied in a non-transitory machine-readable medium, including instructions configured to cause one or more processors to perform operations comprising: ("The memory device 216 may be a non-transitory computer-readable medium … or any other non-transitory medium that may be used to store information including image, data, or computer executable instructions (e.g., stored in any format) capable of being accessed by the processor," par. 34) wherein each image of the plurality of images comprises at least one object of interest, ("The image acquisition device 232 may be configured to acquire one or more images of the patient's anatomy for a region of interest," par. 44) and wherein the at least one object of interest comprises a surgical cavity, a tissue region exhibiting post-operative anatomical distortion, or both the surgical cavity and the tissue region ("Medical images 246 may include information such as imaging data associated with a patient anatomical region, organ, or volume of interest segmentation data. Patient data 245 may include information such as (1) functional organ modeling data (e.g., serial versus parallel organs, appropriate dose response models, etc.); (2) radiation dosage data (e.g., dose-volume histogram (DVH) information; or (3) other clinical information about the patient and course of treatment," par. 31); for each image of the plurality of images, b) receiving a ground-truth pixel-based annotation of the image, wherein the ground-truth pixel-based annotation comprises a ground-truth segmentation mask for the at least one object of interest, ("The 3D ground truth label map may be divided to sequential 2D ground truth label maps, respectively corresponding to the sequential stacks of adjacent 2D images, and pixels of the 2D ground truth label maps are associated with known anatomical structures," par. 59) c) obtaining a predicted segmentation mask ("The segmentation unit 403 may use at least one trained CNN model received from CNN model training unit 402 to predict the anatomical structure each voxel of a 4D image represents," par. 63) by feeding the image to the artificial neural network implementing a prediction function, ("The encoding portion 524 of the CNN model 510 may include one or more convolutional layers 528. Each convolutional layer 528 may have a plurality of parameters, such as the width (“W”) and height (“H”) determined by the upper input layer (e.g., the size of the input of convolutional layer 528), and a count of filters or kernels (“N”) in the layer and their sizes," par. 72) d) calculating a loss using a training function, when the predicted segmentation mask and the ground-truth segmentation mask are given as input to the training function; ("During the training of CNN model 510, the loss layer may determine how the network training penalizes the deviation between the predicted 2D label map and the 2D ground truth label map," par. 81) e) optimizing the model parameters of the prediction function by minimizing the loss with respect to the model parameters for the plurality of the images based on the loss for each image; ("The model 700A parameters may be established from training data, such as by minimizing a loss function," par. 88) f) replacing the model parameters of the prediction function with the optimized model parameters; ("The model 700A parameters may be established from training data, such as by minimizing a loss function," par. 88) and outputting the artificial neural network ("A loss function may be used to establish model 700A parameters as mentioned above in the training arm 772, or other techniques may be used to establish the model 700A parameters. The model may be provided for use in a prediction arm," par. 89).
Han does not explicitly teach all of receiving a plurality of images from Magnetic Resonance Imaging (MRI) brain scan sequences of post-surgical Glioblastoma patients, and wherein model parameters of the prediction function are randomly initialized.
However, Kateb et al. teach receiving a plurality of images from Magnetic Resonance Imaging (MRI) ("first images obtained from biopsy, Infrared Imaging, Ultraviolet Imaging, Diffusion Tensor Imaging (DTI), Computed Tomography (CT), Magnetic Resonance Imaging (MRI)," par. 12) brain scan sequences of post-surgical ("One or more first images can comprise a pre-operative image and/or an intra-operative image and/or a post operative image of one or more patients," par. 13) Glioblastoma patients ("images of the tissue 108 (e.g., the brain, e.g., the same tissue as imaged in FIG. 1A, or different tissue as compared to the tissue imaged in FIG. 1A) including a diseased/abnormal region 110 (e.g., a cancerous tumor)," par. 33) ("patients with glioblastoma," par. 8).
Additionally, Mitchell teach wherein model parameters of the prediction function are randomly initialized ("The segmenting computer 150 initializes each of the multiple instances of the neural network with respectively randomized weight parameters," par. 90).
Han, Kateb et al., and Mitchell are combined as per claim 1.
2nd Claim Rejections - 35 USC § 103
Claim 2 is rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2019 0030371 A1, (Han), US Patent Publication 2016 0035093 A1, (Kateb et al.), and US Patent Publication 2022 0237785 A1, (Mitchell) in view of US Patent Publication 2022 051402 A1, (Dikici et al.).
Claim 2
Regarding Claim 2, Han, Kateb et al., and Mitchell teach the method of claim 1 as noted above.
Han, Kateb et al., and Mitchell do not explicitly teach wherein the plurality of images are obtained from a native Tl-weighted sequence, a post- contrast Tl-weighted sequence, a T2-weighted sequence, a T2-Fluid Attenuated Inversion Recovery MRI sequence, or any combination thereof.
However, Dikici et al. teach wherein the plurality of images are obtained from a native Tl-weighted sequence, a post- contrast Tl-weighted sequence, a T2-weighted sequence, a T2-Fluid Attenuated Inversion Recovery MRI sequence, or any combination thereof ("MRI datasets as input, including post-contrast T1-weighted 3D, T2-weighted 2D fluid-attenuated inversion recovery, and T1-weighted 2D sequences," par. 83).
Therefore, taking the teachings of Han, Kateb et al., and Mitchell as a whole, it would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the segmentation and prediction architecture as taught by Han, post-surgical glioblastoma patient images as taught by Kateb et al., and random parameter initialization as taught by Mitchell. to use the MRI sequence datasets as taught by Dikici et al. The suggestion/motivation for doing so would have been that, “Accordingly, computer-aided detection approaches have been proposed to assist radiologists by automatically segmenting and/or detecting BM in contrast-enhanced Magnetic Resonance Imaging (MRI) sequences, which is the key modality for the detection, characterization, and monitoring of BM. To this end, the most important imaging sequence is a T1-weighted image acquisition following intravenous administration of a gadolinium-based contrast agent. This sequence is particularly helpful for demonstrating vascularity within lesions as seen with BMs” as noted by the Dikici et al. disclosure in paragraph [0080], which also motivates combination because the combination would predictably have a higher accuracy as there is a reasonable expectation that incorporating T1-weighted contrast-enhanced MRI sequences would yield the necessary hyper-intense visual signatures that naturally correspond to tumor margins, allowing the combined architecture to accurately delineate lesion boundaries; and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
3rd Claim Rejections - 35 USC § 103
Claims 3 and 19 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2019 0030371 A1, (Han), US Patent Publication 2016 0035093 A1, (Kateb et al.), and US Patent Publication 2022 0237785 A1, (Mitchell) in view of US Patent Publication 2020 0364509 A1, (Weinzaepfel et al.).
Claim 3
Regarding Claim 3, Han, Kateb et al., and Mitchell teach the method of claim 1 as noted above.
Han teaches wherein each of the plurality of images comprises multiple objects of interest, including a contrast-enhancing tumor, regions of edema, and the surgical cavity, ("Medical images 246 may include information such as imaging data associated with a patient anatomical region, organ, or volume of interest segmentation data. Patient data 245 may include information such as (1) functional organ modeling data (e.g., serial versus parallel organs, appropriate dose response models, etc.); (2) radiation dosage data (e.g., dose-volume histogram (DVH) information; or (3) other clinical information about the patient and course of treatment," par. 31) and wherein the training function is averaged over all objects of interest ("The segmentation unit 403 may use at least one trained CNN model received from CNN model training unit 402 to predict the anatomical structure each voxel of a 4D image represents. When the image segmentation is completed, segmentation unit 403 may output a 3D label map, associating each voxel of the 3D image to an anatomical structure," par. 63) ("Additionally, the ReLu layer may reduce or avoid saturation during a backpropagation training process," par. 75 wherein backpropagation includes averaging the loss over each training function).
Han, Kateb et al., and Mitchell do not explicitly teach all of wherein one prediction function per object of interest and one training function per object of interest are implemented.
However, Weinzaepfel et al. teach wherein one prediction function per object of interest and one training function per object of interest are implemented ("Branch A, after being processed by a convolution neural network (500 of FIG. 7), a convolution neural network being a neural network composed of interleaving convolution layers and rectified linear units, predicts binary segmentation for each object-of-interest (700 of FIG. 7) and x and y reference image coordinates regression, with an object-of-interest specific regressor (800 and 900, respectively, of FIG. 7). In one embodiment, the convolution neural network predicts segmentation and correspondences on a dense grid of size 56×56 in each box which is then interpolated to obtain a per-pixel prediction," par. 57).
Therefore, taking the teachings of Han, Kateb et al., Mitchell, and Weinzaepfel et al. as a whole, it would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the segmentation and prediction architecture as taught by Han, post-surgical glioblastoma patient images as taught by Kateb et al., and random parameter initialization as taught by Mitchell to use the one vs rest classifier as taught by Weinzaepfel et al. The suggestion/motivation for doing so would have been that, “Based upon this evaluation, homography data augmentation, at training, allows the generation of more viewpoints of the objects-of-interest, and thus, enabling improved detection and matching of objects-of-interest at test time with novel viewpoints” as noted by the Weinzaepfel et al. disclosure in paragraph [0093], which also motivates combination because the combination would predictably have a higher accuracy as there is a reasonable expectation that more objects of interest will be detected/predicted in the images; and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 19
Regarding Claim 19, Han, Kateb et al., and Mitchell teach the computer-program product of claim 18 as noted above.
Han teaches wherein each of the plurality of images comprises multiple objects of interest, including a contrast-enhancing tumor, regions of edema, and the surgical cavity, ("Medical images 246 may include information such as imaging data associated with a patient anatomical region, organ, or volume of interest segmentation data. Patient data 245 may include information such as (1) functional organ modeling data (e.g., serial versus parallel organs, appropriate dose response models, etc.); (2) radiation dosage data (e.g., dose-volume histogram (DVH) information; or (3) other clinical information about the patient and course of treatment," par. 31) and wherein the training function is averaged over all objects of interest ("The segmentation unit 403 may use at least one trained CNN model received from CNN model training unit 402 to predict the anatomical structure each voxel of a 4D image represents. When the image segmentation is completed, segmentation unit 403 may output a 3D label map, associating each voxel of the 3D image to an anatomical structure," par. 63) ("Additionally, the ReLu layer may reduce or avoid saturation during a backpropagation training process," par. 75 wherein backpropagation includes averaging the loss over each training function).
Han, Kateb et al., and Mitchell do not explicitly teach all of wherein one prediction function per object of interest and one training function per object of interest are implemented.
However, Weinzaepfel et al. teach wherein one prediction function per object of interest and one training function per object of interest are implemented ("Branch A, after being processed by a convolution neural network (500 of FIG. 7), a convolution neural network being a neural network composed of interleaving convolution layers and rectified linear units, predicts binary segmentation for each object-of-interest (700 of FIG. 7) and x and y reference image coordinates regression, with an object-of-interest specific regressor (800 and 900, respectively, of FIG. 7). In one embodiment, the convolution neural network predicts segmentation and correspondences on a dense grid of size 56×56 in each box which is then interpolated to obtain a per-pixel prediction," par. 57).
Han, Kateb et al., Mitchell, and Weinzaepfel et al. are combined as per claim 3.
4th Claim Rejections - 35 USC § 103
Claims 5, 7, 8, 11, and 20 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2019 0030371 A1, (Han), US Patent Publication 2016 0035093 A1, (Kateb et al.), and US Patent Publication 2022 0237785 A1, (Mitchell) in view of Isenee, F. et al., "nnU-Net for Brain Tumor Segmentation" ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, NY:1-15 (Nov 2, 2020) (Submitted by applicant in IDS dated 6/03/2024).
Claim 5
Regarding Claim 5, Han, Kateb et al., and Mitchell teach the method of claim 4 as noted above.
Han, Kateb et al., and Mitchell do not explicitly teach all of wherein the prediction function is a single ensemble of five confidence-aware nnU-Nets, and wherein the training is performed for each nnU-Net.
[AltContent: textbox (Figure 1 shows the five downsampling operations network architecture.)]
PNG
media_image3.png
256
674
media_image3.png
Greyscale
However, Isenee et al. teach wherein the prediction function is a single ensemble of five confidence-aware nnU-Nets, and wherein the training is performed for each nnU-Net ("A total of five downsampling operations are performed, resulting in a feature map size 4 x 4 x 4 in the bottleneck," sec. 2.2, pg. 4).
Therefore, taking the teachings of Han, Kateb et al., Mitchell, and Isenee et al. as a whole, it would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the segmentation and prediction architecture as taught by Han, post-surgical glioblastoma patient images as taught by Kateb et al., and random parameter initialization as taught by Mitchell to use the nnU-Net training architecture as taught by Isenee et al. The suggestion/motivation for doing so would have been that, “Ve recently proposed nnU-Net [25], a general purpose segmentation method that automatically configures segmentation pipelines for arbitrary biomedical datasets. nnU-Net set new state of the art results on the majority of the 23 datasets it was tested on, underlining the effectiveness of this approach” as noted by the Isenee et al. disclosure in section 1, pg. 2, which also motivates combination because the combination would predictably have a more utility for general segmentation as there is a reasonable expectation that nnU-Net architectures automatically configures segmentation pipelines based on dataset properties; and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 7
Regarding Claim 7, Han, Kateb et al., and Mitchell teach the method of claim 1 as noted above.
Han, Kateb et al., and Mitchell do not explicitly teach all of wherein the training is performed for at least 500 epochs.
However, Isenee et al. teach wherein the training is performed for at least 500 epochs ("Training runs for a total of 1000 epochs, where one epoch is defined as 250 iterations," sec. 2.2, pg.4).
Han, Kateb et al., Mitchell, and Isenee et al. are combined as per claim 5.
Claim 8
Regarding Claim 8, Han, Kateb et al., and Mitchell teach the method of claim 1 as noted above.
Han, Kateb et al., and Mitchell do not explicitly teach all of wherein within one epoch at least 100 batches are processed.
However, Isenee et al. teach wherein within one epoch at least 100 batches are processed ("Training runs for a total of 1000 epochs, where one epoch is defined as 250 iterations," sec. 2.2, pg.4).
Han, Kateb et al., Mitchell, and Isenee et al. are combined as per claim 5.
Claim 11
Regarding Claim 11, Han, Kateb et al., and Mitchell teach the method of claim 10 as noted above.
Han, Kateb et al., and Mitchell do not explicitly teach all of wherein the stochastic gradient descent is performed with Nesterov momentum within a range between 0.9 and 0.99.
However, Isenee et al. teach wherein the stochastic gradient descent is performed with Nesterov momentum within a range between 0.9 and 0.99 ("nnU-Net uses stochastic gradient descent with an initial learning rate of 0.01 and a Nesterov momentum of 0.99," sec 2.2, pg. 4).
Han, Kateb et al., Mitchell, and Isenee et al. are combined as per claim 5.
Claim 20
Regarding Claim 20, Han, Kateb et al., and Mitchell teach the computer-program product of claim 18 as noted above.
Han, Kateb et al., and Mitchell do not explicitly teach all of wherein the prediction function is a single ensemble of five confidence-aware nnU-Nets.
However, Isenee et al. teach wherein the prediction function is a single ensemble of five confidence-aware nnU-Nets ("A total of five downsampling operations are performed, resulting in a feature map size 4 x 4 x 4 in the bottleneck," sec. 2.2, pg. 4).
Han, Kateb et al., Mitchell, and Isenee et al. are combined as per claim 5.
5th Claim Rejections - 35 USC § 103
Claim 9 is rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2019 0030371 A1, (Han), US Patent Publication 2016 0035093 A1, (Kateb et al.), US Patent Publication 2022 0237785 A1, (Mitchell), and Isenee, F. et al., "nnU-Net for Brain Tumor Segmentation" ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, NY:1-15 (Nov 2, 2020) (Submitted by applicant in IDS dated 6/03/2024) in view of US Patent Publication 2022 051402 A1, (Dikici et al.).
Claim 9
Regarding Claim 9, Han, Kateb et al., Mitchell, and Isenee et al. teach the method of claim 8 as noted above.
Han, Kateb et al., Mitchell, and Isenee et al. do not explicitly teach all of wherein the training is performed by applying data augmentation on the images with (i)a random patch scaling within a range between 0.7 and 1.4, (ii) a random rotation, {iii) a random gamma correction within a range between 0.7 and 1.5, or (iv) a random mirroring.
However, Dikici et al. teach wherein the training is performed by applying data augmentation on the images with (i)a random patch scaling within a range between 0.7 and 1.4, (ii) a random rotation, {iii) a random gamma correction within a range between 0.7 and 1.5, or (iv) a random mirroring ("Each positive sample goes through augmentation process: (B-1) mid-axial slice of an original cropped sample, (B-2) random elastic deformation is applied, (B-3) random gamma correction is applied, (B-4) sample volume is randomly flipped, and (B-5) sample volume is randomly rotated," par. 31).
Therefore, taking the teachings of Han, Kateb et al., Mitchell, and Isenee et al. as a whole, it would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the segmentation and prediction architecture as taught by Han, post-surgical glioblastoma patient images as taught by Kateb et al., random parameter initialization as taught by Mitchell, and the nnU-Net training architecture as taught by Isenee et al. to use random rotation as taught by Dikici et al. The suggestion/motivation for doing so would have been that, “In the proposed framework, a form of the region-based strategy is introduced; random gamma corrections are applied to cropped volumetric regions during the augmentation stage [38]. Accordingly, the framework (1) does not make any assumptions about the histogram shape or intensity characteristics of given MRI datasets, and (2) avoids losing or corrupting potentially valuable intensity features, which is a common disadvantage of image intensity normalization-based methods” as noted by the Dikici et al. disclosure in paragraph [0108], which also motivates combination because the combination would predictably have a higher accuracy as there is a reasonable expectation that all valuable features of the dataset will be preserved; and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
6th Claim Rejections - 35 USC § 103
Claim 21 is rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2019 0030371 A1, (Han) and US Patent Publication 2016 0035093 A1, (Kateb et al.) in view of US Patent Publication 2021 090257 A1, (Bhatia et al.).
Claim 21
Regarding Claim 21, Han teaches a system, comprising: one or more processors; ("The image processing device 212 may include a memory device 216, a processor 214 and a communication interface," par. 28) and one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform operations comprising: ("The memory device 216 may store computer-executable instructions," par. 28) generating, by an artificial neural network, a test segmentation mask ("The 3D ground truth label map may be divided to sequential 2D ground truth label maps, respectively corresponding to the sequential stacks of adjacent 2D images, and pixels of the 2D ground truth label maps are associated with known anatomical structures," par. 59) wherein the artificial neural network has been trained by minimizing a loss with respect to predicted and ground-truth segmentation masks ("During the training of CNN model 510, the loss layer may determine how the network training penalizes the deviation between the predicted 2D label map and the 2D ground truth label map," par. 81)("The model 700A parameters may be established from training data, such as by minimizing a loss function," par. 88) of a plurality of training images, and wherein each image depicts at least one object of interest comprises a surgical cavity, a tissue region exhibiting post- operative anatomical distortion, or both the surgical cavity and the tissue region ("Medical images 246 may include information such as imaging data associated with a patient anatomical region, organ, or volume of interest segmentation data. Patient data 245 may include information such as (1) functional organ modeling data (e.g., serial versus parallel organs, appropriate dose response models, etc.); (2) radiation dosage data (e.g., dose-volume histogram (DVH) information; or (3) other clinical information about the patient and course of treatment," par. 31); extracts a plurality of features from the test segmentation mask, (" Feature extraction portion 520 may extract one or more features of an input stack of adjacent 2D images," par. 66) generating an assessment based on the plurality of features for characterizing a disease status or a treatment response ("The segmentation unit 403 may use at least one trained CNN model received from CNN model training unit 402 to predict the anatomical structure each voxel of a 4D image represents," par. 63).
Kateb et al. teach obtaining a test image obtained from MRI brain scan sequences ("first images obtained from biopsy, Infrared Imaging, Ultraviolet Imaging, Diffusion Tensor Imaging (DTI), Computed Tomography (CT), Magnetic Resonance Imaging (MRI)," par. 12) of a post- surgical ("One or more first images can comprise a pre-operative image and/or an intra-operative image and/or a post operative image of one or more patients," par. 13) Glioblastoma test patient; ("images of the tissue 108 (e.g., the brain, e.g., the same tissue as imaged in FIG. 1A, or different tissue as compared to the tissue imaged in FIG. 1A) including a diseased/abnormal region 110 (e.g., a cancerous tumor)," par. 13)("patients with glioblastoma," par. 13) the post-surgical Glioblastoma test patient, and the post-surgical Glioblastoma test patient ("images of the tissue 108 (e.g., the brain, e.g., the same tissue as imaged in FIG. 1A, or different tissue as compared to the tissue imaged in FIG. 1A) including a diseased/abnormal region 110 (e.g., a cancerous tumor)," par. 33) ("patients with glioblastoma," par. 8).
Han and Kateb et al. do not explicitly teach all of the plurality of features comprises at least one of a volumetric measurement of the at least one object of interest and a bidimensional diametrical measurement of the at least one object of interest.
However, Bhatia et al. teach wherein the plurality of features comprises at least one of a volumetric measurement of the at least one object of interest and a bidimensional diametrical measurement of the at least one object of interest ("The feature signature f.sub.ROI may comprise numerous features which as a sum characterize the analyzed region of interest ROI. For instance, the feature signature f.sub.ROI may comprise texture features, shape features, intensity/density features, color or grey scale features, size features, structural features, or localization features," par. 176).
Therefore, taking the teachings of Han, Kateb et al., and Bhatia et al. as a whole, it would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the segmentation and prediction architecture as taught by Han and post-surgical glioblastoma patient images as taught by Kateb et al. to use the particular feature qualities taught by Bhatia et al. The suggestion/motivation for doing so would have been that, “The feature signature f.sub.ROI may comprise numerous features which as a sum characterize the analyzed region of interest” as noted by the Bhatia et al. disclosure in paragraph [0176], which also motivates combination because the combination would predictably have a higher efficiency and accuracy as there is a reasonable expectation that combining precise volumetric/dimensional measurements with a neural network architecture would lead to a robust, automated, and accurate segmentation and classification of the object, reducing manual intervention and measurement error; and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
7th Claim Rejections - 35 USC § 103
Claims 16, 17, and 22 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2019 0030371 A1, (Han), US Patent Publication 2016 0035093 A1, (Kateb et al.), and US Patent Publication 2022 0237785 A1, (Mitchell) in view of US Patent Publication 2021 090257 A1, (Bhatia et al.).
Claim 16
Regarding Claim 16, Han, Kateb et al., and Mitchell teach the method of claim 1 as noted above.
Han teaches generating, by the artificial neural network implementing the prediction function, a test segmentation mask ("The 3D ground truth label map may be divided to sequential 2D ground truth label maps, respectively corresponding to the sequential stacks of adjacent 2D images, and pixels of the 2D ground truth label maps are associated with known anatomical structures," par. 59) and extracts a plurality of features from the test segmentation mask (" Feature extraction portion 520 may extract one or more features of an input stack of adjacent 2D images," par. 66).
Kateb et al. teach obtaining a test image obtained from MRI brain scan sequences ("first images obtained from biopsy, Infrared Imaging, Ultraviolet Imaging, Diffusion Tensor Imaging (DTI), Computed Tomography (CT), Magnetic Resonance Imaging (MRI)," par. 12) of a post-surgical ("One or more first images can comprise a pre-operative image and/or an intra-operative image and/or a post operative image of one or more patients," par. 13) Glioblastoma test patient; and the post-surgical Glioblastoma test patient ("images of the tissue 108 (e.g., the brain, e.g., the same tissue as imaged in FIG. 1A, or different tissue as compared to the tissue imaged in FIG. 1A) including a diseased/abnormal region 110 (e.g., a cancerous tumor)," par. 33) ("patients with glioblastoma," par. 8).
Han, Kateb et al., and Mitchell do not explicitly teach all of wherein the plurality of features comprises at least one of a volumetric measurement of the at least one object of interest and a bidimensional diametrical measurement of the at least one object of interest.
However, Bhatia et al. teach wherein the plurality of features comprises at least one of a volumetric measurement of the at least one object of interest and a bidimensional diametrical measurement of the at least one object of interest ("The feature signature f.sub.ROI may comprise numerous features which as a sum characterize the analyzed region of interest ROI. For instance, the feature signature f.sub.ROI may comprise texture features, shape features, intensity/density features, color or grey scale features, size features, structural features, or localization features," par. 176).
Therefore, taking the teachings of Han, Kateb et al., Mitchell, and Bhatia et al. as a whole, it would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the segmentation and prediction architecture as taught by Han, post-surgical glioblastoma patient images as taught by Kateb et al., and random parameter initialization as taught by Mitchell to use the particular feature qualities taught by Bhatia et al. The suggestion/motivation for doing so would have been that, “The feature signature f.sub.ROI may comprise numerous features which as a sum characterize the analyzed region of interest” as noted by the Bhatia et al. disclosure in paragraph [0176], which also motivates combination because the combination would predictably have a higher efficiency and accuracy as there is a reasonable expectation that combining precise volumetric/dimensional measurements with a neural network architecture would lead to a robust, automated, and accurate segmentation and classification of the object, reducing manual intervention and measurement error; and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 17
Regarding Claim 17, Han, Kateb et al., and Mitchell teach the method of claim 16 as noted above.
Han teaches further comprising classifying a disease status, a type of lesion, or a treatment response based on the plurality of features ("The segmentation unit 403 may use at least one trained CNN model received from CNN model training unit 402 to predict the anatomical structure each voxel of a 4D image represents," par. 63).
Kateb et al. teach the post-surgical Glioblastoma test patient ("One or more first images can comprise a pre-operative image and/or an intra-operative image and/or a post operative image of one or more patients," par. 13) ("patients with glioblastoma," par. 8).
Han, Kateb et al., Mitchell, and Bhatia et al. are combined as per claim 16.
Claim 22
Regarding Claim 22, Han, Kateb et al., and Bhatia et al. teach the system of claim 21 as noted above.
Han teaches for each image of the plurality of images, receiving a ground-truth pixel-based annotation of the image, wherein the ground-truth pixel-based annotation comprises a ground-truth segmentation mask for the at least one object of interest, ("The 3D ground truth label map may be divided to sequential 2D ground truth label maps, respectively corresponding to the sequential stacks of adjacent 2D images, and pixels of the 2D ground truth label maps are associated with known anatomical structures," par. 59) obtaining a predicted segmentation mask ("The segmentation unit 403 may use at least one trained CNN model received from CNN model training unit 402 to predict the anatomical structure each voxel of a 4D image represents," par. 63) by feeding the image to the artificial neural network implementing a prediction function calculating a loss using a training function, ("The encoding portion 524 of the CNN model 510 may include one or more convolutional layers 528. Each convolutional layer 528 may have a plurality of parameters, such as the width (“W”) and height (“H”) determined by the upper input layer (e.g., the size of the input of convolutional layer 528), and a count of filters or kernels (“N”) in the layer and their sizes," par. 72) when the predicted segmentation mask and the ground-truth segmentation mask are given as input to the training function; ("During the training of CNN model 510, the loss layer may determine how the network training penalizes the deviation between the predicted 2D label map and the 2D ground truth label map," par. 81) optimizing the model parameters of the prediction function by minimizing the loss with respect to the model parameters for the plurality of the images based on the loss for each image; and replacing the model parameters of the prediction function with the optimized model parameters ("The model 700A parameters may be established from training data, such as by minimizing a loss function," par. 88).
Kateb et al. teach receiving a plurality of images from Magnetic Resonance Imaging (MRI) brain scan sequences ("first images obtained from biopsy, Infrared Imaging, Ultraviolet Imaging, Diffusion Tensor Imaging (DTI), Computed Tomography (CT), Magnetic Resonance Imaging (MRI)," par. 12) of post-surgical ("One or more first images can comprise a pre-operative image and/or an intra-operative image and/or a post operative image of one or more patients," par. 13) Glioblastoma patients ("images of the tissue 108 (e.g., the brain, e.g., the same tissue as imaged in FIG. 1A, or different tissue as compared to the tissue imaged in FIG. 1A) including a diseased/abnormal region 110 (e.g., a cancerous tumor)," par. 33) ("patients with glioblastoma," par. 8).
Mitchell teaches wherein model parameters of the prediction function are randomly initialized ("The segmenting computer 150 initializes each of the multiple instances of the neural network with respectively randomized weight parameters," par. 90).
Han, Kateb et al., Mitchell, and Bhatia et al. are combined as per claim 16.
8th Claim Rejections - 35 USC § 103
Claim 23 is rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2019 0030371 A1, (Han), US Patent Publication 2016 0035093 A1, (Kateb et al.), US Patent Publication 2022 0237785 A1, (Mitchell), and US Patent Publication 2021 090257 A1, (Bhatia et al.) in view of US Patent Publication 2020 0364509 A1, (Weinzaepfel et al.).
Claim 23
Regarding Claim 23, Han, Kateb et al., and Mitchell teach the system of claim 22 as noted above.
Han teaches wherein each of the plurality of images comprises multiple objects of interest, including a contrast-enhancing tumor, regions of edema, and the surgical cavity, ("Medical images 246 may include information such as imaging data associated with a patient anatomical region, organ, or volume of interest segmentation data. Patient data 245 may include information such as (1) functional organ modeling data (e.g., serial versus parallel organs, appropriate dose response models, etc.); (2) radiation dosage data (e.g., dose-volume histogram (DVH) information; or (3) other clinical information about the patient and course of treatment," par. 31) and wherein the training function is averaged over all objects of interest ("The segmentation unit 403 may use at least one trained CNN model received from CNN model training unit 402 to predict the anatomical structure each voxel of a 4D image represents. When the image segmentation is completed, segmentation unit 403 may output a 3D label map, associating each voxel of the 3D image to an anatomical structure," par. 63) ("Additionally, the ReLu layer may reduce or avoid saturation during a backpropagation training process," par. 75 wherein backpropagation includes averaging the loss over each training function).
Han, Kateb et al., and Mitchell do not explicitly teach all of wherein one prediction function per object of interest and one training function per object of interest are implemented.
However, Weinzaepfel et al. teach wherein one prediction function per object of interest and one training function per object of interest are implemented ("Branch A, after being processed by a convolution neural network (500 of FIG. 7), a convolution neural network being a neural network composed of interleaving convolution layers and rectified linear units, predicts binary segmentation for each object-of-interest (700 of FIG. 7) and x and y reference image coordinates regression, with an object-of-interest specific regressor (800 and 900, respectively, of FIG. 7). In one embodiment, the convolution neural network predicts segmentation and correspondences on a dense grid of size 56×56 in each box which is then interpolated to obtain a per-pixel prediction," par. 57).
Therefore, taking the teachings of Han, Kateb et al., Mitchell, Bhatia et al., and Weinzaepfel et al. as a whole, it would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the segmentation and prediction architecture as taught by Han, post-surgical glioblastoma patient images as taught by Kateb et al., random parameter initialization as taught by Mitchell, and the particular feature qualities taught by Bhatia et al. to use the one vs rest classifier as taught by Weinzaepfel et al. The suggestion/motivation for doing so would have been that, “Based upon this evaluation, homography data augmentation, at training, allows the generation of more viewpoints of the objects-of-interest, and thus, enabling improved detection and matching of objects-of-interest at test time with novel viewpoints” as noted by the Weinzaepfel et al. disclosure in paragraph [0093], which also motivates combination because the combination would predictably have a higher accuracy as there is a reasonable expectation that more objects of interest will be detected/predicted in the images; and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
9th Claim Rejections - 35 USC § 103
Claim 24 is rejected under 35 U.S.C. 103 as obvious over US Patent Publication 2019 0030371 A1, (Han), US Patent Publication 2016 0035093 A1, (Kateb et al.), and US Patent Publication 2021 090257 A1, (Bhatia et al.) in view of Isenee, F. et al., "nnU-Net for Brain Tumor Segmentation" ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, NY:1-15 (Nov 2, 2020) and US Patent Publication 2022 051402 A1, (Dikici et al.).
Claim 24
Regarding Claim 24, Han, Kateb et al., and Bhatia et al. teach the method of claim 21 as noted above.
Han, Kateb et al., and Bhatia et al. do not explicitly teach all of wherein the artificial neural network is a single ensemble of five confidence-aware nnU-Nets.
However, Isenee et al. teach wherein the artificial neural network is a single ensemble of five confidence-aware nnU-Nets ("A total of five downsampling operations are performed, resulting in a feature map size 4 x 4 x 4 in the bottleneck," sec. 2.2, pg. 4).
Therefore, taking the teachings of Han, Kateb et al., Bhatia et al., and Isenee et al. as a whole, it would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to modify the segmentation and prediction architecture as taught by Han, post-surgical glioblastoma patient images as taught by Kateb et al., and the particular feature qualities taught by Bhatia et al. to use the nnU-Net training architecture as taught by Isenee et al. The suggestion/motivation for doing so would have been that, “Ve recently proposed nnU-Net [25], a general purpose segmentation method that automatically configures segmentation pipelines for arbitrary biomedical datasets. nnU-Net set new state of the art results on the majority of the 23 datasets it was tested on, underlining the effectiveness of this approach” as noted by the Isenee et al. disclosure in section 1, pg. 2, which also motivates combination because the combination would predictably have a more utility for general segmentation as there is a reasonable expectation that nnU-Net architectures automatically configures segmentation pipelines based on dataset properties; and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Karsten F. Lantz whose telephone number is (571)272-4564. The examiner can normally be reached Monday-Friday 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ms. Jennifer Mehmood can be reached on 571-272-2976. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Karsten F. Lantz/Examiner, Art Unit 2664
Date: 5/26/2026
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664
Prosecution Insights