DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1, 7, and 10-13 are amended. Claims 1, 3-4, and 6-20 are pending in this application.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3-4, 6, 8-10, and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirsch et al., “Segmentation of MRI head anatomy using deep volumetric networks and multiple spatial priors” in view of Jansen et al., “Patient-specific fine-tuning of convolutional neural networks for follow-up lesion quantification”.
Regarding claim 1, Hirsch discloses an image diagnosis apparatus, comprising:
an acquirer to acquire a tomographic image including a region to be diagnosed of a subject (Section 1 Introduction; first paragraph and Section 2.8 Training and Testing Data; first paragraph; Clinical and basic research require segmentation of magnetic resonance images (MRIs) of human heads, including abnormal anatomies such as tumors or lesions; The training data consist of T1-weighted MRI scans from 4 healthy subjects and 43 individuals who suffered a stroke. The strokes occurred at least 6 month prior to the MRI scan, at which point the lesion is largely replaced by CSF. MRI scans from normal subjects were obtained on a 3T Siemens Trio scanner (Erlangen, Germany). The stroke scans were collected at Georgetown University and the University of North Carolina, Chapel Hill, also on a 3T Siemens Trio scanner. The trained network was also applied to MRI images of 47 patients with disorders of consciousness collected at the Pitié-Salpêtrière University Hospital in Paris, on a 3T General Electric Signa system (Milwaukee, Wisconsin)); and
a drawer to draw, based on the tomographic image acquired by the acquirer, a labeled image including the region to be diagnosed, the labeled image being partitioned according to classes each of which indicates one of a lesion area, a cavity area, a soft tissue area, a bone area, and a background area, the labeled image having a unique pixel value with respect to each class (Section 2.8 Training and Testing Data; second paragraph and Figure 5; Specifically, the 43 stroke heads are first segmented automatically and then manually corrected for errors in particular around the stroke lesions and boundaries between CSF, gray matter, and skull, resulting in seven classes (background, air cavities, skin, bone, CSF, white matter, and gray matter); The manual segmentation is on the first column, followed by the T1-weighted MRI that is used as an input for the network, next are segmentations from (a), (b) the detail + context network and (c) SPM8 compared to the segmentation from the Multiprior. Each color represents one of each of seven tissue-classes used for classification: black, background; brown, skin; yellow, bone/skull; green, air/sinus cavities; light blue, CSF; white, white matter; and gray, gray matter. Notice the large CSF-filled lesion in (a)), wherein
the drawer estimates, based on a model that is generated by machine learning and that, with respect to input of a pixel value of each pixel in a tomographic image, outputs a pixel value of each pixel in a labeled image, a pixel value of each pixel in a labeled image from the tomographic image acquired by the acquirer (Section 2.1 Detail CNN and Figure 1; Multiprior network structure. The detail network (black path) consists of a 3D CNN with eight layers. During training, this network takes as input a patch of 253 voxels around a target patch of 93 to be classified (green cube). The size of the convolutional kernels mapping between layer is indicated by numbers to the right. For instance, 33 × 50 × 30 indicates a 3D convolution kernel of size 33 transforming 30 features to 50 features. The “context” network (red path) is identical in structure to the “detail” network, except that it processes a downsampled version of a larger FOV of 573 voxels during training. It includes an upsampling layer at the end to merge features at the same scale as the detail network. Prior probabilities for the target patch are extracted from a TPM and added as input to the final classification (blue arrow). The “classification” network (purple) takes the concatenated output of all three pathways as input and classifies the target patch with three fully connected layers and no additional spatial mixing (kernel of size 13). After the entire image has been segmented, a 3D CRF processes the resulting output segmentation while taking the original input image into account (green arrow). Arrows indicate copying).
Hirsch discloses claim 1 as enumerated above, but Hirsch does not explicitly disclose the lesion area is assigned a pixel value that is different from pixel values assigned to the cavity area, the soft tissue area, the bone area, and the background area such that the lesion area is distinguished from a non-lesion area in the labeled image as claimed.
However, Jansen discloses the classes were weighted based on class frequencies using a weighted loss function. The class weights were set to 1 for the background class and to 5 for the lesion class. The last two layers combine the feature maps of the previous layers to classify each pixel. For the evaluation, the masked probability output was postprocessed to a binary image. For the liver data set, the resulting binary image was divided into separate objects representing individual lesions (Section 2.2 Base CNN Model; third paragraph, Section 2.3 Patient-Specific Fine-Tuning; second paragraph, and Section 2.4 Lesion Quantification on Follow-Up Scan second paragraph).
Therefore, taking the combined disclosures of Hirsch and Jansen as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the classes were weighted based on class frequencies using a weighted loss function. The class weights were set to 1 for the background class and to 5 for the lesion class. The last two layers combine the feature maps of the previous layers to classify each pixel. For the evaluation, the masked probability output was postprocessed to a binary image. For the liver data set, the resulting binary image was divided into separate objects representing individual lesions as taught by Jansen into the invention of Hirsch for the benefit of improving the lesion quantification performance of general CNNs by exploiting a patient’s previously acquired imaging (Jansen: Abstract; Conclusion).
Regarding claim 3, the image diagnosis apparatus according to claim 1, Hirsch in the combination further discloses wherein the lesion area is a tumor area in a brain, the tomographic image is an image of a cross section of a brain of a subject, the cross section being obtained by slicing the brain in a transverse plane direction at a plurality of points, and the drawer draws a labeled image corresponding to each tomographic image acquired by the acquirer (Figures 3, 5-7, and 11-13).
Regarding claim 4, the image diagnosis apparatus according to claim 3, Hirsch in the combination further discloses wherein the tumor area is an area where a metastatic brain tumor has developed (Figure 5, last sentence).
Regarding claim 6, the image diagnosis apparatus according to claim 1, Hirsch in the combination further discloses wherein the image diagnosis apparatus further includes a trainer to generate the model by machine learning (Section 1 Introduction, last paragraph; Section 2.3 Classification Network; and Section 2.8 Training and Testing Data).
Regarding claim 8, the image diagnosis apparatus according to claim 6, Hirsch in the combination further discloses wherein the trainer generates the model, using teacher data generated based on a plurality of tomographic images that has cross sections obtained by slicing a brain of each of a plurality of subjects in a transverse plane direction at a plurality of points (Section 1 Introduction, last paragraph; Section 2.3 Classification Network; and Section 2.8 Training and Testing Data).
Regarding claim 9, the image diagnosis apparatus according to claim 1, Hirsch in the combination further discloses wherein the image diagnosis apparatus further includes an outputter to color-code a labeled image with respect to each class, the labeled image being drawn by the drawer (Figures 5-7 and 11-13).
Regarding claim 10, this claim recites substantially the same limitations that are performed by claim 1 above, and it is rejected for the same reasons.
Regarding claim 12, this claim recites substantially the same limitations that are performed by claim 1 above, and it is rejected for the same reasons.
Claim(s) 7, 11, and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirsch et al., “Segmentation of MRI head anatomy using deep volumetric networks and multiple spatial priors” in view of Weiss et al., “automated multiclass tissue segmentation of clinical brain MRIs in the presence of lesions”.
Regarding claim 7, Hirsch discloses an image diagnosis apparatus, comprising:
an acquirer to acquire a tomographic image including a region to be diagnosed of a subject (Section 1 Introduction; first paragraph and Section 2.8 Training and Testing Data; first paragraph; Clinical and basic research require segmentation of magnetic resonance images (MRIs) of human heads, including abnormal anatomies such as tumors or lesions; The training data consist of T1-weighted MRI scans from 4 healthy subjects and 43 individuals who suffered a stroke. The strokes occurred at least 6 month prior to the MRI scan, at which point the lesion is largely replaced by CSF. MRI scans from normal subjects were obtained on a 3T Siemens Trio scanner (Erlangen, Germany). The stroke scans were collected at Georgetown University and the University of North Carolina, Chapel Hill, also on a 3T Siemens Trio scanner. The trained network was also applied to MRI images of 47 patients with disorders of consciousness collected at the Pitié-Salpêtrière University Hospital in Paris, on a 3T General Electric Signa system (Milwaukee, Wisconsin));
a drawer to draw, based on the tomographic image acquired by the acquirer, a labeled image including the region to be diagnosed, the labeled image being partitioned according to classes each of which indicates one of at least a lesion area, a normal tissue, and a background area, the labeled image having a unique pixel value with respect to each class (Section 2.8 Training and Testing Data; second paragraph and Figure 5; Specifically, the 43 stroke heads are first segmented automatically and then manually corrected for errors in particular around the stroke lesions and boundaries between CSF, gray matter, and skull, resulting in seven classes (background, air cavities, skin, bone, CSF, white matter, and gray matter); The manual segmentation is on the first column, followed by the T1-weighted MRI that is used as an input for the network, next are segmentations from (a), (b) the detail + context network and (c) SPM8 compared to the segmentation from the Multiprior. Each color represents one of each of seven tissue-classes used for classification: black, background; brown, skin; yellow, bone/skull; green, air/sinus cavities; light blue, CSF; white, white matter; and gray, gray matter. Notice the large CSF-filled lesion in (a)), and
a trainer to generate a model that is generated by machine learning and that, with respect to input of a pixel value of each pixel in a tomographic image, outputs a pixel value of each pixel in a labeled image, wherein the drawer estimates, based on the model generated by the trainer, a pixel value of each pixel in a labeled image from a pixel value of each pixel in the tomographic image acquired by the acquirer (Section 2.1 Detail CNN and Figure 1; Multiprior network structure. The detail network (black path) consists of a 3D CNN with eight layers. During training, this network takes as input a patch of 253 voxels around a target patch of 93 to be classified (green cube). The size of the convolutional kernels mapping between layer is indicated by numbers to the right. For instance, 33 × 50 × 30 indicates a 3D convolution kernel of size 33 transforming 30 features to 50 features. The “context” network (red path) is identical in structure to the “detail” network, except that it processes a downsampled version of a larger FOV of 573 voxels during training. It includes an upsampling layer at the end to merge features at the same scale as the detail network. Prior probabilities for the target patch are extracted from a TPM and added as input to the final classification (blue arrow). The “classification” network (purple) takes the concatenated output of all three pathways as input and classifies the target patch with three fully connected layers and no additional spatial mixing (kernel of size 13). After the entire image has been segmented, a 3D CRF processes the resulting output segmentation while taking the original input image into account (green arrow). Arrows indicate copying), and
the trainer generates the model, using teacher data that include a pixel value of each pixel in a tomographic image as input data and a pixel value of each pixel in a labeled image as output data, the labeled image being generated based on the tomographic image (Section 2.7 Cost Function and Section 2.8 Training and Testing Data; third paragraph; Training was set to reduce the generalized Dice loss between the predicted segmentation by the network and the ground-truth provided by the manual segmentations…The inner product · sums over all elements of the 3D volume. C is the total number of classes (C = 7 in this case); During network training, four heads were kept out for validation purposes, measuring generalization performance during training epochs and used to define the stopping point, i.e., the epoch with maximum Dice score on the validation set. This procedure was used for training all convolutional neural networks (CNNs) (Multiprior, DeepMedic, and U-Net variants)).
Hirsch discloses claim 7 as enumerated above, but Hirsch does not explicitly disclose a weight of each class adjusted based on a number of counted pixels of the class so as to adjust a contribution of the class during generation of the model as claimed.
However, Weiss discloses for median frequency weighting experiments class weights were determined by wi = nb/ni where wi is the weight of the ith class and nb and ni are the median number of voxels in the background class and ith class, respectively, across the entire training set. When training models with categorical crossentropy loss we utilized softmax activation at the end of the network and 7 output channels, 1 for each tissue type and 1 for background (not brain tissue), as the objective of the categorical cross entropy loss is to determine the most probable label for each voxel (Section 2.6 Loss function optimization).
Therefore, taking the combined disclosures of Hirsch and Weiss as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate for median frequency weighting experiments class weights were determined by wi = nb/ni where wi is the weight of the ith class and nb and ni are the median number of voxels in the background class and ith class, respectively, across the entire training set. When training models with categorical crossentropy loss we utilized softmax activation at the end of the network and 7 output channels, 1 for each tissue type and 1 for background (not brain tissue), as the objective of the categorical cross entropy loss is to determine the most probable label for each voxel as taught by Weiss into the invention of Hirsch for the benefit of clinical decision support and quantitative analysis of clinical brain MRIs in the presence of lesions (Weiss: Abstract; last sentence).
Regarding claim 11, this claim recites substantially the same limitations that are performed by claim 7 above, and it is rejected for the same reasons.
Regarding claim 13, this claim recites substantially the same limitations that are performed by claim 7 above, and it is rejected for the same reasons.
Claim(s) 14-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hirsch et al., “Segmentation of MRI head anatomy using deep volumetric networks and multiple spatial priors” in view of Jansen et al., “Patient-specific fine-tuning of convolutional neural networks for follow-up lesion quantification” and further in view of Floyd et al., “Artificial Neural Networks for SPECT Image Reconstruction with Optimized Weighted Backprojection”.
Regarding claim 14, the image diagnosis apparatus of claim 6, Hirsch and Jansen in the combination further disclose wherein the drawer draws, using a neural network (Hirsch: Section 2.1 Detail CNN and Figure 1).
Hirsch and Jansen in the combination disclose claim 14 as enumerated above, but Hirsch and Jansen do not explicitly disclose an input layer including a plurality of neurons for input, an output layer including a plurality of neurons for output, and at least one intermediate layer, including a plurality of intermediate neurons, positioned between the input layer and the output layer, wherein connection states exist between the input layer, the output layer, and the intermediate layer as claimed.
However, Floyd discloses artificial neural network includes the input layer, the hidden layer, and the output layer. The input layer connected to the hidden layer and the hidden later connected to the output layer (Fig. 1; Section: Method, First and Second paragraphs).
Therefore, taking the combined disclosures of Hirsch, Jansen, and Floyd as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate artificial neural network includes the input layer, the hidden layer, and the output layer. The input layer connected to the hidden layer and the hidden later connected to the output layer as taught by Floyd into the inventions of Hirsch and Jansen for the benefit of representing the tomographic reconstruction process using the artificial neural network architecture (Floyd: Section: Discussion, First paragraph).
Regarding claim 15, the image diagnosis apparatus of claim 14, Floyd in the combination further discloses wherein a number of the plurality of neurons in the input layer corresponds to a number of pieces of input data (Fig. 1; Section: Method, First and Second paragraphs).
Regarding claim 16, the image diagnosis apparatus of claim 14, Floyd in the combination further discloses wherein a number of the plurality of neurons in the output layer corresponds to a number of pieces of output data (Fig. 1; Section: Method, First and Second paragraphs).
Regarding claim 17, the image diagnosis apparatus of claim 14, Floyd in the combination further discloses wherein each piece of input data in the input layer is input to respective intermediate neurons in the intermediate layer (Fig. 1; Section: Method, First and Second paragraphs).
Regarding claim 18, the image diagnosis apparatus of claim 14, Floyd in the combination further discloses wherein the trainer uses a plurality of data sets to adjust weighting coefficients that indicate the connection states between the input layer, the output layer, and the intermediate layer (Section: Introduction).
Regarding claim 19, the image diagnosis of claim 18, Floyd in the combination further discloses wherein based on the pixel values of the respective pixels in the labeled image (estimated values of output data) output from the neurons for output by inputting the pixel values of the respective pixels in the tomographic image (input data) to the neurons for input in the input layer and the pixel values of the respective pixels in the labeled image (set values of output data), a mean squared error (MSE) between the estimated values and the set values of the output data is calculated (Sections: Method and Result).
Regarding claim 20, the image diagnosis of claim 19, Floyd in the combination further discloses wherein by minimizing the MSE the weighting coefficients of the connection states between the input layer, the output layer, and the intermediate layer of the neural network are optimized (Sections: Introduction, Method, and Result).
Response to Arguments
Applicant's arguments with respect to claims 1, 3-4, and 6-20 have been considered but are moot in view of the new ground(s) of rejection.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VAN D HUYNH whose telephone number is (571)270-1937. The examiner can normally be reached 8AM-6PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen R Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VAN D HUYNH/Primary Examiner, Art Unit 2665
Prosecution Insights