DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1, 5,11, and 15 have been amended.
Claims 1-2, 5-6, 8-12, 15-16 and 18-26.
Response to Arguments
On page 2 of the “Remarks” filed on Feb 03, 2026 Applicant asserts “Neither Huang nor Wu teaches or suggests a second model taking as input the "one or more attention features maps and the medical image" and outputting "an enhanced medical image." Huang is directed to an end-to-end framework where the final output is a segmentation mask……. Huang does not teach a second enhancement model whose output is an enhanced clinical image trained against ground- truth high-quality images”.
Response: Huang does not rely upon solely for the final enhanced-image output. Huang is relied upon for teaching that segmentation probability maps may be used as attention/anatomical-prior inputs in the reconstruction path. Fig. 1 (d) Recurrent model acts as a secondary stage following the initial reconstruction and segmentation, the Recurrent module takes the output from the (c) Attention module (the "attention map") as an input for its AttReg block; within module (d), the AttReg and DC blocks work together to refine the image. While the final output of the entire pipeline is a "Refined mask" (a probability map), the intermediate output of the DC block within module (d) is a refined medical image. This enhanced image is then fed into the UNet/ConvLSTM component to generate the final mask, confirming that the enhancement process results in an image, not a probability map. The new reference, Sun et al. “Joint CS-MRI Reconstruction and Segmentation with a Unified Deep Network”, supplies the standard/high-quality ground truth reconstruction training and reconstruction MRI output.
Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-2, 5-6, 9-12, 15-16, and 19-21, are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. NPL “Brain Segmentation from k-space with End-to-end Recurrent Attention Network” in view of NPL Sun et al. “Joint CS-MRI Reconstruction and Segmentation with a Unified Deep Network”.
Regarding claim 1, Huang et al. teaches
PNG
media_image1.png
452
756
media_image1.png
Greyscale
comprising: (a) receiving a medical image of a subject wherein the medical image is acquired using a medical imaging apparatus with shortened scanning time or reduced amount of tracer dose (see Fig, 1 diagram labels above disclose “The proposed model takes the under-sampled k-space data as input” Note: Under-sampled k-space data implies to MRI data where a reduced portion of the raw k-space data is collected, significantly decreasing scan times by not collecting all necessary spatial frequencies”), (b) applying a first model to the medical image for segmentation of the medical image by outputting one or more attention feature maps comprising one or more probability maps (Fig. 1 (c) disclose “segmentation probability map (initial or refined)” page 4, 1st para; “The segmentation map itself is already a probability map. After a softmax layer σ(·) that ensures the sum of the four different classes to be 1, the maps can be utilized directly for attention. Each of the four segmentation probability maps are element-wise multiplied with the input image features xt−1 ∈ R 2×w×h to generate new features..” see also page 3, 1st para; “b) We introduce a novel attention module that guides the network to generate segmentation-driven image features to improve the segmentation performance” Note: Probability maps are the direct output of a segmentation network and the attention feature maps often include or are derived from probability maps); and (c) applying a second model to the one or more attention features maps and the medical image for enhancing a quality of the medical image by outputting an enhanced medical image, wherein the enhanced medical image is not a probability map (see page 3, 1st para; “The intermediate segmentation is recurrently exploited as anatomical prior that provides guidance to recover segmentation-driven image features from the raw data”, And Fig. 1 (d) Recurrent model acts as a secondary stage following the initial reconstruction and segmentation, the Recurrent module takes the output from the (c) Attention module (the "attention map") as an input for its AttReg block; within module (d), the AttReg and DC blocks work together to refine the image. While the final output of the entire pipeline is a "Refined mask" (a probability map), the intermediate output of the DC block within module (d) is a refined medical image. This enhanced image is then fed into the UNet/ConvLSTM component to generate the final mask, confirming that the enhancement process results in an image, not a probability map). Huang et al. built to improve segmentation quality not image quality. However, Huang et al. does not specifically teach a computer-implemented method for improving image quality, wherein the first model and the second model are trained in an end-to-end training process using training data comprising at least ground truth medical images having standard pr high quality.
In the same field of endeavor Sun et al. teaches a computer-implemented method for improving image quality (see page 3, section 3.2; “As more blocks are stacked in a cascaded manner, the quality of the reconstructed MRI of each block can be gradually improved”), wherein the first model and the second model are trained in an end-to-end training process using training data comprising at least ground truth medical images having standard pr high quality (see page 4, section 3.3; “We pre-train the MRNN with under-sampled and fully-sampled MRI training pairs. Similarly, we pre-train the MSN with fully-sampled MRI and their corresponding segmentation labels…. After pre-training separately and initializing the remaining parts, the parame ters of SegNetMRIN (with N blocks in the MRN portion, but a single segmentation decoder duplicated N times) are then fine-tuned. Therefore, both the reconstruction and seg mentation tasks share the same encoders, but have separate decoders for their respective tasks….. where xfs i (2) and xi are the ith full-sampled training image and the output of the MRN, respectively.. we then con struct and fine-tune SegNetMRI with the following loss function LSegNetMRI = LMRN +λLOMSN”). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify a method of a novel learning framework that performs magnetic resonance brain image segmentation directly from k-space data of Huang et al. in view of the use of Joint CS-MRI reconstruction and segmentation with a unified deep network of Sun et al. in order to improve both the reconstruction and segmentation performance when using compressive measurements (see page 3, section 3.2).
Regarding claim 2, the rejection of claim 1 is incorporated herein.
Sun et al. in the combination further teach wherein the first model is a deep learning network model comprising a Res-Net architecture (see page 2, left col., last para; “Adeep residual architecture was also proposed for this same mapping” Note: the deep residual network implies Res-Net architecture).
Regarding claim 5, the rejection of claim 1 is incorporated herein.
Sun et al. in the combination further teach wherein the first model and the second model comprise a Res-Net architecture (see page 2, left col., last para; “A deep residual architecture was also proposed for this same mapping” Note: the deep residual network implies Res-Net architecture).
Regarding claim 6, the rejection of claim 1 is incorporated herein.
Huang et al. in the combination further teach wherein the second model is trained to adapt to the one or more attention feature maps (see page 4, 1st para; “By explicitly utilizing intermediate segmentation results for reconstruction, or more precisely image feature extraction, segmentation-driven features will be generated, which in turn improves segmentation performance during training with back-propagation algorithm”, see also page 5; “The attention module fAttReg takes the initial reconstruction feature x N−1 0 (feature from the N −1 reconstruction block) and segmentation probability maps st−1 as input and generates new image xt”, Note: the reconstruction/attention subnetwork (AttReg +DC) ingests the probability maps and is trained end-to-end under segmentation loss, so its weights adapts to those maps through back-propagation).
Regarding claim 9, the rejection of claim 1 is incorporated herein.
Huang et al. in the combination further teach wherein the medical imaging apparatus is a transforming magnetic resonance (MR) device or a Positron Emission Tomography (PET) device (see page 1, section 1; “. Magnetic Resonance Imaging (MRI), for example, acquires data in the spatial-frequency domain (the so called k-space) and the MR images need to be reconstructed from the k-space data before further analysis”).
Regarding claim 10, the rejection of claim 1 is incorporated herein.
Sun et al. in the combination further teach wherein the enhanced medical image has a higher resolution or improved signal-noise ratio (see also page 6, 1st para; “We also give averaged reconstruction performance measures in Table 2 using peak signal-to-noise ratio (PSNR)”).
Regarding claim 11, the scope of claim 11 is fully incorporated in claim 1, and the rejection of claim 1 is equally applicable here (see Huang et al. further disclose computer-implemented CNNs for medical image segmentation and reconstruction. These require storage of program instructions on a transitory medium executed by processor).
Regarding claim 12, the rejection of claim 11 is incorporated herein.
Sun et al. in the combination further teach wherein the first model is a deep learning network model comprising a Res-Net architecture (see page 2, left col., last para; “Adeep residual architecture was also proposed for this same mapping” Note: the deep residual network implies Res-Net architecture).
Regarding claim 15, the rejection of claim 11 is incorporated herein.
Sun et al. in the combination further teach wherein the first model and the second model comprise a Res-Net architecture (see page 2, left col., last para; “A deep residual architecture was also proposed for this same mapping” Note: the deep residual network implies Res-Net architecture).
Regarding claim 16, the rejection of claim 11 is incorporated herein.
Huang et al. in the combination further teach wherein the second model is trained to adapt to the one or more attention feature maps (see page 4, 1st para; “By explicitly utilizing intermediate segmentation results for reconstruction, or more precisely image feature extraction, segmentation-driven features will be generated, which in turn improves segmentation performance during training with back-propagation algorithm”, see also page 4 and 5; “The attention module fAttReg takes the initial reconstruction feature x N−1 0 (feature from the N −1 reconstruction block) and segmentation probability maps st−1 as input and generates new image xt”, Note: the reconstruction/attention subnetwork (AttReg +DC) ingests the probability maps and is trained end-to-end under segmentation loss, so its weights adapts to those maps through back-propagation).
Regarding claim 19, the rejection of claim 11 is incorporated herein.
Huang et al. in the combination further teach wherein the medical imaging apparatus is a transforming magnetic resonance (MR) device or a Positron Emission Tomography (PET) device (see page 1, section 1; “. Magnetic Resonance Imaging (MRI), for example, acquires data in the spatial-frequency domain (the so called k-space) and the MR images need to be reconstructed from the k-space data before further analysis”).
Regarding claim 20, the rejection of claim 11 is incorporated herein.
Sun et al. in the combination further teach wherein the enhanced medical image has a higher resolution or improved signal-noise ratio (see also page 6, 1st para; “We also give averaged reconstruction performance measures in Table 2 using peak signal-to-noise ratio (PSNR)”).
Regarding claim 21, the rejection of claim 1 is incorporated herein.
Sun et al. in the combination further teach further comprising receiving a user input indicative of a selection of a region of interest (ROI), generating pathology information in the ROI, improving a quality of the ROI, or improving a quality of the medical image (see page 2, Last para; “These deep learning based CS MRImodels have achieved higher reconstruction quality”, see also page 3, left col. last para; “This experiment shows that while CS-MRI can substantially improve the reconstruction quality visually”).
Claims 8, 18 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. in view of Sun et al. as applied in claim 1 above, and further in view of Zhou et al. (US 10430946 B1).
Regarding claim 8, the rejection of claim 1 is incorporated herein. The combination of Huang et al. and Sun et al. as a whole does not teach wherein the one or more attention feature maps include a noise map or lesion map.
Zhou et asl. teach wherein the one or more attention feature maps include a noise map or lesion map (see col. 18, lines 7-13; “Moreover, the lesion attention maps generated by the lesion attention model 470 can be used as pseudo masks to refine the multi-lesion masks generator 435 using large-scale, image-level annotation data in a semi-supervised manner. The tasks of generating lesion masks 445 and grading diseases can be jointly optimized in an end-to-end network” Note: the limitation includes “or” option). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify a method of a novel learning framework that performs magnetic resonance brain image segmentation directly from k-space data of Huang et al. in view of the use of Joint CS-MRI reconstruction and segmentation with a unified deep network of Sun et al. and a neural network architecture accurately perform object segmentation and grading functions of Zhou et al. in order to improve the final disease grading performance of the lesion attention model (see col. 18, lines 7-13).
Regarding claim 18, the rejection of claim 11 is incorporated herein.
Zhou et al. in the combination further teach wherein the one or more attention feature maps include a noise map or lesion map (see col. 18, lines 7-13; “Moreover, the lesion attention maps generated by the lesion attention model 470 can be used as pseudo masks to refine the multi-lesion masks generator 435 using large-scale, image-level annotation data in a semi-supervised manner. The tasks of generating lesion masks 445 and grading diseases can be jointly optimized in an end-to-end network” Note: the limitation includes “or” option”). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify a method of a novel learning framework that performs magnetic resonance brain image segmentation directly from k-space data of Huang et al. in view of the use of Joint CS-MRI reconstruction and segmentation with a unified deep network of Sun et al. and a neural network architecture accurately perform object segmentation and grading functions of Zhou et al. in order to improve the final disease grading performance of the lesion attention model (see col. 18, lines 7-13).
Regarding claim 24, the rejection of claim 23 is incorporated herein.
Sun et al. in the combination further teach wherein the first model comprises a segmentation model trained to generate the one or more attention feature maps (see page 3, 1st para; “We introduce a novel attention module that guides the network to generate segmentation-driven image features to improve the segmentation performance”).
Zhou et al. in the combination further teach comprising a lesion map based on the two or more MR imaging pulse sequences (see col. 18, lines 7-13; “Moreover, the lesion attention maps generated by the lesion attention model 470 can be used as pseudo masks to refine the multi-lesion masks generator 435 using large-scale, image-level annotation data in a semi-supervised manner. The tasks of generating lesion masks 445 and grading diseases can be jointly optimized in an end-to-end network”). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify a method of a novel learning framework that performs magnetic resonance brain image segmentation directly from k-space data of Huang et al. in view of the use of Joint CS-MRI reconstruction and segmentation with a unified deep network of Sun et al. and a neural network architecture accurately perform object segmentation and grading functions of Zhou et al. in order to improve the final disease grading performance of the lesion attention model (see col. 18, lines 7-13).
Claims 22 are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. in view of Sun et al. as applied in claim 1 above, and further in view of Wang et al. (US 20200167930 A1).
Regarding claim 22, the rejection of claim 21 is incorporated herein. The combination of Huang et al. and Sun et al. as a whole does not teach
In the same field of endeavor Wang et al. teach wherein the one or more attention feature maps are generated based at least in part on the user input (see para [0071]; “The machine learning-based segmentation is improved through integrating user interactions or indications, e.g. clicks and/or scribbles..,,user interactions are combined with the machine learning system based on geodesic distance maps. Geodesic distances of each pixel, or voxel for 3-dimensional images, to user interactions of each class are calculated and used as extra input for the machine learning system….. Such a geodesic distance transform [40, 6] encodes spatial regularization and contrast-sensitivity (and has not previously been used or considered for CNNs)”). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify a method of a novel learning framework that performs magnetic resonance brain image segmentation directly from k-space data of Huang et al. in view of the use of Joint CS-MRI reconstruction and segmentation with a unified deep network of Sun et al. and generating a segmentation of input image using a first machine learning system of Wang et al. in order to increase interaction efficiency (see para [0071]).
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. in view of Sun et al. as applied in claim 1 above, and further in view of Wu et al. NPL “Self-attention convolutional neural network for improved MR image reconstruction”.
Regarding claim 23, the rejection of claim 1 is incorporated herein. The combination of Huang et al. and Sun et al. does not teach wherein the medical image comprises two or more MR imaging pulse sequences
In the same field of endeavor Wu et al. teach wherein the medical image comprises two or more MR imaging pulse sequences (see page 319, section 2.1.; “The data were acquired on a 3T scanner (GE Healthcare, Waukesha, WI) using an adiabatic inversion recovery spin-lock prepared UTE sequence with different numbers of IR spin-lock pulses (2, 4, 6, 8, 12, and 16)”, see also page 327, section 5. Conclusion; “A self-attention convolutional neural network framework was developed for the reconstruction of sparsely sampled MRI, aiming to provide improved image fidelity for accelerated MR image acquisition”). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify a method of a novel learning framework that performs magnetic resonance brain image segmentation directly from k-space data of Huang et al. in view of the use of Joint CS-MRI reconstruction and segmentation with a unified deep network of Sun et al. and Self-attention convolutional neural network for improved MR image reconstruction of Wu et al. in order to guide reconstruction to improve image quality under constrained acquisition (see page 319, section 2.1).
Claims 25 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al. in view of Sun et al. as applied in claim 1 above, and further in view of Park et al. (US 20220215646 A1).
Regarding claim 25, the rejection of claim 1 is incorporated herein. The combination of Huang et al. and Sun et al. does not teach wherein the end-to-end training process comprises adjusting an optimal loss function according to noise distribution.
In the same field of endeavor Park et al. teach wherein the end-to-end training process comprises adjusting an optimal loss function according to noise distribution (see para [0046]; “The system 300 can iteratively update the parameters of the one or more convolutional layers 132a, 132b, 132c for different training inputs until the loss function is optimized. An example loss function is described later below. [0047] In one implementation, to jointly optimize first stage 110 and stage 140, a loss function that estimates parameters aiming at estimating parameters Om for the first probability map and Θ.sup.(2) for the second probability map, W, and b by optimizing the function: J=h.sup.(1)J.sup.(1)(Θ.sup.(1))+h.sup.(2)J.sup.(2)(Θ.sup.(2),W,b), where, W indicates convolutional filters, for example, whose dimension is (5×5 |L|), b is bias, Θ.sup.(1) includes one or more performance parameters of the first stage of the machine learning model”). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify a method of a novel learning framework that performs magnetic resonance brain image segmentation directly from k-space data of Huang et al. in view of the use of Joint CS-MRI reconstruction and segmentation with a unified deep network of Sun et al. and apparatus for segmenting internal structures depicted in an image of Park et al. in order to improve the performance relative to implementations (see para [0046]).
Regarding claim 26, the rejection of claim 1 is incorporated herein.
Park et al. in the combination further teach wherein the end-to-end training process comprises summing a first loss for the first trained model and a second loss for the second trained model (see para [0047]; “In one implementation, to jointly optimize first stage 110 and stage 140, a loss function that estimates parameters aiming at estimating parameters Om for the first probability map and Θ.sup.(2) for the second probability map, W, and b by optimizing the function… is the total loss function that is the weighted average of loss functions J.sup.1 for the first stage of the machine learning mode and J.sup.2 for the second stage of the machine learning model. J.sup.1 is dependent on Θ.sup.(1) and J.sup.2 is dependent on Θ.sup.(2)”). Accordingly, it would have been obvious to one of ordinary skill in the art before the effecting filling date of the invention to modify a method of a novel learning framework that performs magnetic resonance brain image segmentation directly from k-space data of Huang et al. in view of the use of Joint CS-MRI reconstruction and segmentation with a unified deep network of Sun et al. and apparatus for segmenting internal structures depicted in an image of Park et al. in order to improve the performance relative to implementations (see para [0046]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WINTA GEBRESLASSIE whose telephone number is (571)272-3475. The examiner can normally be reached Monday-Friday9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at 571-270-5180. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WINTA GEBRESLASSIE/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights