DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on November 27, 2024 is in compliance with 37 CFR 1.97 and 1.98 and therefore has been considered by the examiner and placed in the file.
Drawings
The drawings are objected to because lines and text in Fig. 2 are not sufficiently dense, dark and thick to give them satisfactory reproduction characteristics, as required by 37 CFR 1.84(l).
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The specification is objected to because in the first line of para. [0034] of the present specification, “114” should be changed to - -112- - to make the present specification consistent with the description of Fig. 2.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The BRIs are used for purposes of searching for prior art, but cannot be incorporated into the claims. Claim limitations must be given their plain meaning unless such meaning is inconsistent with the specification. MPEP 2111.01. BRIs for some of the claim limitations are provided below. Should Applicant believe that other interpretations are warranted, Applicant should point to the portions of the present disclosure that clearly show that a different interpretation is appropriate.
The BRI of a claim element is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked. As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claim 10 recites several “means for” limitations. Use of the word “means” followed by functional language (“storing program code instructions”) creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. Because claim 10 does not recite any structure for performing the function, the presumption is not rebutted, and therefore the claim limitations are interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Likewise, claims 12 and 13 recite means limitations that are also interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, for the same reason that claim 10 is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Because these limitations are being interpreted under 35 U.S.C. 112(f), they are being interpreted to cover the corresponding structure described in the specification as performing the claimed functions, and equivalents thereof. The examiner finds that the specification discloses sufficient structure (i.e., processor executing computer instructions stored in memory) for performing the functions. Therefore, the BRI for these limitations are computer instructions stored in a read from memory and executed by a processor to perform the functions, and equivalents thereof.
If applicant does not intend to have this limitation interpreted under 35 U.S.C. 112(f), applicant may: (1) amend the claim limitation to avoid it being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation recites sufficient structure to perform the claimed function so as to avoid it being interpreted under 35 U.S.C. 112(f).
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-6, 8 and 10-14 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by an article entitled “Adjacent slices feature transformer network for single anisotropic 3D brain MRI image super-resolution”, by Wang et al., published November 19, 2021 in Elsevier, Biomedical Signal Processing and Control (hereinafter referred to as “Wang”)
Regarding claim 1, Wang discloses a computer-implemented method (The method performed by adjacent slice feature transformer (AFST) network shown in Fig. 1 is a computer-implemented method) comprising:
receiving one or more 3D (three-dimensional) medical images each comprising a plurality of 2D (two-dimensional) slices (Section 2.1 Network Architecture, discloses that the adjacent slice feature transformer (ASFT) network shown in Fig. 1 receives a low resolution (LR) 3D MRI anisotropic image, ILR, of size WxHxD, as indicated at the top of the left-hand column of page 3: “Input: LR MRI image with size WxHxD: ILR; anisotropic scale factor: s”. The LR MRI input image comprises 2D slices, namely, the upper, U, and nether, N, adjacent slices, which the ASFT network processes to generate a target slice in between the adjacent slices at each and uses the U, N and target slices to reconstructs “anisotropic 3D brain MRI scans slice by slice”);
extracting a first set of features from the plurality of 2D slices of the one or more 3D medical images using a machine learning based encoding network (Top of left column of page 3: “[a]s shown in Fig. 1, each branch first uses one convolution layer to extract initial features X1 = {xU1, x1, xN1} from the input slices XIn”… Then, X1 is fed into the FMNet to learn the mapping from initial features to deep features. As shown in Fig. 1, FMNet contains a series of stacked multi-branch (i.e, MFTE) blocks to extract and rectify rich hierarchical features that benefit to the brain MRI SR task.”);
resampling each respective feature of the first set of features based on a spatial location of one or more pixels of the 2D slices from which the respective feature was extracted (The BRI for this limitation, based on para. [0028] of the present specification, is that it means sampling the extracted features of the 2D slices of the received 3D image to form 2D slices in feature space that are aligned with the sampled 2D slices. From the bottom of the left column of page 3 to the top of the right column of page 3, Wang discloses that once the features of adjacent slices, referred to in Wang as the upper, U, and nether, N, slices, are extracted by the FMNet of the ASFT network shown in Fig. 1 to form the adjacent 2D slices in feature space, the spatial locations of the features in the adjacent U and N slices are sampled in the MFTE blocks of the FMNet and further in the RecNet to form a 2D slice in feature space in between the adjacent slices, which means that the features of the adjacent slices and of the slice in between are aligned in feature space);
encoding the resampled first set of features into a second set of features (The BRI for the limitation “encoding the resampled first set of features into a second set of features”, based on para. [0030] of the present specification, is that the resampled first set of features is processed in some way to convert the resampled first set of features into a second set of features. Sections 2.1 and 2.2 of Wang disclose that the extracted and enriched features that are output from the U, N and main branches of the MFTE blocks of the FMNet shown in Fig. 1 are concatenated together to form a tensor that is then fed into the channel attention (CA) unit of the RecNet portion of the ASFT network shown in Fig. 1. This concatenation that produces a tensor embedding constitutes encoding the sampled first set of features into a second set of features.);
performing a medical imaging analysis task based on the second set of features (The BRI for this limitation, based on paras. [0032]-[0033], is that some further processing of the second set of features is performed, e.g., “the medical imaging analysis task is image synthesis for generating a synthetic image from the one or more medical images. The synthetic image may be in a domain, acquisition orientation, or resolution that is absent from the one or more 3D medical images.” Section 2.1 of Wang meets this limitation because it discloses that the RecNet of the ASFT network processes the concatenation comprising the second feature set with a function FRec(·) to produce a synthetic super resolution (SR) 3D isotropic MRI image: “where FRec(⋅) represents the function of RecNet. When the given input anisotropic image is SR reconstructed slice by slice using the proposed ASFT network, we can get the final isotropic 3D brain MRI image ISR.”); and
outputting results of the medical imaging analysis task (The output of the RecNet is the super resolution image shown in Fig. 1, ISR. Fig. 7 shows the output image in the bottom right of page 7).
Regarding claim 2, para. [0019] of the present specification indicates that when multiple 3D medical images are received, the images can have different resolutions, different acquisition orientations, or different domains. Neither claim 1 nor claim 2 requires receiving more than one 3D medical image. Therefore, the BRI for this limitation is that a single 3D medical image can be received, in which case the image has only has a single resolution, a single acquisition orientation, and a single domain. Wang discloses that the input 3D medical MRI image, ILR, that is received by the ASFT network shown in Fig. 1 has a low resolution and is acquired with some acquisition orientation in an MRI domain.
Regarding claim 3, as indicated above in the rejection of claim 1, Wang discloses sampling the extracted features of the adjacent U and N 2D slices in feature space to form the in-between target 2D slice in the feature space, in which case all three of the 2D slices in feature space are aligned. The term “uniform grid” is not explicitly defined in the present specification, but para. [0028] describes it as the 2D slice in feature space formed by resampling the extracted 2D slices in feature space. Therefore, the BRI for a uniform grid is that it means a 2D slice in feature space formed by sampling the extracted features of 2D slices. Wang discloses this limitation because Wang discloses that the in-between 2D slice is formed in feature space from the features that have been extracted from the adjacent U and N 2D slices in feature space.
Regarding claim 4, the BRI for the limitation “stacking the 2D images in the feature space to form one or more 3D isotropic images in the feature space”, based para. [0028] of the present specification, is that it means that the resampled extracted features of the 2D slices that form the 2D slice uniform grid are combined in feature space with the 2D slice uniform grid to form a 3D isotropic image. Section 2.1 of Wang discloses that the extracted features of the 2D adjacent U and N slices that are resampled to form the features of the target 2D slice, i.e., the in-between 2D slice, are combined with the features of the target 2D slice via concatenation and that the combination is fed into the RecNet of the ASFT network shown in Fig. 1 and processed by a function FRec(·) to “get the final isotropic 3D brain MRI image ISR.”
Regarding claim 5, in Wang, the spatial locations of the pixels of the 2D slices have to comprise coordinates in order to align the extracted features of the U and N adjacent slices to generate the corresponding features of the target in-between slice. This is confirmed by the fact that Wang uses 2D convolution kernels “to capture the spatial continuity and similarity between MRI slices”, as discussed in section 3.2. Also, the spatial attention (SA) modules of the ASFT network apply spatial attention weights, as discussed in section 2.2, which also indicates that the pixels of the image data comprising the 2D slices comprise coordinates.
Regarding claim 6, the BRI for the term “positional embeddings” is based on its plain meaning because the term is not explicitly defined in the present specification. In the art of machine learning, an embedding is a vector output from an encoder. A positional embedding is a vector output from an encoder that includes positional information. In Wang, the extracted features of the U and N adjacent 2D slices that are resampled in the FMNet, which is an encoder that performs a chain of convolution operations on the U and N 2D slices to perform extraction and resampling, form the target 2D slice features. These features of the U, N and target 2D slices in feature space are concatenated to form a vector that is fed into the CA unit of the RecNet, as discussed in section 2.2. This vector, or embedding, necessarily includes positional information so that the features of the U and N adjacent slices and of the target 2D slice can be combined during reconstruction to form the 3D isotropic MRI image that is output from the ASFT network. Therefore, Wang discloses the positional embedding limitation of claim 6.
Regarding claim 8, the ASFT network is a transformer that performs encoding and decoding (Abstract and lower left-hand column of page 2: “we propose a novel brain MRI adjacent slices feature transformer (ASFT) SR network”), and therefore the encoding of the resampled first set of features is performed using a transformer encoder. The operations performed by the FMNet of the ASFT network are transformer encoding operations. Also, the self-attention (SA) module of the ASFT transformer of Wang performs SA encoding operations. As is well known in the art, SA is a core feature of transformer encoders used to emphasize meaningful features during feature extraction.
Regarding claim 10, as indicated above, all of the means limitations are interpreted as invoking 35 U.S.C. 112(f) and therefore are interpreted as operations being performed by a processor executing instructions stored in a memory device, and equivalents thereof. Wang does not explicitly state that the ASFT transformer network is implemented by a processor executing instructions, or that it is implemented in hardware or in a combination of hardware and software. Machine learning operations of the type described in Wang are typically performed in software executed by one or more processors. Regardless of the implementation details of the ASFT transformer network of Wang, the ASFT transformer network meets the BRI of a processor, memory and equivalents thereof. Since claim 10 recites the same functions recited in claim 1, the rejection of claim 1 applies mutatis mutandis to claim 10.
Regarding claim 11, the rejection of claim 2 applies mutatis mutandis to claim 11.
As indicated above, claims 12 and 13 are also interpreted as invoking 35 U.S.C. 112(f) and therefore are interpreted as operations being performed by a processor executing instructions stored in a memory device, and equivalents thereof. Accordingly, the rejections of claims 3 and 4 apply mutatis mutandis to claims 12 and 13, respectively, since the ASFT transformer network of Wang meets the BRI of a processor configured to execute instructions stored in memory and equivalents thereof.
Regarding claim 14, the rejection of claim 5 applies mutatis mutandis to claim 14.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of an article entitled “Transformers In Medical Image Analysis”, by He et al., published by Elsevier B.V. on behalf of Chinese Medical Association in Intelligent Medicine 3 (2023) 59–78 (hereinafter referred to as “He”).
Regarding claim 9, Wang does not explicitly disclose that the image analysis task is performed using at least one of a transformer encoder or a state space model. He, in the same field of endeavor, discloses using transformer encoders to perform image analysis tasks (Section 2. Transformers and 2.2.1 discussing the transformer encoder) and provides a comparative analysis of the benefits of using various types of transformer encoders.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system of Wang such that the ASFT transformer encoder operations that are used for the image analysis tasks are performed in a transformer encoder as taught by He. One of ordinary skill in the art would have been motivated to make the modification to take advantage of the self-attention mechanism of the transformer encoder at emphasizing important or meaningful features during feature extraction. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (implementing a transformer encoder within or at the output of the RecNet of the ASFT transformer to perform the medical image analysis task) to yield predictable results.
Claims 15-17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of U.S. Publ. Appl. No. 2024/0046453 A1 to Jacob et al. (hereinafter referred to as “Jacob”).
Regarding claim 15, since claim 15 recites the same operations that are recited in claim 1, the rejection of claim 1 applies mutatis mutandis to claim 15. Wang does not explicitly disclose a non-transitory computer readable medium comprising instructions which, when executed by a computer, cause the computer to carry out operations. Jacob, in the same field of endeavor, discloses a non-transitory computer readable medium comprising instructions which, when executed by a computer, cause the computer to carry out operations for performing machine learning image analysis tasks (Fig. 6, processor 604 and memory 610, para. [0071], Abstract).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system of Wang such that the ASFT transformer operations are performed in software being executed by a processor as taught by Jacob. One of ordinary skill in the art would have been motivated to make the modification to take advantage of the well-known benefits of performing operations in software executed by a processor since machine learning models are typically implemented in software executed by one or more processor. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (implementing the ASFT transformer in software executed by a processor) to yield predictable results.
Regarding claims 16 and 17, the rejections of claims 2 and 6 apply mutatis mutandis to claims 16 and 17, respectively.
Regarding claim 19, the rejection of claim 8 applies mutatis mutandis to claim 19.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Jacob as applied to claims 15-17 and 19 and further in view of He. The rejection of claim 9 applies mutatis mutandis to claim 20.
Allowable Subject Matter
Claims 7 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claims 7 and 18 recite that the resampling of the respective features of the first set of features comprises encoding of the first set of features with modality embeddings. Although prior art exists that teaches encoding features into embeddings that include modality information, none of the prior art teaches or suggests this limitation in combination with the resampling of extracted features and the other limitations recited in base claims 1 and 15.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. Publ. Appl. No. 2026/0094267 A1 discloses a multi-modal transformer neural network that processes (i) frame tokens, (ii) electronic health data, and/or (iii) interpretable feature(s). For example, the input to the multi-modal transformer neural network may include the frame tokens appended to the embedded electronic health data and/or the embedded interpretable feature(s). The electronic health data and interpretable feature(s) may be embedded by linear projection, for example. In some embodiments, the multi-modal input and a learnable class token are added to a learnable temporal embedding and passed through the multi-modal transformer neural network. The multi-modal transformer neural network may apply temporal attention to the multi-modal input. The multi-modal transformer neural network may include a video transformer (e.g., ViViT) modified to allow multi-modal inputs.
An article entitled “Slice Imputation: Intermediate Slice Interpolation for Anisotropic 3D Medical Image Segmentation”, by Wu et al., published March 21, 2022 in arXiv:2203.10773v1, discloses a frame-interpolation-based method for slice imputation to improve segmentation accuracy for anisotropic 3D medical images, in which the number of slices and their corresponding segmentation labels can be increased between two consecutive slices in anisotropic 3D medical volumes. The proposed multitask inter-slice imputation method, in particular, incorporates a smoothness loss function to evaluate the smoothness of the interpolated 3D medical volumes in the through-plane direction (sagittal and coronal). It not only improves the resolution of the interpolated 3D medical volumes in the through-plane direction
but also transforms them into isotropic representations, which leads to better segmentation performances.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights