DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on November 20, 2024 is in compliance with 37 CFR 1.97 and 1.98 and therefore has been considered and placed in the file.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The BRIs are used for purposes of searching for prior art, but cannot be incorporated into the claims. Claim limitations must be given their plain meaning unless such meaning is inconsistent with the specification. MPEP 2111.01. BRIs for some of the claim limitations are provided below. Should Applicant believe that other interpretations are warranted, Applicant should point to the portions of the present disclosure that clearly show that a different interpretation is appropriate.
Claim Objections
Claim 5 is objected to because of the following informalities: the term “(VOIR))” in line 5 of claim 5 should be changed to --(VOIR)--. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 7, 9, 10, 15 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over an article entitled “Deep Learning for Pulmonary Embolism Detection: Tackling the RSNA 2020 AI Challenge”, by Ian Pan, published in 2021 in Radiology: Artificial Intelligence 2021; 3(5):e210068 (hereinafter referred to as “Pan”) in view of an article entitled “Medical Image Classification based on an Adaptive Size Deep Learning Model”, by Liu et al., published October 26, 2021 in ACM Trans. Multimedia Comput. Commun. Appl., Vol. 17, No. 3s, Article 102 (hereinafter referred to as “Liu”).
Regarding claim 1, Pan discloses a system (Fig. 3) comprising:
a memory to store instructions (Page 4, section entitled “Building the Model” discusses memory requirements of the system shown in Fig. 3. Page 3, section entitled “Development Environment” discloses using Nvidia Quadro RTX 6000 GPUs, each with 24-GB VRAM for implementing the machine learning models shown in Fig. 3);
a processor to execute the instructions stored in the memory to perform the following operations (As indicated above, Page 3 discloses using Nvidia Quadro RTX 6000 GPUs, each with 24-GB VRAM for implementing the machine learning models shown in Fig. 3):
receiving a plurality of Computed Tomography Pulmonary Angiography (CTPA) exams (hereinafter the ''exams'') as input image data, each exam in the plurality of exams comprising a varying plurality of individual images (hereinafter the ''slices'') (Pan discloses that the system shown in Fig. 3 is trained and tested using CTPA examinations, each of which comprises a plurality of individual images: “This challenge included 9446 CT pulmonary angiographic (CTPA) examinations, expertly annotated by thoracic radiologists from the STR. The data were divided into training (n = 7279), public test (n = 650), and private test (n =1517) sets”. The RSNA challenge required determining slice-level classification (referred to in Pan as “image-level” classification) as well as exam-level classification, which means that a plurality of slices of each of the examinations are classified and that each examination is also classified, as discussed on Page 4, Building the Model, bullet points 1. and 2. Pan does not explicitly disclose that the exams comprise varying pluralities of individual images);
annotating each of the slices with one of a pulmonary embolism (PE) present label or PE absent label (Page 1, section entitled “Understanding the Challenge”: “[i]n this competition, 10 classification labels were used as part of the scoring and are summarized in Table 1. These labels indicated the following: whether pulmonary embolism (PE) was present on a specific image”);
annotating each of the exams with one of a plurality of labels each indicating a different PE state and location (Page 1, section entitled “Understanding the Challenge”: “[i]n this competition, 10 classification labels were used as part of the scoring and are summarized in Table 1. These labels indicated … whether PE was present in a single examination”);
performing a slice-level PE classification to determine the presence or absence of PE for each of the slices (Page 4, section entitled “Image-Level Feature Extraction”: “[t]he feature extractor was trained to predict one image-level label (presence of PE)”); and
performing and outputting, via an Embedding-based Vision Transformer (E-ViT), an exam-level diagnosis using the slice-level classifications (Page 4, “Building the Model” section, bullet point 2: “Sequence modeling for both refinement of image-level PE predictions and prediction of examination-level PE labels”. The section entitled “Sequence Modeling” discloses using vision transformers (referred to in Pan as “image transformers”) to perform exam-level diagnosis using the slice-level classifications. The BRI for the term “embedding-based vision transformer” is a transformer that receives embeddings, i.e., vectors, as input. The exam-level vision transformer of Pan is embedding-based because it uses embeddings corresponding the feature vectors extracted from all images, as discussed in the “Sequence Modeling” section).
As indicated above, Pan does not explicitly disclose that the exams comprise varying pluralities of individual images. The BRI for this limitation, based on para. [0072] of the present specification, is that the received images of each exam have varying sizes and/or that the exams have different numbers of images. Pan does disclose, on page 5, resizing the model input to 32 images by 416 x 416 pixels and cropping the images to 364 x 364 pixels. Therefore, Pan does suggest that the system receives exams comprising varying pluralities of images. Nevertheless, Pan does not explicitly disclose that the exams comprise varying pluralities of images.
Liu, in the same field of endeavor, discloses addressing the problem of using AI machine learning models to perform classification tasks on multi-size medical image datasets by using an adaptive size deep learning model that receives datasets having varying pluralities of images and adapts to the sizes of the medical image datasets (Introduction of Liu).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system and method of Pan to be capable of adapting to exams having varying pluralities of images as taught by Liu. One of ordinary skill in the art would have been motivated to make the modification to improve versatility of the system by allowing it to be trained on and to perform classification on varying pluralities of images. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the GPUs of Pan to perform image size/number adaptation).
Regarding claim 2, Pan discloses that the plurality of exams comprises one of the Radiological Society of North America (RSNA) Pulmonary Embolism (PE) CT dataset and the Computer Aided Diagnosis-Pulmonary Embolism (CAD-PE) Challenge dataset, and an in-house PE-CAD dataset (Page 1 of Pan discloses that the solution that Pan used in the RSNA 2020 challenge required training and testing using the RSNA PE CT dataset).
Regarding claim 7, Pan discloses that performing and outputting an exam-level diagnosis using E-ViT comprises generating predictions for a collection of the slices and assigning a label to each corresponding exam (Fig. 3 shows generated slice predictions for a collection of slices and the block labeled “Final Label (9)” corresponds to the final label for the exam, as discussed in the section entitled “Building the Model”, bullet points 2 and 3).
Regarding claim 9, the rejection of claim 1 applies mutatis mutandis to claim 9. Regarding the recited non-transitory computer readable storage media, the 24-GB VRAMs of the Nvidia GPUs constitute non-transitory computer readable storage media for storing instructions executed by the GPUs to perform the recited operations. (Section entitled “Development Environment”).
Regarding claim 10, the rejection of claim 2 applies mutatis mutandis to claim 10.
Regarding the rejections of claim 15, the rejection of claim 7 applies mutatis mutandis to claim 15.
Regarding claims 17 and 18, the rejections of claims 1 and 2 apply mutatis mutandis to claims 17 and 18, respectively.
Claims 3, 11 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Pan in view of Liu as applied to claims 1, 2, 7, 9-10, 15 and 17-18 and further in view of an article entitled “Computer Aided Detection for Pulmonary Embolism Challenge (CAD-PE)”, by Gozalez et al., published March 30, 2020 in arXiv:2003.13440 [eess.IV] (hereinafter referred to as “Gonzalez”).
Regarding claim 3, the combined teachings of Pan and Liu do not explicitly teach that performing slice-level PE classification to determine the presence or absence of PE for each of the slices comprises pre-processing steps of lung localization to focus on a region of interest in the slices, and windowing to highlight pixel intensities within a range of 100-700 Hounsfield Units.
Gonzalez, in the same field of endeavor, discloses a computer-aided detection (CAD) system for detecting PE in CPTA images that uses a Hounsfield units (HU) thresholding method of windowing to highlight pixel intensities within a range of 100-700 HUs (Third paragraph of section entitled “III. Materials and Methods” and section entitled “Universidad de Alicante”: “[i]nput images are clipped between the values [-200,500] HU and normalized in the range [0,1].”).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system and method of Pan as modified by Liu further based on the teachings of Gonzalez to perform the Hounsfield Hounsfield windowing process of Gonzalez. One of ordinary skill in the art would have been motivated to make the modification to remove irrelevant image data while emphasizing regions of interest, thereby reducing processing overhead while improving performance. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the GPUs of Pan to perform Hounsfield windowing).
Regarding claims 11 and 19, the rejection of claim 3 applies mutatis mutandis to claims 11 and 19.
Claims 4, 12 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Pan in view of Liu as applied to claims 1, 2, 7, 9-10, 15 and 17-18 and further in view of an article entitled “Seeking an Optimal Approach for Computer-Aided Pulmonary Embolism Detection”, by Islam et al., published September 21, 2021 in Mach Learn Med Imaging. 2021 Sep 21;12966:692–702. doi: 10.1007/978-3-030-87589-3_71 (hereinafter referred to as “Islam”) and U.S. Publ. Appl. No. 2023/0147286 A1 to Hsiao (hereinafter referred to as “Hsiao”).
Regarding claim 4, the combined teachings of Pan and Liu do not explicitly teach that performing slice-level PE classification to determine the presence or absence of PE for each of the slices comprises performing slice-level classification using an ensemble of CNN-based architectures including Xception, SeXception and SeResNext50.
Islam, in the same field of endeavor, discloses that CNN architectures including Xception, SeXception and SeResNext50 can be used for determining the presence or absence of PE in CPTA image slices and provides a comparative performance analysis of these three different architectures (Section 3.1 Image-level Classification and Tables 2 and 3). However, Islam does not explicitly disclose using an ensemble of these three CNN architectures for this purpose. Hsiao, in the same field of endeavor, discloses that using an ensemble of CNN architectures is beneficial and explicitly discloses one of the architectures being Xception (paras. [0055] and [0063]).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system and method of Pan as modified by Liu further based on the teachings of Islam and Hsiao to use an ensemble of CNN architectures including Xception, SeXception and SeResNext50. One of ordinary skill in the art would have been motivated to make the modification because Hsiao teaches the desirability of using CNN architecture ensembles in classifying features such as PEs in CT images (paras. [0041]-[0042]) and Islam teaches the performance benefits of using Xception, SeXception and SeResNext50 architectures for slice-level and exam-level PE classification. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the GPUs of Pan to implement an ensemble of Xception, SeXception and SeResNext50 architectures for performing image-level PE detection).
Regarding claims 12 and 20, the rejection of claim 4 applies mutatis mutandis to claims 12 and 20.
Claims 5 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Pan in view of Liu as applied to claims 1, 2, 7, 9-10, 15 and 17-18 and further in view of U.S. Publ. Appl. No. 2020/0074701 A1 to Liang et al. (hereinafter referred to as “Liang”).
The combined teachings of Pan and Liu do not explicitly teach that the plurality of exams comprises the CAD-PE Challenge dataset, and wherein performing slice-level PE classification to determine the presence or absence of PE for each of the slices comprises a pre-processing step of representing the input image data using a three-dimensional (3D) vessel-oriented image representation (VOIR)) to create 3D VOIR data.
Liang, in the same field of endeavor, discloses using the exams comprising CAD-PE Challenge dataset (Para. [0071]: “[e]mbodiments of the invention have also evaluated using the entire 20 CTPA test datasets from the PE challenge”) and that when performing image-level PE classification to determine the presence or absence of PE in the images (para. [0029]), a preprocessing step is performed to represent the image data using a three-dimensional (3D) vessel-oriented image representation (VOIR) to create 3D VOIR data (Para. [0030], Figs. 7 and 8).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system and method of Pan as modified by Liu further based on the teachings of Liang to use a preprocessing step to represent the input image data by using 3D VOIR to create 3D VOIR data as taught by Liang. One of ordinary skill in the art would have been motivated to make the modification to benefit from the advantages of using 3D VOIR representations, including
“efficiency and compactness—concisely summarizing three-dimensional (3D) contextual information around an embolus in only three image channels. VOIR is compact because it concisely summarizes the 3D contextual information around emboli in only three image channels”, as taught by Liang (Para. [0030]). The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the GPUs of Pan to implement 3D VOIR).
Regarding claim 13, the rejection of claim 5 applies mutatis mutandis to claim 13.
Claims 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Pan in view of Liu and Liang as applied to claims 5 and 13 and further in view of an article entitled “Discriminative, Restorative, and Adversarial Learning: Stepwise Incremental Pretraining”, by Guo et al., published September 15, 2022 in Domain Adaptation and Representation Transfer (DART) 2022 (hereinafter referred to as “Guo”).
Regarding claim 6, the combined teachings of Pan, Liu and Liang do not explicitly teach using self-supervised TransVW model with ((D)+R)+A) on the 3D VOIR data to improve PE false positive reduction performance for subsequent slice-level PE classification. The BRI for this limitation is based on para. [00138] of the present specification, which describes this limitation as meaning discriminative encoder in combination with a restorative decoder and that combination being used in combination with an adversarial encoder to form a transferable visual word (TransVW) model.
Guo, in the same field of endeavor, discloses uniting three self-supervised learning (SSL) ingredients (discriminative, D, restorative, R, and adversarial, A, learning) to obtain a transferable visual word (TransVW) model that enables collaborative representation learning and yields three transferable components: a discriminative encoder, a restorative decoder, and an adversary encoder (Figs. 1 and 2(e) show the (D+R)+A TransVW architecture described on pages 67-68. Guo discloses at the bottom of page 68 that uniting these components results in “pulmonary embolism false positive reduction as shown in Table 2”).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system and method of Pan as modified by Liu and Liang further based on the teachings of Guo to use the (D+R)+A TransVW architecture and method of Guo on the 3D VOIR data of Liang to achieve a reduction in false positive PE detection as taught by Guo. One of ordinary skill in the art would have been motivated to make the modification to achieve a reduction in false positive PE detection. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the GPUs of Pan to implement (D+R)+A processing on the 3D VOIR data).
Regarding claim 14, the rejection of claim 6 applies mutatis mutandis to claim 14.
Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Pan in view of Liu as applied to claims 1, 2, 7, 9-10, 15 and 17-18 and further in view of an article entitled “Models Genesis”, by Zhou et al., published in January 2021 in Med Image Anal. 2021 January (hereinafter referred to as “Zhou”).
Regarding claim 8, the combined teachings of Pan and Liu do not explicitly disclose using a Models Genesis self-supervised learning method to improve PE false positive reduction performance for subsequent slice-level PE classification.
Zhou, in the same field of endeavor, discloses using Models Genesis in 3D medical imaging for processing chest CT slices (bottom of page 2-top of page 3. Sections 4.3 and 4.4 of Zhou disclose that the Models Genesis achieves pulmonary embolism false positive reduction while also reducing annotation efforts).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the present disclosure, to modify the system and method of Pan as modified by Liu further based on the teachings of Zhou to use the Models Genesis self-supervised learning method of Zhou to improve false positive reduction performance for slice-level PE classification. One of ordinary skill in the art would have been motivated to make the modification to achieve a reduction in false positive PE detection while also reducing annotation efforts. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the GPUs of Pan to implement Models Genesis self-supervised learning).
Regarding claim 16, the rejection of claim 8 applies mutatis mutandis to claim 16.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL J SANTOS whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella can be reached at (571)272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL J. SANTOS/ Examiner, Art Unit 2667
/MATTHEW C BELLA/ Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights