DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: a segmentation and annotation module and annotation module in claim 21.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-13 and 15-23 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Taerum (US 20200085382).
Regarding claim 1:
Taerum teaches: a method, comprising:
establishing one or more measurements and one or more phenotypical characteristics of a lesion using a first machine learning model operating on a machine segmentation of a lesion indicated in a medical image (¶ [0142] “…Possible segmentation architectures include but are not limited to ENet, U-Net, and their variants… 2D or 3D FCNs are utilized.”; ¶ [0176] “…the exact FCN that is used for segmentation may vary as long as it performs pixelwise segmentation. 3D extensions of ENet, U-Net, and their variants are all possible”; ¶ [0363] “The system can automatically measure the volume of the nodules…From the volume of each nodule, the maximum diameter in the axial plane and its orthogonal diameter are mathematically calculated and reported”; ¶ [0391] “The system can automatically define features of liver lesions in the different series, comprising enhancement, washout, and corona presence, and then calculates the corresponding LI-RADS score”; ¶ [0461] “…the CNN may have been trained to predict one or more of a variety of different objectives from patient medical images, including but not limited to: features of potentially cancerous lesions, e.g., size, shape, spiculations; features of the surrounding organ, e.g., texture, other (possibly non-cancer) disease; lesion malignancy; changes to any of the above metrics over time,” Taerum discloses ML segmentation (FCN/U-net), then establishes measurements (volume and diameters) and phenotypical characteristics);
using the first machine learning model or a second machine learning model, providing a medical prediction concerning the lesion using one or more of the one or more measurements, the one or more phenotypical characteristics, or features extracted from the machine segmentation (¶ [0471] “…One or more models may be combined into an ensemble of models. Each model may be any machine learning model that accepts structured features and performs classification or regression”; ¶ [0475] “…The features include demographic features of the patients (age, sex, etc.), features from histopathological assessment of lesion biopsy (tumor stage, grade, presence of lymph node metastases), features related to medical procedures and complications in the preceding 12 months, and image features from the most recent thoracic CT exam (current tumor size, change in tumor size since the previous thoracic CT exam, CNN-extracted features for a CNN that was trained to distinguish lesions from blood vessels in CT images e.g., following [Berens 2016])”; ¶ [0457] “…In the inference phase 5640 of this implementation, initially a patient is selected for whom inference is to be performed at 5614. Patient data is loaded for the selected patient at 5616 and features are extracted at 5618 in the same manner as they were extracted during training at 5606. Inference is performed with the trained machine learning models 5612 and input features 5618 to predict outcomes for the patient under one or more different treatment scenarios 5620”; Taerum discloses ML model(S) that uses measurements (tumor size, change), segmentation-derived feature (CNN features), and lesion characteristics to provide medical predictions); and
outputting the medical prediction with at least one of the one or more measurements or at least one of the one or more phenotypical characteristics (¶ [0374] – ¶ [0375] “The system can automatically report findings and their characterizations… The automatic report can also be a graphical report containing tables and images that describe the evolution of the findings over time.”; ¶ [0399] “…FIG. 53 is a GUI 5300 that shows an excerpt of an automated report that collects all characteristics of each finding”; ¶ [0457] “...The results of inference are then displayed to the user 5622 on a display 5624.”; ¶ [0482] – ¶ [0484] “…Outcome predictions are then returned 5908 and displayed to the user on a display 5912” and FIGs. 60 and 61).
Regarding claim 2:
Taerum teaches the limitations of claim 1 as applied above.
Taerum further teaches: obtaining an indication of interest from a trained person and segmenting the medical image to create the machine segmentation based on the indication of interest (¶ [0338] “…the system allows a user to move an adjustable radius sphere (or cylinder), also referred to herein as an editing tool, within a volume in order to add voxels to a segmentation…The placement and movement of the sphere may be controlled by the user clicking and dragging (e.g., via a mouse or other pointer) on such an MPR representation of the volume. By alternating between adjusting the position and orientation of the MPR and using an editing tool of the system, the user is able to quickly segment a region of interest as defined by the current application. “; ¶ [0343] “when a user clicks on some point within one of the MPRs, a sphere 3602 is initialized at that point within the 3D volume…As the sphere is moved through the volume guided by the user dragging the sphere's center point over the MPR, the voxels that come in contact with the sphere are added to the segmentation. This feature is shown in the screenshot 3700 of FIG. 37 and the screenshot 3800 of FIG. 38…The segmentation grows as the sphere follows the mouse movement until the mouse button is released.”; ¶ [0344] “As the current segmentation is edited, the MPRs are continually updated in order to display the intersection of the MPR with the segmented volume.)
Regarding claim 3:
Taerum teaches the limitations of claim 2 as applied above.
Taerum further teaches: wherein said segmenting comprises segmenting the medical image with a deep learning model trained to distinguish the lesion from other structures (¶ [0176] “..the exact FCN that is used for segmentation may vary as long as it performs pixelwise segmentation. 3D extensions of ENet, U-Net, and their variants are all possible”; ¶ [0181] “…The network is trained only on 3D patches that contain lesions, though in some implementations non-lesions are also included… The 3D image patches are matched with 3D boolean masks representing whether each pixel within the 3D patch is in a lesion”; ¶ [0200] “…he general idea behind fully convolutional networks (FCNs) is to use a downsampling path to learn relevant features at a variety of spatial scales followed by an upsampling path to combine the features for pixelwise prediction… Upsampling the activation volumes back to the original resolution is necessary in a fully convolutional network for pixel-wise segmentation”).
Regarding claim 4:
Taerum teaches the limitations of claim 3 as applied above.
Taerum further teaches: wherein the deep learning model comprises a U-net (¶ [0176] “..the exact FCN that is used for segmentation may vary as long as it performs pixelwise segmentation. 3D extensions of ENet, U-Net, and their variants are all possible”; ¶ [0203] FIG. 24 shows a schematic representation of the U-Net convolutional neural network architecture 2400 according to at least some implementations of the present disclosure.”).
Regarding claim 5:
Taerum teaches the limitations of claim 3 as applied above.
Taerum further teaches: wherein said providing the medical prediction comprises using the features extracted from the machine segmentation (¶ [0461] “The CNN used for feature extraction may be any of a variety of forms of CNN, including but not limited to: a classification network; an object detection network; a semantic segmentation network; or any combination of the above”; ¶ [0461] “…the outputs of these intermediate layers contain a representation of the input that is relevant for quantifying its properties (e.g., malignancy), so it is reasonable to think of the outputs of intermediate layers as relevant “features” of the lesion”; ¶ [0475] “..image features from the most recent thoracic CT exam (current tumor size, change in tumor size since the previous thoracic CT exam, CNN-extracted features for a CNN that was trained to distinguish lesions from blood vessels in CT images e.g., following [Berens 2016])”
Regarding claim 6:
Taerum teaches the limitations of claim 3 as applied above.
Taerum further teaches: wherein the features extracted from the machine segmentation comprise radiomic features (¶ [0363] “The system can automatically measure the volume of the nodules that were detected either automatically or manually. From the volume of each nodule, the maximum diameter in the axial plane and its orthogonal diameter are mathematically calculated and reported. All of these measurements can be edited by the user. Furthermore, from the volume of each nodule, the density of the nodule can also be calculated and displayed in an editable fashion.”).
Regarding claim 7:
Taerum teaches the limitations of claim 3 as applied above.
Taerum further teaches: wherein the features extracted from the machine segmentation comprise deep features extracted from the deep learning model (FIG. 32 and ¶ [0329] “…In this implementation, three dimensions, including “size,” “average intensity” and “deep learning” are shown.”; ¶ [0475] “..image features from the most recent thoracic CT exam (current tumor size, change in tumor size since the previous thoracic CT exam, CNN-extracted features for a CNN that was trained to distinguish lesions from blood vessels in CT images e.g., following [Berens 2016])”
).
Regarding claim 8:
Taerum teaches the limitations of claim 2 as applied above.
Taerum further teaches: wherein said obtaining comprises allowing the trained person to indicate a point or region of interest on the medical image using a graphical user interface and encoding that point or region of interest as the indication of interest (¶ [0313] “…The query lesion could be selected in many different ways, including but not limited to: a user clicking on or tapping a lesion when viewing a radiological study (such as an MR or CT study); a user selecting a lesion from a list of previously identified lesions; via an automated system; or some combination of the above”; ¶ [0343] “…when a user clicks on some point within one of the MPRs, a sphere 3602 is initialized at that point within the 3D volume… As the sphere is moved through the volume guided by the user dragging the sphere's center point over the MPR, the voxels that come in contact with the sphere are added to the segmentation”; ¶ [0338] “…the system allows a user to move an adjustable radius sphere (or cylinder), also referred to herein as an editing tool, within a volume in order to add voxels to a segmentation”; ¶ [0344] “As the current segmentation is edited, the MPRs are continually updated in order to display the intersection of the MPR with the segmented volume.”).
Regarding claim 9:
Taerum teaches the limitations of claim 1 as applied above.
Taerum further teaches: wherein the one or more phenotypical characteristics comprise one or more of subtlety, structure, calcification, sphericity, margin, lobulation, spiculation, and texture (¶ [0289] “…Once a clinically relevant set of features—including, for example, spiculations—is identified, one can create a training dataset with lesions and their ground truth annotations (including, e.g., the degree of spiculation for each lesion), design a CNN model to predict the annotations, and train it on the training dataset.”).
Regarding claim 10:
Taerum teaches the limitations of claim 1 as applied above.
Taerum further teaches: wherein the one or more measurements comprise one or more of lesion diameter, short axis, area, volume, and conformity to a shape (FIG. 45, ¶ [0349] “…The application further displays values associated with the physical extent of the segmentation, such as volume of the segmentation, the longest diameter of the segmentation, etc., as shown in the box 4502 on the right side of the screenshot 4500. In at least some implementations, the MPR displays the major diameter and the orthogonal diameter as lines 4504 and 4506, respectively, on a selected segmentation 4508.”).
Regarding claim 11:
Taerum teaches the limitations of claim 1 as applied above.
Taerum further teaches: wherein the medical prediction comprises a diagnosis concerning the lesion; a classification of the lesion according to a phenotype or genotype; a prediction of disease progression; a prediction of whether the lesion is likely to respond to a particular treatment; a prediction of whether an apparent growth of the lesion during a treatment represents a true progression or a pseudo-progression caused by the treatment; or a prediction of whether a particular patient is likely to experience a particular side effect (¶ 0323] “the classifier may classify the malignancy, lesion type, cancer subtype or prognosis of the query lesion. “; ¶ 0471] “…one or more models are trained at 5716 to predict patient outcomes. One or more models may be combined into an ensemble of models. Each model may be any machine learning model that accepts structured features and performs classification or regression, including but not limited to: random forests; gradient boosted decision trees; multi-layer perceptrons; or any combination of the above.”; ¶ [0478] “…Outcome predictions would then be separately available under the conditions that one of treatment combination A or treatment combination B is used”; ¶ 0484] “…A table 6002 of treatments along with the associated probability 6006 of lung cancer-associated death for each treatment 6004 is shown”).
Regarding claim 12:
Taerum teaches the limitations of claim 1 as applied above.
Taerum further teaches: wherein said providing the medical prediction uses the second machine learning mode (¶ 0471] “…one or more models are trained at 5716 to predict patient outcomes. One or more models may be combined into an ensemble of models. Each model may be any machine learning model that accepts structured features and performs classification or regression, including but not limited to: random forests; gradient boosted decision trees; multi-layer perceptrons; or any combination of the above.”).
Regarding claim 13:
Taerum teaches the limitations of claim 1 as applied above.
Taerum further teaches: wherein the second machine learning model is a deep learning model ((¶ [0176] “..the exact FCN that is used for segmentation may vary as long as it performs pixelwise segmentation. 3D extensions of ENet, U-Net, and their variants are all possible”; ¶ [0203] FIG. 24 shows a schematic representation of the U-Net convolutional neural network architecture 2400 according to at least some implementations of the present disclosure.”; FIG. 32 and ¶ [0329] “…In this implementation, three dimensions, including “size,” “average intensity” and “deep learning” are shown.”; ¶ [0475] “..image features from the most recent thoracic CT exam (current tumor size, change in tumor size since the previous thoracic CT exam, CNN-extracted features for a CNN that was trained to distinguish lesions from blood vessels in CT images e.g., following [Berens 2016])”).
Regarding claim 15:
Taerum teaches the limitations of claim 1 as applied above.
Taerum further teaches: wherein the medical prediction is a longitudinal medical prediction based on multiple medical images taken over time (¶ [0461] “…the CNN may have been trained to predict one or more of a variety of different objectives from patient medical images, including but not limited to: features of potentially cancerous lesions, e.g., size, shape, spiculations; features of the surrounding organ, e.g., texture, other (possibly non-cancer) disease; lesion malignancy; changes to any of the above metrics over time, using images acquired over time (e.g., over the course of days, months or years)”; ¶ [00475] “…features related to medical procedures and complications in the preceding 12 months, and image features from the most recent thoracic CT exam (current tumor size, change in tumor size since the previous thoracic CT exam, CNN-extracted features for a CNN that was trained to distinguish lesions from blood vessels in CT images e.g., following [Berens 2016]).”; ¶ [0371] “…Measurement may comprise linear dimensions, areas, volumes, and pixel density. These measurements are then compared mathematically to assess changes in size or presentation of the finding, and calculate growth or shrinkage of a finding over time”; ¶ [0375] “…The automatic report can be structured so that findings are presented based on urgency and severity. The automatic report can also be a graphical report containing tables and images that describe the evolution of the findings over time”).
Regarding claim 16:
Taerum teaches the limitations of claim 1 as applied above.
Taerum further teaches: A method, comprising:
accepting an indication of interest indicating a point or region in a medical image using a graphical user interface (¶ [0343] “…when a user clicks on some point within one of the MPRs, a sphere 3602 is initialized at that point within the 3D volume”; ¶ [0338] “…the system allows a user to move an adjustable radius sphere (or cylinder)... The placement and movement of the sphere may be controlled by the user clicking and dragging (e.g., via a mouse or other pointer) on such an MPR representation of the volume. By alternating between adjusting the position and orientation of the MPR and using an editing tool of the system, the user is able to quickly segment a region of interest as defined by the current application”);
based on the indication of interest, segmenting the medical image or a plurality of medical images for a single patient to identify a lesion in the medical image or the plurality of medical images using a segmentation machine learning model trained to identify the lesion (¶ [0343] “…As the sphere is moved through the volume guided by the user dragging the sphere's center point over the MPR, the voxels that come in contact with the sphere are added to the segmentation”; ¶ [0141] “…a fully convolutional network (FCN) is utilized for segmentation to locate as many lesion candidates”; ¶ [0142] “…Various styles of FCN may be chosen, as long as the FCN performs pixelwise segmentation. Possible segmentation architectures include but are not limited to ENet, U-Net, and their variants”; ¶ 0181] “The network is trained only on 3D patches that contain lesions, though in some implementations non-lesions are also included …The 3D image patches are matched with 3D boolean masks representing whether each pixel within the 3D patch is in a lesion.”);
extracting image features descriptive of the lesion from the medical image, the plurality of medical images, or the segmentation machine learning model (¶ [0460] – ¶ [0463] “…The CNN used for feature extraction may be any of a variety of forms of CNN, including but not limited to: a classification network; an object detection network; a semantic segmentation network; or any combination of the above… outputs of intermediate layers as relevant “features” of the lesion… These feature maps can be used as features to help predict objectives for which the model was not explicitly trained”; ¶ [0475] “…The features include…image features from the most recent thoracic CT exam (current tumor size, change in tumor size since the previous thoracic CT exam, CNN-extracted features for a CNN that was trained to distinguish lesions from blood vessels in CT images e.g., following [Berens 2016]”).
establishing one or more measurements and one or more phenotypical characteristics of the lesion using one or more models (¶ [0142] “…Possible segmentation architectures include but are not limited to ENet, U-Net, and their variants… 2D or 3D FCNs are utilized.”; ¶ [0176] “…the exact FCN that is used for segmentation may vary as long as it performs pixelwise segmentation. 3D extensions of ENet, U-Net, and their variants are all possible”; ¶ [0363] “The system can automatically measure the volume of the nodules…From the volume of each nodule, the maximum diameter in the axial plane and its orthogonal diameter are mathematically calculated and reported”; ¶ [0391] “The system can automatically define features of liver lesions in the different series, comprising enhancement, washout, and corona presence, and then calculates the corresponding LI-RADS score”; ¶ [0461] “…the CNN may have been trained to predict one or more of a variety of different objectives from patient medical images, including but not limited to: features of potentially cancerous lesions, e.g., size, shape, spiculations; features of the surrounding organ, e.g., texture, other (possibly non-cancer) disease; lesion malignancy; changes to any of the above metrics over time,” Taerum discloses ML segmentation (FCN/U-net), then establishes measurements (volume and diameters) and phenotypical characteristics);
generating a medical prediction concerning the lesion with a predictive machine learning model trained to use one or more of the image features descriptive of the lesion, the one or more measurements, or the one or more phenotypical characteristics (¶ [0471] “…One or more models may be combined into an ensemble of models. Each model may be any machine learning model that accepts structured features and performs classification or regression”; ¶ [0475] “…The features include demographic features of the patients (age, sex, etc.), features from histopathological assessment of lesion biopsy (tumor stage, grade, presence of lymph node metastases), features related to medical procedures and complications in the preceding 12 months, and image features from the most recent thoracic CT exam (current tumor size, change in tumor size since the previous thoracic CT exam, CNN-extracted features for a CNN that was trained to distinguish lesions from blood vessels in CT images e.g., following [Berens 2016])”; ¶ [0457] “…In the inference phase 5640 of this implementation, initially a patient is selected for whom inference is to be performed at 5614. Patient data is loaded for the selected patient at 5616 and features are extracted at 5618 in the same manner as they were extracted during training at 5606. Inference is performed with the trained machine learning models 5612 and input features 5618 to predict outcomes for the patient under one or more different treatment scenarios 5620”; Taerum discloses ML model(S) that uses measurements (tumor size, change), segmentation-derived feature (CNN features), and lesion characteristics to provide medical predictions); and
outputting the medical prediction, the one or more measurements, and the one or more phenotypical characteristics (¶ [0374] – ¶ [0375] “The system can automatically report findings and their characterizations… The automatic report can also be a graphical report containing tables and images that describe the evolution of the findings over time.”; ¶ [0399] “…FIG. 53 is a GUI 5300 that shows an excerpt of an automated report that collects all characteristics of each finding”; ¶ [0457] “...The results of inference are then displayed to the user 5622 on a display 5624.”; ¶ [0482] – ¶ [0484] “…Outcome predictions are then returned 5908 and displayed to the user on a display 5912” and FIGs. 60 and 61).
Regarding claim 17:
Taerum teaches the limitations of claim 16 as applied above.
Taerum further teaches: wherein the medical prediction is a longitudinal prediction (¶ [0461] “…the CNN may have been trained to predict one or more of a variety of different objectives from patient medical images, including but not limited to: features of potentially cancerous lesions, e.g., size, shape, spiculations; features of the surrounding organ, e.g., texture, other (possibly non-cancer) disease; lesion malignancy; changes to any of the above metrics over time, using images acquired over time (e.g., over the course of days, months or years)”; ¶ [00475] “…features related to medical procedures and complications in the preceding 12 months, and image features from the most recent thoracic CT exam (current tumor size, change in tumor size since the previous thoracic CT exam, CNN-extracted features for a CNN that was trained to distinguish lesions from blood vessels in CT images e.g., following [Berens 2016]).”; ¶ [0371] “…Measurement may comprise linear dimensions, areas, volumes, and pixel density. These measurements are then compared mathematically to assess changes in size or presentation of the finding, and calculate growth or shrinkage of a finding over time”; ¶ [0375] “…The automatic report can be structured so that findings are presented based on urgency and severity. The automatic report can also be a graphical report containing tables and images that describe the evolution of the findings over time”).
Regarding claim 18:
Taerum teaches the limitations of claim 16 as applied above.
Taerum further teaches: wherein the medical prediction comprises a diagnosis concerning the lesion; a classification of the lesion according to a phenotype or genotype; a prediction of disease progression; a prediction of whether the lesion is likely to respond to a particular treatment; a prediction of whether an apparent growth of the lesion during a treatment represents a true progression or a pseudo-progression caused by the treatment; or a prediction of whether a particular patient is likely to experience a particular side effect (¶ 0323] “the classifier may classify the malignancy, lesion type, cancer subtype or prognosis of the query lesion. “; ¶ 0471] “…one or more models are trained at 5716 to predict patient outcomes. One or more models may be combined into an ensemble of models. Each model may be any machine learning model that accepts structured features and performs classification or regression, including but not limited to: random forests; gradient boosted decision trees; multi-layer perceptrons; or any combination of the above.”; ¶ [0478] “…Outcome predictions would then be separately available under the conditions that one of treatment combination A or treatment combination B is used”; ¶ 0484] “…A table 6002 of treatments along with the associated probability 6006 of lung cancer-associated death for each treatment 6004 is shown”; ¶ [0475] “…The features include…image features from the most recent thoracic CT exam (current tumor size, change in tumor size since the previous thoracic CT exam, CNN-extracted features for a CNN that was trained to distinguish lesions from blood vessels in CT images e.g., following [Berens 2016])”).
Regarding claim 19:
Taerum teaches the limitations of claim 16 as applied above.
Taerum further teaches: wherein the segmentation machine learning model is a deep learning machine model and the one or more image features are deep features extracted from the deep learning machine model (¶ [0176] “..the exact FCN that is used for segmentation may vary as long as it performs pixelwise segmentation. 3D extensions of ENet, U-Net, and their variants are all possible”; ¶ [0181] “…The network is trained only on 3D patches that contain lesions, though in some implementations non-lesions are also included… The 3D image patches are matched with 3D boolean masks representing whether each pixel within the 3D patch is in a lesion”; ¶ [0460] – ¶ [0463] “…The CNN used for feature extraction may be any of a variety of forms of CNN, including but not limited to: a classification network; an object detection network; a semantic segmentation network; or any combination of the above… outputs of intermediate layers as relevant “features” of the lesion… These feature maps can be used as features to help predict objectives for which the model was not explicitly trained”).
Regarding claim 20:
Taerum teaches the limitations of claim 16 as applied above.
Taerum further teaches: wherein the features extracted from the machine segmentation comprise radiomic features (¶ [0363] “The system can automatically measure the volume of the nodules that were detected either automatically or manually. From the volume of each nodule, the maximum diameter in the axial plane and its orthogonal diameter are mathematically calculated and reported. All of these measurements can be edited by the user. Furthermore, from the volume of each nodule, the density of the nodule can also be calculated and displayed in an editable fashion.”).
Regarding claim 21:
Taerum teaches: A system (FIG. 62) comprising:
a segmentation and annotation module having at least one segmentation machine learning model that accepts a medical image and an indication of interest, segments the medical image at least in a vicinity of the indication of interest to identify a lesion, the segmentation and annotation module producing annotations of the lesion using output of the at least one segmentation machine learning model (FIGS. 18-19, FIGS 47-48, and FIG. 51, ¶ [0343] “when a user clicks on some point within one of the MPRs, a sphere 3602 is initialized at that point within the 3D volume…As the sphere is moved through the volume guided by the user dragging the sphere's center point over the MPR, the voxels that come in contact with the sphere are added to the segmentation. This feature is shown in the screenshot 3700 of FIG. 37 and the screenshot 3800 of FIG. 38…The segmentation grows as the sphere follows the mouse movement until the mouse button is released.”; ¶ [0338] “…the system allows a user to move an adjustable radius sphere (or cylinder), also referred to herein as an editing tool, within a volume in order to add voxels to a segmentation…The placement and movement of the sphere may be controlled by the user clicking and dragging (e.g., via a mouse or other pointer) on such an MPR representation of the volume. By alternating between adjusting the position and orientation of the MPR and using an editing tool of the system, the user is able to quickly segment a region of interest as defined by the current application. “; ¶ [0349] “…The application further displays values associated with the physical extent of the segmentation, such as volume of the segmentation, the longest diameter of the segmentation, etc., as shown in the box 4502 on the right side of the screenshot 4500. In at least some implementations, the MPR displays the major diameter and the orthogonal diameter as lines 4504 and 4506, respectively, on a selected segmentation 4508.” ) and
a phenotyping module having at least one phenotyping machine learning model that uses image features descriptive of the lesion in the medical image as input to produce measurements or scores for at least one phenotypical characteristic of the lesion (FIGS. 47-48, and FIG. 51, ¶ [0460] – ¶ [0463] “…The CNN used for feature extraction may be any of a variety of forms of CNN, including but not limited to: a classification network; an object detection network; a semantic segmentation network; or any combination of the above… outputs of intermediate layers as relevant “features” of the lesion… These feature maps can be used as features to help predict objectives for which the model was not explicitly trained”; ¶ [0363] “The system can automatically measure the volume of the nodules…From the volume of each nodule, the maximum diameter in the axial plane and its orthogonal diameter are mathematically calculated and reported”; ¶ [0391] “The system can automatically define features of liver lesions in the different series, comprising enhancement, washout, and corona presence, and then calculates the corresponding LI-RADS score”).
Regarding claim 22:
Taerum teaches the limitations of claim 21 as applied above.
Taerum further teaches: wherein the image features are radiomic features extracted from the medical image one or both of within or around the lesion (¶ [0401] “…Qualitative assessments include the texture, shape, brightness relative to other tissue, and change in brightness over time in cases where contrast is injected into the patient and a time series of scans are available….It is also possible to quantitatively assess texture, shape, and brightness with specialized software”; ¶ [0389] “FIG. 51 shows a screenshot 5100 of segmentation of the liver and calculation of the longest linear diameter 5104 of a lesion 5102. Other measurements the system can capture comprise of liver fat content, fibrosis and texture, as well as measurements of surrounding organs and tissues”; ¶ [0363] “The system can automatically measure the volume of the nodules that were detected either automatically or manually. From the volume of each nodule, the maximum diameter in the axial plane and its orthogonal diameter are mathematically calculated and reported. All of these measurements can be edited by the user. Furthermore, from the volume of each nodule, the density of the nodule can also be calculated and displayed in an editable fashion.”)
Regarding claim 23:
Taerum teaches the limitations of claim 21 as applied above.
Taerum further teaches: wherein the at least one segmentation machine learning model is a deep learning segmentation model and the image features are deep features extracted from the deep learning segmentation. (¶ [0176] “..the exact FCN that is used for segmentation may vary as long as it performs pixelwise segmentation. 3D extensions of ENet, U-Net, and their variants are all possible”; ¶ [0181] “…The network is trained only on 3D patches that contain lesions, though in some implementations non-lesions are also included… The 3D image patches are matched with 3D boolean masks representing whether each pixel within the 3D patch is in a lesion”; ¶ [0460] – ¶ [0463] “…The CNN used for feature extraction may be any of a variety of forms of CNN, including but not limited to: a classification network; an object detection network; a semantic segmentation network; or any combination of the above… outputs of intermediate layers as relevant “features” of the lesion… These feature maps can be used as features to help predict objectives for which the model was not explicitly trained”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 14 is rejected under 35 U.S.C. 103 as being unpatentable over Taerum (US 20200085382) In view of Lou (US 20190371450).
Regarding claim 14:
Taerum teaches the limitations of claim 1 as applied above.
Taerum does not expressly teaches: wherein the first machine learning model and the second machine learning model are trained using multi-task learning.
However, in a related field, Lou teaches: wherein the first machine learning model and the second machine learning model are trained using multi-task learning (FIG.s 7-8 and ¶ [0127]).
Therefore, it would have been obvious to a person of ordinary skill in the art prior to the effective filing date of the claimed invention to have modified Taerum to incorporate the teachings of Lou by including: wherein the first machine learning model and the second machine learning model are trained using multi-task learning in order to regularize and improve prediction when labeled outcome data are scarce.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WASSIM MAHROUKA whose telephone number is (571)272-2945. The examiner can normally be reached Monday-Thursday 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WASSIM MAHROUKA/Primary Examiner, Art Unit 2665