DETAILED ACTION
Acknowledgements
This office action is in response to the claims filed December 01, 2025.
Claims 1, 3-5, 9, 10-12, 13-16, and 18 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment(s)
Claims 1, 3-5, 9, 10-12, 13-16, and 18 are pending.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 4, 5, 10, 11, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over BRYNOLFSSON et. al (hereinafter BRYNOLFSSON) (WO2022008374A1) in view of Browning et. al (hereinafter Browning (US12039728B2) and in further view Tian, Z., Shen, C., Chen, H., & He, T. (2019). Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9627-9636).
As per claim 1, BRYNOLFSSON teaches:
A method for classifying a malignancy risk of a kidney, the method comprising: (“abstract discloses, “The machine learning modules may be used not only to detect presence and locations of such regions within an image, but also to segment the region corresponding to the lesion and/or classify such hotspots based on the likelihood that they are indicative of a true, underlying cancerous lesion.”)
providing imaging data of an anatomy of a subject patient, the imaging data based on computer tomography (CT) and/or magnet resonance imaging (MRI), ([0267] discloses, “Example process 1700 receives as input, and operates on, a 3D PET image 1702 and a 3D CT image 1704. CT image 1704 is input to a first, organ segmentation, machine learning module 1706 that performs segmentation to identify 3D volumes in the CT image that represent particular tissue regions and/or organs of interest, or anatomical groupings of multiple (e.g., related) tissue regions and/or organs. Organ segmentation machine learning module 1706 is, accordingly, used to generate a 3D segmentation map 1708 that identifies, within the CT image, the particular tissue regions and/or organs of interest or anatomical groupings thereof. For example, in certain embodiments segmentation map 1708 identifies two volumes of interest corresponding two anatomical groupings of organs corresponding to an anatomical grouping of high uptake soft-tissue organs comprising a liver, spleen, kidneys, and a urinary bladder, and a second corresponding to an aorta (e.g., thoracic and abdominal part), which is a low uptake soft tissue organ. In certain embodiments, organ segmentation machine learning module 1706 generates an initial segmentation map as output that identifies various individual organs, including those that make up the anatomical groupings of segmentation map 1708, as well as, in certain embodiments, others, and segmentation map 1708 is created from the initial segmentation map (e.g., by assigning volumes corresponding to individual organs of an anatomical grouping a same label).”)
wherein the imaging data comprises at least partially a representation of the kidney of the subject patient; ([0267] discloses, “Example process 1700 receives as input, and operates on, a 3D PET image 1702 and a 3D CT image 1704. CT image 1704 is input to a first, organ segmentation, machine learning module 1706 that performs segmentation to identify 3D volumes in the CT image that represent particular tissue regions and/or organs of interest, or anatomical groupings of multiple (e.g., related) tissue regions and/or organs. Organ segmentation machine learning module 1706 is, accordingly, used to generate a 3D segmentation map 1708 that identifies, within the CT image, the particular tissue regions and/or organs of interest or anatomical groupings thereof. For example, in certain embodiments segmentation map 1708 identifies two volumes of interest corresponding two anatomical groupings of organs corresponding to an anatomical grouping of high uptake soft-tissue organs comprising a liver, spleen, kidneys, and a urinary bladder, and a second corresponding to an aorta (e.g., thoracic and abdominal part), which is a low uptake soft tissue organ. In certain embodiments, organ segmentation machine learning module 1706 generates an initial segmentation map as output that identifies various individual organs, including those that make up the anatomical groupings of segmentation map 1708, as well as, in certain embodiments, others, and segmentation map 1708 is created from the initial segmentation map (e.g., by assigning volumes corresponding to individual organs of an anatomical grouping a same label).”)
detecting,..[…]…anatomical landmarks in the imaging data, ([0011] discloses, “In certain embodiments, the Al-based lesion detection technique described herein augment the functional information obtained from nuclear medicine images with anatomical information obtained from anatomical images, such as x-ray computed tomography (CT) images. For example, machine learning modules utilized in the approaches described herein may receive multiple channels of input, including a first channel corresponding to a portion of a functional, nuclear medicine, image (e.g., a PET image; e.g., a SPECT image), as well as additional channels corresponding to a portion of a co-aligned anatomical (e.g., CT) image and/or anatomical information derived therefrom. Adding anatomical context in this manner may improve accuracy of lesion detection approaches. Anatomical information may also be incorporated into lesion classification approaches applied following detection. For example, in addition to computing lesion index values based on intensities of detected hotspots, hotspots may also be assigned an anatomical label based on their location. For example, detected hotspots may be automatically assigned an label (e.g., an alphanumeric label) based on whether their locations correspond to locations within a prostate, pelvic lymph node, non pelvic lymph node, bone, or a soft-tissue region outside the prostate and lymph nodes.”)
…[…]…segmenting using a first neural network at least one region of the kidney representation based on the imaging data and the anatomical landmarks, wherein the anatomical landmarks are used initialize and constrain the first neural network for segmentation; ([0020] discloses, “In certain embodiments, the machine learning module generates, as output, the 3D hotspot map [e.g., wherein the machine learning module implements a machine learning algorithm (e.g., an artificial neural network (ANN)) trained to segment the 3D functional image (e.g., based at least in part on intensities of voxels of the 3D functional image) to identify the 3D hotspot volumes of the 3D hotspot map (e.g., the 3D hotspot map delineating, for each hotspot, a 3D boundary (e.g., an irregular boundary) of the hotspot, thereby identifying the 3D hotspot volumes (e.g., enclosed by the 3D hotspot boundaries)); e.g., wherein the machine learning module implements a machine learning algorithm trained to determine, for each voxel of at least a portion of the 3D functional image, a hotspot likelihood value representing a likelihood that the voxel corresponds to a hotspot (e.g., and step (b) comprises performing one or more subsequent post-processing steps, such as -9- thresholding, to identify the 3D hotspot volumes of the 3D hotspot map using the hotspot likelihood values (e.g., the 3D hotspot map delineating, for each hotspot, a 3D boundary (e.g., an irregular boundary) of the hotspot, identifying the 3D hotspot volumes (e.g., enclosed by the 3D hotspot boundaries)))].” And see [0029] discloses, “In certain embodiments, the method comprises determining, by the processor (e.g., automatically), for each hotspot, an anatomical classification corresponding to a particular anatomical region and/or group of anatomical regions within the subject in which the potential cancerous lesion that the hotspot represents is determined [e.g., by the processor (e.g., based on a received and/or detennined 3D segmentation map)] to be located [e.g., within a prostate, a pelvic lymph node, a non-pelvic lymph node, a bone (e.g., a bone metastatic region), and a soft tissue region not situated in prostate or lymph node].”)
detecting using a second neural network one or more suspected lesions of the segmented kidney representation; [0022] discloses, “In certain embodiments, step (d) comprises using a second machine learning module to determine, for each hotspot of the portion, the lesion likelihood classification [e.g., wherein the machine learning module implements a machine learning algorithm trained to detect hotspots (e.g., to generate, as output, the hotspot list and/or the 3D hotspot map) and to determine, for each hotspot, the lesion likelihood classification for the hotpot]. In certain embodiments, step (d) comprises using a second machine learning module (e.g., a hotspot classification module) to determine the lesion likelihood classification for each hotspot [e.g., based at least in part on one or more members selected from the group consisting of: intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map; e.g., wherein the second machine learning module receives one or more channels of input corresponding to one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map].” And see [0035] discloses, “[e.g., wherein the second machine learning module implements a machine learning algorithm (e.g., an artificial neural network (ANN)) trained to segment the 3D functional image based at least in part on the hotspot list along with intensities of voxels of the 3D functional image to identify the 3D hotspot volumes of the 3D hotspot map;”)
and classifying the detected suspected lesion with the malignancy risk using, …[…]… classifies the malignancy risk based on the imaging data and at least histopathologic data;([0036] discloses, “In certain embodiments, step (e) comprises using a third machine learning module (e.g., a hotspot classification module) to determine the lesion likelihood classification for each hotspot [e.g., based at least in part on one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map; e.g., wherein the third machine learning module receives one or more channels of input corresponding to one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map].” And see [0037] discloses, “In certain embodiments, the method comprises: (f) selecting, by the processor, based at least in part on the lesion likelihood classifications for the hotspots, a subset of the one or more hotspots corresponding to hotspots having a high likelihood of corresponding to cancerous lesions (e.g., for inclusion in a report; e.g., for use in computing one or more risk index values for the subject).” And see [0210] discloses, “In certain embodiments, as described herein, machine learning modules implement one or more machine learning techniques, such as random forest classifiers, artificial neural networks (ANNs), convolutional neural networks (CNNs), and the like.” And see [0196]-[0202]/ examiner notes instant app defines in [015]-[016] a deep profiler such as with encoder within CNN)
wherein training data for configured at least two of the reinforcement learning, first neural network, second neural network, or deep profiler comprises training data fragments based on a same subject patient. [0302] discloses, “This study used a subset of the data that contained only lesions with at least one other lesion of the same type in the same patient. This resulted in a dataset with 684 manually segmented lesion uptake volumes (278 in bone, 357 in lymph nodes, 49 in prostate) across 92 patients. Automatic refinement by thresholding was performed, and the output was compared to the original volumes. Performance was measured by a weighted average of rank correlations between refined volumes and original volumes within a patient and tissue type, with the weight given by the number of segmented hotspots volumes in the patient. This performance measure indicates whether the relative sizes between segmented hotspot volumes have been preserved, but disregards absolute sizes, which are subjectively defined since uptake volumes do not have clear boundaries. However, for a particular patient and tissue type, the same nuclear medicine reader made all annotations, and they can hence be assumed to have been made in a systematic manner, with a smaller lesion annotation actually reflecting a smaller uptake volume compared to a larger lesion annotation.” And see [0352] discloses, “Training the CNN models includes an iterative minimization problem where the training algorithm updates model parameters to lower the segmentation error. Segmentation error is defined as the deviation from a perfect overlap between manual segmentation and the CNN-model segmentation. Each neural network used for organ segmentation was trained to configure optimal parameters and weights. The training data for developing the neural networks for aPROMISE, as described above, consists of low dose CT images with manually segmented and labelled body parts.”)
However, BRYNOLFSSON does not teach the underlined portions:
detecting, using an agent and reinforcement learning, anatomical landmarks in the imaging data,
wherein the agent learns using an action-value function to search for objects in the image based on an exploration of an environment using an off-policy E-greedy approach;
and classifying the detected suspected lesion with the malignancy risk using a third neural network, wherein the third neural network is a deep profiler, wherein the deep profiler comprises a task-specific network configured to generate an image signature, wherein the third neural network classifies the malignancy risk based on the imaging data and at least histopathologic data;
However, Browning does teach the underlined portions:
detecting, using an agent and reinforcement learning, anatomical landmarks in the imaging data, (Col. 5 lines 2-7)
wherein the agent learns using an action-value function to search for objects in the image based on an exploration of an environment using an off-policy E-greedy approach; (Table 1 “training exploration policy epsilon greedy” and see Col. 13 lines 16-19 and see Col. 15 lines 65-67 and Col. 16 lines 1-65)
However, Browning also does not teach the underlined portions:
and classifying the detected suspected lesion with the malignancy risk using a third neural network, wherein the third neural network is a deep profiler, wherein the deep profiler comprises a task-specific network configured to generate an image signature, wherein the third neural network classifies the malignancy risk based on the imaging data and at least histopathologic data;
However, Tian does teach the underlined portions:
and classifying the detected suspected lesion with the malignancy risk using a third neural network, wherein the third neural network is a deep profiler, wherein the deep profiler comprises a task-specific network configured to generate an image signature, wherein the third neural network classifies the malignancy risk based on the imaging data and at least histopathologic data;
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions and Browning’s teachings of utilizing using an agent and reinforcement learning with active function using off -policy E-greedy approach with Tian’s teachings of FCOS and other techniques such as Fast RCNN as cited, the motivation being BRYNOLFSSON teaches that there is a significant need for improved tools that facilitate and improve accuracy of image review and analysis for cancer diagnosis and treatment, (see [0008]) and Tian teaches the challenge of segmentation (see introduction) therefore it would only optimize making better choices which more efficiently and accurately a determine malignancy risk and lesions without rendering the primary reference inoperable.
As per claim 3, BRYNOLFSSON further teaches:
The method of claim 1, wherein classifying comprises classifying using the deep profiler, the deep profiler comprising an encoder for extracting imaging features. (;([0036] discloses, “In certain embodiments, step (e) comprises using a third machine learning module (e.g., a hotspot classification module) to determine the lesion likelihood classification for each hotspot [e.g., based at least in part on one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map; e.g., wherein the third machine learning module receives one or more channels of input corresponding to one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map].” And see [0037] discloses, “In certain embodiments, the method comprises: (f) selecting, by the processor, based at least in part on the lesion likelihood classifications for the hotspots, a subset of the one or more hotspots corresponding to hotspots having a high likelihood of corresponding to cancerous lesions (e.g., for inclusion in a report; e.g., for use in computing one or more risk index values for the subject).” And see [0210] discloses, “In certain embodiments, as described herein, machine learning modules implement one or more machine learning techniques, such as random forest classifiers, artificial neural networks (ANNs), convolutional neural networks (CNNs), and the like.”)
However, BRYNOLFSSON and Browning do not teach the underlined portions:
The method of claim 1, wherein classifying comprises classifying using the deep profiler, the deep profiler comprising an encoder for extracting imaging features.
However, Tian does teach the underlined portions:
The method of claim 1, wherein classifying comprises classifying using the deep profiler, the deep profiler comprising an encoder for extracting imaging features. (see related work discloses, “Anchor-based detectors inherit the ideas from traditional sliding-window and proposal based detectors such as Fast R-CNN [6]. In anchor-based detectors, the anchor boxes can be viewed as pre-defined sliding windows or proposals, which are classified as positive or negative patches, with an extra offsets regression to refine the prediction of bounding box locations. Therefore, the anchor boxes in these detectors may be viewed as training samples. Unlike previous detectors like Fast RCNN, which compute image features for each sliding window/ proposal repeatedly, anchor boxes make use of the feature maps of CNNs and avoid repeated feature computation, speeding up detection process dramatically.” / Fast R-CNN to one of ordinary skill under BRI is interpreted to be an encoder)
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings and of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions and Browning’s teachings of the uncertainty of landmark detection (see Col. 2) with Tian’s teachings of FCOS and Fast RCNN as cited for the same reasons given in claim 13.
As per claim 4, BRYNOLFSSON and Browning do not explicitly teach:
The method of claim 3, wherein the encoder is a convolutional neural network.;
However, Tian does explicitly teach:
The method of claim 3, wherein the encoder is a convolutional neural network.; (see related work discloses, “Anchor-based detectors inherit the ideas from traditional sliding-window and proposal based detectors such as Fast R-CNN [6]. In anchor-based detectors, the anchor boxes can be viewed as pre-defined sliding windows or proposals, which are classified as positive or negative patches, with an extra offsets regression to refine the prediction of bounding box locations. Therefore, the anchor boxes in these detectors may be viewed as training samples. Unlike previous detectors like Fast RCNN, which compute image features for each sliding window/ proposal repeatedly, anchor boxes make use of the feature maps of CNNs and avoid repeated feature computation, speeding up detection process dramatically.” / Fast R-CNN to one of ordinary skill under BRI is interpreted to be an encoder)
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings and of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions and Browning’s teachings of the uncertainty of landmark detection (see Col. 2) with Tian’s teachings of FCOS and Fast RCNN as cited for the same reasons given in claim 13.
As per claim 5, BRYNOLFSSON further teaches:
The method of claim 3, wherein classifying comprises estimating at least one malignancy risk indicator. ([0036] discloses, “In certain embodiments, step (e) comprises using a third machine learning module (e.g., a hotspot classification module) to determine the lesion likelihood classification for each hotspot [e.g., based at least in part on one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map; e.g., wherein the third machine learning module receives one or more channels of input corresponding to one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map].” And see [0037] discloses, “In certain embodiments, the method comprises: (f) selecting, by the processor, based at least in part on the lesion likelihood classifications for the hotspots, a subset of the one or more hotspots corresponding to hotspots having a high likelihood of corresponding to cancerous lesions (e.g., for inclusion in a report; e.g., for use in computing one or more risk index values for the subject).” And see [0210] discloses, “In certain embodiments, as described herein, machine learning modules implement one or more machine learning techniques, such as random forest classifiers, artificial neural networks (ANNs), convolutional neural networks (CNNs), and the like.”)
As per claim 10, BRYNOLFSSON and Browning do not teach:
The method claim 1, wherein detecting the one or more suspected lesions comprises detecting based on a fully convolutional one-stage object detection of the second neural network.
However, Tian does teach:
The method claim 1, wherein detecting the one or more suspected lesions comprises detecting based on a fully convolutional one-stage object detection of the second neural network.(abstract discloses, “We propose a fully convolutional one-stage object detec tor (FCOS) to solve object detection in a per-pixel predic tion fashion, analogue to semantic segmentation. Almost all state-of-the-art object detectors such as RetinaNet, SSD, YOLOv3, and Faster R-CNN rely on pre-defined anchor boxes. In contrast, our proposed detector FCOS is anchor box free, as well as proposal free. By eliminating the pre defined set of anchor boxes, FCOS completely avoids the complicated computation related to anchor boxes such as calculating overlapping during training. More importantly, wealso avoid all hyper-parameters related to anchor boxes, which are often very sensitive to the final detection perfor mance.” And see section materials discloses, “The liver data comes from the public dataset Sliver 07 [30]. The dataset consists of abdominal CT scans from 20 anesthetized patients (more than 5500 liver slices) and regional annotations. The CT scans are in grayscale with 512 × 512 pixels.”)
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings and of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions and Browning’s teachings of the uncertainty of landmark detection (see Col. 2) with Tian’s teachings of FCOS and Fast RCNN as cited for the same reasons given in claim 13.
As per claim 11, BRYNOLFSSON further teaches:
The method of claim 1, wherein providing the imaging data comprises at least one of the following: providing based on computer tomography and/or magnet resonance imaging, and/or providing at least partially a 3D illustration of the anatomy of the subject patient. ([0267] discloses, “Example process 1700 receives as input, and operates on, a 3D PET image 1702 and a 3D CT image 1704. CT image 1704 is input to a first, organ segmentation, machine learning module 1706 that performs segmentation to identify 3D volumes in the CT image that represent particular tissue regions and/or organs of interest, or anatomical groupings of multiple (e.g., related) tissue regions and/or organs. Organ segmentation machine learning module 1706 is, accordingly, used to generate a 3D segmentation map 1708 that identifies, within the CT image, the particular tissue regions and/or organs of interest or anatomical groupings thereof. For example, in certain embodiments segmentation map 1708 identifies two volumes of interest corresponding two anatomical groupings of organs corresponding to an anatomical grouping of high uptake soft-tissue organs comprising a liver, spleen, kidneys, and a urinary bladder, and a second corresponding to an aorta (e.g., thoracic and abdominal part), which is a low uptake soft tissue organ. In certain embodiments, organ segmentation machine learning module 1706 generates an initial segmentation map as output that identifies various individual organs, including those that make up the anatomical groupings of segmentation map 1708, as well as, in certain embodiments, others, and segmentation map 1708 is created from the initial segmentation map (e.g., by assigning volumes corresponding to individual organs of an anatomical grouping a same label).”)
As per claim 12, BRYNOLFSSON further teaches:
The method of claim 1, further comprising: converting the imaging data from at least a partially 3D illustration of the anatomy of the subject patient to a 2D illustration of the anatomy of the subject patient. ([0262] discloses, “For example, the viewer may allow a user to select and view various 2D slices, along particular (e.g., selected) cross-sectional planes, of 3D images. In certain embodiments, the viewer allows a user to view a maximum intensity projection (MIP) of 3D image data.”)
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over BRYNOLFSSON et. al (hereinafter BRYNOLFSSON) (WO2022008374A1) in view of Browning et. al (hereinafter Browning (US12039728B2), in view of Tian, Z., Shen, C., Chen, H., & He, T. (2019). Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9627-9636)., and in further view of Yang, D., Xu, D., Zhou, S. K., Georgescu, B., Chen, M., Grbic, S., ... & Comaniciu, D. (2017, September). Automatic liver segmentation using an adversarial image-to-image network. In International conference on medical image computing and computer-assisted intervention (pp. 507-515). Cham: Springer International Publishing
As per claim 9, BRYNOLFSSON, Browning, and Tian do not explicitly teach:
The method of claim 1, wherein the first neural network is a convolutional encoder-decoder architecture or a multi-level feature concatenation and deep supervision architecture
However, Yang does explicitly teach:
The method of claim 1, wherein the first neural network is a convolutional encoder-decoder architecture or a multi-level feature concatenation and deep supervision architecture.(see abstract discloses, “In this paper, we propose an automatic and efficient algorithm to segment liver from 3D CT volumes. Adeep image-to-image network (DI2IN) is first deployed to generate the liver segmentation, employing a convolutional encoder-decoder architecture combined with multi-level feature concatenation and deep super vision. Then an adversarial network is utilized during training process to discriminate the output of DI2IN from ground truth, which further boosts the performance of DI2IN.”)
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions and Browning’s teachings, and Tian’s teachings with Yang’s teachings of liver segmentation with DI2IN as cited, the motivation being BRYNOLFSSON teaches that there is a significant need for improved tools that facilitate and improve accuracy of image review and analysis for cancer diagnosis and treatment, (see [0008]) therefore it would only optimize making better choices which more efficiently and accurately a determine malignancy risk and lesions without rendering the primary reference inoperable.
Claims 13, 14, 15, 16, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over BRYNOLFSSON et. al (hereinafter BRYNOLFSSON) (WO2022008374A1) in view of Yang, D., Xu, D., Zhou, S. K., Georgescu, B., Chen, M., Grbic, S., ... & Comaniciu, D. (2017, September). Automatic liver segmentation using an adversarial image-to-image network. In International conference on medical image computing and computer-assisted intervention (pp. 507-515). Cham: Springer International Publishing., and in further view of Tian, Z., Shen, C., Chen, H., & He, T. (2019). Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9627-9636).
As per claim 13, BRYNOLFSSON teaches:
A system for classifying a malignancy risk scoring of a kidney, (“abstract discloses, “The machine learning modules may be used not only to detect presence and locations of such regions within an image, but also to segment the region corresponding to the lesion and/or classify such hotspots based on the likelihood that they are indicative of a true, underlying cancerous lesion.”)
the system comprising: an interface configured to provide imaging data of an anatomy of a subject patient ([0012] discloses, “In certain embodiments, detected hotspots and associated information, such as computed lesion index values and anatomical labeling, are displayed with an interactive graphical user interface (GUI) so as to allow for review by a medical professional, such as a physician, radiologist, technician, etc.”)
the imaging data based on computer tomography (CT) and/or magnet resonance imaging(MR), wherein the imaging data comprises at least partially a representation of a kidney of the subject patient; ([0267] discloses, “Example process 1700 receives as input, and operates on, a 3D PET image 1702 and a 3D CT image 1704. CT image 1704 is input to a first, organ segmentation, machine learning module 1706 that performs segmentation to identify 3D volumes in the CT image that represent particular tissue regions and/or organs of interest, or anatomical groupings of multiple (e.g., related) tissue regions and/or organs. Organ segmentation machine learning module 1706 is, accordingly, used to generate a 3D segmentation map 1708 that identifies, within the CT image, the particular tissue regions and/or organs of interest or anatomical groupings thereof. For example, in certain embodiments segmentation map 1708 identifies two volumes of interest corresponding two anatomical groupings of organs corresponding to an anatomical grouping of high uptake soft-tissue organs comprising a liver, spleen, kidneys, and a urinary bladder, and a second corresponding to an aorta (e.g., thoracic and abdominal part), which is a low uptake soft tissue organ. In certain embodiments, organ segmentation machine learning module 1706 generates an initial segmentation map as output that identifies various individual organs, including those that make up the anatomical groupings of segmentation map 1708, as well as, in certain embodiments, others, and segmentation map 1708 is created from the initial segmentation map (e.g., by assigning volumes corresponding to individual organs of an anatomical grouping a same label).”)
a processor configured to…[…]… to segment at least one region of the kidney representation which is based on the imaging data, ([0017] discloses, “In certain embodiments, the method comprises automatically segmenting, by the processor, the 3D anatomical image, thereby creating the 3D segmentation map.” And see [0020] discloses, “In certain embodiments, the machine learning module generates, as output, the 3D hotspot map [e.g., wherein the machine learning module implements a machine learning algorithm (e.g., an artificial neural network (ANN)) trained to segment the 3D functional image (e.g., based at least in part on intensities of voxels of the 3D functional image) to identify the 3D hotspot volumes of the 3D hotspot map (e.g., the 3D hotspot map delineating, for each hotspot, a 3D boundary (e.g., an irregular boundary) of the hotspot, thereby identifying the 3D hotspot volumes (e.g., enclosed by the 3D hotspot boundaries)); e.g., wherein the machine learning module implements a machine learning algorithm trained to determine, for each voxel of at least a portion of the 3D functional image, a hotspot likelihood value representing a likelihood that the voxel corresponds to a hotspot (e.g., and step (b) comprises performing one or more subsequent post-processing steps, such as -9- thresholding, to identify the 3D hotspot volumes of the 3D hotspot map using the hotspot likelihood values (e.g., the 3D hotspot map delineating, for each hotspot, a 3D boundary (e.g., an irregular boundary) of the hotspot, identifying the 3D hotspot volumes (e.g., enclosed by the 3D hotspot boundaries)))].” And see [0029] discloses, “In certain embodiments, the method comprises determining, by the processor (e.g., automatically), for each hotspot, an anatomical classification corresponding to a particular anatomical region and/or group of anatomical regions within the subject in which the potential cancerous lesion that the hotspot represents is determined [e.g., by the processor (e.g., based on a received and/or detennined 3D segmentation map)] to be located [e.g., within a prostate, a pelvic lymph node, a non-pelvic lymph node, a bone (e.g., a bone metastatic region), and a soft tissue region not situated in prostate or lymph node].”)
configured to …[…]…to detect one or more suspected lesions of the segmented kidney representation, ([0021] discloses, “In certain embodiments, the method comprises: (d) determining, by the processor, for each hotspot of at least a portion of the hotspots, a lesion likelihood classification corresponding to a likelihood of the hotspot representing a lesion within the subject [e.g., a binary classification indicative of whether the hotspot is a true lesion or not; e.g., a likelihood value on a scale (e.g., a floating point value ranging from zero to one) representing a likelihood of the hotspot representing a true lesion].” And see [0022] discloses, “In certain embodiments, step (d) comprises using a second machine learning module to determine, for each hotspot of the portion, the lesion likelihood classification [e.g., wherein the machine learning module implements a machine learning algorithm trained to detect hotspots (e.g., to generate, as output, the hotspot list and/or the 3D hotspot map) and to determine, for each hotspot, the lesion likelihood classification for the hotpot]. In certain embodiments, step (d) comprises using a second machine learning module (e.g., a hotspot classification module) to determine the lesion likelihood classification for each hotspot [e.g., based at least in part on one or more members selected from the group consisting of: intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map; e.g., wherein the second machine learning module receives one or more channels of input corresponding to one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map].” And see [0035] discloses, “[e.g., wherein the second machine learning module implements a machine learning algorithm (e.g., an artificial neural network (ANN)) trained to segment the 3D functional image based at least in part on the hotspot list along with intensities of voxels of the 3D functional image to identify the 3D hotspot volumes of the 3D hotspot map;”)
and configured …[…]…classify the detected suspected lesion with a malignancy risk. ;([0028] discloses, “In certain embodiments, the method comprises using the determined lesion index values compute (e.g., automatically, by the processor) an overall risk index for the subject, indicative of a caner status and/or risk for the subject.” And see [0036] discloses, “In certain embodiments, step (e) comprises using a third machine learning module (e.g., a hotspot classification module) to determine the lesion likelihood classification for each hotspot [e.g., based at least in part on one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map; e.g., wherein the third machine learning module receives one or more channels of input corresponding to one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map].” And see [0037] discloses, “In certain embodiments, the method comprises: (f) selecting, by the processor, based at least in part on the lesion likelihood classifications for the hotspots, a subset of the one or more hotspots corresponding to hotspots having a high likelihood of corresponding to cancerous lesions (e.g., for inclusion in a report; e.g., for use in computing one or more risk index values for the subject).” And see [0210] discloses, “In certain embodiments, as described herein, machine learning modules implement one or more machine learning techniques, such as random forest classifiers, artificial neural networks (ANNs), convolutional neural networks (CNNs), and the like.” And see [0196]-[0202]/ examiner notes instant app defines in [015]-[016] a deep profiler such as with encoder within CNN)
However, BRYNOLFSSON does not teach the underlined portions:
a memory configured to store a first neural network comprising a convolutional encoder-decoder architecture with multi-level feature concatenation and deep supervision and further comprising adversarial image-to-image refinement by a discriminator network,
a second neural network configured as a fully convolutional one-stage object detection network that is anchor-free and proposal-free
and a third neural network comprising a deep profiler comprising a task-specific network configured to generate an image signature,
wherein at least two of the neural networks are trained using training data fragments based on a same patient;
a processor configured to use the first neural network to segment at least one region of the kidney representation which is based on the imaging data,
configured to use the second neural network to detect one or more suspected lesions of the segmented kidney representation,
and configured to implement the third neural network comprising a deep profiler to classify the detected suspected lesion with a malignancy risk.
However, Yang does teach the underlined portions:
a memory configured to store a first neural network comprising a convolutional encoder-decoder architecture with multi-level feature concatenation and deep supervision and further comprising adversarial image-to-image refinement by a discriminator network,
a processor configured to use the first neural network to segment at least one region of the kidney representation which is based on the imaging data, (see abstract discloses, “In this paper, we propose an automatic and efficient algorithm to segment liver from 3D CT volumes. Adeep image-to-image network (DI2IN) is first deployed to generate the liver segmentation, employing a convolutional encoder-decoder architecture combined with multi-level feature concatenation and deep super vision. Then an adversarial network is utilized during training process to discriminate the output of DI2IN from ground truth, which further boosts the performance of DI2IN.”)
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions with Yang’s teachings of liver segmentation with DI2IN as cited, the motivation being BRYNOLFSSON teaches that there is a significant need for improved tools that facilitate and improve accuracy of image review and analysis for cancer diagnosis and treatment, (see [0008]) therefore it would only optimize making better choices which more efficiently and accurately a determine malignancy risk and lesions without rendering the primary reference inoperable.
Yang also does not teach the underlined portions:
a second neural network configured as a fully convolutional one-stage object detection network that is anchor-free and proposal-free
and a third neural network comprising a deep profiler comprising a task-specific network configured to generate an image signature,
wherein at least two of the neural networks are trained using training data fragments based on a same patient;
configured to use the second neural network to detect one or more suspected lesions of the segmented kidney representation,
and configured to implement the third neural network comprising a deep profiler to classify the detected suspected lesion with a malignancy risk.
However, Tian does teach the underlined portions
a second neural network configured as a fully convolutional one-stage object detection network that is anchor-free and proposal-free, (abstract discloses, “We propose a fully convolutional one-stage object detec tor (FCOS) to solve object detection in a per-pixel predic tion fashion, analogue to semantic segmentation. Almost all state-of-the-art object detectors such as RetinaNet, SSD, YOLOv3, and Faster R-CNN rely on pre-defined anchor boxes. In contrast, our proposed detector FCOS is anchor box free, as well as proposal free. By eliminating the pre defined set of anchor boxes, FCOS completely avoids the complicated computation related to anchor boxes such as calculating overlapping during training. More importantly, wealso avoid all hyper-parameters related to anchor boxes, which are often very sensitive to the final detection perfor mance.” And see section materials discloses, “The liver data comes from the public dataset Sliver 07 [30]. The dataset consists of abdominal CT scans from 20 anesthetized patients (more than 5500 liver slices) and regional annotations. The CT scans are in grayscale with 512 × 512 pixels.”)
and a third neural network comprising a deep profiler comprising a task-specific network configured to generate an image signature, (see related work discloses, “Anchor-based detectors inherit the ideas from traditional sliding-window and proposal based detectors such as Fast R-CNN [6]. In anchor-based detectors, the anchor boxes can be viewed as pre-defined sliding windows or proposals, which are classified as positive or negative patches, with an extra offsets regression to refine the prediction of bounding box locations. Therefore, the anchor boxes in these detectors may be viewed as training samples. Unlike previous detectors like Fast RCNN, which compute image features for each sliding window/ proposal repeatedly, anchor boxes make use of the feature maps of CNNs and avoid repeated feature computation, speeding up detection process dramatically.” / examiner notes features space plus feature vectors used in anchor based CNN is interpreted under BRI as a type of image signature.)
wherein at least two of the neural networks are trained using training data fragments based on a same patient; (abstract discloses, “We propose a fully convolutional one-stage object detec tor (FCOS) to solve object detection in a per-pixel predic tion fashion, analogue to semantic segmentation. Almost all state-of-the-art object detectors such as RetinaNet, SSD, YOLOv3, and Faster R-CNN rely on pre-defined anchor boxes. In contrast, our proposed detector FCOS is anchor box free, as well as proposal free. By eliminating the pre defined set of anchor boxes, FCOS completely avoids the complicated computation related to anchor boxes such as calculating overlapping during training. More importantly, wealso avoid all hyper-parameters related to anchor boxes, which are often very sensitive to the final detection perfor mance.” And see section materials discloses, “The liver data comes from the public dataset Sliver 07 [30]. The dataset consists of abdominal CT scans from 20 anesthetized patients (more than 5500 liver slices) and regional annotations. The CT scans are in grayscale with 512 × 512 pixels.”)
configured to use the second neural network to detect one or more suspected lesions of the segmented kidney representation, (abstract discloses, “We propose a fully convolutional one-stage object detec tor (FCOS) to solve object detection in a per-pixel predic tion fashion, analogue to semantic segmentation. Almost all state-of-the-art object detectors such as RetinaNet, SSD, YOLOv3, and Faster R-CNN rely on pre-defined anchor boxes. In contrast, our proposed detector FCOS is anchor box free, as well as proposal free. By eliminating the pre defined set of anchor boxes, FCOS completely avoids the complicated computation related to anchor boxes such as calculating overlapping during training. More importantly, wealso avoid all hyper-parameters related to anchor boxes, which are often very sensitive to the final detection perfor mance.” And see section materials discloses, “The liver data comes from the public dataset Sliver 07 [30]. The dataset consists of abdominal CT scans from 20 anesthetized patients (more than 5500 liver slices) and regional annotations. The CT scans are in grayscale with 512 × 512 pixels.”)
and configured to implement the third neural network comprising a deep profiler to classify the detected suspected lesion with a malignancy risk. (see related work discloses, “Anchor-based detectors inherit the ideas from traditional sliding-window and proposal based detectors such as Fast R-CNN [6]. In anchor-based detectors, the anchor boxes can be viewed as pre-defined sliding windows or proposals, which are classified as positive or negative patches, with an extra offsets regression to refine the prediction of bounding box locations. Therefore, the anchor boxes in these detectors may be viewed as training samples. Unlike previous detectors like Fast RCNN, which compute image features for each sliding window/ proposal repeatedly, anchor boxes make use of the feature maps of CNNs and avoid repeated feature computation, speeding up detection process dramatically.” / examiner notes features space plus feature vectors used in anchor based CNN is interpreted under BRI as a type of image signature.)
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings and of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions and Yang’s teachings of liver segmentation with DI2IN with Tian’s teachings of FCOS and fast RCNN as cited, the motivation being BRYNOLFSSON teaches that there is a significant need for improved tools that facilitate and improve accuracy of image review and analysis for cancer diagnosis and treatment, (see [0008]), Yang teaches the complex nature of organ segmentation (see abstract), and Tian teaches complexity of segmentation (see introduction) therefore it would only optimize the predictions and which more efficiently and accurately a determine malignancy risk and lesions without rendering the primary reference inoperable as it stems from convolutional networks.
As per claim 14, BRYNOLFSSON teaches:
The system of claim 13, …[…]…is configured to classify the malignancy risk based on imaging data and non-imaging data, wherein the non-imaging data comprises at least histopathologic data. ;([0036] discloses, “In certain embodiments, step (e) comprises using a third machine learning module (e.g., a hotspot classification module) to determine the lesion likelihood classification for each hotspot [e.g., based at least in part on one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map; e.g., wherein the third machine learning module receives one or more channels of input corresponding to one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map].” And see [0037] discloses, “In certain embodiments, the method comprises: (f) selecting, by the processor, based at least in part on the lesion likelihood classifications for the hotspots, a subset of the one or more hotspots corresponding to hotspots having a high likelihood of corresponding to cancerous lesions (e.g., for inclusion in a report; e.g., for use in computing one or more risk index values for the subject).” And see [0210] discloses, “In certain embodiments, as described herein, machine learning modules implement one or more machine learning techniques, such as random forest classifiers, artificial neural networks (ANNs), convolutional neural networks (CNNs), and the like.” And see [0196]-[0202])
However, BRYNOLFSSON and Yang do not explicitly teach the underlined portion:
The system of claim 13, wherein the deep profiler is configured to classify the malignancy risk based on imaging data and non-imaging data, wherein the non-imaging data comprises at least histopathologic data.
However, Tian does teach the underlined portion:
The system of claim 13, wherein the deep profiler is configured to classify the malignancy risk based on imaging data and non-imaging data, wherein the non-imaging data comprises at least histopathologic data. ((see related work discloses, “Anchor-based detectors inherit the ideas from traditional sliding-window and proposal based detectors such as Fast R-CNN [6]. In anchor-based detectors, the anchor boxes can be viewed as pre-defined sliding windows or proposals, which are classified as positive or negative patches, with an extra offsets regression to refine the prediction of bounding box locations. Therefore, the anchor boxes in these detectors may be viewed as training samples. Unlike previous detectors like Fast RCNN, which compute image features for each sliding window/ proposal repeatedly, anchor boxes make use of the feature maps of CNNs and avoid repeated feature computation, speeding up detection process dramatically.” / examiner notes features space plus feature vectors used in anchor based CNN is interpreted under BRI as a type of image signature.)
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings and of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions and Yang’s teachings of liver segmentation with DI2IN with Tian’s teachings of FCOS and other teachings such fast CNN as cited for the same reasons given in claim 13.
As per claim 15, BRYNOLFSSON further teaches:
The system of claim 13, wherein …[…]…to extract imaging features. ([0036] discloses, “In certain embodiments, step (e) comprises using a third machine learning module (e.g., a hotspot classification module) to determine the lesion likelihood classification for each hotspot [e.g., based at least in part on one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map; e.g., wherein the third machine learning module receives one or more channels of input corresponding to one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map].” And see [0037] discloses, “In certain embodiments, the method comprises: (f) selecting, by the processor, based at least in part on the lesion likelihood classifications for the hotspots, a subset of the one or more hotspots corresponding to hotspots having a high likelihood of corresponding to cancerous lesions (e.g., for inclusion in a report; e.g., for use in computing one or more risk index values for the subject).” And see [0210] discloses, “In certain embodiments, as described herein, machine learning modules implement one or more machine learning techniques, such as random forest classifiers, artificial neural networks (ANNs), convolutional neural networks (CNNs), and the like.”)
However, BRYNOLFSSON and Yang do not teach the underlined portions:
The system of claim 13, wherein the deep profiler further comprises an encoder to extract imaging features.
However, Tian does teach the underlined portions:
The system of claim 13, wherein the deep profiler further comprises an encoder to extract imaging features.(see related work discloses, “Anchor-based detectors inherit the ideas from traditional sliding-window and proposal based detectors such as Fast R-CNN [6]. In anchor-based detectors, the anchor boxes can be viewed as pre-defined sliding windows or proposals, which are classified as positive or negative patches, with an extra offsets regression to refine the prediction of bounding box locations. Therefore, the anchor boxes in these detectors may be viewed as training samples. Unlike previous detectors like Fast RCNN, which compute image features for each sliding window/ proposal repeatedly, anchor boxes make use of the feature maps of CNNs and avoid repeated feature computation, speeding up detection process dramatically.” / Fast R-CNN to one of ordinary skill under BRI is interpreted to be a decoder)
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings and of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions and Yang’s teachings of liver segmentation with DI2IN with Tian’s teachings of FCOS and Fast RCNN as cited for the same reasons given in claim 13.
As per claim 16, BRYNOLFSSON further teaches:
The system of claim 13, wherein the deep profiler further comprises a decoder configured to estimate at least one malignancy risk indicator. ([0036] discloses, “In certain embodiments, step (e) comprises using a third machine learning module (e.g., a hotspot classification module) to determine the lesion likelihood classification for each hotspot [e.g., based at least in part on one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map; e.g., wherein the third machine learning module receives one or more channels of input corresponding to one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map].” And see [0037] discloses, “In certain embodiments, the method comprises: (f) selecting, by the processor, based at least in part on the lesion likelihood classifications for the hotspots, a subset of the one or more hotspots corresponding to hotspots having a high likelihood of corresponding to cancerous lesions (e.g., for inclusion in a report; e.g., for use in computing one or more risk index values for the subject).” And see [0210] discloses, “In certain embodiments, as described herein, machine learning modules implement one or more machine learning techniques, such as random forest classifiers, artificial neural networks (ANNs), convolutional neural networks (CNNs), and the like.”)
However, BRYNOLFSSON and Yang do not teach the underlined portions:
The system of claim 13, wherein the deep profiler further comprises a decoder configured to estimate at least one malignancy risk indicator.
However, Tian does teach the underlined portions:
The system of claim 13, wherein the deep profiler further comprises a decoder configured to estimate at least one malignancy risk indicator.(see related work discloses, “Anchor-based detectors inherit the ideas from traditional sliding-window and proposal based detectors such as Fast R-CNN [6]. In anchor-based detectors, the anchor boxes can be viewed as pre-defined sliding windows or proposals, which are classified as positive or negative patches, with an extra offsets regression to refine the prediction of bounding box locations. Therefore, the anchor boxes in these detectors may be viewed as training samples. Unlike previous detectors like Fast RCNN, which compute image features for each sliding window/ proposal repeatedly, anchor boxes make use of the feature maps of CNNs and avoid repeated feature computation, speeding up detection process dramatically.” / Fast R-CNN to one of ordinary skill under BRI is interpreted to be a decoder)
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings and of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions and Yang’s teachings of liver segmentation with DI2IN with Tian’s teachings of FCOS and Fast RCNN as cited for the same reasons given in claim 13.
As per claim 18, BRYNOLFSSON teaches:
A method for training a machine learning algorithm to classify a malignancy risk of a kidney, the method comprising: (“abstract discloses, “The machine learning modules may be used not only to detect presence and locations of such regions within an image, but also to segment the region corresponding to the lesion and/or classify such hotspots based on the likelihood that they are indicative of a true, underlying cancerous lesion.”)
…[…]…with first training data including computer tomography (CT) and/or magnet resonance imaging (MR) imaging data of an anatomy of at least one subject patient, wherein the imaging data comprises at least partially a representation of one or more kidneys …[…]…([0047] discloses, “In another aspect, the invention is directed to a system for automatically processing 3D images of a subject to identify and/or characterize (e.g., grade) cancerous lesions within the subject, the system comprising: a processor of a computing device; and a memory having instructions stored thereon, wherein the instructions, when executed by the processor, cause the processor to: (a) receive (e.g., and/or access) a 3D functional image of the subject obtained using a functional imaging modality [e.g., positron emission tomography (PET); e.g., single-photon emission computed tomography (SPECT)][e.g., wherein the 3D functional image comprises a plurality of voxels, each representing a particular physical volume within the subject and having an intensity value (e.g., standard uptake value (SUV)) that represents detected radiation emitted from the particular physical volume, wherein at least a portion of the plurality of voxels of the 3D functional image represent physical volumes within the target tissue region]; (b) automatically detect, using a machine learning module [e.g., a pre trained machine learning module (e.g., having pre-determined (e.g., and fixed) parameters having been determined via a training procedure)], one or more hotspots within the 3D functional image, each hotspot corresponding to a local region of elevated intensity with respect to its surrounding and representing (e.g., indicative of) a potential cancerous lesion within the subject, thereby creating one or both of (i) and (ii) as follows: (i) a hotspot list [e.g., a list of coordinates (e.g., image coordinates; e.g., physical space coordinates); e.g., a mask identifying voxels of the 3D functional image, each voxel corresponding to a location (e.g., a center of mass) of a detected hotspot] identifying, for each hotspot, a location of the hotspot, and (ii) a 3D hotspot map, identifying, for each hotspot, a coTesponding 3D hotspot volume within the 3D functional image {e.g., wherein, the 3D hotspot map is a segmentation map (e.g., comprising one or more segmentation masks) identifying, for each hotspot, voxels within the 3D functional image corresponding to the 3D hotspot volume of each hotspot [e.g., wherein the 3D hotspot map is obtained via artificial intelligence-based segmentation of the functional image (e.g., using a machine-learning module that receives, as input, at least the 3D functional image and generates the 3D hotspot map as output, thereby segmenting hotspots)]; e.g., wherein the 3D hotspot map delineates, for each hotspot, a 3D boundary (e.g., an irregular boundary) of the hotspot (e.g., the 3D boundary enclosing the 3D hotspot volume, e.g., and distinguishing voxels of the 3D functional image that make up the 3D hotspot volume from other voxels of the 3D functional image)}; and (c) store and/or provide, for display and/or further processing, the hotspot list and/or the 3D hotspot map.” And [0214] discloses, “For example, by collecting a dataset of PET/CT images in which hotspots that represent lesions have been manually detected and segmented, training material for Al-based lesion detection algorithms can be obtained.” And see [0106] discloses, “ In certain embodiments, step (b) comprises segmenting the anatomical image such that the 3D segmentation map identifies one or more organ volumes corresponding to soft tissue organs of the subject [e.g., left/right lungs, left/right gluteus maximus, urinary bladder, liver, left/right kidney, gallbladder, spleen, thoracic and abdominal aorta and, optionally (e.g., for patients not having undergone radical prostatectomy, a prostate], “)
training a second neural network with second training data including one or more detected lesions of one or more segmented kidney representations training a second neural network with second training data including one or more detected lesions of one or more segmented kidney representations, wherein the second neural network is configured as a fully convolutional one-stage object detection network that is anchor-free and proposal-free; and training a third neural network with third training data of one or more lesions classified with a malignancy risk, wherein the third neural network comprises atask-specific network configured to generate an image signature; wherein at least two of the first training data, second training data, or third training data are based on data fragments from the same subject patient;( ([0022] discloses, “ In certain embodiments, step (d) comprises using a second machine learning module to determine, for each hotspot of the portion, the lesion likelihood classification [e.g., wherein the machine learning module implements a machine learning algorithm trained to detect hotspots (e.g., to generate, as output, the hotspot list and/or the 3D hotspot map) and to determine, for each hotspot, the lesion likelihood classification for the hotpot].” And see [0214] discloses, “In certain embodiments, the systems and methods described herein utilize one or more machine learning modules to analyze intensities of 3D functional images and detect hotspots representing potential lesions. For example, by collecting a dataset of PET/CT images in which hotspots that represent lesions have been manually detected and segmented, training material for Al-based lesion detection algorithms can be obtained. These manually labeled images can be used to train one or more machine learning algorithms to automatically analyze functional images (e.g., PET images) to accurately detect and segment hotspots corresponding to cancerous lesions.” And see [0106] discloses, “ In certain embodiments, step (b) comprises segmenting the anatomical image such that the 3D segmentation map identifies one or more organ volumes corresponding to soft tissue organs of the subject [e.g., left/right lungs, left/right gluteus maximus, urinary bladder, liver, left/right kidney, gallbladder, spleen, thoracic and abdominal aorta and, optionally (e.g., for patients not having undergone radical prostatectomy, a prostate], “)
and training a third neural network with third training data of one or more lesions classified with a malignancy risk; ([0035] discloses, “[0036] discloses, “In certain embodiments, step (e) comprises using a third machine learning module (e.g., a hotspot classification module) to determine the lesion likelihood classification for each hotspot [e.g., based at least in part on one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map; e.g., wherein the third machine learning module receives one or more channels of input corresponding to one or more members selected from the group consisting of intensities of the 3D functional image, the hotspot list, the 3D hotspot map, intensities of a 3D anatomical image, and a 3D segmentation map].” And see [0037] discloses, “In certain embodiments, the method comprises: (f) selecting, by the processor, based at least in part on the lesion likelihood classifications for the hotspots, a subset of the one or more hotspots corresponding to hotspots having a high likelihood of corresponding to cancerous lesions (e.g., for inclusion in a report; e.g., for use in computing one or more risk index values for the subject).” And see [0210] discloses, “In certain embodiments, as described herein, machine learning modules implement one or more machine learning techniques, such as random forest classifiers, artificial neural networks (ANNs), convolutional neural networks (CNNs), and the like.” And see “[0196]-[0202]” And see [0214] discloses, “In certain embodiments, the systems and methods described herein utilize one or more machine learning modules to analyze intensities of 3D functional images and detect hotspots representing potential lesions. For example, by collecting a dataset of PET/CT images in which hotspots that represent lesions have been manually detected and segmented, training material for Al-based lesion detection algorithms can be obtained. These manually labeled images can be used to train one or more machine learning algorithms to automatically analyze functional images (e.g., PET images) to accurately detect and segment hotspots corresponding to cancerous lesions.” And see [0106] discloses, “ In certain embodiments, step (b) comprises segmenting the anatomical image such that the 3D segmentation map identifies one or more organ volumes corresponding to soft tissue organs of the subject [e.g., left/right lungs, left/right gluteus maximus, urinary bladder, liver, left/right kidney, gallbladder, spleen, thoracic and abdominal aorta and, optionally (e.g., for patients not having undergone radical prostatectomy, a prostate], “)
wherein at least two of the first training data, second training data, or third training data are based on data fragments from the same subject patient. ([0302] discloses, “This study used a subset of the data that contained only lesions with at least one other lesion of the same type in the same patient. This resulted in a dataset with 684 manually segmented lesion uptake volumes (278 in bone, 357 in lymph nodes, 49 in prostate) across 92 patients. Automatic refinement by thresholding was performed, and the output was compared to the original volumes. Performance was measured by a weighted average of rank correlations between refined volumes and original volumes within a patient and tissue type, with the weight given by the number of segmented hotspots volumes in the patient. This performance measure indicates whether the relative sizes between segmented hotspot volumes have been preserved, but disregards absolute sizes, which are subjectively defined since uptake volumes do not have clear boundaries. However, for a particular patient and tissue type, the same nuclear medicine reader made all annotations, and they can hence be assumed to have been made in a systematic manner, with a smaller lesion annotation actually reflecting a smaller uptake volume compared to a larger lesion annotation.” And see [0352] discloses, “Training the CNN models includes an iterative minimization problem where the training algorithm updates model parameters to lower the segmentation error. Segmentation error is defined as the deviation from a perfect overlap between manual segmentation and the CNN-model segmentation. Each neural network used for organ segmentation was trained to configure optimal parameters and weights. The training data for developing the neural networks for aPROMISE, as described above, consists of low dose CT images with manually segmented and labelled body parts.”)
However, BRYNOLFSSON does not explicitly teach the underlined portions:
training a first neural network with first training data including computer tomography (CT) and/or magnet resonance imaging (MR) imaging data of an anatomy of at least one subject patient, wherein the imaging data comprises at least partially a representation of one or more kidneys wherein the first neural network comprises a convolutional encoder- decoder architecture with multi-level feature concatenation and deep supervision, wherein training comprises adversarial image-to-image refinement by a discriminator network;
training a second neural network with second training data including one or more detected lesions of one or more segmented kidney representations training a second neural network with second training data including one or more detected lesions of one or more segmented kidney representations, wherein the second neural network is configured as a fully convolutional one-stage object detection network that is anchor-free and proposal-free; and training a third neural network with third training data of one or more lesions classified with a malignancy risk, wherein the third neural network comprises atask-specific network configured to generate an image signature; wherein at least two of the first training data, second training data, or third training data are based on data fragments from the same subject patient;
However, Yang does explicitly teach the underlined portions:
training a first neural network with first training data including computer tomography (CT) and/or magnet resonance imaging (MR) imaging data of an anatomy of at least one subject patient, wherein the imaging data comprises at least partially a representation of one or more kidneys wherein the first neural network comprises a convolutional encoder- decoder architecture with multi-level feature concatenation and deep supervision, wherein training comprises adversarial image-to-image refinement by a discriminator network; (see abstract discloses, “In this paper, we propose an automatic and efficient algorithm to segment liver from 3D CT volumes. Adeep image-to-image network (DI2IN) is first deployed to generate the liver segmentation, employing a convolutional encoder-decoder architecture combined with multi-level feature concatenation and deep super vision. Then an adversarial network is utilized during training process to discriminate the output of DI2IN from ground truth, which further boosts the performance of DI2IN.”)
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions with Yang’s teachings of liver segmentation with DI2IN as cited, the motivation being BRYNOLFSSON teaches that there is a significant need for improved tools that facilitate and improve accuracy of image review and analysis for cancer diagnosis and treatment, (see [0008]) therefore it would only optimize making better choices which more efficiently and accurately a determine malignancy risk and lesions without rendering the primary reference inoperable.
However, Yang also does not explicitly teach the underlined portions:
training a second neural network with second training data including one or more detected lesions of one or more segmented kidney representations training a second neural network with second training data including one or more detected lesions of one or more segmented kidney representations, wherein the second neural network is configured as a fully convolutional one-stage object detection network that is anchor-free and proposal-free; and training a third neural network with third training data of one or more lesions classified with a malignancy risk, wherein the third neural network comprises atask-specific network configured to generate an image signature; wherein at least two of the first training data, second training data, or third training data are based on data fragments from the same subject patient;
However, Tian does teach the underlined portions:
training a second neural network with second training data including one or more detected lesions of one or more segmented kidney representations training a second neural network with second training data including one or more detected lesions of one or more segmented kidney representations, wherein the second neural network is configured as a fully convolutional one-stage object detection network that is anchor-free and proposal-free; (abstract discloses, “We propose a fully convolutional one-stage object detec tor (FCOS) to solve object detection in a per-pixel predic tion fashion, analogue to semantic segmentation. Almost all state-of-the-art object detectors such as RetinaNet, SSD, YOLOv3, and Faster R-CNN rely on pre-defined anchor boxes. In contrast, our proposed detector FCOS is anchor box free, as well as proposal free. By eliminating the pre defined set of anchor boxes, FCOS completely avoids the complicated computation related to anchor boxes such as calculating overlapping during training. More importantly, wealso avoid all hyper-parameters related to anchor boxes, which are often very sensitive to the final detection perfor mance.” And see section materials discloses, “The liver data comes from the public dataset Sliver 07 [30]. The dataset consists of abdominal CT scans from 20 anesthetized patients (more than 5500 liver slices) and regional annotations. The CT scans are in grayscale with 512 × 512 pixels.”)
and training a third neural network with third training data of one or more lesions classified with a malignancy risk, wherein the third neural network comprises atask-specific network configured to generate an image signature; (see related work discloses, “Anchor-based detectors inherit the ideas from traditional sliding-window and proposal based detectors such as Fast R-CNN [6]. In anchor-based detectors, the anchor boxes can be viewed as pre-defined sliding windows or proposals, which are classified as positive or negative patches, with an extra offsets regression to refine the prediction of bounding box locations. Therefore, the anchor boxes in these detectors may be viewed as training samples. Unlike previous detectors like Fast RCNN, which compute image features for each sliding window/ proposal repeatedly, anchor boxes make use of the feature maps of CNNs and avoid repeated feature computation, speeding up detection process dramatically.” / examiner notes features space plus feature vectors used in anchor based CNN is interpreted under BRI as a type of image signature.)
wherein at least two of the first training data, second training data, or third training data are based on data fragments from the same subject patient; (abstract discloses, “We propose a fully convolutional one-stage object detec tor (FCOS) to solve object detection in a per-pixel predic tion fashion, analogue to semantic segmentation. Almost all state-of-the-art object detectors such as RetinaNet, SSD, YOLOv3, and Faster R-CNN rely on pre-defined anchor boxes. In contrast, our proposed detector FCOS is anchor box free, as well as proposal free. By eliminating the pre defined set of anchor boxes, FCOS completely avoids the complicated computation related to anchor boxes such as calculating overlapping during training. More importantly, wealso avoid all hyper-parameters related to anchor boxes, which are often very sensitive to the final detection perfor mance.” And see section materials discloses, “The liver data comes from the public dataset Sliver 07 [30]. The dataset consists of abdominal CT scans from 20 anesthetized patients (more than 5500 liver slices) and regional annotations. The CT scans are in grayscale with 512 × 512 pixels.”)
It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine BRYNOLFSSON’s teachings and of utilizing CNN’s to classify malignancy risk of the kidney and segment images to detect lesions and Yang’s teachings of liver segmentation with DI2IN with Tian’s teachings of FCOS as cited for the same reasons given in claim 13.
Response to Arguments Regarding 35 U.S.C § 102/103 Rejections
Applicant’s arguments on pages 1-5 of remarks have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Examiner maintains the 35 U.S.C § 103 rejection.
Prior Art not cited but made of record
Annangi et. al (hereinafter Annangi)(US2023/0052078Al)
Liu, X., Guo, S., Yang, B., Ma, S., Zhang, H., Li, J., ... & Fu, Y. (2018). Automatic organ segmentation for CT scans based on super-pixel and convolutional neural networks. Journal of digital imaging, 31(5), 748-760
WO2022232172A1- Murphy et. al
[0007] Advantageously, the present disclosure provides robust techniques for identifying a risk of kidney graft failure for a subject. The following presents a summary of the invention in order to provide a basic understanding of some of the aspects of the invention. This summary is not an extensive overview of the invention. It is not intended to identify key/critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some of the concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later.
WO2023014495A1– SELLERGREN et. al
It is desirable in many tasks to use available training data (e.g., medical images and related information) to train a machine learning model (e.g., an artificial neural network) to classify inputs or to generate some other prediction or output based on inputs. In practice, larger models (e.g., having more trainable parameters) are able to achieve higher performance (e.g., sensitivity, specificity). However, the training of such larger models often requires more training examples and for the set of training examples to be ‘higher quality’ (e.g., spanning a wider variety of potential inputs and outputs in a manner that is less biased).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ashley Elizabeth Evans whose telephone number is (571) 270-0110. The examiner can normally be reached Monday – Friday 8:00 AM – 5:00 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mamon Obeid can be reached on (571) 270-1813. The fax phone number for the organization where this application or proceeding is assigned 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center. Should you have questions on access to the Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/ASHLEY ELIZABETH EVANS/Examiner, Art Unit 3687
/MAMON OBEID/Supervisory Patent Examiner, Art Unit 3687
Prosecution Insights