DETAILED ACTION
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 15-20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Xie et al (US 20220230310).
Regarding claim 15, Xie discloses a non-transitory computer-readable medium having instructions that, upon execution by a computing device (para [0150]), cause the computing device to perform operations comprising:
receiving an image comprising a pathological lesion (Fig. 1, para [0093] "Annotation can be performed manually by one or more humans (annotators such as a radiologists or pathologists) confirming the presence of one or more objects of interest in each image of the subset of images 145a and providing labels 150 to the one or more objects of interest. For example, if an object of interest is a tumor or lesion, then annotation data may indicate a type of tumor or lesion, such as a tumor or lesion in a liver, a lung, a pancreas, and/or a kidney.");
ingesting, by a model trained to detect lesions based on a marking, the image (Fig. 1, para [0093] "Annotation can be performed manually by one or more humans (annotators such as a radiologists or pathologists) confirming the presence of one or more objects of interest in each image of the subset of images 145a and providing labels 150 to the one or more objects of interest For example, if an object of interest is a tumor or lesion, then annotation data may indicate a type of tumor or lesion, such as a tumor or lesion in a liver, a lung, a pancreas, and/or a kidney."; [0094] “In some instances, a subset of images 145 may be transmitted to an annotator device 155 to be included within a training data set (i.e., the subset of images 145a). Input may be provided (e.g., by a radiologist) to the annotator device 155 using (for example) a mouse, track pad, stylus and/or keyboard that indicates (for example) whether the image depicts an object of interest (e.g., a lesion, an organ, etc.); a number of objects of interest depicted within the image; and a perimeter (bounding box or segmentation boundary) of each depicted object of interest within the image. Annotator device 155 may be configured to use the provided input to generate labels 150 for each image.”);
generating, based on the model trained to detect lesions, a marking for one or more unannotated lesions of the image (Fig. 1, para [0102] "Map processing controller 170 includes processes for overlaying the bounding box or segmentation mask corresponding to the detected object of interest from the first image onto the same of object of interest as depicted in the second image. In instances in which the segmentation mask is determined, the segmentation mask is projected onto the second image (e.g., two-dimensional slices of the second image) such that the boundaries of the segmentation mask can be used to define a rectangular bounding box enclosing a region of interest corresponding to the object of interest within the second image.");
defining a volumetric segmentation corresponding to the marking (Figs. 3-4; para [0121]); refining the volumetric segmentation based on volumetric segmentation training data to define a refined volumetric segmentation (para [0122]);
reconciling the refined volumetric segmentation with another volumetric description corresponding to the one or more unannotated lesions (Figs. 3-4; para [0123]);
generating a presentment image, based on the reconciliation (Fig. 4, para [0126] "For example, (i) the portion of the second image with the estimated segmentation boundary around the object of interest, and/or (ii) a size, surface area, and/or volume of the object of interest, may be stored in a storage device and/or displayed on a user device."); and
causing a display of the presentment image via a graphical user interface (Fig. 4, para [0126] "For example, (i) the portion of the second image with the estimated segmentation boundary around the object of interest, and/or (ii) a size, surface area, and/or volume of the object of interest, may be stored in a storage device and/or displayed on a user device.").
Regarding claim 16, Xie discloses the computer-readable medium of claim 15, wherein the marking comprises at least one of: a bounding box (Fig. 1, para [0093]); or a line annotation.
Regarding claim 17, Xie discloses the computer-readable medium of claim 15, comprising instructions to cause the computing device to perform operations comprising:
determining a parallel volumetric segmentation (Fig. 1, para [0086] "In some instances, multiple images 135 depict an object of interest, such that each of the multiple images 135 may correspond to a virtual "slice" of the object of interest. Each of the multiple images 135 may have a same viewing angle, such that each image 135 depicts a plane that it parallel to other planes depicted in other images 135 corresponding to the subject and object of interest."); and
reconciling the refined volumetric segmentation with the parallel volumetric segmentation to define the presentment image (Figs. 3-4; para [0123]).
Regarding claim 18, Xie discloses the computer-readable medium of claim 15, wherein at least one of the one or more unannotated lesions comprise a tumor and the image is one or more slices of a magnetic resonance imaging (MRI) image (Fig. 1, para [0089] "The subset of images 145a are acquired from one or more imaging modalities (e.g., MRI and CT). Each image depicts one or more objects of interest such as a cephalic region, a chest region, an abdominal region, a pelvic region, a spleen, a liver, a kidney, a brain, a tumor, a lesion, or the like.").
Regarding claim 19, Xie discloses the computer-readable medium of claim 15, wherein: the image comprises a plurality of two-dimensional slices of a three-dimensional structure (Fig. 4, para [0121] "In some instances, the segmentation mask is projected onto two-dimensional slices of the second image, such that the boundaries of the segmentation mask within a two-dimensional space can be used to define a rectangular bounding box enclosing a region of interest corresponding to the detected object of interest."); and the volumetric segmentation is a three dimensional volumetric segmentation (Fig. 4, para [0123] "At block 430, the portion of the second image is input into a three-dimensional neural network model constructed for volumetric segmentation using a weighted loss function (e.g., a modified 3D U-Net model).").
Regarding claim 20, Xie discloses the computer-readable medium of claim 15, wherein the instructions to refine the volumetric segmentation comprise instructions that, upon execution by the computing device, cause the computing device to perform operations comprising: iteratively refining the volumetric segmentation based on the volumetric segmentation training data, each iterative refinement comprising a generation of an iterative segmentation of the pathological lesion (Fig. 1, para [0095]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xie et al (US 20220230310) in view of Swinburne et al (NPL: Semisupervised Training of a Brain MRI Tumor Detection Model Using Mined Annotations, see IDS).
Regarding claim 1, Xie discloses a system for automated segmentation of pathological lesions (para [0045] automated object segmentation of medical images), the system comprising:
one or more processors coupled to a non-transitory memory (para [0150]), the one or more processors configured to:
receive a plurality of first images, a first portion of the plurality of first images comprising annotations corresponding to one or more pathological lesions (Fig. 1, para [0093] "Annotation can be performed manually by one or more humans (annotators such as a radiologists or pathologists) confirming the presence of one or more objects of interest in each image of the subset of images 145a and providing labels 150 to the one or more objects of interest. For example, if an object of interest is a tumor or lesion, then annotation data may Indicate a type of tumor or lesion, such as a tumor or lesion in a liver, a lung, a pancreas, and/or a kidney.");
generate, based on the lesion detection model, a third bounding box for one or more unannotated lesions of a second image (Fig. 1, para [0102] "Map processing controller 170 includes processes for overlaying the bounding box or segmentation mask corresponding to the detected object of interest from the first image onto the same of object of interest as depicted in the second image. In instances in which the segmentation mask is determined, the segmentation mask is projected onto the second image (e.g., two-dimensional slices of the second image) such that the boundaries of the segmentation mask can be used to define a rectangular bounding box enclosing a region of interest corresponding to the object of Interest within the second image.");
define a volumetric segmentation corresponding to the third bounding box (Figs. 3-4; para [0121], [0123]);
refine the volumetric segmentation based on volumetric segmentation training data to define a refined volumetric segmentation (para [0122], [0123], [0124]);
reconcile the refined volumetric segmentation with another volumetric description corresponding to at least one of the one or more unannotated lesions (Figs. 3-4; para [0123], [0124]);
generate a presentment image, based on the reconciliation (Fig. 4, para [0126] "For example, (i) the portion of the second image with the estimated segmentation boundary around the object of interest, and/or (ii) a size, surface area, and/or volume of the object of interest, may be stored in a storage device and/or displayed on a user device."); and
cause a display of the presentment image via a graphical user interface (Fig. 4. para [0126] "For example, (i) the portion of the second image with the estimated segmentation boundary around the object of interest, and/or (ii) a size, surface area, and/or volume of the object of interest, may be stored in a storage device and/or displayed on a user device.").
Xie does not teach line annotations corresponding to one or more pathological lesions; convert a first plurality of line annotations to a first plurality of corresponding bounding boxes; train a lesion detection model to generate a second plurality of bounding boxes, based on the first plurality of bounding boxes.
However, Swinburne teaches line annotations corresponding to one or more pathological lesions (page 81 identify all brain MRI scans acquired at our center from January 2012 to December 2017 that contained at least one line annotation);
convert a first plurality of line annotations to a first plurality of corresponding bounding boxes (page 80-81 – Materials and Methods: Line annotations were converted to boxes, excluding boxes shorter than 1 cm or longer than 7 cm; Automated Data Curation and Bounding Box Generation: automated data curation pipeline was constructed to isolate line annotation data referencing axial T1-weighted postcontrast images, geometrically square the lines to generate bounding boxes);
train a lesion detection model to generate a second plurality of bounding boxes, based on the first plurality of bounding boxes (Fig. 1; page 80 - Materials and Methods: The resulting boxes were used for supervised training of object detection models using RetinaNet and Mask region-based convolutional neural network (R- CNN) architectures).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the automated segmentation of pathological lesions taught by Xie with the bounding box generation taught by Swinburne since doing so would improve systems and methods for automatically identifying and converting the line annotations to bounding boxes for efficient volumetric segmentation.
Regarding claim 2, Xie in view of Swinburne discloses the system of claim 1, Xie further teaches wherein the one or more
processors are configured to: determine a parallel volumetric segmentation (Fig. 1, para [0086] "In some instances, multiple images 135 depict an object of interest, such that each of the multiple images 135 may correspond to a virtual "slice" of the object of interest. Each of the multiple images 135 may have a same viewing angle, such that each image 135 depicts a plane that it parallel to other planes depicted in other images 135 corresponding to the subject and object of interest."); and reconcile the refined volumetric segmentation with the parallel volumetric segmentation to define the presentment image (Figs. 3-4; para [0123]).
Regarding claim 3, Xie in view of Swinburne discloses the system of claim 1, Xie further teaches wherein at least one of the one or more unannotated lesions comprise a tumor and the second image is one or more slices of a magnetic resonance imaging (MRI) image (Fig. 1, para [0089] "The subset of images 145a are acquired from one or more imaging modalities (e.g., MRI and CT) Each image depicts one or more objects of interest such as a cephalic region, a chest region, an abdominal region, a pelvic region, a spleen, a liver, a kidney, a brain, a tumor, a lesion, or the like.").
Regarding claim 4, Xie in view of Swinburne discloses the system of claim 1, Xie further teaches wherein at least one of the one or more unannotated lesions is lung cancer and the second image is a computed topography (CT) scan (Fig. 1, para [0089] "The subset of images 145a are acquired from one or more imaging modalities (e.g., MRI and CT).. Each image depicts one or more objects of interest such as a cephalic region, a chest region, an abdominal region, a pelvic region, a spleen, a liver, a kidney, a brain, a tumor, a lesion, or the like.").
Regarding claim 5, Xie in view of Swinbume discloses the system of claim 1, Swinburne further teaches wherein the plurality of first images includes a second portion of the plurality of first Images lacking the line annotations of the one or more pathological lesions (Fig. 2). The motivation to combine the references is discussed above in the rejection for claim 1.
Regarding claim 6, Xie in view of Swinburne discloses the system of claim 1, wherein: Xie further teaches the second image comprises a plurality of two-dimensional slices of a three dimensional structure (Fig. 4, para [0121] "In some instances, the segmentation mask is projected onto two-dimensional slices of the second image, such that the boundaries of the segmentation mask within a two-dimensional space can be used to define a rectangular bounding box enclosing a region of interest corresponding to the detected object of interest."); and
the volumetric segmentation is a three-dimensional volumetric segmentation (Fig. 4, para [0123] "At block 430, the portion of the second image is input Into a three-dimensional neural network model constructed for volumetric segmentation using a weighted loss function (e.g., a modified 3D U-Net model).").
Regarding claim 7, Xie in view of Swinburne discloses the system of claim 1, Xie further teaches wherein, to reconcile the refined volumetric segmentation, the one or more processors are configured to iteratively refine the volumetric segmentation based on the volumetric segmentation training data, each iterative refinement comprising a generation of an iterative segmentation of the one or more pathological lesions (Fig. 1, para [0095]).
Regarding claim 8, Xie discloses a method for segmenting pathological lesions (para [0045]), the method comprising:
receiving, by one or more processors coupled to a non-transitory memory (para [0150]), a plurality of first images, a first portion of the plurality of first images comprising annotations corresponding to one or more first pathological lesions (Fig. 1, para [0093] "Annotation can be performed manually by one or more humans (annotators such as a radiologists or pathologists) confirming the presence of one or more objects of interest in each image of the subset of images 145a and providing labels 150 to the one or more objects of interest For example, if an object of interest is a tumor or lesion, then annotation data may indicate a type of tumor or lesion, such as a tumor or lesion in a liver, a lung, a pancreas, and/or a kidney.");
defining, by the one or more processors, a second bounding box for one or more second pathological lesions of a second image using a detection model (Fig. 1, para [0093]);
defining, by the one or more processors, a volumetric segmentation corresponding to each bounding box of the second image (Figs. 3-4; para [0121]);
refining, by the one or more processors, the volumetric segmentation based on volumetric segmentation training data to define a refined volumetric segmentation (para [0122]);
reconciling, by the one or more processors, the refined volumetric segmentation with another volumetric segmentation corresponding to at least one of the one or more second pathological lesions (Figs. 3-4; para [0123]);
generating, by the one or more processors, a presentment image, based on the reconciliation (Fig. 4, para [0126] "For example, (i) the portion of the second image with the estimated segmentation boundary around the object of interest, and/or (ii) a size, surface area, and/or volume of the object of interest, may be stored in a storage device and/or displayed on a user device."); and
causing, by the one or more processors, a display of the presentment image via a graphical user interface (Fig. 4, para [0126] "For example, (i) the portion of the second image with the estimated segmentation boundary around the object of interest, and/or (ii) a size, surface area, and/or volume of the object of interest, may be stored in a storage device and/or displayed on a user device.").
Xie does not teach line annotations corresponding to one or more pathological lesions;
defining, by the one or more processors, a plurality of first bounding boxes converted from the line annotations of the plurality of first images; train a lesion detection model to generate a second plurality of bounding boxes, based on the first plurality of bounding boxes.
However, Swinburne teaches line annotations corresponding to one or more pathological lesions (page 5 identify all brain MRI scans acquired at our center from January 2012 to December 2017 that contained at least one line annotation);
defining, by the one or more processors, a plurality of first bounding boxes converted from the line annotations of the plurality of first images (page 2 and 5 Line annotations were converted to boxes, excluding boxes shorter than 1 cm or longer than 7 cm; automated data curation pipeline was constructed to isolate line annotation data referencing axial T1-weighted postcontrast images, geometrically square the lines to generate bounding boxes);
train a lesion detection model to generate a second plurality of bounding boxes, based on the first plurality of bounding boxes (Fig. 1; page 2 The resulting boxes were used for supervised training of object detection models using RetinaNet and Mask region-based convolutional neural network (R-CNN) architectures).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the automated segmentation of pathological lesions taught by Xie with the bounding box generation taught by Swinburne since doing so would improve systems and methods for automatically identifying and converting the line annotations to bounding boxes for efficient volumetric segmentation.
Regarding claim 9, Xie in view of Swinburne discloses the method of claim 8, comprising: Xie further teaches
determining, by the one or more processors, a parallel volumetric segmentation (Fig. 1, para [0086] "In some instances, multiple images 135 depict an object of interest, such that each of the multiple images 135 may correspond to a virtual "slice" of the object of interest. Each of the multiple images 135 may have a same viewing angle, such that each image 135 depicts a plane that it parallel to other planes depicted in other images 135 corresponding to the subject and object of interest."); and
reconciling, by the one or more processors, the refined volumetric segmentation with the parallel volumetric segmentation to define the presentment image (Figs. 3-4; para [0123]).
Regarding claim 10, Xie in view of Swinburne discloses the method of claim 8, Xie further teaches wherein at least one of
the one or more second pathological lesions comprise a tumor and the second image is one or more slices of a magnetic resonance imaging (MRI) image (Fig. 1, para [0089] "The subset of images 145a are acquired from one or more imaging modalities (e.g., MRI and CT) Each image depicts one or more objects of interest such as a cephalic region, a chest region, an abdominal region, a pelvic region, a spleen, a liver, a kidney, a brain, a tumor, a lesion, or the like.").
Regarding claim 11, Xie in view of Swinburne discloses the method of claim 8, Xie further teaches wherein at least one of the one or more second pathological lesions is lung cancer and the second image is a computed topography (CT) scan (Fig. 1, para [0089] "The subset of images 145a are acquired from one or more imaging modalities (e.g., MRI and CT). Each image depicts one or more objects of interest such as a cephalic region, a chest region, an abdominal region, a pelvic region, a spleen, a liver, a kidney, a brain, a tumor, a lesion, or the like.").
Regarding claim 12, Xie in view of Swinburne discloses the method of claim 8, Swinburne further teaches wherein the plurality of first images includes a second portion of the plurality of first images lacking the line annotations of the one or more first pathological lesions (Fig. 2). The motivation to combine the references is discussed above in the rejection for claim 8.
Regarding claim 13, Xie in view of Swinburne discloses the method of claim 8, Xie further teaches wherein: the second image comprises a plurality of two-dimensional slices of a three dimensional structure (Fig. 4, para [0121] "In some instances, the segmentation mask is projected onto two-dimensional slices of the second image, such that the boundaries of the segmentation mask within a two-dimensional space can be used to define a rectangular bounding box enclosing a region of interest corresponding to the detected object of interest."); and
the volumetric segmentation is a three dimensional volumetric segmentation (Fig. 4, para [0123] "At block 430, the portion of the second image is input into a three-dimensional neural network model constructed for volumetric segmentation using a weighted loss function (e.g., a modified 3D U-Net model).").
Regarding claim 14, Xie in view of Swinburne discloses the method of claim 8, Xie further teaches wherein, to reconcile the refined volumetric segmentation, the one or more processors are configured to: iteratively refine the volumetric segmentation based on the volumetric segmentation training data, each iterative refinement comprising a generation of an iterative segmentation of the one or more first pathological lesions (Fig. 1, para [0095]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEVIN KY whose telephone number is (571)272-7648. The examiner can normally be reached Monday-Friday 9-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEVIN KY/ Primary Examiner, Art Unit 2671
Prosecution Insights