Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1, and 10 have been amended.
Claim 23 has been cancelled.
Claims 31-33 have been newly added.
Claims 1-22, and 24-33 are still pending for consideration.
Response to Arguments
Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-2, 4-8, 12-13, 16, 20, 26 and 33 are rejected under 35 U.S.C. 103 as being unpatentable over Eastwood et al. (US 20180137689 A1) in view of Georgescu et al. (US 20190206056 A1), and further in view of Magee et al. NPL “Histopathology in 3D: From three-dimensional reconstruction to multi-stain and multi-modal analysis”.
Regarding claim 1, Eastwood et al. disclose a method for generating a digital reconstruction of tissue (see para [0072]; “3D reconstruction from 2D histological tissue section images”), the method comprising: receiving, at a computing system, image data of a tissue sample, wherein multiple sections of the tissue sample are stained with hematoxylin and eosin (H&E) (see Abstract; “creating a 3D volume image of a tissue block from a series of images of histological sections taken from the tissue block”, see also para [0106]; “the sections are mounted on slides, the sections are stained”, see para [0041]; “the section is picked up on a glass slide and stained (usually with hematoxylin and eosin, the H&E stain)”); registering, by the computing system, the image data to generate registered image data based on mapping independent serial images of the image data to a common coordinate system using non-linear image registration (see para [0074]; “uses multi-scale image registration that utilizes image symmetry, overall image shape, and image content to compile serial sections into a 3D volume. The HIC software uses multiple stage, multiple resolution registration algorithms for each adjacent pair of section images in properly ordered series of section images, for example, as generated using section-ordering processes”, see para [0129]; “The present inventors have discovered that this correction is essential for successful 2D and 3D linear and nonlinear registration of sections and brains to reference brains (and their atlases) for varying sources”); determining, by the computing system, a digital volume of the tissue sample in three dimensional (3D) space based on the annotated image data (see para [0045]; “computer-implemented methods that allow a user to quickly and easily create a 3D histological image, or “3D composite histological image,” from images of multiple histological sections… algorithms for efficiently identifying and generating the 3D composite histological image”), and returning, by the computing system, the digital volume of the tissue sample in 3D space to be presented in a graphical user interface (GUI) display at a user computing device (see para [0047]; The display of anatomical information from a tissue-block atlas may result from any one of user interaction with a displayed experimental section image, user interaction with a displayed atlas image, and user interaction with a graphically displayed ontological hierarchy of anatomical regions of the tissue block under consideration”). However, Eastwood et al. does not teach identifying, by the computing system, tissue subtypes based on application of a machine learning model to the registered image data.
In the same field of endeavor, Georgescu et al. teaches identifying, by the computing system, tissue subtypes based on application of a machine learning model to the registered image data (see para [0063]; “The proposed computer-automated method for tumor finding, outlining and classifying uses a convolutional neural network (CNN) to find each nuclear pixel on the WSI and then to classify each such pixel into one of a non-tumor class and one of a plurality of tumor classes, in our current implementation breast tumor classes”); the tissue subtypes including a particular type of tissue (see para [0069]; “the input image patch and a number of channels (i.e. number of feature maps) equal to the number of tissue classes to be detected, i.e. a non-tumorous type and one or more tumor types”, see also para [0098]; “with reference to Tile D, Tile C also shows results from our CNN method (first areas shaded pink with pink perimeter lines correspond to a first tumor type, i.e. the tumor type shown red in Tile D; second areas shaded yellow with pink perimeter lines correspond to a second tumor type, i.e. the tumor type shaded blue in Tile D)”); annotating, by the computing system, the identified tissue subtypes to generate annotated image data (see para [0078]; “The CNN thus labels each pixel as non-cancerous or belonging to one or more of several different cancer (tumor) types”, see also para [0098]; “with reference to Tile D, Tile C also shows results from our CNN method (first areas shaded pink with pink perimeter lines correspond to a first tumor type, i.e. the tumor type shown red in Tile D; second areas shaded yellow with pink perimeter lines correspond to a second tumor type, i.e. the tumor type shaded blue in Tile D). [0099] Tile D is a tumor probability heatmap generated by our CNN”). Accordingly, it has been obvious to one ordinary skill in the art before the effective filed of the invention to modify a method creating a 3D volume image of a tissue block from a series of images of histological sections taken from the tissue block of Eastwood et al. in view of the use of a convolutional neural network (CNN) is applied to identifying tumors in a histological image of Georgescu et al. in order to automate and improve identification and annotation of tissue types in the reconstructed image (see para [0153]). However, the combination of Eastwood et al. and Georgescu et al. as a whole does not teach and measuring, by the computing system, 3D connectivity of the particular type of tissue in the tissue sample, using the digital volume of the tissue sample in 3D space.
In the same field of endeavor, Magee et al. teaches and measuring, by the computing system, 3D connectivity of the particular type of tissue in the tissue sample, using the digital volume of the tissue sample in 3D space (see page 4, right col. 2nd para; “We used the original volumetric reconstruction algorithm (Section 2) to generate volumes from liver tissue with five different types of liver disease…… The size, shape, and connectivity of nodules were quantified… a 3D sub-voxel anisotropic morphological opening procedure, and (iii) assigning statistics to connected 3D components (size, elongation, etc.)….. The number of connected components and size of connected components showed a statistically significant variation between diseases”, Note: "Digital volume of the tissue sample in 3D space" corresponds to the "volumetric reconstruction... to generate volumes from liver tissue" and "Measuring 3D connectivity... of a particular type of tissue" corresponds to quantifying the "connectivity of nodules" and applying a "3D sub-voxel anisotropic morphological opening procedure... and assigning statistics to connected 3D components"). Accordingly, it has been obvious to one ordinary skill in the art before the effective filed of the invention to modify a method creating a 3D volume image of a tissue block from a series of images of histological sections taken from the tissue block of Eastwood et al. in view of the use of a convolutional neural network (CNN) is applied to identifying tumors in a histological image of Georgescu et al. and further in view of Histopathology in 3D: From three-dimensional reconstruction to multi-stain and multi-modal analysis of Magee et al. in order to transform dissimilar images into a multi-channel representation derived from co-occurrence statistics between roughly aligned images (see page 4, right col. 2nd para).
Regarding claim 2, the rejection of claim 1 is incorporated herein.
Georgescu et al. in the combination further teach wherein the tissue sample is at least one of a pancreatic tissue sample, a skin tissue sample, a breast tissue sample, a lung tissue sample, and a small intestines tissue sample (see para [0078]; “The cancer of particular interest is breast cancer, but the method is also applicable to histology images of other cancers, such as cancer of the bladder, colon, rectum, kidney, blood (leukemia), endometrium, lung, liver, skin, pancreas, prostate, brain, spine and thyroid”).
Regarding claim 4, the rejection of claim 1 is incorporated herein.
Eastwood et al. in the combination further teach wherein the image data is between lx and 40x magnification, wherein lateral x and y resolution is between 0.2pm and 10pm and axial z resolution is between 0.5pm and 40pm (see para [0174]; “automatically move the stage, increase the magnification, and/or automatically focus the imaging system to effectively zoom into the contoured region. Those skilled in the art will readily understand that many variations of selection and zooming in and zooming out can be devised that are under the control of the AIM software in response to user inputs directed to any one of the displayed images and/or the graphically displayed ontological hierarchy. Automated annotation of anatomies on a live-view image of a specimen slide that is at a low resolution provided by AIM software of the present disclosure, and the AIM software's ability to keep track of location on a slide while switching objective lenses (e.g., to a higher magnification lens) enables precise measurements within objectively determined regions of the slide-mounted tissue section at issue”).
Regarding claim 5, the rejection of claim 1 is incorporated herein.
Eastwood et al. in the combination further teach wherein registering, by the computing system, the image data to generate registered image data further comprises: identifying, as a point of reference, a center image of the image data (see para [0075]; “a first pair of section images in the ordered series of section images, and the first section image is the fixed image (a/k/a “reference” or “source”) …this first registration stage… a centered rigid transform based on center of rotation, angle of rotation, and translation”); and calculating global registration for each of the image data based on the point of reference (see para [0134]; “Registration is run starting at each candidate location to adjust parameters of a centered Euler transform (3D translation and rotation about a point) at a coarse resolution (˜128 pixels along the shortest axis, in one example)”, see also para [0179], Note; a specific search strategy that can be used as a method to achieve global registration).
Regarding claim 6, the rejection of claim 5 is incorporated herein.
Eastwood et al. in the combination further teach wherein calculating global registration further comprises iteratively calculating registration angle and translation for each of the image data (see para [0137]-[0145]; “Registration is run starting at each candidate location to adjust parameters of a centered Euler transform (3D translation and rotation about a point) at a coarse resolution (˜128 pixels along the shortest axis, in one example).. This process is repeated until the spacing between candidate transforms drops below a threshold (e.g., 80 μm)…. that refines favorable candidate transforms allowing for oblique angles and scale differences… Registration is run starting at each candidate location to adjust parameters of a scale-versor transform (translation, rotation, and non-uniform scaling)).
Regarding claim 7, the rejection of claim 6 is incorporated herein.
Eastwood et al. in the combination further teach further comprising calculating elastic registration for each of the image data based on calculating rigid registration of cropped image tiles of each of the globally registered image data at intervals that range between 0.1mm and 5mm (see para [0116]; “register the individual brain image to the template using multiple stage registration up to a nonlinear (BSpline) transform”, see also para [0069]; “To create a series of section images from each slide image, the HIC software uses the contour(s) it draws around each section image to determine a crop-box sized to completely contain the image region(s) as defined by the contour(s). The HIC software then stores each cropped section image in a distinct image file and causes these section image files to be ordered in serial order using the selected ordering information described above. the resolution is limited to 1,000 pixels×2,000 pixels, but other resolutions can certainly be used….the image registration techniques described herein typically work very well using these limited-resolution images”, Note: nonlinear registration implies elastic registration and nonlinear B-spline are fundamentally interval-based).
Regarding claim 8, the rejection of claim 1 is incorporated herein.
Georgescu et al. in the combination further teach wherein the tissue sample includes at least one of normal human tissue, precancerous human tissue, and cancerous human tissue (see para [0033]; “There are various options for setting the tissue classes, but most if not all embodiments will have in common that a distinction will be made in the classes between non-tumorous and tumorous tissue. The non-tumorous tissue classes may include one, two or more classes. The tumorous tissue classes may also include one, two or more classes. For example, in our current implementation we have three tissue classes, one for non-tumorous tissue and two for tumorous tissue, wherein the two tumorous tissue classes are for invasive tumors and in situ tumors”).
Regarding claim 12, the rejection of claim 1 is incorporated herein.
Georgescu et al. in the combination further teach wherein the machine learning model was trained, by the computing system, with manual annotations of one or more tissue subtypes in a plurality of training tissue image data wherein the machine learning model is at least one of a deep learning semantic segmentation model, a convolutional neural network (CNN), and a U-net structure (see para [0124]; “FIG. 4 is a flow diagram showing the steps involved in training the CNN.[0125] In Step S40, training data is retrieved containing WSIs for processing which have been annotated by a clinician to find, outline and classify tumors. The clinician's annotations represent the ground truth data”, see also para [0088]; “Training the network may be done on a GPU, CPU or a FPGA using any one of several available deep learning frameworks”, and para [0129]; “each of a batch of input image patches is input into the CNN and processed to find, outline and classify the patches on a pixel-by-pixel basis as described further above with reference to FIGS. 1A and 1B” implies semantic segmentation ).
Regarding claim 13, the rejection of claim 12 is incorporated herein.
Georgescu et al. in the combination further teach further comprising training, by the computing system, the machine learning model based on randomly overlaying extracted annotated regions of one or more tissue samples on a training image (see para [0129]-[0031]; “each of a batch of input image patches is input into the CNN and processed….. the CNN output image patches are compared with the ground truth data…. the probability map is presented on the display as a semi-transparent overlay to the WSI… the CNN then learns from this comparison and updated the CNN weights”) and cutting the training image into the plurality of training tissue image data (see para [0126]; “In Step S41, the WSIs are broken down into image patches, which are the input image patches for the CNN. That is, image patches are extracted from the WSI”, see also page 6, last para; “we create our training dataset by semi-randomly overlaying extracted annotated regions on a large image, then cutting this large image into many training and validation images”.
Regarding claim 16, the rejection of claim 1 is incorporated herein.
Georgescu et al in the combination further teach wherein the machine learning model is trained, by the computing system (see para [0102]; “CNN design which can be programmed to and trained for classifying tissue into any number of different types as desired and clinically relevant”), to identify at least one of inflammation, cancer cells, and extracellular matrix (ECM) in the image data (see page 6, 3rd para; “a computer-automated tumor finding method which detects and outlines invasive and in situ breast cancer cell nuclei automatically”).
Regarding claim 20, the rejection of claim 1 is incorporated herein.
Georgescu et al. in the combination further teach further comprising classifying, by the computing system, the image data based on pixel resolution, annotation tissue classes, color definitions for labeling of tissue classes, and names of tissue subtypes corresponding to labels associated with each class of tissue subtypes (see para [0035]; “With the results from the CNN, the method may be extended to include a scoring process based on the pixel classification and the tumors that are defined from that classification with reference to the probability map. For example, the method may further comprise: defining areas in the histological image that correspond to tumors according to the probability map; scoring each tumor according to a scoring algorithm to assign a score to each tumor; and storing the scores into the record in the data repository. The scoring thus takes place on the histological image, but is confined to those areas identified by the probability map as containing tumorous tissue”, see also para [0036]; “The tumor scores may also be displayed in some convenient manner, e.g. with text labels on or pointing to the tumors, or alongside the image”, and para [0099]; “Tile D is a tumor probability heatmap generated by our CNN. It can be seen how our approach of pixel-level prediction produces areas with smooth perimeter outlines. For our heatmap, different (arbitrarily chosen) colors indicate different classes, namely green for non-tumor, red for a first tumor type and blue for a second tumor type”).
Regarding claim 26, the rejection of claim 1 is incorporated herein.
Georgescu et al. in the combination further teach further comprising generating, by the computing system, immune cell heatmaps of pancreatic cancer precursor legions based on the digital volume of the tissue sample and using at least one of H&E, immunocytochemistry (HC),immunofluorescence (IF), imaging mass cytometry (IMC), and spatial transcriptomics (see para [0099]; “For our heatmap, different (arbitrarily chosen) colors indicate different classes, namely green for non-tumor, red for a first tumor type and blue for a second tumor type”, see also para [0105]; “For example, there could be 5 adjacent sections, each with a different stain, such as ER, PR, p53, HER2, H&E and Ki-67”, and para [0010]; “On the glass slide, hormone-specific antibodies are applied using immunohistochemical (IHC) techniques to a formalin-fixed paraffin-embedded breast tissue section from the patient”).
Regarding claim 33, the scope of claim 33 is fully encompassed by the scope of claim 1, accordingly, the rejection of claim 1 is fully applicable here.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Eastwood et al. and Georgescu et al. and further in view of Magee et al. as applied in claim 1 above, and further in view of Burmer et al. (US 20020186875 A1)
Regarding claim 3, the rejection of claim 1 is incorporated herein. The combination of Eastwood et al. and Georgescu et al. does not teach further comprising determining, by the computing system, 3D radial density of each identified tissue subtype and each cell in the digital volume of the tissue sample.
In the same field of endeavor, Burmer et al. teach further comprising determining, by the computing system, 3D radial density of each identified tissue subtype and each cell in the digital volume of the tissue sample (see para [0089]; “The measurement of the fractal probability density function is accomplished by applying the radial-density distribution law, d=Cr(D.sup.-2), where d is the density of tissue/cell pattern points at a given location”, see also para [0063]; “The pixel intensity data values are used in parameter extraction algorithms that operate in two or three dimensions”). Accordingly, it has been obvious to one ordinary skill in the art before the effective filed of the invention to modify a method creating a 3D volume image of a tissue block from a series of images of histological sections taken from the tissue block of Eastwood et al. in view of the use of a convolutional neural network (CNN) is applied to identifying tumors in a histological image of Georgescu et al. and further in view of Histopathology in 3D: From three-dimensional reconstruction to multi-stain and multi-modal analysis of Magee et al. and method for image recognition optimized for the repeating patterns characteristic of organic material of Burmer et al. in order to compare overall cell densities between samples (see para [0089]).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Eastwood et al. and Georgescu et al. and further in view of Magee et al. as applied in claim 1 above, and further in view of Hodnelandet al. (US 20090081775 A1).
Regarding claim 21, the rejection of claim 1 is incorporated herein. The combination of Eastwood et al. Georgescu et al. and Magee et al. as a whole does not teach further comprising, for each tissue subtype: summing, by the computing system, pixels of the tissue sample in a z dimension; generating, by the computing system, a projection of a volume of the tissue sample on an xy axis; normalizing, by the computing system, the projection based on the projection's maximum; and visualizing, by the computing system, the projection using a same color scheme created for visualization of the tissue sample in the 3D space.
In the same field of endeavor, Hodneland et al. teach further comprising, for each tissue subtype: summing, by the computing system, pixels of the tissue sample in a z dimension; generating, by the computing system, a projection of a volume of the tissue sample on an xy axis (see page 26, last para; “The 3D labelled matrices of each patient case were used to construct z-projections of each tissue subtype. For each tissue subtype, the pixels of the 3D matrix corresponding to that subtype were summed in the z-dimension, creating a projection of the volume on the xy axis”); normalizing, by the computing system, the projection based on the projection's maximum; and visualizing, by the computing system, the projection using a same color scheme created for visualization of the tissue sample in the 3D space (see page 26, last para; “The projections were normalized by their maximum and visualized using the imagesc command in MATLAB 2020b using the same color scheme created for visualization of the 3D tissue”). Accordingly, it has been obvious to one ordinary skill in the art before the effective filed of the invention to modify a method creating a 3D volume image of a tissue block from a series of images of histological sections taken from the tissue block of Eastwood et al. in view of the use of a convolutional neural network (CNN) is applied to identifying tumors in a histological image of Georgescu et al. and further in view of Histopathology in 3D: From three-dimensional reconstruction to multi-stain and multi-modal analysis of Magee et al. and device for automated cell analysis and determination of transport and communication between living cells of Hodnelandet al. in order to create visualization of the 3D tissue (see page 26, last para).
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Eastwood et al. and Georgescu et al. and further in view of Magee et al. as applied in claim 1 above, and further in view of Varekamp al. (US 20190294859 A1).
Regarding claim 22, the rejection of claim 1 is incorporated herein. The combination of Eastwood et al. Georgescu et al. and Magee et al. as a whole does not teach further comprising calculating, by the computing system, cell density of each tissue subtype in the tissue sample using the digital volume of the tissue sample.
In the same field of endeavor, Varekamp et al. teach further comprising calculating, by the computing system, cell density of each tissue subtype in the tissue sample using the digital volume of the tissue sample (see para [0020]; “the at least one quantification of the first subtype of cell comprises a density map.[0021] In other words, an image map of the relative contribution of a particular subtype within a tissue sample can be determined”, see also para [0027]; “absolute and relative cell densities can be used to determine if there is a mono-culture of a particular subtype at particular areas of a tissue sample”). Accordingly, it has been obvious to one ordinary skill in the art before the effective filed of the invention to modify a method creating a 3D volume image of a tissue block from a series of images of histological sections taken from the tissue block of Eastwood et al. in view of the use of a convolutional neural network (CNN) is applied to identifying tumors in a histological image of Georgescu et al. and further in view of Histopathology in 3D: From three-dimensional reconstruction to multi-stain and multi-modal analysis of Magee et al. and an apparatus (10) for determining cellular composition in one or more tissue sample microscopic images of Varekamp et al. in order to classify image features from one or more test patients into a low-risk or high-risk group (see para [0020).
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over Eastwood et al. and Georgescu et al. and further in view of Magee et al. as applied in claim 1 above, and further in view of Van et al. (US 20140003685 A1).
Regarding claim 24, the rejection of claim 1 is incorporated herein. The combination of Eastwood et al. Georgescu et al. and Magee et al. as a whole does not teach comprising calculating, by the computing system, collagen fiber alignment in the tissue sample using the digital volume of the tissue sample.
In the same field of endeavor, Van et al. teach further comprising calculating, by the computing system, collagen fiber alignment in the tissue sample using the digital volume of the tissue sample (see para [0028]; “calculating a variance of the recorded angles to thereby quantify a collagen fiber alignment of the sample of periprosthetic tissue”). Accordingly, it has been obvious to one ordinary skill in the art before the effective filed of the invention to modify a method creating a 3D volume image of a tissue block from a series of images of histological sections taken from the tissue block of Eastwood et al. in view of the use of a convolutional neural network (CNN) is applied to identifying tumors in a histological image of Georgescu et al. and further in view of Histopathology in 3D: From three-dimensional reconstruction to multi-stain and multi-modal analysis of Magee et al. and A method for quantifying collagen fiber alignment in periprosthetic tissue in a mammal of Van et al. in order to encourage optimal tissue integration and potentially reduce capsule formation and collagen fiber alignment (see para [0028]).
Regarding claim 25, the rejection of claim 1 is incorporated herein.
Kiemen et al. in the combination further teach further comprising calculating, by the computing system, a fibroblast aspect ratio of the tissue sample based on measuring a length of major and minor axis of nuclei in a ductal submucosa in the digital volume of the tissue sample (see page 28, last para; “Calculation of collagen fiber alignment and fibroblast aspect ratio ….measured the length of major and minor axis of nuclei in the ductal submucosa to calculate aspect ratios using ImageJ. In total, we measured 1546 nuclei. Violin plots were constructed from data using code available in ref. (47)”). Accordingly, it has been obvious to one ordinary skill in the art before the effective filed of the invention to modify a method creating a 3D volume image of a tissue block from a series of images of histological sections taken from the tissue block of Eastwood et al. in view of the use of a convolutional neural network (CNN) is applied to identifying tumors in a histological image of Georgescu et al. and method to reconstruct three-dimensional (3D) centimeter-scale tissues utilizing deep learning approaches of Kiemen et al. in order to create collagen alignment around normal pancreatic duct (see page 28, last para).
Claims 17, 27-30 are rejected under 35 U.S.C. 103 as being unpatentable over Eastwood et al. and Georgescu et al. and further in view of Magee et al. as applied in claim 1, and further in view of Duric et al. (US 20220323043 A1).
Regarding claim 17, the rejection of claim 1 is incorporated herein. The combination of Eastwood et al. Georgescu et al. and Magee et al. does not teach wherein the tissue subtypes include at least one of normal ductal epithelium, pancreatic intraepithelial neoplasia, intraductal papillary mucinous neoplasm, PDAC, smooth muscle and nerves, acini, fat, ECM, and islets of Langerhans.
In the same end of endeavor, Duric et al. wherein the tissue subtypes include at least one of normal ductal epithelium, pancreatic intraepithelial neoplasia, intraductal papillary mucinous neoplasm, PDAC, smooth muscle and nerves, acini, fat, ECM, and islets of Langerhans (see para [0016]; “The image partitioning process to generate tissue groups by the first type of tissue and the second type of tissue can comprise using a sound speed image. The first type of tissue can be fibroglandular tissue. The second type of tissue can be fatty tissue”). Accordingly, it has been obvious to one ordinary skill in the art before the effective filed of the invention to modify a method creating a 3D volume image of a tissue block from a series of images of histological sections taken from the tissue block of Eastwood et al. in view of the use of a convolutional neural network (CNN) is applied to identifying tumors in a histological image of Georgescu et al. and further in view of Histopathology in 3D: From three-dimensional reconstruction to multi-stain and multi-modal analysis of Magee et al. and method of analyzing an image of a volume of tissue to determine a risk of developing breast cancer of Duric et al. in order to monitor breast cancer during and/or after a treatment protocol (see para [0016]).
Regarding claim 27, the rejection of claim 1 is incorporated herein.
Georgescu et al. in the combination further teach further comprising: retrieving, by the computing system and from a data store (see para [0019]; “receiving a histological image or set thereof from a record stored in a data repository”), one or more deep learning models that were trained using patient tissue training data (see para [0125]; “In Step S40, training data is retrieved containing WSIs for processing which have been annotated by a clinician to find, outline and classify tumors. The clinician's annotations represent the ground truth data”, see also para [0059]; “For the deepest convolutional layer C10, three stages of deconvolution are needed, via D1 and D2 to layer D3. The result is three arrays D3, D5, D6 of equal size to the input patch”), wherein the one or more deep learning models are configured to (i) generate multi-dimensional volumes of patient tissue from patient tissue image data; wherein the patient tissue training data is different than the tissue sample and wherein the patient tissue image data is different than the image data; generating, by the computing system, the digital volume of the tissue sample in 3D space based on applying the one or more deep learning models to the image data (see para [0132]-[0137]; “After training, the CNN can be applied to WSIs independently of any ground truth data, i.e. in live use for prediction FIG. 5 is a flow diagram showing the steps involved in prediction using the CNN. In Step S50, one or more WSIs are retrieved for processing, e.g. from a laboratory information system (LIS) or other histological data repository. The WSIs are pre-processed, for example as described above. In Step S51, image patches are extracted from the or each WSI. The patches may cover the entire WSI or may be a random or non-random selection. In Step S52, the image patches are pre-processed, for example as described above. In Step S53, each of a batch of input image patches is input into the CNN and processed to find, outline and classify the patches on a pixel-by-pixel basis as described further above with reference to FIGS. 1A and 1B”, see also para [0106]-[0107]; “The different images are then aligned, warped or otherwise pre-processed to map the coordinates of any given feature on one image to the same feature on the other images… a coordinate mapping between different WSIs of a set comprising differently stained adjacent sections, the WSIs can be merged into a single composite WSI from which composite patches may be extracted for processing by the CNN”).
Duric et al. in the combination further teach (ii) determine stiffness measurements of tissue components in the multi- dimensional volumes of patient tissue (see para [0087]; “Volumetric stiffness measurements may further stratify this risk, particularly for the denser fibroglandular/stromal tissues”, see also para [0199]; “UST stiffness measurements by SoftVue extracted information on the tissue bulk modulus which was then converted to an index of relative tissue stiffness (from 0=very soft to 1=extremely stiff)”); determining, by the computing system, stiffness measurements of the tissue components of the tissue sample based on applying the one or more deep learning models to the digital volume of the tissue sample (see Abstract; “method of analyzing an image of a volume of tissue to determine a risk of developing breast cancer”, see also para [0097]; “machine learning and/or employing a mask to selectively identify more grouped regions of stiffness >5 mm (i.e., representing potential mass for evaluation)”); and returning, by the computing system, the determined stiffness measurements for the tissue components of the tissue sample (see para [0267]; “SoftVue UST is unique in its ability to display a whole-breast distribution of tissue stiffness, including masses”, see also para [0276]; “Quantitative stiffness values of large and small masses, as displayed by the unfiltered and spatially filtered algorithms, are shown in Table 12”). Accordingly, it has been obvious to one ordinary skill in the art before the effective filed of the invention to modify a method creating a 3D volume image of a tissue block from a series of images of histological sections taken from the tissue block of Eastwood et al. in view of the use of a convolutional neural network (CNN) is applied to identifying tumors in a histological image of Georgescu et al. and further in view of Histopathology in 3D: From three-dimensional reconstruction to multi-stain and multi-modal analysis of Magee et al. and method of analyzing an image of a volume of tissue to determine a risk of developing breast cancer of Duric et al. in order to monitor breast cancer during and/or after a treatment protocol (see page 28, last para).
Regarding claim 28, the rejection of claim 27 is incorporated herein.
Duric et al. in the combination further teach wherein the tissue sample is a breast tissue (see para [0015]; “stiffness of a region of interest within the volume of breast tissue”).
Regarding claim 29, the rejection of claim 27 is incorporated herein.
Duric et al. in the combination further teach wherein determining, by the computing system, stiffness measurements of the tissue components of the tissue sample comprises determining Pearson or Spearman correlation and statistical significance for each of the tissue components in the digital volume of the tissue sample (see para [0139]; “Pearson correlation coefficients, or analysis of variance (ANOVA) as appropriate”, see also para [0148]; “Spearman correlation coefficients were calculated to determine the strength of the correlations between the Volpara and UST assessment of breast density”).
Regarding claim 30, the rejection of claim 27 is incorporated herein.
Duric et al. in the combination further teach wherein the stiffness measurements correspond to at least one of (i) resistances of the tissue components of the tissue sample to deformation, (ii) elastic modulus, and (iii) Young's modulus (see para [0083]; “The primary method by which to assess breast density with ultrasound tomography is through the measurement of Sound Speed. The average speed of sound (s) through human tissue is related to tissue density and elasticity as: ….. In human breast tissue, the elastic constant scales as c is proportion to ρ.sup.3. Substitution into the above equation for sound speed allows us to factor out the dependence on elasticity”, see also para [0096]; “Tissue properties expressed by the bulk modulus thus describe material resistance to uniform compression and associated volume changes. The bulk modulus also has a larger dynamic range than either Young's or shear modulus, allowing greater likelihood of tissue differentiation”).
Claims 31-32 are rejected under 35 U.S.C. 103 as being unpatentable over Eastwood et al. and Georgescu et al. and further in view of Magee et al. as applied in claim 1, and further in view of Tolkach et al. NPL “Three-dimensional reconstruction of prostate cancer architecture with serial immunohistochemical sections: hallmarks of tumour growth, tumour compartmentalisation, and implications for grading and heterogeneity”
Regarding claim 31, the rejection of claim 1 is incorporated herein. The combination of Eastwood et al. Georgescu et al. and Magee et al. does not teach wherein: measuring the 3D connectivity of the particular type of tissue in the tissue sample includes determining that multiple distinct instances of the particular type of tissue present in a two dimensional (2D) independent serial image of the image data belong to separate branches of a single 3D structure of the particular type of tissue: and wherein the 2D independent serial image of the image data is one of the independent serial images and represents a corresponding section of the multiple sections of the tissue sample.
In the same field of endeavor, Tolkach et al. teaches wherein: measuring the 3D connectivity of the particular type of tissue in the tissue sample includes determining that multiple distinct instances of the particular type of tissue present in a two dimensional (2D) independent serial image of the image data belong to separate branches of a single 3D structure of the particular type of tissue (see page 1051, Method and results; “Seventy-five consecutive histological sections (5 lm) of a typical prostate adenocarcinoma…. continuity of all glands, with the tumour being an integrated system …for further 3D reconstructions with FIJI/IMAGEJ software… the coexistence of independent terminal units of such branches in one 2D slice”); and wherein the 2D independent serial image of the image data is one of the independent serial images and represents a corresponding section of the multiple sections of the tissue sample (see page 1051, Method and results; “Seventy-five consecutive histological sections (5 lm) of a typical prostate adenocarcinoma…. continuity of all glands, with the tumour being an integrated system …for further 3D reconstructions with FIJI/IMAGEJ software… the coexistence of independent terminal units of such branches in one 2D slice”). Accordingly, it has been obvious to one ordinary skill in the art before the effective filed of the invention to modify a method creating a 3D volume image of a tissue block from a series of images of histological sections taken from the tissue block of Eastwood et al. in view of the use of a convolutional neural network (CNN) is applied to identifying tumors in a histological image of Georgescu et al. and further in view of Histopathology in 3D: From three-dimensional reconstruction to multi-stain and multi-modal analysis of Magee et al. and Three-dimensional reconstruction of prostate cancer architecture with serial immunohistochemical sections: hallmarks of tumour growth, tumour compartmentalisation, and implications for grading and heterogeneity of Tolkach et al. et al. in order to examine the feasibility of three-dimensional (3D) reconstruction of tumour morphology based on multiple consecutive histological sections and to decipher relevant features of prostate cancer architecture (see page 28, last para).
Regarding claim 32, the rejection of claim 31 is incorporated herein.
Tolkach et al. in the combination further teach wherein: the multiple distinct instances of the particular type of tissue present in the 2D independent serial image represent cancer or precancer lesions that are distinct from each other in the 2D independent serial image (see page 1051, Method and results; “Seventy-five consecutive histological sections (5 lm) of a typical prostate adenocarcinoma (Gleason score of 3 + 4 = 7) were immune stained (pan-cytokeratin)….. independent terminal units of such branches in one 2D slice explaining intratumoral heterogen ity”); and the single 3D structure of the particular type of tissue represents a 3D cancer or precancer lesion represented within the digital volume of the tissue sample in 3D space (see page 1051, Method and results; “a tumour building a tree-like branched structure with very ‘plastic’ branches….continuity of all glands, with the tumour being an integrated system …for further 3D reconstructions with FIJI/IMAGEJ software… the coexistence of independent terminal units of such branches in one 2D slice”).
Allowable Subject Matter
Claims 9-11, 14-15 and 18-19 are objected to as being dependent upon a rejected base claim, but
would be allowable if rewritten in independent form including all of the limitations of the base
claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claim 9 further recites “further comprising normalizing, by the computing system, the registered image data to generate normalized image data based on: correcting two dimensional (2D) serial cell counts based on in-situ measured nuclear diameter of cells in the tissue sample; locating nuclei in each histological section of the registered image data based on color deconvolution; for each located nuclei, measuring in-situ diameters of each cell type; mapping the nuclei in a serial 2D z plane; and extrapolating true cell counts from the serial 2D z plane” which is neither anticipated nor rendered obvious by the prior art of record.
Claim 10 further recites “further comprising normalizing, by the computing system, the registered image data to generate normalized image data based on: extracting, using color deconvolution, a hemotoxylin channel from each of the image data depicting the one or more sections of the tissue samples stained with H&E; and for each of the image data depicting the one or more sections of the tissue samples stained with H&E: identifying a tissue region in the image data based on detecting regions of the image data with low green channel intensity and high red-green-blue (rbg) standard deviation; converting rgb channels in the image data to optical density; identifying clusters, based on k means clustering, to represent one or more optical densities of the image data; and deconvolving the image data, based on the one or more optical densities, into hemotoxylin, eosin, and background channel images”.
Claim 11 is objected to since it depends on base claim 10.
Claim 14 further recites “wherein training the machine learning model further comprises: identifying, by the computing system, bounding boxes around each annotated region of the one or more tissue samples; and randomly overlaying each identified bounding box containing a least represented tissue subtype on a blank image tile until the tile is at least 65% full of annotated regions of the one or more tissue samples”.
Claim 15 is objected to since it depends on base claim 14.
Claim 18 further recites “wherein determining, by the computing system, the digital volume of the tissue sample in 3D space based on the annotated image data comprises consolidating multi-labeled image data into a 3D matrix based on registering (i) the annotated image data and (ii) cell coordinates counted on unregistered histological sections of the annotated image data”.
Claim 19 is objected to since it depends on base claim 14.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WINTA GEBRESLASSIE whose telephone number is (571)272-3475. The examiner can normally be reached Monday-Friday9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at 571-270-5180. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WINTA GEBRESLASSIE/Examiner, Art Unit 2677
