Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2, 4-14, 17-18 and 20-23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Molin (US 20180089496 A1), in view of Hall (US 20150279026 A1), in view of Yang (US 20200402215 A1), and further in view of Avanaki (US 20200359887 A1).
Regarding claims 1 and 18, Molin discloses a display configured to display a user interface (fig. 7, ref 10 (display));
a memory (fig. 14, ref 136 (memory)); and
a processor operatively coupled to the display and the memory, the processor configured to (fig. 14, ref 148):
PNG
media_image1.png
457
397
media_image1.png
Greyscale
receive (fig. 11, ref 400
[0106] FIG. 9 illustrates that the circuit 10c can calculate probability using a machine learning system 66 that can evaluate measured properties of respective patches 25 input to the machine learning system 66, such as shape, intensity, volume, area, color);
PNG
media_image2.png
555
370
media_image2.png
Greyscale
identify, using a machine learning algorithm, a set of patches of the image data ("[0095] The term “positivity” and derivatives thereof refers to a probability of a positive association with a clinical disease state such as cancer, for example, that can be identified by a stained color of a nuclei. The typical case is for cancer detection, diagnosis or evaluation, where nuclei relevant for disease assessment have been given a distinct staining, often a brown color.
[0106] FIG. 9 illustrates that the circuit 10c can calculate probability using a machine learning system 66 that can evaluate measured properties of respective patches 25 input to the machine learning system 66, such as shape, intensity, volume, area, color and the like and output a calculated probability including, for example, a nucleus probability (a probability that the detected nuclei is actually a nuclei) and a positivity probability (a probability that the detected nuclei is positively stained)."),
each patch from the set of patches includes a suspected abnormality ([0104] Again, each randomly selected patch from this class (block 214) can be sorted by the same defined common sorting criteria, shown as by nucleus and positivity probability (block 234). Sorted patches from each of the three different classes can then be presented in at least one patch gallery in a sorted order (block 240) of probability of positivity.
[0105] This sorting procedure is representative of a positive count evaluation of cells but other sorting procedures can be used for other evaluations of a patch gallery content, such as mitosis as shown in FIGS. 5A and 5B where there is no particular order but detected mitoses can be shown side by side in the patches displayed to exclude false positives.
[0181] A machine learning algorithm 66 can be used to divide a plurality of patches 25 into a patch gallery 25g. The patches can be divided into three groups, Positive, Negative and Excluded, similar to the division of nuclei in FIG. 6, which as discussed above is divided into Positive, Negative and Non-tumor. Non-tumor is the same as Excluded. Dividing the patches 25 into three groups is the same as classifying into the three groups and/or nominally sorting the patches along a cell type feature, which is a compound feature 25c as in FIG. 9.
PNG
media_image3.png
304
722
media_image3.png
Greyscale
[0189] FIG. 17 is a flow chart of actions/operations that can occur to reclassify patches 25 based on user-input and can optionally retrain a machine learning model for the classification according to embodiments of the present invention.);
generate a consolidated view of patches selected identified by the machine learning algorithm as a suspected abnormality, wherein the consolidated view includes ([0104] The patches 25 can be electronically divided and/or pre-sorted into a plurality of different primary classes based on the presence of a cell nucleus and a positivity probability and may exclude nuclei outside a visible span (block 200).
[0105] This sorting procedure is representative of a positive count evaluation of cells but other sorting procedures can be used for other evaluations of a patch gallery content, such as mitosis as shown in FIGS. 5A and 5B where there is no particular order but detected mitoses can be shown side by side in the patches displayed to exclude false positives.
[0106] FIG. 9 illustrates that the circuit 10c can calculate probability using a machine learning system 66 that can evaluate measured properties of respective patches 25 input to the machine learning system 66, such as shape, intensity, volume, area, color and the like and output a calculated probability including, for example, a nucleus probability (a probability that the detected nuclei is actually a nuclei) and a positivity probability (a probability that the detected nuclei is positively stained).)), the consolidated view being arranged according to a predefined layout ([0040] The at least one panel of the patch gallery can be arranged in an order from left to right and/or up to down, with decreasing probability values of positivity and/or cell nuclei decreasing to the right and/or down.);
display the consolidated view of the set of patches on the user interface (fig. 4a);
PNG
media_image4.png
371
534
media_image4.png
Greyscale
and display a two-dimensional view of the volume of the tissue sample in the user interface ([0048] The method can further include concurrently displaying a zoomed or normal main digital pathology or cytology image view with the patch gallery and the patches in the patch gallery can be presented in decreasing value from top to bottom and/or left to right on the display.
[0049] The WSI image can be a two-dimensional (2-D) WSI image having between about 1×10.sup.6 pixels to about 1×10.sup.12 pixels.),
wherein: the two-dimensional view is from a user-selected depth in the volume of the tissue sample ([0092] The term “semi-automated” refers to an image processing system, method, module or circuit that employs user (e.g., pathologist) input to adjust a feature dimension of the image to explore image feature variability generated by automated systems.
[0109] The patch gallery 25g can provide separation zones 25z between adjacent patches 25 and a user interface feature 31 which can reside in one or more of the zones 25z. The user interface feature 31 can be electronically selected and/or moved relative to a first position on the display, in the patch gallery user interface 30, to adjust a feature dimension in the patches 25.).
Molin does not disclose wherein each patch from the set of patches is associated with a spatial location in the tissue sample and the processor is configured to display markings in the consolidated view to associate patches from the set of patches that have proximate spatial locations with one another and link portions of the consolidated view to portions of the image data to switch between the consolidated view and a portion of the image data including a patch included in the consolidated view.
In a similar field of endeavor of medical microscopy imaging, Hall teaches wherein each patch from the set of patches is associated with a spatial location in the tissue sample ([0077] Thus, for example, different tissue samples from different cut locations can have unique electronic identifiers that correlate a tissue sample Ts with a physical cut location 10 and/or the corresponding virtual cut location 110.)
and the processor is configured to display markings in the consolidated view to associate patches from the set of patches that have proximate spatial locations with one another ("Fig. 4
[0077] The circuit 30 can be configured to correlate virtual cut mark locations 110 on the macroscopic map M (FIG. 4) to the tissue samples Ts taken from the respective cut locations 10 on the specimen G. Thus, for example, different tissue samples from different cut locations can have unique electronic identifiers that correlate a tissue sample Ts with a physical cut location 10 and/or the corresponding virtual cut location 110.") and link portions of the consolidated view to portions of the image data to switch between the consolidated view and a portion of the image data including a patch included in the consolidated view ([0081] The cut location marks 110 on a respective macroscopic image or model M may be configured as active or inactive objects or links. For active objects or links, a user may select (e.g., click or touch) a particular cut location mark 110 on image or model M on the display 35 and the viewer V can automatically present WSI slides S associated with the selected cut location mark. The resulting presentation on the display 35 can be to provide the relevant slides S in a new larger viewing window, in a concurrent adjacent or overlay window and/or along one side of the macroscopic image M. A set of thumbnail WSI images 200 of different slides S may be concurrently displayed along a side, top or bottom of the screen. The slide Sa being currently viewed by the viewer V can be highlighted, shown as color coded perimeter matching a color of the visually enhanced cut 110a (cut A). The set of slides 200 may be displayed via a pull down option, for example, or in subsets of slides, relative to a particular block B or a particular cut location, for example.
[0083] Still referring to FIG. 4, the viewer V may optionally also be configured to provide a reference window Wref which shows an overview of the entire currently viewed slide Sa.
[0108] FIGS. 8 and 10 illustrate an optional use of tether lines 88 that connect a cut location 110 on the macroscopic image M to a thumbnail slide image T and/or the larger displayed slide sample S. The visual tethers may be faded, dimmed or selected via a UI by a user. Typically, the tethers 88 are not required and may be used via a selection by a user or may be omitted as an option from the viewer V. Where used, the tethers may be just visual tethers or may be active links that allow for more information to be provided to a user for a particular cut location and/or slide.
[0109] FIGS. 9 and 10 illustrate that a viewer V can show two large views of slides S.sub.1, S.sub.2 in high magnification and the corresponding cut location marks 110a (“A” and “E”) can be visually enhanced relative to the other cut mark locations. The cut location marks 110a that are related to the current large slide views S.sub.1, S.sub.2 can be shown as bold, with increased intensity and/or in solid line and/or with a different color relative to the other cut location marks 110 which can also be shown in broken lines. Here, each relevant cut mark location 110a is shown in a brighter, increased intensity and in a different solid line color on the image M.).
PNG
media_image5.png
607
497
media_image5.png
Greyscale
(in fig 9, t1 and t2 are different views of the different portions of a cell. They are in a consolidated view (portions of images stitched together in a row). T1 is shown as s1 (can be seen as a patch of the reference image with the red arrows) while t2 is shown as s2.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin’s disclosure of tissue image patching, with Hall’s teaching of spatial location of cell patches, in order to automatically generate physical or digital markings on the macroscopic images to reduce time and for better accuracy ([0006]). Molin and Hall do explicitly not disclose or teach and wherein: the machine learning algorithm is trained, using previously labeled volumetric imaging data of benign and malignant tissue, to identify features associated with malignant tissue.
Molin implicitly discloses the machine learning algorithm is trained, using previously labeled volumetric imaging data of benign and malignant tissue, to identify features associated with malignant tissue ([0189] FIG. 17 is a flow chart of actions/operations that can occur to reclassify patches 25 based on user-input and can optionally retrain a machine learning model for the classification according to embodiments of the present invention.).
However, in a similar field of endeavor of endeavor class-aware adversarial pulmonary nodule synthesis, Yang explicitly teaches, in better detail, and the machine learning algorithm is trained, using previously labeled volumetric imaging data of benign and malignant tissue, to identify features associated with malignant tissue ([0023] FIG. 1 shows an exemplary computed tomography (CT) medical image 100 of a patient showing pulmonary nodule 102. Pulmonary nodule 102 in image 100 is a malignant pulmonary nodule. Image 100 may be part of a training dataset for training a machine learning network for performing an image analysis task such as, e.g., classifying a nodule as benign or malignant.), wherein said machine learning algorithm distinguishes between benign and malignant tissue features ([0038] At step 404, the target nodule is classified using a trained machine learning network trained based on a synthesized medical image patch. The target nodule may be classified as one of malignant or benign. In one embodiment, the trained machine learning network is a deep 3D CNN pre-trained for natural video classification, however any suitable machine learning network may be employed. In one embodiment, the synthesized medical image patch is generated according to the steps of method 200 of FIG. 2.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin and Hall’s disclosure of spatial location of cell patches, with Yang’s teaching of benign and malignant classification, by using a synthesized medical image patch that includes the unmasked portion of the initial medical image patch and a synthesized nodule replacing the masked portion of the initial medical image patch in order to reduce processing time (abstract and [0003]). Molin, Hall and Yang do not disclose optical coherence tomography B-scan, the machine learning algorithm is trained, using previously labeled volumetric imaging data of benign and malignant tissue, to identify features associated with malignant tissue, and the suspected abnormality is identified by the machine learning algorithm as having features associated with malignant tissue.
In a similar field of endeavor of identifying, detecting and/or diagnosing cancer using OCT features, Avanaki teaches receive optical coherence tomography B-scan image data (B-scan image data) of a tissue sample ([0089] A region of interest is specified in an OCT B-scan image. …. By repetition for several regions of interest (ROIs), which are averaged, and standard deviations calculated, optical radiomic features can be derived for that tissue: mean and standard deviation of scattering and absorption coefficients, and anisotropy factor.
[0094] Different ways of choosing the ROI may be investigated on optical coherence tomography (OCT) images of milk phantoms: (1) a median filter may be initially applied on a stack of 170 OCT images acquired from the same cross section, the extracted optical properties may be averaged over several ROIs chosen in the resultant image);
the machine learning algorithm is trained, using previously labeled volumetric imaging data of benign and malignant tissue, to identify features associated with malignant tissue ([0035] OCT systems may provide high spatial resolution (<10 microns), intermediate penetration depth (˜1.5 to 2 mm), and volumetric imaging capability. This disclosure provides a diagnostic-assistant modality in dermatology especially to detect and diagnose benign skin tumors
[0052] The system 100 may include a priori knowledge of OCT images and, healthy and melanoma tissue histology, to allow training of heuristics (e.g., a machine-learning kernel) with improved specificity and detection accuracy than traditional systems and classifiers.
[0053] Referring again to FIG. 2, process 200 may include one or multiple phases such as Training Phase 201 and Test Phase 203. For Training Phase 201, the optical radiomic features and their labels (e.g., histology results) are input to provide heuristics, e.g., machine learning. For Test Phase 203, OCT images of a suspect skin area will be analyzed by the trained heuristics (e.g., machine-learning kernel) with the selected optical radiomic features, e.g., optical radiomic signatures, and indicate or display diagnostic results using display 111. Diagnostic results may include the tissue status such as healthy, non-cancerous, non-melanoma, or benign tissue (e.g., “Tissue sample is consistent with healthy tissue”), or unhealthy, cancerous, melanoma or malignant tissue (e.g., “Tissue sample exhibits characteristics consistent with melanoma”).).
each patch from the set of patches only includes a suspected abnormality ([0041] The operations may include, for example, obtain a first image of a suspect region-of-interest (SROI) for a suspect lesion, obtain a second image of a healthy region-of-interest (HROI) near the suspect lesion, classify the extracted optical properties to generate an issue status including as at least one of melanoma tissue and benign tissue, and display the issue status indicating the at least one of melanoma tissue and benign tissue. The operations may further include to normalize optical properties from the SROI and from the HROI, obtain an averaged A-line of the SROI and the HROI, generate a set of normalized optical radiomic features from an averaged A-line of the SROI and the HROI, evaluate the set of normalized optical radiomic features to distinguish whether the suspect lesion is consistent with the melanoma tissue and the benign tissue, and display optical information including at least one of optical properties, normalized optical properties, and classified optical properties indicating the least one of melanoma tissue and benign tissue.);
the suspected abnormality is identified by the machine learning algorithm as having features associated with malignant tissue ([0092] The heuristics may include one or multiple phases including, for example, (i) a Training Phase, and (ii) a Test Phase. In the Training Phase, the optical radiomic features and their labels (histology results) are input to the heuristics. In the Test Phase, OCT images of a suspect skin area will be analyzed by the trained heuristics (e.g., machine-learning kernel) with the selected optical radiomic features, e.g., optical radiomic signatures. The system 100 may display the status of the tissue associated with healthy or benign tissue, e.g., “Tissue sample is consistent with healthy tissue” or cancerous or melanoma tissue, e.g., “Tissue sample exhibits characteristics consistent with melanoma”.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to combine the known method of benign or malignant tissue determination using a computed tomography medical image of Molin, Hall and Yang’s disclosure with the known techniques of optical coherence tomography B-scan images as taught by Avanaki, in order to yield predictable results a non-invasive, no contact way of imaging a tissue to generate cross-sectional, ultra-high resolution images, which results in an improved system.
Regarding claim 2, Molin discloses wherein the image data includes a set of two-dimensional scans of the tissue sample ([0026] The obtained WSI image can be a two-dimensional (2-D) WSI having between about 1?10.sup.6 pixels to about 1?10.sup.12 pixels.),
the processor further configured to process the set of two-dimensional scans of the tissue sample into a plurality of patches each having the same predefined dimensions ([0101] The patches 25 in a respective patch gallery 25g view can have the same area (height by width dimensions) and each patch 25 can correspond to the same size area of the WSI image.),
each patch from the plurality of patches being a different portion of a two-dimensional scan from the set of two-dimensional scans ([0099] In a respective patch gallery 25g, the patches 25 can be associated with different areas in an ROI 15r. The ROI 15r can be many times (10?-100? or more) larger in size than the patches 25 of the WSI image 15.),
the set of patches being patches from the plurality of patches that include at least one feature associated with an abnormality ("[0039] The electronically classifying the patches can be carried out to classify the patches as either (i) comprising positive nuclei, (ii) comprising negative nuclei, or (iii) comprising non-nuclei.
[0104] The number of samples N can be the same or different for each of the positive and negative patch classes.").
Regarding claim 4, Molin does not disclose but Hall teaches wherein the processor is configured to display (1) a first marking having a first color proximate to each patch from a first subset of the set of patches ([0082] Thus, each mark 110 can be shown with a defined color that corresponds to the slide color indicia, e.g., each mark 110 and its associated slides S can be shown all in green, red, fuchsia, pink, yellow and the like.) and
(2) a second marking having a second color different from the first color proximate to each patch from a second subset of the set of patches ([0082] Thus, each mark 110 can be shown with a defined color that corresponds to the slide color indicia, e.g., each mark 110 and its associated slides S can be shown all in green, red, fuchsia, pink, yellow and the like.),
each patch from the first subset of patches having a spatial location that is adjacent to at least one other patch from the first subset of patches, and each patch from the second subset of patches having a spatial location that is adjacent to at least one other patch from the second subset of patches (fig. 8, ref m, ref 88 and [0082] For example, all slides from the same cut location, typically some with different stains, may be grouped together, e.g., adjacent each other, shown only in the thumbnail set 200 (with other slides from other cut locations omitted) and/or concurrently shown with all slides but emphasized with a common color background and/or perimeter, with an adjacent icon or with an overlay.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin’s disclosure of tissue image patching, with Hall’s teaching of spatial location of cell patches, in order to automatically generate physical or digital markings on the macroscopic images ([0006]).
Regarding claims 5 and 21, Molin discloses wherein the processor is configured to display the consolidated view of the set of patches in a first area of the user interface (fig. 1, ref 25g (set of patches)),
the processor further configured to display a perspective view of the tissue sample in a second area of the user interface such that the perspective view of the tissue sample is displayed together with at least a portion of the consolidated view of the set of patches (fig. 1, ref 16 (second area), ref 25g (set of patches)).
Regarding claims 6 and 22, Molin does not disclose but Hall teaches wherein the processor is further configured to display a two-dimensional view of the tissue sample at a predefined depth in a third area of the user interface such that the two-dimensional view of the tissue sample is displayed together with the perspective view and at least a portion of the consolidated view of the set of patches (fig 4, ref wref (full view of current slide), ref 5 (view of small area of wref), ref 200 (views of many small areas in ref 5)).
PNG
media_image6.png
519
411
media_image6.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin’s disclosure of tissue image patching, with Hall’s teaching of spatial location of cell patches, in order to automatically generate physical or digital markings on the macroscopic images ([0006]).
Regarding claims 7 and 23, Molin does not disclose but Hall teaches wherein each patch from the set of patches is associated with a spatial location in the tissue sample (fig. 8, ref m, ref 88 and [0082] For example, all slides from the same cut location, typically some with different stains, may be grouped together, e.g., adjacent each other, shown only in the thumbnail set 200 (with other slides from other cut locations omitted) and/or concurrently shown with all slides but emphasized with a common color background and/or perimeter, with an adjacent icon or with an overlay.),
the processor further configured to display a first marking proximate to a patch from the set of patches in the consolidated view and a second marking in at least one of the perspective view or the two-dimensional view that is indicative of the spatial location of the patch (fig. 8, ref m, ref 88 and [0082] For example, all slides from the same cut location, typically some with different stains, may be grouped together, e.g., adjacent each other, shown only in the thumbnail set 200 (with other slides from other cut locations omitted) and/or concurrently shown with all slides but emphasized with a common color background and/or perimeter, with an adjacent icon or with an overlay.),
the first and second markings sharing a common characteristic (fig. 8, ref m, ref 88 and [0082] For example, all slides from the same cut location, typically some with different stains, may be grouped together, e.g., adjacent each other, shown only in the thumbnail set 200 (with other slides from other cut locations omitted) and/or concurrently shown with all slides but emphasized with a common color background and/or perimeter, with an adjacent icon or with an overlay.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin’s disclosure of tissue image patching, with Hall’s teaching of spatial location of cell patches, in order to automatically generate physical or digital markings on the macroscopic images ([0006]).
Regarding claim 8, Molin does not disclose but Hall teaches wherein the first and second markings have the same color ([0082] In some embodiments, each slide S associated with a respective cut location mark 110 can be color-coded to the mark of that location, e.g., the object overlay line can have a color that is the same as that of the slide background and/or perimeter border, for example.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin’s disclosure of tissue image patching, with Hall’s teaching of spatial location of cell patches, in order to automatically generate physical or digital markings on the macroscopic images ([0006]).
Regarding claim 9, Molin discloses wherein the processor is configured to, in response to receiving an input indicating a different predefined depth, display a two-dimensional view of the tissue sample at the different predefined depth ([0101] The patches 25 in a respective patch gallery 25g view can have the same area (height by width dimensions) and each patch 25 can correspond to the same size area of the WSI image. Alternatively, one or more of the patches 25 can have a larger or smaller area in the patch gallery view on the display 10 than one or more others.).
Regarding claim 10, Molin discloses wherein the processor is configured to, in response to detecting a selection of a patch from the set of patches, display a two-dimensional scan of the tissue sample that includes the patch selected from the set of patches ([0193] table 1, click on patch, navigation to location of patch in high magnification).
Regarding claim 11, Molin does not disclose but Hall teaches wherein the processor is further configured to display a marking proximate to a portion of the two-dimensional scan that corresponds to a location of the patch selected from the set of patches ([0079] The viewer V may be configured to show only the relevant cut mark 110a and omit or greatly reduce the intensity and/or visual prominence of the non-relevant cut marks (the cut marks not associated with the current slide Sa on the display 35).).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin’s disclosure of tissue image patching, with Hall’s teaching of spatial location of cell patches, in order to automatically generate physical or digital markings on the macroscopic images ([0006]).
Regarding claims 12 and 17, Molin discloses wherein the processor is further configured to, in response to receiving an input indicating that a patch from the set of patches is not associated with an abnormality ([0115] Adjustment to move the lines 34 to form a different border in the patch gallery 25g allows a user to adjust the inclusion parameters applied by an automated circuit and can automatically remove the positive count of nuclei in patches associated with/from the sub-segment of the ROI 18 as non-relevant patches 25, which in the gallery 25g, reside closer to the “non-tumor” and “negative” sides of the patch gallery 25g.):
remove the patch from the set of patches ([0115] Adjustment to move the lines 34 to form a different border in the patch gallery 25g allows a user to adjust the inclusion parameters applied by an automated circuit and can automatically remove the positive count of nuclei in patches associated with/from the sub-segment of the ROI 18 as non-relevant patches 25, which in the gallery 25g, reside closer to the “non-tumor” and “negative” sides of the patch gallery 25g.); and
after removing the patch from the set of patches, generate an updated consolidated view of the set of patches ([0115] The view and/or count in a normal image view 15 on the display 10 can be updated automatically based on this user adjustment, typically after accepting the adjustment via user interface 30i.).
Regarding claim 13, Molin does not disclose but Hall teaches the processor operatively coupled to the imaging device ([0014] Other embodiments are directed to a grossing workstation. The workstation includes: a cut mark location identification circuit; at least one camera over a workspace at the workstation in communication with the cut mark location identification circuit; and a cutting instrument in communication with the cut mark location identification circuit.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin’s disclosure of tissue image patching, with Hall’s teaching of spatial location of cell patches, in order to automatically generate physical or digital markings on the macroscopic images ([0006]).
Regarding claim 14, Molin discloses a memory (fig. 14, ref 136 (memory)); and
a processor operatively coupled to the memory, the processor configured to (fig. 14, ref 148):
receive a set of two-dimensional image scans of a three-dimensional tissue sample ([0027] The WSI image can be a three-dimensional (3-D) WSI with a z extent that has a plurality of slices across a depth of a tissue section with less pixels in the z extent relative to x and y extents.);
process the set of two-dimensional image scans to produce a set of patches each having the same dimensions ([0101] The patches 25 in a respective patch gallery 25g view can have the same area (height by width dimensions) and each patch 25 can correspond to the same size area of the WSI image.),
each patch from the set of patches including image data from a different portion of the set of two-dimensional image scans ([0099] In a respective patch gallery 25g, the patches 25 can be associated with different areas in an ROI 15r. The ROI 15r can be many times (10?-100? or more) larger in size than the patches 25 of the WSI image 15.);
identify, using a machine learning algorithm ("[0095] The term “positivity” and derivatives thereof refers to a probability of a positive association with a clinical disease state such as cancer, for example, that can be identified by a stained color of a nuclei. The typical case is for cancer detection, diagnosis or evaluation, where nuclei relevant for disease assessment have been given a distinct staining, often a brown color.
[0106] FIG. 9 illustrates that the circuit 10c can calculate probability using a machine learning system 66 that can evaluate measured properties of respective patches 25 input to the machine learning system 66, such as shape, intensity, volume, area, color and the like and output a calculated probability including, for example, a nucleus probability (a probability that the detected nuclei is actually a nuclei) and a positivity probability (a probability that the detected nuclei is positively stained).") and
(2) include at least one feature that is a suspected ([0189] FIG. 17 is a flow chart of actions/operations that can occur to reclassify patches 25 based on user-input and can optionally retrain a machine learning model for the classification according to embodiments of the present invention.); and
generate a consolidated view of such that each patch in the consolidated view includes at least one feature that is a suspected abnormality ([0104] The patches 25 can be electronically divided and/or pre-sorted into a plurality of different primary classes based on the presence of a cell nucleus and a positivity probability and may exclude nuclei outside a visible span (block 200).
[0105] This sorting procedure is representative of a positive count evaluation of cells but other sorting procedures can be used for other evaluations of a patch gallery content, such as mitosis as shown in FIGS. 5A and 5B where there is no particular order but detected mitoses can be shown side by side in the patches displayed to exclude false positives.
[0106] FIG. 9 illustrates that the circuit 10c can calculate probability using a machine learning system 66 that can evaluate measured properties of respective patches 25 input to the machine learning system 66, such as shape, intensity, volume, area, color and the like and output a calculated probability including, for example, a nucleus probability (a probability that the detected nuclei is actually a nuclei) and a positivity probability (a probability that the detected nuclei is positively stained).)), in which the subset of patches are arranged according to a predefined layout ([0040] The at least one panel of the patch gallery can be arranged in an order from left to right and/or up to down, with decreasing probability values of positivity and/or cell nuclei decreasing to the right and/or down.). Molin does not disclose or teach and wherein each patch from the set of patches is associated with a spatial location in the tissue sample,
the subset of patches arranged in the consolidated view such that patches from the subset of patches having spatial locations proximate to one another are arranged adjacent to one another and link portions of the consolidated view to portions of the image data to switch between the consolidated view and a portion of the image data including a patch included in the consolidated view.
In a similar field of endeavor of medical microscopy imaging, Hall teaches and wherein each patch from the set of patches is associated with a spatial location in the tissue sample ([0077] Thus, for example, different tissue samples from different cut locations can have unique electronic identifiers that correlate a tissue sample Ts with a physical cut location 10 and/or the corresponding virtual cut location 110.),
the subset of patches arranged in the consolidated view such that patches from the subset of patches having spatial locations proximate to one another are arranged adjacent to one another ([0082] For example, all slides from the same cut location, typically some with different stains, may be grouped together, e.g., adjacent each other, shown only in the thumbnail set 200 (with other slides from other cut locations omitted) and/or concurrently shown with all slides but emphasized with a common color background and/or perimeter, with an adjacent icon or with an overlay.); and link portions of the consolidated view to portions of the image data to switch between the consolidated view and a portion of the image data including a patch included in the consolidated view ([0081] The cut location marks 110 on a respective macroscopic image or model M may be configured as active or inactive objects or links. For active objects or links, a user may select (e.g., click or touch) a particular cut location mark 110 on image or model M on the display 35 and the viewer V can automatically present WSI slides S associated with the selected cut location mark. The resulting presentation on the display 35 can be to provide the relevant slides S in a new larger viewing window, in a concurrent adjacent or overlay window and/or along one side of the macroscopic image M. A set of thumbnail WSI images 200 of different slides S may be concurrently displayed along a side, top or bottom of the screen. The slide Sa being currently viewed by the viewer V can be highlighted, shown as color coded perimeter matching a color of the visually enhanced cut 110a (cut A). The set of slides 200 may be displayed via a pull down option, for example, or in subsets of slides, relative to a particular block B or a particular cut location, for example.
[0108] FIGS. 8 and 10 illustrate an optional use of tether lines 88 that connect a cut location 110 on the macroscopic image M to a thumbnail slide image T and/or the larger displayed slide sample S. The visual tethers may be faded, dimmed or selected via a UI by a user. Typically, the tethers 88 are not required and may be used via a selection by a user or may be omitted as an option from the viewer V. Where used, the tethers may be just visual tethers or may be active links that allow for more information to be provided to a user for a particular cut location and/or slide.
[0109] FIGS. 9 and 10 illustrate that a viewer V can show two large views of slides S.sub.1, S.sub.2 in high magnification and the corresponding cut location marks 110a (“A” and “E”) can be visually enhanced relative to the other cut mark locations. The cut location marks 110a that are related to the current large slide views S.sub.1, S.sub.2 can be shown as bold, with increased intensity and/or in solid line and/or with a different color relative to the other cut location marks 110 which can also be shown in broken lines. Here, each relevant cut mark location 110a is shown in a brighter, increased intensity and in a different solid line color on the image M.).
PNG
media_image5.png
607
497
media_image5.png
Greyscale
(in fig 9, t1 and t2 are different views of the different portions of a cell. They are in a consolidated view (portions of images stitched together in a row). T1 is shown as s1 (can be seen as a patch of the reference image with the red arrows) while t2 is shown as s2.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin’s disclosure of tissue image patching, with Hall’s teaching of spatial location of cell patches, in order to automatically generate physical or digital markings on the macroscopic images ([0006]). Molin and Hall do not explicitly disclose or teach and wherein: the machine learning algorithm is trained, using previously labeled volumetric imaging data of benign and malignant tissue, to identify features associated with malignant tissue.
Molin implicitly discloses wherein: the machine learning algorithm is trained, using previously labeled volumetric imaging data of benign and malignant tissue, to identify features associated with malignant tissue ([0189] FIG. 17 is a flow chart of actions/operations that can occur to reclassify patches 25 based on user-input and can optionally retrain a machine learning model for the classification according to embodiments of the present invention.).
However, in a similar field of endeavor of endeavor class-aware adversarial pulmonary nodule synthesis, Yang explicitly teaches, in better detail, and wherein: the machine learning algorithm is trained, using previously labeled volumetric imaging data of benign and malignant tissue, to identify features associated with malignant tissue, ([0023] FIG. 1 shows an exemplary computed tomography (CT) medical image 100 of a patient showing pulmonary nodule 102. Pulmonary nodule 102 in image 100 is a malignant pulmonary nodule. Image 100 may be part of a training dataset for training a machine learning network for performing an image analysis task such as, e.g., classifying a nodule as benign or malignant.), wherein said machine learning algorithm distinguishes between benign and malignant tissue features ([0038] At step 404, the target nodule is classified using a trained machine learning network trained based on a synthesized medical image patch. The target nodule may be classified as one of malignant or benign. In one embodiment, the trained machine learning network is a deep 3D CNN pre-trained for natural video classification, however any suitable machine learning network may be employed. In one embodiment, the synthesized medical image patch is generated according to the steps of method 200 of FIG. 2.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin and Hall’s disclosure of spatial location of cell patches, with Yang’s teaching of benign and malignant classification, by using a synthesized medical image patch that includes the unmasked portion of the initial medical image patch and a synthesized nodule replacing the masked portion of the initial medical image patch in order to reduce processing time (abstract and [0003]). Molin, Hall and Yang do not explicitly disclose each patch from the subset of patches includes a suspected abnormality, and the suspected abnormality is identified by the machine learning algorithm as having features associated with malignant tissue.
each patch from the subset of patches includes a suspected abnormality (abstract: A system and method of optical coherence tomography includes defining a suspect region-of-interest (SROI) for a suspect lesion in a first OCT B-scan image, defining a healthy region-of-interest (HROI) near the suspect lesion in a second OCT B-scan image), and the suspected abnormality is identified by the machine learning algorithm as having features associated with malignant tissue ([0092] The heuristics may include one or multiple phases including, for example, (i) a Training Phase, and (ii) a Test Phase. In the Training Phase, the optical radiomic features and their labels (histology results) are input to the heuristics. In the Test Phase, OCT images of a suspect skin area will be analyzed by the trained heuristics (e.g., machine-learning kernel) with the selected optical radiomic features, e.g., optical radiomic signatures. The system 100 may display the status of the tissue associated with healthy or benign tissue, e.g., “Tissue sample is consistent with healthy tissue” or cancerous or melanoma tissue, e.g., “Tissue sample exhibits characteristics consistent with melanoma”.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to combine the known method of benign or malignant tissue determination using a computed tomography medical image of Molin, Hall and Yang’s disclosure with the known techniques of optical coherence tomography B-scan images as taught by Avanaki, in order to yield predictable results a non-invasive, no contact way of imaging a tissue to generate cross-sectional, ultra-high resolution images, which results in an improved system.
Regarding claims 20, Molin does not disclose but Hall teaches wherein the consolidated view of the set of patches is displayed in a first area of the user interface, the method further comprising (fig. 10, ref t1, t2):
displaying in one or more second areas of the user interface at least one of: a perspective view of the tissue sample, or a two-dimensional view of the tissue sample at a predefined depth (fig. 10, ref s1, s2 (close up view of t1, t2)).
PNG
media_image7.png
299
473
media_image7.png
Greyscale
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin’s disclosure of tissue image patching, with Hall’s teaching of spatial location of cell patches, in order to automatically generate physical or digital markings on the macroscopic images ([0006]).
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Molin (US 20180089496 A1), in view of Hall (US 20150279026 A1), in view of Yang (US 20200402215 A1), in view of Avanaki (US 20200359887 A1), and further in view of Agus (US 20200388028 A1).
Regarding claim 16, Molin, Hall, Yang and Avanaki do not disclose or teach wherein the consolidated view of the subset of patches includes information associated with each patch from the subject of patches, the information including at least one of: a ductal carcinoma in situ (DCIS) score, a confidence value associated with the DCIS score, spatial location information.
In a similar field of endeavor of diagnosing, theragnosing, and classifying types of cancer, Agus teaches wherein the consolidated view of the subset of patches includes information associated with each patch from the subject of patches, the information including at least one of: a ductal carcinoma in situ (DCIS) score ([0073] The network's performance on the DCIS dataset was intriguing. While the network was trained on IDC images, the high accuracy on DCIS images may be explained by a several factors. Biologically, it has been noted that there are morphometric similarities between DCIS and IDC17, thus patterns learned on IDC may apply to DCIS. Another explanation is that the co-occurrence of DCIS and IDC in some of the IDC training images allowed the network to learn patterns for DCIS and IDC simultaneously. An alternative explanation is that the higher accuracy on the DCIS dataset was due to the method of dataset preparation. Images in the DCIS dataset were carefully chosen by pathologists and contain little stroma relative to epithelial tissue. On the other hand, images in the IDC datasets were obtained from a commercial tissue microarray supplier. These cores are large, diverse (containing a mixture of stromal and epithelial tissue), and noisy (exhibiting varying degrees of staining artifact). Thus, biologic or region-selection factors may explain the accuracy on the DCIS dataset.), a confidence value associated with the DCIS score, spatial location information.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine Molin, Hall, Yang and Avanaki’s disclosure of benign and malignant classification using OCT b-scan images, with Agus’s teaching of a ductal carcinoma in situ diagnosis, in order to extract latent information from basic imaging techniques to predict molecular level information that captures the underlying biology of the cancer ([0005]).
Claim(s) 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Molin (US 20180089496 A1), in view of Hall (US 20150279026 A1), in view of Yang (US 20200402215 A1), in view of Avanaki (US 20200359887 A1), and further in view of Ha (US 20200364855 A1).
Regarding claim 24, Molin, Hall, Yang, and Avanaki do not explicitly disclose but in a similar field of endeavor of tissue classification, Ha teaches wherein the B-scan image data relates to ductal carcinoma in situ (DCIS) ([0067] FIG. 5 shows a diagram of exemplary OCT image acquisition and labeling according to an exemplary embodiment of the present disclosure. Each A-line 505 within every OCT B-scan (obtained using image acquisition procedure 510) was manually labeled 515 into four tissue types. Labels were carried out manually, after images 520 were matched with corresponding histology slides 525 and labeled as stroma, adipose, IDC, and DCIS as these are the most common features of breast tissue.), and the suspected abnormality includes one or more features previously identified in tissue affected by ductal carcinoma in situ ([0082] Each A-line was labeled for six tissue types: (i) IDC, (ii) DCIS, (iii) mucinous carcinoma, (iv) Phyllodes sarcoma, (v) stroma, (vi) adipose. Labeling procedures used a custom graphical user interface (“GUI”). Two volumes/patient were labeled for 23 patients, resulting in 37 k B-scans and 29.5 million A-lines.
[0088] binary of differentiation of malignancy from non-malignant tissues yielded five-fold cross-validation Dice scores of 0.85-0.92 as shown in the graphs of FIGS. 12 and 13. The convergence of the binary classification illustrates the plots of the validation sets 1205 and 1305, and the training sets 1210 and 1310.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to combine the known systems of benign and malignant classification using OCT b-scan images, as disclosed by Molin, Hall, Yang and Avanaki, with the known technique of DCIS abnormality detection, as taught by Ha, in order to yield the predictable results of real-time, label-free cross-sectional imaging right in the operating room allows the surgeon to inspect the margins of the specimen immediately and excise more tissue if residual DCIS is spotted.
Response to Arguments
Applicant's arguments filed 07/27/2026 have been fully considered but they are not persuasive.
Regarding the applicant’s argument that prior arts do not disclose or teach the limitations of wherein the consolidated view includes only patches that include a suspected abnormality, the examiner most respectfully disagrees.
The instant application teaches:
[0031] In some embodiments, the data analyzer 115 can identify in the imaging data where areas of anomalous or suspicious data has been detected. For example, the data analyzer 115 can use a suitable annotation tool (e.g., colored marking, arrows, outline, etc.) to flag or mark portions of the imaging data detected to include suspicious features. In some embodiments, the data analyzer 115 can generate a dataset including patches of imaged areas having fixed width that include suspicious areas, and pass this dataset onto the data visualizer 116. In some embodiments, the data analyzer 115 can make in each patch where suspicious features were identified (e.g., a portion of a lesion) that led to each patch being identified as suspicious.
Molin discloses:
[0081] The cut location marks 110 on a respective macroscopic image or model M may be configured as active or inactive objects or links. For active objects or links, a user may select (e.g., click or touch) a particular cut location mark 110 on image or model M on the display 35 and the viewer V can automatically present WSI slides S associated with the selected cut location mark. [0096] Generally stated, embodiments of the invention are directed to user interfaces and associated image processing circuits that provide interactive patch galleries on a display (typically of a viewer) that can allow a user to visually evaluate and/or adjust automated views of ROIs and/or total or parts of WSI images to explore variability and/or reliability in automatically generated visualizations/views.
[0104] Again, each randomly selected patch from this class (block 214) can be sorted by the same defined common sorting criteria, shown as by nucleus and positivity probability (block 234). Sorted patches from each of the three different classes can then be presented in at least one patch gallery in a sorted order (block 240) of probability of positivity.
Avanaki teaches:
[0041] Scanning probe 123 and computing device 103 maybe communicatively connected to each other to provide operation. The operations may include, for example, obtain a first image of a suspect region-of-interest (SROI) for a suspect lesion, obtain a second image of a healthy region-of-interest (HROI) near the suspect lesion, classify the extracted optical properties to generate an issue status including as at least one of melanoma tissue and benign tissue, and display the issue status indicating the at least one of melanoma tissue and benign tissue.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to combine the known system of tissue analysis using whole slide images with an image gallery, as disclosed by Molin, with the known techniques of optical coherence tomography B-scan images and detection of suspicious tissue, as taught by Avanaki, in order to yield predictable results a non-invasive, no contact way of imaging a tissue to generate cross-sectional, ultra-high resolution images, which results in an improved system.
Regarding the argument that prior arts do not disclose the two-dimensional view is from a user-selected depth in the volume of the tissue sample, the applicant most respectfully disagrees.
Molin discloses:
[0092] The term “semi-automated” refers to an image processing system, method, module or circuit that employs user (e.g., pathologist) input to adjust a feature dimension of the image to explore image feature variability generated by automated systems.
[0109] The patch gallery 25g can provide separation zones 25z between adjacent patches 25 and a user interface feature 31 which can reside in one or more of the zones 25z. The user interface feature 31 can be electronically selected and/or moved relative to a first position on the display, in the patch gallery user interface 30, to adjust a feature dimension in the patches 25.
The instant application teaches:
[0041] For example, a user can input a selected depth, and the compute device can display a view of the tissue sample at that selected depth. In some embodiments, as depicted in FIGS. 5 and 7, the user interface can include a plurality of portions that each display a different view of the tissue sample, including portions that display the consolidated view of diagnostically relevant image patches, perspective views of the tissue, or larger 2D scans of the tissue at different depths and/or along different directions.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this invention to associate a adjusting a feature dimension of the image, as taught by Molin to be the same as user can input a selected depth, as disclosed by the instant application.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AHMED A NASHER whose telephone number is (571)272-1885. The examiner can normally be reached Mon - Fri 0800 - 1700.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AHMED A NASHER/Examiner, Art Unit 2675
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights