DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/23/2024 has been made record of and considered by the examiner.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: #908 of FIG. 9. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: process 306 (see [0067]), processes 314 and 316 [see [0068]), process 304 (see [0069]). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claim 11 is objected to because of the following informalities: “A method applying,” is recited, which should be corrected to “A method for applying.” Appropriate correction is required.
Obviousness Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for
reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 12,159,329. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims in the instant application are broader than their corresponding version in U.S. Patent No. 12,159,329, as shown below. Note that the segment-specific transformation of U.S. Patent No. 12,159,329 is understood to “increase a perceptual quality of the plurality of segments” as recited in the instant claims.
U.S. Patent Number 12,159,329
Application Number 18/924,614
Claim 1. A system for color gamut normalization for pathology slides, the system comprising at least a computing device, wherein the at least a computing device is configured to:
receive a whole slide image including a magnification level of the whole slide image; and
generate a plurality of segments associated with the whole slide image as a function of one or more biological tissue type variabilities and the magnification level of the whole slide image, wherein generating the plurality of segments associated with the whole slide image comprises: performing biologically relevant segmentation of the whole slide image according to the magnification level;
apply a segment-specific transformation to each segment of the plurality of segments;
create a user interface data structure, wherein the user interface data structure comprises the plurality of segments; and
display the plurality of segments through a graphical user interface as a function of the user interface data structure.
Claim 1. A system for applying transformations to pathology slides, the system comprising at least a computing device, wherein the computing device is configured to:
receive a whole slide image;
generate a plurality of segments associated with the whole slide image as a function of one or more biological tissue type variabilities;
apply at least a segment-specific transformation to each segment of the plurality of segments, wherein the at least a segment-specific transformation is configured to increase a perceptual quality of the plurality of segments; and
display the plurality of segments through a graphical user interface as a function of a user interface data structure.
Likewise, similar “method” claim 11 in the instant application are also rejected as non-obvious variants of “method” claim 11 in U.S. Patent No. 12,159,329.
Claims 2 and 12 of the present application are anticipated by claim 9 of the patented claims.
Claims 3 and 13 of the present application are anticipated by claims 1 and 5 of the patented claims.
Claims 4 and 14 of the present application are anticipated by claim 9, FIG. 3, of the patented claims.
Claims 5 and 15 of the present application are anticipated by claim 9, FIG. 3, of the patented claims.
Claims 6 and 16 of the present application are anticipated by claim 4 of the patented claims.
Claims 7 and 17 of the present application are anticipated by claims 8 and 10 of the patented claims.
Claims 8 and 18 of the present application are anticipated by claims 1, 5, and 10 of the patented claims.
Claims 9 and 19 of the present application are anticipated by 1, 5, and 10, FIG. 1, of the patented claims.
Claims 10 and 20 of the present application are anticipated by claims 1, 8, and 10, FIG. 1, of the patented claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-5, 7-8, 10-15, 17-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Saltz (US 2020/0388029 A1), in further view of Alemi (US 11,455,753 B1).
Consider claims 1 and 11, Saltz discloses a system/method for applying transformations to pathology slides, the system comprising at least a computing device, wherein the computing device is configured to (FIGs. 16-19):
receive a whole slide image (¶173; “step 531 of workflow 530, a new diagnostic pathology slide(s) is scanned by a whole slide pathology scanner (step 532) that results in one or more digitized whole slide images (533)”);
generate a plurality of segments associated with the whole slide image as a function of one or more biological tissue type variabilities (¶25, 111, 117-118, 122-124; “The necrosis segmentation CNN outputs pixel-wise segmentation results. An example module to implement this task is DeconvNet because it is designed to predict pixel-wise class labels and handle structures and objects at multiple scales (which is more suitable for segmentation than patch-level classification)… the system trains a necrosis segmentation CNN to classify each pixel as inside or outside a necrosis region… If over half of a 50x50 patch intersects with a necrotic region, the patch is classified as non-lymphocyte-infiltrated”; ¶132-134; “The fully unsupervised autoencoder 70 in FIG . 2A first decomposes or segments an input histopathology image patch 71 into foreground (e.g. nuclei) 73 and background (e.g. cytoplasm) 74”; ¶168-169, 202-207);
apply at least a segment-specific transformation to each segment of the plurality of segments, wherein the at least a segment-specific transformation is configured to increase a (¶207; “A color normalization step is applied during the model training phase to account for stain variations … data augmentation is applied … which also perturbs the colors of each patch slightly, to generate multiple versions of the same patch.”); and
display the plurality of segments through a graphical user interface as a function of a user interface data structure (FIGs. 8E, 8G, ¶223-224; “each patch in a WSI is represented as a rectangle and associated with a classification label and the probability value computed by the CNN. This information is stored as a data element (document) in FeatureDB and indexed to speed up queries by the TIL-Map editor to retrieve and display subsets of patches.”; ¶265-266; “core user interface allows a user to view a whole slide tissue image and supports interactive panning of the image and zooming of the image regions of interest. It also can display an overlay on an image of image segmentation results as polygons…”; ¶294-296).
Saltz fails to explicitly disclose wherein the at least a segment-specific transformation is configured to increase a perceptual quality of the plurality of segments.
In related art, Alemi discloses apply at least a segment-specific transformation to each segment of the plurality of segments, wherein the at least a segment-specific transformation is configured to increase a perceptual quality of the plurality of segments (Alemi Col. 10 lines 12-23; “The adjusted image 212 may include an adjusted color, an adjusted amount of a particular stain, an adjusted brightness, an adjusted sharpness, and/or adjusted contrast, among other adjustments. In some examples, indications of one or more regions of the input image 210 to be adjusted may also be received as input and only those one or more regions (e.g., rather than the entire image) may be adjusted in the adjusted image 212. Further inputs utilized by (e.g., specific to) one or more of the modules 202, 204, 206, 208, described in detail in FIGS. 2B through 2E below, may be received and applied to adjust the attributes of the input image 210 accordingly.”; Col. 23 lines 5-67 – Col. 24 lines 1-57).
The goal of Saltz is to “further refine respective tumoral classification and prognosis of tumoral tissue samples (Saltz ¶3).” Saltz further states, “Achieving accurate and robust segmentation results is desirable in cancer diagnostics because of image noise, such as image acquisition artifacts, differences in staining, and variability in nuclear morphology within and across tissue specimens (Saltz ¶8).” Alemi states, “When pathologists review an image of a pathology slide on a microscope, they cannot adjust attributes (e.g., the global or local properties) of that image beyond magnification. With digital pathology, a pathologist may be given tools to alter semantically meaningful, attributes of a digital whole slide image, including one or more stains used to prepare the slide (Alemi Col. 1 lines 25-31).” Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the region-specific image enhancement techniques of Alema into the segmentation/classification methods of Saltz to improve visualization of the classified regions of interest, and therefore provide improved diagnostic classifications with greater efficiency (Saltz ¶3-10; Alemi Col. 1, Col. 4 lines 52-67).
Consider claims 2 and 12, Saltz, as modified by Alemi, discloses the claimed invention wherein the computing device is further configured to determine the at least a segment-specific transformation as a function of the plurality of segments (Saltz ¶99, 117-118, 202-207; Alemi Col. 1, Col. 11 lines 59-67 – Col. 12 lines 1-17).
Consider claims 3 and 13, Saltz, as modified by Alemi, discloses the claimed invention wherein the computing device is communicatively connected to one or more scanners configured to capture the whole slide image from a pathology slide (Saltz FIG. 3B, ¶173, 390-393).
Consider claims 4 and 14, Saltz, as modified by Alemi, discloses the claimed invention wherein: applying the at least a segment-specific transformation to each segment of the plurality of segments comprises generating a processed image as a function of the at least a segment-specific transformation (Saltz ¶207; Alemi Col. 10 lines 12-23; Col. 23 lines 5-67 – Col. 24 lines 1-57); and
the computing device is further configured to upload the processed image to a database (Saltz ¶173, 182).
Consider claims 5 and 15, Saltz, as modified by Alemi, discloses the claimed invention wherein: applying the at least a segment-specific transformation to each segment of the plurality of segments comprises generating a processed image as a function of the at least a segment-specific transformation (Saltz ¶207; Alemi Col. 10 lines 12-23; Col. 23 lines 5-67 – Col. 24 lines 1-57); and
displaying the plurality of segments through the graphical user interface comprises visualizing the processed image (Saltz FIGs. 8E, 8G, ¶223-224, ¶265-266; Alemi Col. 10 lines 12-23; Col. 23 lines 5-67 – Col. 24 lines 1-57).
Consider claims 7 and 17, Saltz, as modified by Alemi, discloses the claimed invention wherein generating the plurality of segments associated with the whole slide image as a function of one or more biological tissue type variabilities comprises generating the plurality of segments associated with the whole slide image using a computer vision model (Saltz ¶25, 111, 117-118, 122-124, 132-134, 168-169, 202-207; Alemi Col. 9 lines 50-60).
Consider claims 8 and 18, Saltz, as modified by Alemi, discloses the claimed invention wherein: the computing device is further configured to determine a current magnification level (Saltz ¶114-116, 224, 266; Alemi Col. 23 lines 40-57); and
generating the plurality of segments associated with the whole slide image using a computer vision model comprises segmenting the whole slide image based on semantic meaning of the current magnification level (Saltz ¶114-116, 250-251).
Consider claims 10 and 20, Saltz, as modified by Alemi, discloses the claimed invention wherein: generating the plurality of segments associated with the whole slide image as a function of the one or more biological tissue type variabilities comprises determining a segment bounding path associated with the plurality of segments (Saltz ¶223-224, 266; Alemi Col. 10 lines 12-23; Col. 23 lines 5-67 – Col. 24 lines 1-57); and
the computing device is further configured to store the segment bounding path (Saltz ¶182, 223-224, 266; Alemi Col. 23 lines 5-67).
Claims 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Saltz, in further view of Alemi, as applied to claims 1-5, 7-8, 10-15, 17-18, and 20 above, and further in view of Sato (US 2015/0153559 A1).
Consider claims 6 and 16, Saltz, as modified by Alemi, fails to explicitly disclose wherein displaying the plurality of segments through the graphical user interface comprises displaying multiple views of a specimen within a Z-stack.
In related art, Sato discloses wherein displaying the plurality of segments through the graphical user interface comprises displaying multiple views of a specimen within a Z-stack (Sato ¶59, 71, 143).
Saltz discloses, “The lym-CNN operates at 100×100 pixels and a 20× magnification. At this size and magnification there is not enough contextual information for distinguishing neutrophils vs. lymphocytes. Therefore a second CNN is used that helps identify regions of necrosis. This CNN operates at a lower magnification (more zoomed out) than 20× (10×, 6.6×, 5× . . . ), because at these resolutions there is more contextual information which helps discriminate between lymphocytes and neutrophils (Saltz ¶250).” Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the z-stack display through the GUI of Sato into the display of Saltz, to predictably yield sequentially viewing images/segments in a z-stack by means of user input such as a mouse scroll (Saltz ¶223-224, 241; Sato ¶59, 71, 143).
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Saltz, in further view of Alemi, as applied to claims 1-5, 7-8, 10-15, 17-18, and 20 above, and further in view of Yip (US 2020/0211189 A1).
Consider claims 9 and 19, Saltz, as modified by Alemi, discloses the claimed invention wherein:
the current magnification level comprises a 400x magnification level (Saltz ¶134, 359, 223-224, 265-266; The specification explains that 400x magnification is the level required to achieve single cell visualization (see [0044]-[0045]), and higher magnification is required to see cell organelles, such as the nucleus. Without explicitly stating the use of a 400x magnification level, Saltz clearly visualizes on the tissue, cell, and organelle levels, and allows the user to further zoom in on regions of interest.).
However, Saltz, as modified by Alemi, fails to explicitly disclose segmenting the whole slide image based on semantic meaning of the current magnification level comprises:
training the computer vision model to segment the whole slide image into regions with different cell types; and
segmenting the whole slide image into regions with different cell types using the trained computer vision model.
In related art, Yip discloses the current magnification level comprises a 400x magnification level (Yip ¶83-87); and
segmenting the whole slide image based on semantic meaning of the current magnification level comprises (Yip ¶62-87):
training the computer vision model to segment the whole slide image into regions with different cell types (Yip ¶62-82); and
segmenting the whole slide image into regions with different cell types using the trained computer vision model (Yip ¶62-82).
Saltz discloses, “The lym-CNN operates at 100×100 pixels and a 20× magnification. At this size and magnification there is not enough contextual information for distinguishing neutrophils vs. lymphocytes. Therefore a second CNN is used that helps identify regions of necrosis. This CNN operates at a lower magnification (more zoomed out) than 20× (10×, 6.6×, 5× . . . ), because at these resolutions there is more contextual information which helps discriminate between lymphocytes and neutrophils (Saltz ¶250).” Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the segmentation based on semantic meaning of the current magnification level of Yip into the segmentation methods of Saltz, as modified by Alemi, to analyze the slides and classify the tissue components by tissue class and cell type (Yip ¶7).
Relevant Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Heller (‘EpiTools: an open-source image analysis toolkit for quantifying epithelial growth dynamics’).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ASHLEY HYTREK whose telephone number is (703)756-4562. The examiner can normally be reached M-F 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Steve Koziol can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ASHLEY HYTREK/ Examiner, Art Unit 2665
/Stephen R Koziol/ Supervisory Patent Examiner, Art Unit 2665
Prosecution Insights