Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The Information Disclosure Statement(s) filed on 16 September 2025 and 05 June 2026, has been considered by the Examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Claims 1, 10 and 15 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claims recite methods and system for analyzing a multi-resolution image. The limitations of:
Claim 1
[… obtaining …] at least one multi-resolution image; [… obtaining …] at least one input that includes at least one natural language query from a user and a plurality of parameters; generating one or more outputs by executing a […] algorithm to analyze the at least one multi-resolution image based on the at least one natural language query, wherein the one or more outputs comprise at least one natural language answer corresponding to the at least one natural language query and wherein the at least one natural language answer is provided for each of the plurality of parameters; and [… outputting …], the at least one multi-resolution image and the one or more outputs.
Claim 15, which is representative of claims 10
[… having …] instructions for generating and [… outputting …] a natural language answer corresponding to a natural language query from a user; […]; and […] perform operations including: [… obtaining …] one or more multi-resolution images and the natural language query; automatically generating, by executing a […] algorithm to analyze the one or more multi-resolution image at a plurality of magnification levels, a natural language answer for the respective magnification levels that correspond to the natural language query; and [… outputting …] the one or more multiresolution images and the natural language answer.
, as drafted, is a system, which under its broadest reasonable interpretation, covers a method of organizing human activity (i.e., managing personal behavior including following rules or instructions) via human interaction with generic computer components. That is, by a human user interacting with a display interface (claims 1 and 10), a computer-readable storage medium, a display interface and one or more processors (claim 15), the claimed invention amounts to managing personal behavior or interaction between people, the Examiner notes as stated in 2106.04(a)(2), “certain activity between a person and a computer… may fall within the "certain methods of organizing human activity" grouping”. For example, via human interaction with a display interface (claims 1 and 10), a computer-readable storage medium, a display interface and one or more processors (claim 15), the claim encompasses a human user providing a digital pathology multi-resolution image, providing user selections and a question and organizing the collected data to provide a result to provide to the human user a result. If a claim limitation, under its broadest reasonable interpretation, covers managing personal behavior or interactions between people but for the recitation of generic computer components, then it falls within the “method of organizing human activity” grouping of abstract ideas. Accordingly, the claim recites an abstract idea.
This judicial exception is not integrated into a practical application. In particular, the claim recites the additional elements of various units, which implements the abstract idea. The display interface (claims 1 and 10), a computer-readable storage medium, a display interface and one or more processors (claim 15) are recited at a high-level of generality (i.e., a general-purpose computers/ computer components implementing generic computer functions; see Applicant's Specification Fig. 2, paragraph [0049]) such that it amounts no more than mere instructions to apply the exception using generic computer components. Accordingly, these additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim is directed to an abstract idea.
The claim recites the additional elements of “receiving…”, “executing a machine learning algorithm”, “displaying…” and “storing…”. The “receiving…” steps are recited at a high-level of generality (i.e., as a general means of receiving/transmitting data) and amounts to the mere transmission and/or receipt of data, which is a form of extra-solution activity. The “executing a machine learning algorithm” steps are recited at a high-level of generality (i.e., using a generic off-the shelf model) and amounts to generally linking the abstract idea to a particular technological environment. The “displaying…” is recited at a high-level of generality (i.e., as a general displaying data) and amounts to merely linking of the abstract idea to particular technological environment. The “storing…” is recited at a high-level of generality (i.e., as a general means of storing data) and amounts to the mere storage of data, which is a form of extra-solution activity. Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application. The claim is directed to an abstract idea.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of a display interface (claims 1 and 10), a computer-readable storage medium, a display interface and one or more processors (claim 15), to perform the noted steps amounts to no more than mere instructions to apply the exception using generic hardware components. Mere instructions to apply an exception using a generic hardware component cannot provide an inventive concept (“significantly more”).
Also as discussed above with respect to integration of the abstract idea into a practical application, the additional elements of “receiving…”, “executing a machine learning algorithm”, “displaying…” and “storing…” were considered extra-solution activity and/or generally linking to a particular technological environment. The “receiving…” has been re-evaluated under the "significantly more" analysis and determined to amount to be well-understood, routine, and conventional elements/functions. As described in MPEP 2106.0S(d)(II)(i) "Receiving or transmitting data over a network" is well-understood, routine, and conventional. The “executing a machine learning algorithm” has been re-evaluated under the "significantly more" analysis and determined to amount to be well-understood, routine, and conventional elements/functions. As described in Iftikhar (20250245827): Figs. 1-4, paragraphs [0005]-[0008]; Yip (20250245827): paragraphs [0161]-[0164]; Locke (20210073984): paragraphs [0007]-[0009]; use of a machine learning model is well-understood, routine and conventional. The “displaying…” has been re-evaluated under the “significantly more” analysis and determined to amount to be well-understood, routine, and conventional elements/functions. As described in Iftikhar (20250245827): Figs. 1-4, paragraphs [0036]; Yip (20250245827): paragraphs [0061]-[0064], [0191]; Locke (20210073984): paragraphs [0044], [0064]; displaying data is well-understood, routine, and conventional elements/functions. The “storing…” has been re-evaluated under the “significantly more” analysis and determined to amount to be well-understood, routine, and conventional elements/functions. As described in MPEP 2106.05(d)(II)(iv) “Storing and retrieving information in memory” is well-understood, routine, and conventional. Well-understood, routine, and conventional elements/functions cannot provide “significantly more.” As such the claim is not patent eligible.
Claims 2-9, 11-14 and 16-20 are similarly rejected because either further define the abstract idea and/or do not further limit the claim to a practical application or provide as inventive concept such that the claims are subject matter eligible.
Claims 2-3, 8, 11 and 16-17 recite the labels describing data, however the labels of data are not additional elements capable of providing an additional element and/or significantly more.
Claims 4-6, 12 and 19-20 further describe the displayed data, however display of data on a user interface was already considered above and is incorporated herein.
Claim 7, 9, 14 and 18 further describe providing data to a machine learning algorithm for analysis, however use of a machine learning algorithm was already considered above and is incorporated herein.
Claim 13 recites “has been trained”, and is not actually an active training step (i.e., an additional element), the claim amounts to describing the labels used to create the model, the labels of data used are not additional elements, while the claim may recite a machine learning model (i.e., an additional element), use of a machine learning model was already considered above and is incorporated herein.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-3, 5-7 and 9-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Pub. No. 20250245827 (hereafter “Iftikhar”), in view of U.S. Patent Pub. No. 20200211189 (hereafter “Yip”).
Regarding claim 1, Iftikhar teaches a method for analyzing a multi-resolution image (Iftikhar: Figs. 1-5, paragraphs [0005]-[0006], “one or more computing devices, methods, and non-transitory computer-readable media that may be utilized to train one or more machine-learning models to generate a prediction of one or more tile-level class labels for a histopathology image based on a slide-level class label preassigned to the histopathology image… the one or more computing devices may then input the number of image tiles at different magnifications into respective machine-learning models (e.g., an ensemble of machine-learning models) each trained to generate a prediction of a tile-level class label for the respective image tiles at different magnifications utilizing the image tile and the slide-level class label preassigned to the histopathology image”), comprising:
receiving at least one multi-resolution image (Iftikhar: Figs. 1-5, paragraphs [0007]-[0008], “one or more computing devices may access a histopathology image, in which the histopathology image comprises a slide-level class label… extract, based on the histopathology image, a plurality of regions of pixels of the histopathology image at a plurality of magnifications… for each of the extracted plurality of regions of pixels, the one or more computing devices may input the region of pixels into a machine-learning model”, paragraph [0027], “The image scanner 124 may capture the digital image at multiple magnifications (e.g., utilizing a 2× objective, a 5× objective, a 10× objective, a 20× objective, a 40× objective, a 100× objective, a 200× objective, a 500× objective, and so forth)”, paragraph [0037], “a user (e.g., histopathologist or clinician) may access a user device 130 that is in communication with the whole slide image processing system 110 and provide a query image for analysis”);
receiving at least one input that includes at least one natural language query from a user and a [… user selection …] (Iftikhar: Figs. 1-5, 9, paragraph [0027], “Manipulation of the image may be used to capture a selected portion of the sample at the desired range of magnifications. Image scanner 124 may further capture annotations and/or morphometrics identified by a human operator”, paragraph [0036], “an output generating module 114 of the whole slide image processing system 110 may generate output corresponding to result tile and result WSI datasets based on user request… the output may include a variety of visualizations, interactive graphics, and reports based upon the type of request and the type of data that is available… The output may be based on existence of and access to the appropriate data, so the output generating module 116 may be empowered to access metadata and anonymized patient information as needed”, paragraphs [0068]-[0074], “the AI architecture 902 may include machine learning (ML) models 904, natural language processing (NLP) models 906, expert systems 908, computer-based vision models 910, speech recognition models 912, planning models 914, and robotics models 916… the NLP models 906 may include any algorithms or functions that may be suitable for automatically manipulating natural language, such as speech and/or text. For example, in some embodiments, the NLP models 906 may include content extraction models 924, classification models 926, machine translation models 928, question answering (QA) models 930, and text generation models 932. In certain embodiments, the content extraction models 924 may include a means for extracting text or images from electronic documents (e.g., webpages, text editor documents, and so forth) to be utilized… The QA models 930 may include any algorithms or functions that may be suitable for automatically answering questions posed by humans in, for example, a natural language, such as that performed by voice-controlled personal assistant devices. The text generation models 932 may include any algorithms or functions that may be suitable for automatically generating natural language texts”. The Examiner notes a human operator may pose a natural language question and user requests (i.e., selections), which teaches what is required under the broadest reasonable interpretation);
generating one or more outputs by executing a machine learning algorithm to analyze the at least one multi-resolution image based on the at least one natural language query (Iftikhar: Figs. 1-5, paragraph [0008], “for each of the extracted plurality of regions of pixels, the one or more computing devices may input the region of pixels into a machine-learning model trained to generate a prediction of a class label for the region of pixels based on the region of pixels and the slide-level class label, and to output, by the machine-learning model, the prediction of the class label for the region of pixels… For example, in some embodiments, the prediction of the one or more tile-level class labels may include an identification of one or more biomarkers associated with tissues or cells included within the histopathology image”, paragraphs [0043]-[0044], “the machine-learning models 206A, 206B, 206C, and 206D may each include a convolutional neural network (CNN) or a deep neural network (DNN)”. The Examiner notes that “to analyze the at least one multi-resolution image based on the at least one natural language query” is an intended use of the generation of one or more outputs that is not required to occur. This feature has been fully considered by the Examiner; however, the limitation does not provide patentable distinction over the cited prior art because it is an intended use or result of the generation of one or more outputs),
wherein the one or more outputs comprise at least one natural language answer corresponding to the at least one natural language query and wherein the at least one natural language answer is provided for each of the plurality of parameters (Iftikhar: Figs. 1-5, paragraph [0008], “For example, in some embodiments, the prediction of the one or more tile-level class labels may include an identification of one or more biomarkers associated with tissues or cells included within the histopathology image”, paragraphs [0043]-[0044], “the machine-learning models 206A, 206B, 206C, and 206D may each include a convolutional neural network (CNN) or a deep neural network (DNN)… the machine-learning model 206A may be trained to generate a tile-level class label prediction 208A for the image tile extracted and downsampled to N× magnification utilizing the slide-level class label associated with the histopathology image 202. Similarly, in certain embodiments, the machine-learning model 206B may be trained to generate a tile-level class label prediction 208B for the image tile extracted and downsampled to M× magnification utilizing the slide-level class label associated with the histopathology image 202. The machine-learning model 206C may be trained to generate a tile-level class label prediction 208C for the image tile extracted and downsampled to Y× magnification utilizing the slide-level class label associated with the histopathology image 202. Lastly, the machine-learning model 206D may be trained to generate a tile-level class label prediction 208D for the image tile extracted and downsampled to Z× magnification utilizing the slide-level class label with the histopathology image 202”, paragraphs [0068]-[0074], “the AI architecture 902 may include machine learning (ML) models 904, natural language processing (NLP) models 906, expert systems 908, computer-based vision models 910, speech recognition models 912, planning models 914, and robotics models 916… the NLP models 906 may include any algorithms or functions that may be suitable for automatically manipulating natural language, such as speech and/or text. For example, in some embodiments, the NLP models 906 may include content extraction models 924, classification models 926, machine translation models 928, question answering (QA) models 930, and text generation models 932. In certain embodiments, the content extraction models 924 may include a means for extracting text or images from electronic documents (e.g., webpages, text editor documents, and so forth) to be utilized… The QA models 930 may include any algorithms or functions that may be suitable for automatically answering questions posed by humans in, for example, a natural language, such as that performed by voice-controlled personal assistant devices. The text generation models 932 may include any algorithms or functions that may be suitable for automatically generating natural language texts”); and
displaying, via a display interface, […] the one or more outputs (Iftikhar: Figs. 1-5, paragraphs [0008]-[0010], “output, by the machine-learning model, the prediction of the class label for the region of pixels… the one or more computing devices may generate a report based on the prediction of the one or more tile-level class labels for the histopathology image. the one or more computing devices may cause a human machine interface (HMI) associated with a pathologist or a clinician to display the report”, paragraph [0036], “an output generating module 114 of the whole slide image processing system 110 may generate output corresponding to result tile and result WSI datasets based on user request… the output may include a variety of visualizations, interactive graphics, and reports based upon the type of request and the type of data that is available”).
Iftikhar may not explicitly teach (underlined below for clarity):
receiving at least one input that includes at least one natural language query from a user and a plurality of parameters;
displaying, via a display interface, the at least one multi-resolution image and the one or more outputs.
Yip teaches receiving at least one input that includes at least one natural language query from a user and a plurality of parameters (Yip: paragraph [0061], “the tissue class overlay map displays the probabilities for each grid tile for the tissue class selected by the user”, paragraph [0085], “user-selected resolution criteria to select a layer with optimal resolution for analysis”, paragraph [0142], “dimensions selected by the user, and may contain weight values selected by the user”, paragraph [0151], “Each factor may be selected by the user”);
displaying, via a display interface, the at least one multi-resolution image and the one or more outputs (Yip: Figs. 3-5, 9, paragraphs [0010]-[0011], “creating an overlay map on a digital image of a slide comprises: receiving the digital image; separating the digital image into a plurality of tiles; and identifying the majority class of tissue visible within each tile in the plurality of tiles, based on a multi-tile analysis”, paragraph [0014], “determining the predicted content for each tile using a classification model configured as a multi-resolution fully convolutional network, the multi-resolution fully convolution network configured to perform classification on digital images of different zoom levels”, paragraph [0025], “generating a digital overlay drawing for the digital image, wherein the digital overlay drawing includes the digital image; and displaying the digital overlay drawing”, paragraph [0062], “The overlay map generator may display the digital overlays as transparent or opaque layers that cover the slide image, aligned such that the slide location shown in the overlay and the slide image are in the same location on the display. The overlay map may have varying degrees of transparency. The degree of transparency may be adjustable by the user. The overlay map generator may report the percentage of the labeled tiles that are associated with each tissue class label, ratios of the number of tiles classified under each tissue class, the total area of all grid tiles classified as a single tissue class, and ratios of the areas of tiles classified under each tissue class.”, paragraph [0084], “each digital image file received by the digital tissue segmenter 201 contains multiple versions of the same image content, and each version has a different resolution. The file stores these copies in stacked layers, arranged by resolution such that the highest resolution image containing the greatest number of bytes is the bottom layer”. Also see, paragraphs [0083]-[0091]).
One of ordinary skill in the art before the effective filing date would have found it obvious to include receiving selection of a plurality of parameters and displaying the multi-resolution image along with outputs from a multi-resolution machine learning model as taught by Yip within the use of a multi-resolution machine learning model with natural language functionality as taught by Iftikhar with the motivation of “reduces computational redundancy and results in greater processing efficiency” (Yip: paragraph [0058]).
Regarding claim 2, Iftikhar and Yip teach the limitations of claim 1, and further teach wherein the plurality of parameters are selected from the group consisting of a region of interest (ROI), one or more magnification levels, additional samples or images, patient information, or an information tier (Iftikhar: Figs. 1-5, 9, paragraphs [0005]-[0006], “a number of different magnifications (e.g., ranging from low-magnification, medium-magnification, and up to high-magnification)… bounding geometry identifying a specific region”, paragraph [0010], “the one or more computing devices may then access a second histopathology image, input the second histopathology image into the trained one or more machine-learning models to generate a prediction of one or more tile-level class labels for the second histopathology image, and output, by the one or more machine-learning models, the prediction of the one or more tile-level class labels for the second histopathology image”, paragraph [0036], “an output generating module 114 of the whole slide image processing system 110 may generate output corresponding to result tile and result WSI datasets based on user request… the output may include a variety of visualizations, interactive graphics, and reports based upon the type of request and the type of data that is available… The output may be based on existence of and access to the appropriate data, so the output generating module 116 may be empowered to access metadata and anonymized patient information as needed”, paragraph [0040], “histopathologist or other expert human annotator to further annotate the low-level features (e.g., assign a class label or bounding geometry identifying a specific region”. Also see, paragraph [0042]).
The motivation to combine is the same as in claim 1, incorporated herein.
Regarding claim 3, Iftikhar and Yip teach the limitations of claim 2, and further teach wherein the plurality of parameters include a ROI and a plurality of magnification levels (Iftikhar: Figs. 1-5, 9, paragraphs [0005]-[0006], “a number of different magnifications (e.g., ranging from low-magnification, medium-magnification, and up to high-magnification)… bounding geometry identifying a specific region”, paragraph [0036], “an output generating module 114 of the whole slide image processing system 110 may generate output corresponding to result tile and result WSI datasets based on user request… the output may include a variety of visualizations, interactive graphics, and reports based upon the type of request and the type of data that is available”, paragraph [0040], “histopathologist or other expert human annotator to further annotate the low-level features (e.g., assign a class label or bounding geometry identifying a specific region”. Also see, paragraph [0042]. Additionally, the Examiner notes that “ROI” and “a plurality of magnification levels” are recited in the alternative in the parent claim (i.e., claim 2), the prior art need not disclose every alternative to be encompassed by the claim limitation).
The motivation to combine is the same as in claim 1, incorporated herein.
Regarding claim 5, Iftikhar and Yip teach the limitations of claim 1, and further teach wherein the one or more outputs have a visual indicator for each of the plurality of parameters (Yip: Fig. 3, paragraphs [0061]-[0062], “The overlay map generator and metric calculator may retrieve the stored 3-dimensional probability data array from the tissue class locator, and convert it into an overlay map that displays the assigned tissue class label for each tile. The assigned tissue class for each tile may be displayed as a transparent color that is unique for each tissue class… a tumor probability overlay heatmap map”, paragraph [0112], “displaying a grid-based digital overlay map in which each tissue class is represented by a unique color”, paragraph [0177], “an exemplary overlay may represent individual lymphocytes by coloring them red to distinguish between clusters of lymphocytes and individual lymphocytes infiltrating other tissues. In one example, individual lymphocytes detected within tumor tiles represent tumor-infiltrating lymphocytes which would be red dots within dark blue or green tiles”).
The motivation to combine is the same as in claim 1, incorporated herein.
Regarding claim 6, Iftikhar and Yip teach the limitations of claim 1, and further teach wherein the visual indicator is a change in color (Yip: Fig. 3, paragraphs [0061]-[0062], “The overlay map generator and metric calculator may retrieve the stored 3-dimensional probability data array from the tissue class locator, and convert it into an overlay map that displays the assigned tissue class label for each tile. The assigned tissue class for each tile may be displayed as a transparent color that is unique for each tissue class… a tumor probability overlay heatmap map”, paragraph [0112], “displaying a grid-based digital overlay map in which each tissue class is represented by a unique color”, paragraph [0177], “an exemplary overlay may represent individual lymphocytes by coloring them red to distinguish between clusters of lymphocytes and individual lymphocytes infiltrating other tissues. In one example, individual lymphocytes detected within tumor tiles represent tumor-infiltrating lymphocytes which would be red dots within dark blue or green tiles”).
The motivation to combine is the same as in claim 1, incorporated herein.
Regarding claim 7, Iftikhar and Yip teach the limitations of claim 1, and further teach wherein the at least one input includes a language parameter, and wherein the machine learning algorithm adjusts the natural language of the at least one natural language answer to correspond to the language parameter (Iftikhar: Figs. 1-5, paragraph [0048], “the one or more tile-level class label predictions 214 for the histopathology image 202 may be utilized in one or more downstream tasks… the report may include a clinical report that may be associated with one or more cancer patients to be provided and displayed, for example, to a histopathologist or a clinician (e.g., oncologist) for purposes of research and/or the diagnosis, prognosis, and treatment of the one or more patients. In another embodiment, the report may include an interpretability and/or explainability report that may be associated with the machine-learning models 206A, 206B, 206C, and 206D to be provided and displayed, for example, to one or more data scientists or developers for purposes of ascertaining and elucidating the prediction and decision-making behaviors of the machine-learning models 206A, 206B, 206C, and 206D”, paragraphs [0068]-[0074], “the AI architecture 902 may include machine learning (ML) models 904, natural language processing (NLP) models 906, expert systems 908, computer-based vision models 910, speech recognition models 912, planning models 914, and robotics models 916… the NLP models 906 may include any algorithms or functions that may be suitable for automatically manipulating natural language, such as speech and/or text. For example, in some embodiments, the NLP models 906 may include content extraction models 924, classification models 926, machine translation models 928, question answering (QA) models 930, and text generation models 932. In certain embodiments, the content extraction models 924 may include a means for extracting text or images from electronic documents (e.g., webpages, text editor documents, and so forth) to be utilized… The QA models 930 may include any algorithms or functions that may be suitable for automatically answering questions posed by humans in, for example, a natural language, such as that performed by voice-controlled personal assistant devices. The text generation models 932 may include any algorithms or functions that may be suitable for automatically generating natural language texts”. The Examiner interprets the user is able to specify the type of report output, which teaches what is required under the broadest reasonable interpretation).
The motivation to combine is the same as in claim 1, incorporated herein.
Regarding claim 9, Iftikhar and Yip teach the limitations of claim 1, and further teach wherein the machine learning algorithm used to analyze the at least one multi-resolution image is provided one or more machine learning data sets that includes at least one of an analysis of a second multi-resolution image, a patient history, a family history, or community health data (Iftikhar: Figs. 1-5, paragraph [0010], “the one or more computing devices may then access a second histopathology image, input the second histopathology image into the trained one or more machine-learning models to generate a prediction of one or more tile-level class labels for the second histopathology image, and output, by the one or more machine-learning models, the prediction… the one or more computing devices may generate a report based on the prediction of the one or more tile-level class labels for the histopathology image. In one embodiment, the one or more computing devices may cause a human machine interface (HMI) associated with a pathologist or a clinician to display the report”, paragraph [0036], “The output may be based on existence of and access to the appropriate data, so the output generating module 116 may be empowered to access metadata and anonymized patient information as needed. As with the other modules of the whole slide image processing system 110, the output generating module 114 may be updated and improved in a modular fashion, so that new output features may be provided to users without requiring significant downtime”).
The motivation to combine is the same as in claim 1, incorporated herein.
Regarding claim 10, Iftikhar teaches a method for analyzing a multi-resolution image (Iftikhar: Figs. 1-5, paragraphs [0005]-[0006], “one or more computing devices, methods, and non-transitory computer-readable media that may be utilized to train one or more machine-learning models to generate a prediction of one or more tile-level class labels for a histopathology image based on a slide-level class label preassigned to the histopathology image… the one or more computing devices may then input the number of image tiles at different magnifications into respective machine-learning models (e.g., an ensemble of machine-learning models) each trained to generate a prediction of a tile-level class label for the respective image tiles at different magnifications utilizing the image tile and the slide-level class label preassigned to the histopathology image”) comprising:
receiving at least one multi-resolution image (Iftikhar: Figs. 1-5, paragraphs [0007]-[0008], “one or more computing devices may access a histopathology image, in which the histopathology image comprises a slide-level class label… extract, based on the histopathology image, a plurality of regions of pixels of the histopathology image at a plurality of magnifications… for each of the extracted plurality of regions of pixels, the one or more computing devices may input the region of pixels into a machine-learning model”, paragraph [0027], “The image scanner 124 may capture the digital image at multiple magnifications (e.g., utilizing a 2× objective, a 5× objective, a 10× objective, a 20× objective, a 40× objective, a 100× objective, a 200× objective, a 500× objective, and so forth)”, paragraph [0037], “a user (e.g., histopathologist or clinician) may access a user device 130 that is in communication with the whole slide image processing system 110 and provide a query image for analysis”);
receiving at least one input that includes at least one natural language query from a user (Iftikhar: Figs. 1-5, 9, paragraph [0027], “Manipulation of the image may be used to capture a selected portion of the sample at the desired range of magnifications. Image scanner 124 may further capture annotations and/or morphometrics identified by a human operator”, paragraph [0036], “an output generating module 114 of the whole slide image processing system 110 may generate output corresponding to result tile and result WSI datasets based on user request”, paragraphs [0068]-[0074], “the AI architecture 902 may include machine learning (ML) models 904, natural language processing (NLP) models 906, expert systems 908, computer-based vision models 910, speech recognition models 912, planning models 914, and robotics models 916… the NLP models 906 may include any algorithms or functions that may be suitable for automatically manipulating natural language, such as speech and/or text. For example, in some embodiments, the NLP models 906 may include content extraction models 924, classification models 926, machine translation models 928, question answering (QA) models 930, and text generation models 932. In certain embodiments, the content extraction models 924 may include a means for extracting text or images from electronic documents (e.g., webpages, text editor documents, and so forth) to be utilized… The QA models 930 may include any algorithms or functions that may be suitable for automatically answering questions posed by humans in, for example, a natural language, such as that performed by voice-controlled personal assistant devices. The text generation models 932 may include any algorithms or functions that may be suitable for automatically generating natural language texts”. The Examiner notes a human operator may pose a natural language question, which teaches what is required under the broadest reasonable interpretation);
generating one or more outputs, by executing a machine learning algorithm to analyze the at least one multi-resolution image at a plurality of magnification levels based on the at least one natural language query (Iftikhar: Figs. 1-5, paragraph [0008], “for each of the extracted plurality of regions of pixels, the one or more computing devices may input the region of pixels into a machine-learning model trained to generate a prediction of a class label for the region of pixels based on the region of pixels and the slide-level class label, and to output, by the machine-learning model, the prediction of the class label for the region of pixels… For example, in some embodiments, the prediction of the one or more tile-level class labels may include an identification of one or more biomarkers associated with tissues or cells included within the histopathology image”, paragraphs [0043]-[0044], “the machine-learning models 206A, 206B, 206C, and 206D may each include a convolutional neural network (CNN) or a deep neural network (DNN)”. The Examiner notes that “to analyze the at least one multi-resolution image at a plurality of magnification levels based on the at least one natural language query” is an intended use of the generation of one or more outputs that is not required to occur. This feature has been fully considered by the Examiner; however, the limitation does not provide patentable distinction over the cited prior art because it is an intended use or result of the generation of one or more outputs),
wherein the one or more outputs comprises a plurality of natural language answers that correspond to the at least one natural language query, and wherein each answer of the plurality of natural language answers is specific to a respective level of the plurality of magnification levels (Iftikhar: Figs. 1-5, paragraph [0008], “For example, in some embodiments, the prediction of the one or more tile-level class labels may include an identification of one or more biomarkers associated with tissues or cells included within the histopathology image”, paragraphs [0043]-[0044], “the machine-learning models 206A, 206B, 206C, and 206D may each include a convolutional neural network (CNN) or a deep neural network (DNN)… the machine-learning model 206A may be trained to generate a tile-level class label prediction 208A for the image tile extracted and downsampled to N× magnification utilizing the slide-level class label associated with the histopathology image 202. Similarly, in certain embodiments, the machine-learning model 206B may be trained to generate a tile-level class label prediction 208B for the image tile extracted and downsampled to M× magnification utilizing the slide-level class label associated with the histopathology image 202. The machine-learning model 206C may be trained to generate a tile-level class label prediction 208C for the image tile extracted and downsampled to Y× magnification utilizing the slide-level class label associated with the histopathology image 202. Lastly, the machine-learning model 206D may be trained to generate a tile-level class label prediction 208D for the image tile extracted and downsampled to Z× magnification utilizing the slide-level class label with the histopathology image 202”, paragraphs [0068]-[0074], “the AI architecture 902 may include machine learning (ML) models 904, natural language processing (NLP) models 906, expert systems 908, computer-based vision models 910, speech recognition models 912, planning models 914, and robotics models 916… the NLP models 906 may include any algorithms or functions that may be suitable for automatically manipulating natural language, such as speech and/or text. For example, in some embodiments, the NLP models 906 may include content extraction models 924, classification models 926, machine translation models 928, question answering (QA) models 930, and text generation models 932. In certain embodiments, the content extraction models 924 may include a means for extracting text or images from electronic documents (e.g., webpages, text editor documents, and so forth) to be utilized… The QA models 930 may include any algorithms or functions that may be suitable for automatically answering questions posed by humans in, for example, a natural language, such as that performed by voice-controlled personal assistant devices. The text generation models 932 may include any algorithms or functions that may be suitable for automatically generating natural language texts”); and
displaying, via a display interface, the at least one multi-resolution image and the one or more outputs (Iftikhar: Figs. 1-5, paragraphs [0008]-[0010], “output, by the machine-learning model, the prediction of the class label for the region of pixels… the one or more computing devices may generate a report based on the prediction of the one or more tile-level class labels for the histopathology image. the one or more computing devices may cause a human machine interface (HMI) associated with a pathologist or a clinician to display the report”, paragraph [0036], “an output generating module 114 of the whole slide image processing system 110 may generate output corresponding to result tile and result WSI datasets based on user request… the output may include a variety of visualizations, interactive graphics, and reports based upon the type of request and the type of data that is available”).
Iftikhar may not explicitly teach (underlined below for clarity):
displaying, via a display interface, the at least one multi-resolution image and the one or more outputs.
Yip teaches displaying, via a display interface, the at least one multi-resolution image and the one or more outputs (Yip: Figs. 3-5, 9, paragraphs [0010]-[0011], “creating an overlay map on a digital image of a slide comprises: receiving the digital image; separating the digital image into a plurality of tiles; and identifying the majority class of tissue visible within each tile in the plurality of tiles, based on a multi-tile analysis”, paragraph [0014], “determining the predicted content for each tile using a classification model configured as a multi-resolution fully convolutional network, the multi-resolution fully convolution network configured to perform classification on digital images of different zoom levels”, paragraph [0025], “generating a digital overlay drawing for the digital image, wherein the digital overlay drawing includes the digital image; and displaying the digital overlay drawing”, paragraph [0062], “The overlay map generator may display the digital overlays as transparent or opaque layers that cover the slide image, aligned such that the slide location shown in the overlay and the slide image are in the same location on the display. The overlay map may have varying degrees of transparency. The degree of transparency may be adjustable by the user. The overlay map generator may report the percentage of the labeled tiles that are associated with each tissue class label, ratios of the number of tiles classified under each tissue class, the total area of all grid tiles classified as a single tissue class, and ratios of the areas of tiles classified under each tissue class.”, paragraph [0084], “each digital image file received by the digital tissue segmenter 201 contains multiple versions of the same image content, and each version has a different resolution. The file stores these copies in stacked layers, arranged by resolution such that the highest resolution image containing the greatest number of bytes is the bottom layer”. Also see, paragraphs [0083]-[0091]).
One of ordinary skill in the art before the effective filing date would have found it obvious to include displaying the multi-resolution image along with outputs from a multi-resolution machine learning model as taught by Yip within the use of a multi-resolution machine learning model with natural language functionality as taught by Iftikhar with the motivation of “reduces computational redundancy and results in greater processing efficiency” (Yip: paragraph [0058]).
Regarding claim 11, Iftikhar and Yip teach the limitations of claim 10, and further teach wherein the at least one input includes a region of interest (Iftikhar: Figs. 1-5, 9, paragraph [0006], “bounding geometry identifying a specific region”, paragraph [0036], “an output generating module 114 of the whole slide image processing system 110 may generate output corresponding to result tile and result WSI datasets based on user request… the output may include a variety of visualizations, interactive graphics, and reports based upon the type of request and the type of data that is available”, paragraph [0040], “histopathologist or other expert human annotator to further annotate the low-level features (e.g., assign a class label or bounding geometry identifying a specific region”).
The motivation to combine is the same as in claim 10, incorporated herein.
Regarding claim 12, Iftikhar and Yip teach the limitations of claim 10, and further teach wherein the one or more outputs have a visual indicator for the respective magnification levels (Yip: Fig. 3, paragraphs [0061]-[0062], “The overlay map generator and metric calculator may retrieve the stored 3-dimensional probability data array from the tissue class locator, and convert it into an overlay map that displays the assigned tissue class label for each tile. The assigned tissue class for each tile may be displayed as a transparent color that is unique for each tissue class… a tumor probability overlay heatmap map”, paragraph [0112], “displaying a grid-based digital overlay map in which each tissue class is represented by a unique color”, paragraph [0177], “an exemplary overlay may represent individual lymphocytes by coloring them red to distinguish between clusters of lymphocytes and individual lymphocytes infiltrating other tissues. In one example, individual lymphocytes detected within tumor tiles represent tumor-infiltrating lymphocytes which would be red dots within dark blue or green tiles”).
The motivation to combine is the same as in claim 10, incorporated herein.
Regarding claim 13, Iftikhar and Yip teach the limitations of claim 10, and further teach wherein the machine learning algorithm has been trained using different data sets for the respective magnification levels (Iftikhar: Figs. 1-5, 9, paragraph [0034], “Training the tile embedding network using specialized or customized sets of images may allow the tile embedding network to identify finer differences between tiles which may result in more detailed and accurate distances between tiles in the feature embedding space at the cost of additional time to acquire the images and the computational and economic cost of training multiple tile generating networks for use by the tile embedding module 112”, paragraphs [0044]-[0047], “training the machine-learning models (e.g., an ensemble of machine-learning models) to predict tile-level class labels at different magnifications, once trained, the machine-learning models (e.g., an ensemble of machine-learning models) may be better suited for predicting tile-level class labels for features of histopathology images at different magnifications and/or resolutions (e.g., similar to the manner in which a histopathologist would analyze and classify features of histopathology images)”. The Examiner interprets each model is trained with a set at the specific magnification to be analyzed).
The motivation to combine is the same as in claim 10, incorporated herein.
Regarding claim 14, Iftikhar and Yip teach the limitations of claim 10, and further teach wherein the machine learning algorithm used to analyze the at least one multi-resolution image at a plurality of magnification levels is provided one or more machine learning data sets that includes at least one of an analysis of a second multi-resolution image, a patient history, a family history, or community health data (Iftikhar: Figs. 1-5, paragraph [0010], “the one or more computing devices may then access a second histopathology image, input the second histopathology image into the trained one or more machine-learning models to generate a prediction of one or more tile-level class labels for the second histopathology image, and output, by the one or more machine-learning models, the prediction… the one or more computing devices may generate a report based on the prediction of the one or more tile-level class labels for the histopathology image. In one embodiment, the one or more computing devices may cause a human machine interface (HMI) associated with a pathologist or a clinician to display the report”, paragraph [0036], “The output may be based on existence of and access to the appropriate data, so the output generating module 116 may be empowered to access metadata and anonymized patient information as needed. As with the other modules of the whole slide image processing system 110, the output generating module 114 may be updated and improved in a modular fashion, so that new output features may be provided to users without requiring significant downtime”).
The motivation to combine is the same as in claim 10, incorporated herein.
REGARDING CLAIM(S) 15
Claim(s) 15 is/are analogous to Claim(s) 10, thus Claim(s) 15 is/are similarly analyzed and rejected in a manner consistent with the rejection of Claim(s) 10.
Regarding claim 16, Iftikhar and Yip teach the limitations of claim 15, and further teach wherein the natural language query comprises a selection of one or more parameters (Iftikhar: Figs. 1-5, 9, paragraph [0027], “Manipulation of the image may be used to capture a selected portion of the sample at the desired range of magnifications. Image scanner 124 may further capture annotations and/or morphometrics identified by a human operator”, paragraph [0036], “an output generating module 114 of the whole slide image processing system 110 may generate output corresponding to result tile and result WSI datasets based on user request… the output may include a variety of visualizations, interactive graphics, and reports based upon the type of request and the type of data that is available… The output may be based on existence of and access to the appropriate data, so the output generating module 116 may be empowered to access metadata and anonymized patient information as needed”, paragraph [0040], “histopathologist or other expert human annotator to further annotate the low-level features (e.g., assign a class label or bounding geometry identifying a specific region”, Also see, paragraphs [0068]-[0074]; Yip: paragraph [0061], “the tissue class overlay map displays the probabilities for each grid tile for the tissue class selected by the user”, paragraph [0085], “user-selected resolution criteria to select a layer with optimal resolution for analysis”, paragraph [0142], “dimensions selected by the user, and may contain weight values selected by the user”, paragraph [0151], “Each factor may be selected by the user”).
The motivation to combine is the same as in claim 15, incorporated herein.
Regarding claim 17, Iftikhar and Yip teach the limitations of claim 16, and further teach wherein the one or more parameters comprise a region of interest (Iftikhar: Figs. 1-5, 9, paragraph [0006], “bounding geometry identifying a specific region”, paragraph [0036], “an output generating module 114 of the whole slide image processing system 110 may generate output corresponding to result tile and result WSI datasets based on user request… the output may include a variety of visualizations, interactive graphics, and reports based upon the type of request and the type of data that is available”, paragraph [0040], “histopathologist or other expert human annotator to further annotate the low-level features (e.g., assign a class label or bounding geometry identifying a specific region”).
The motivation to combine is the same as in claim 15, incorporated herein.
Regarding claim 18, Iftikhar and Yip teach the limitations of claim 15, and further teach wherein the one or more parameters comprises one or more language parameters, and wherein the machine learning algorithm adjusts the language of the natural language answer to correspond to the one or more language parameters (Iftikhar: Figs. 1-5, paragraph [0048], “the one or more tile-level class label predictions 214 for the histopathology image 202 may be utilized in one or more downstream tasks… the report may include a clinical report that may be associated with one or more cancer patients to be provided and displayed, for example, to a histopathologist or a clinician (e.g., oncologist) for purposes of research and/or the diagnosis, prognosis, and treatment of the one or more patients. In another embodiment, the report may include an interpretability and/or explainability report that may be associated with the machine-learning models 206A, 206B, 206C, and 206D to be provided and displayed, for example, to one or more data scientists or developers for purposes of ascertaining and elucidating the prediction and decision-making behaviors of the machine-learning models 206A, 206B, 206C, and 206D”, paragraphs [0068]-[0074], “the AI architecture 902 may include machine learning (ML) models 904, natural language processing (NLP) models 906, expert systems 908, computer-based vision models 910, speech recognition models 912, planning models 914, and robotics models 916… the NLP models 906 may include any algorithms or functions that may be suitable for automatically manipulating natural language, such as speech and/or text. For example, in some embodiments, the NLP models 906 may include content extraction models 924, classification models 926, machine translation models 928, question answering (QA) models 930, and text generation models 932. In certain embodiments, the content extraction models 924 may include a means for extracting text or images from electronic documents (e.g., webpages, text editor documents, and so forth) to be utilized… The QA models 930 may include any algorithms or functions that may be suitable for automatically answering questions posed by humans in, for example, a natural language, such as that performed by voice-controlled personal assistant devices. The text generation models 932 may include any algorithms or functions that may be suitable for automatically generating natural language texts”. The Examiner interprets the user is able to specify the type of report output, which teaches what is required under the broadest reasonable interpretation).
The motivation to combine is the same as in claim 15, incorporated herein.
Regarding claim 19, Iftikhar and Yip teach the limitations of claim 15, and further teach wherein the natural language answer comprises one or more visual indicators to visually differentiate the natural language answer for the respective magnification levels (Yip: Fig. 3, paragraphs [0061]-[0062], “The overlay map generator and metric calculator may retrieve the stored 3-dimensional probability data array from the tissue class locator, and convert it into an overlay map that displays the assigned tissue class label for each tile. The assigned tissue class for each tile may be displayed as a transparent color that is unique for each tissue class… a tumor probability overlay heatmap map”, paragraph [0112], “displaying a grid-based digital overlay map in which each tissue class is represented by a unique color”, paragraph [0177], “an exemplary overlay may represent individual lymphocytes by coloring them red to distinguish between clusters of lymphocytes and individual lymphocytes infiltrating other tissues. In one example, individual lymphocytes detected within tumor tiles represent tumor-infiltrating lymphocytes which would be red dots within dark blue or green tiles”).
The motivation to combine is the same as in claim 15, incorporated herein.
Regarding claim 20, Iftikhar and Yip teach the limitations of claim 19, and further teach wherein the one or more visual indicators is a change in color (Yip: Fig. 3, paragraphs [0061]-[0062], “The overlay map generator and metric calculator may retrieve the stored 3-dimensional probability data array from the tissue class locator, and convert it into an overlay map that displays the assigned tissue class label for each tile. The assigned tissue class for each tile may be displayed as a transparent color that is unique for each tissue class… a tumor probability overlay heatmap map”, paragraph [0112], “displaying a grid-based digital overlay map in which each tissue class is represented by a unique color”, paragraph [0177], “an exemplary overlay may represent individual lymphocytes by coloring them red to distinguish between clusters of lymphocytes and individual lymphocytes infiltrating other tissues. In one example, individual lymphocytes detected within tumor tiles represent tumor-infiltrating lymphocytes which would be red dots within dark blue or green tiles”).
The motivation to combine is the same as in claim 15, incorporated herein.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Pub. No. 20250245827 (hereafter “Iftikhar”) and U.S. Patent Pub. No. 20200211189 (hereafter “Yip”) as applied to claim 1 above, and further in view of U.S. Patent Pub. No. 20210073984 (hereafter “Locke”).
Regarding claim 4, Iftikhar and Yip teach the limitations of claim 2, but may not explicitly teach wherein the plurality of parameters are selected by a toggle switch displayed on a screen.
Locke teaches wherein the plurality of parameters are selected by a toggle switch displayed on a screen (Locke: Figs. 3-12, paragraph [0097], “The prediction heat map overlay visualization may be toggled on and off using the navigation menu 301. For example, a user may select an overlay icon on the navigation menu to toggle on or off a heat map overlay”, paragraph [0104], “The user may quickly switch or toggle the prediction heat map overlay on and off while viewing through the inspect tool 111”).
One of ordinary skill in the art before the effective filing date would have found it obvious to include using a toggle switch on a display screen as taught by Locke with the display of multi-resolution images and analyses as taught by Iftikhar and Yip with the motivation of “expedite and improve a pathologist's work solutions” (Locke: paragraph [0004]).
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Patent Pub. No. 20250245827 (hereafter “Iftikhar”) and U.S. Patent Pub. No. 20200211189 (hereafter “Yip”) as applied to claim 1 above, and further in view of U.S. Patent Pub. No. 20160246777 (hereafter “Moldoveanu”).
Regarding claim 8, Iftikhar and Yip teach the limitations of claim 7, but may not explicitly teach wherein the language parameter is a patient-oriented mode or a teacher mode.
Moldoveanu teaches wherein the language parameter is a patient-oriented mode or a teacher mode (Moldoveanu: paragraph [0076], “Depending on the particular state of the communication platform (100), different interface elements may be present. Accordingly, the interface may be configured for the implementation of various modes of operation and interactions, including the receiving and displaying of statements, queries, challenges, answers, questions, answers, suggestions, modified statements, etc”, paragraph [0151], “A particular mode may be selected by a user (e.g., a student or teacher) during operation from available modes, or may be selected by an administrator of platform (100)”).
One of ordinary skill in the art before the effective filing date would have found it obvious to use a student/teacher (i.e., patient/teacher) mode as taught by Moldoveanu with the changing of the format of the report as taught by Iftikhar and Yip with the motivation of “improve the relevance and responsiveness of suggestions generated by the platform” (Moldoveanu: paragraph [0094]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. Patent Pub. No. 20220051364 (hereafter “Kirszenberg”) teaches magnification of multi-resolution pathology images.
U.S. Patent Pub. No. 20230061428 (hereafter “Raciti”) teaches pathology image analysis at multiple resolutions.
U.S. Patent Pub. No. 20230245303 (hereafter “Miller”) teaches a multi-resolution classification model for whole slide image (WSI) analysis.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Andrew E Lee whose telephone number is (571)272-8323. The examiner can normally be reached M-Th 9-5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Shahid Merchant can be reached on 571-270-1360. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.E.L./Examiner, Art Unit 3684
/Shahid Merchant/Supervisory Patent Examiner, Art Unit 3684
Prosecution Insights