DETAIL OFFICE ACTIONS
The United States Patent & Trademark Office appreciates the response filed for the current application that is submitted on 01/08/2026. The United States Patent & Trademark Office reviewed the following documents submitted and has made the following comments below.
Continued Examination
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 01/08/2026 has been entered.
Amendment
Applicant submitted amendments on 01/08/2026. The Examiner acknowledges the amendment and has reviewed the claims accordingly.
Drawings
Applicant submitted corrected drawings on 08/21/2025. The Examiner acknowledges the drawings and has withdrawn the objection.
Priority
Applicant claims the benefit of US Provisional Application No. 63/282,249, filed 11/23/2021. Claims 1, 4-9, 11-18 and 20-26 have been afforded the benefit of this filing date.
Information Disclosure Statement
The IDS dated 02/26/2023, 04/03/2023, 04/13/2023, and 09/23/2025 has been previously considered and placed in the application file.
Overview
Claims 1-9 and 11-25 are pending in this application and have been considered below.
Claims 2, 3, and 19 have been cancelled.
Claims 1, 4-9, 11-18 and 20-26 are rejected.
Applicant Arguments:
In regards to Argument 1, Applicant states that “the cited passages fail to teach the workflow actually recited by claim 1. As illustrated by the text shown above, claim 1 recites a workflow comprising two different machine learning models (i.e., the preanalytical machine learning model and the secondary machine learning model)” therefore, the rejection of 35 U.S.C. 103 should be withdrawn (See Remarks, page 15, paragraph 3).
In regards to Argument 2, Applicant states that “Support does not appear to be present in any of the cited passages of either reference, considered alone or in combination, for the extraction of a feature map from a first model, and the use of that feature map to train a secondary model, in the context of the present claims.” therefore, the rejection of 35 U.S.C. 103 should be withdrawn (See Remarks, page 15, paragraph 4).
In regards to Argument 3, Applicant states that “the Office Action has failed to articulate a motivation to integrate this element into a system produced by the combined teachings of the relied upon references. Given the fact that this training element is not satisfied, it followed that the combined teachings of Bauer and Saltz also fail to lead to a system or method that utilizes this trained secondary model, as recited by claim 1 (i.e., wherein "the secondary machine learning model generates the outcome of the target secondary indication in response to an input of the target image and a target feature map extracted from a hidden layer of the preanalytical machine learning model fed the target image").” therefore, the rejection of 35 U.S.C. 103 should be withdrawn (See Remarks, page 15-16, paragraph 1).
In regards to Argument 4, Applicant states that “Analogous arguments apply to independent claim 22, which recites substantially similar elements” therefore, the rejection of 35 U.S.C. 103 should be withdrawn (See Remarks, page 16, paragraph 3).
In regards to Argument 5, Applicant states that “When the actual parameters and workflow are considered, it is evident that Klaiman fails to teach each and every element of the image correction and image translation machine learning models, which are both now incorporated into independent claims 18 and 20.” therefore, the rejection of 35 U.S.C. 103 should be withdrawn (See Remarks, page 16, paragraph 3).
In regards to Argument 6, Applicant states that “Independent claims 23 and 25 are presently amended and now recite that the at least one preanalytical factor comprises a plurality of factors (e.g., fixation time, and additionally comprises a tissue thickness, or a warm ischemic time). It is respectfully submitted that these specific combinations are not taught, suggested, or otherwise disclosed in the asserted prior art of record.” therefore, the rejection of 35 U.S.C. 103 should be withdrawn (See Remarks, page 16, paragraph 3).
Examiner’s Responses:
In response to Argument 1, Applicant’s arguments, see Remarks, filed 01/08/2026, with
respect to the rejection(s) of claim(s) 1 under 35 U.S.C. 103 have been fully considered and are not persuasive. Therefore, the rejection has been maintained under 35 U.S.C. 103 in view of Bauer et al (WO Patent Pub No. 2021/037875 A1, hereafter referred to as Bauer) in view of Saltz et al (U.S Patent Pub. No US 2020/0388029 A1, hereafter referred to as Saltz).
The examiner finds that Bauer teaches on workflow that is recited in Claim 1. Applicant argues “the cited passages fail to teach the workflow actually recited by claim 1. As illustrated by the text shown above, claim 1 recites a workflow comprising two different machine learning models (i.e., the preanalytical machine learning model and the secondary machine learning model)”. Specifically the examiner finds that Bauer does teach two models in the workflow specifically in Figure 2, 201 and ¶0139 where the fixation estimation engine is adapted to operate in a training mode and provide the model with training data based on different algorithms. The examiner will maintain prior art Bauer and Saltz and details of the rejection are below.
In response to Argument 2, Applicant’s arguments, see Remarks, filed 01/08/2026, with
respect to the rejection(s) of claim(s) 1 under 35 U.S.C. 103 have been fully considered and are not persuasive. Therefore, the rejection has been maintained under 35 U.S.C. 103 in view of Bauer et al (WO Patent Pub No. 2021/037875 A1, hereafter referred to as Bauer) in view of Saltz et al (U.S Patent Pub. No US 2020/0388029 A1, hereafter referred to as Saltz).
The examiner finds that Bauer and Saltz teach on feature extraction model and the feature extraction model being used as an input as recited in Claim 1. Applicant argues “Support does not appear to be present in any of the cited passages of either reference, considered alone or in combination, for the extraction of a feature map from a first model, and the use of that feature map to train a secondary model, in the context of the present claims.”. Specifically the examiner finds that Saltz teaches a CNN that extracts the features from tissue images in ¶0018 and detect and encode nuclei into feature maps in ¶0018 and ¶0031. Bauer then teaches using the output on one model to be the input of another in Figure 2 ¶0133-¶0139. The examiner will maintain prior art Bauer and Saltz and details of the rejection are below.
In response to Argument 3, Applicant’s arguments, see Remarks, filed 01/08/2026, with
respect to the rejection(s) of claim(s) 1 under 35 U.S.C. 103 have been fully considered and are not persuasive. Therefore, the rejection has been maintained under 35 U.S.C. 103 in view of Bauer et al (WO Patent Pub No. 2021/037875 A1, hereafter referred to as Bauer) in view of Saltz et al (U.S Patent Pub. No US 2020/0388029 A1, hereafter referred to as Saltz).
The examiner finds that Bauer and Saltz teaches on feature extraction model and the feature extraction model being used an input and its motivation to be integrated as an input for a machine learning model as recited in Claim 1. Applicant argues “the Office Action has failed to articulate a motivation to integrate this element into a system produced by the combined teachings of the relied upon references. Given the fact that this training element is not satisfied, it followed that the combined teachings of Bauer and Saltz also fail to lead to a system or method that utilizes this trained secondary model, as recited by claim 1 (i.e., wherein "the secondary machine learning model generates the outcome of the target secondary indication in response to an input of the target image and a target feature map extracted from a hidden layer of the preanalytical machine learning model fed the target image"). Specifically the examiner finds Saltz teaches a CNN that extracts the features from tissue images in ¶0018 and detect and encode nuclei into feature maps in ¶0018 and ¶0031. Bauer then teaches using the output on one model to be the input of another in Figure 2 ¶0133-¶0139). Further, motivation was provided in all present and previous combinations of references. Although a specific motivation may not have been explicitly stated within one of the references, the motivation was not improper, and provided in accordance with the Teaching-Suggestion-Motivation Test (TSM). As such, Examiner's use of these facts as a motivation statement is in compliance with the requirements of the TSM test, since the Teaching-Suggestion-Motivation (TSM) test should be flexibly applied and the teaching, suggestion, or motivation need not be written within the reference. See KSR Int'l Co. v. Teleflex Inc., 82 USPQ2d 1385 (US 2007); Ortho-McNeil Pharm., Inc. v. Mylan Lab., Inc., 520 F.3d 1358, 86 U.S.P.Q.2d 1196 (Fed. Cir. 2008); Ex Parte Kubin, 83 USPQ2d 1410 (Bd. Pat. App. & Int. 2007). The examiner will maintain prior art Bauer and Saltz and details of the rejection are below.
In response to Argument 4, Applicant’s arguments, see Remarks, filed 01/08/2026, with
respect to the rejection(s) of claim(s) 22 under 35 U.S.C. 103 have been fully considered and are not persuasive. Therefore, the rejection has been maintained under 35 U.S.C. 103 in view of Bauer et al (WO 2021/037875 A1, hereafter referred to as Bauer) in view of Saltz et al (U.S Patent Pub. No 2020/0388029, hereafter referred to as Saltz).
The examiner finds that Bauer and Saltz teach on the features of claim 22 as expanded on in responses to Arguments 1-3.
In response to Argument 5, Applicant’s arguments, see Remarks, filed 01/08/2026, with
respect to the rejection(s) of claim(s) 18 and 20 under 35 U.S.C. 103 have been fully considered and are not persuasive. Therefore, the rejection has been maintained under 35 U.S.C. 103 in view of Bauer et al (WO 2021/037875 A1, hereafter referred to as Bauer) in view of Saltz et al (U.S Patent Pub. No 2020/0388029, hereafter referred to as Saltz) in further view of Klaiman (U.S Patent Pub. No 20210005308 A1, hereafter referred to as Klaiman).
The examiner finds that Klaiman teaches on each and every element of the image correction and image translation machine learning models recited in Claims 18 and 20. Applicant argues “When the actual parameters and workflow are considered, it is evident that Klaiman fails to teach each and every element of the image correction and image translation machine learning models, which are both now incorporated into independent claims 18 and 20.” Specifically the examiner finds that Klaiman teaches an model that performs image to image translation in ¶0083, ¶0094, and the image translation model trained on multiple transformed images of tissue samples in ¶0087. The examiner will maintain prior art Bauer and Saltz and Klaiman and details of the rejection are below.
In response to Argument 6, with respect to the rejection(s) of claim(s) 23 and 25 under 35 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn due to the amendment. However, upon further consideration, a new ground(s) of rejection is made for claims under 35 U.S.C. 103 in view of Bauer et al (WO Patent Pub No. 2021/037875 A1, hereafter referred to as Bauer) in view of Saltz et al (U.S Patent Pub. No US 2020/0388029 A1, hereafter referred to as Saltz) in further view of Lovborg et al. (WO 2010/066252 A1, hereafter referred to as Lovborg).
The examiner finds that Lovborg teaches on or a warm ischemic time as recited in Claims 23 and 25. Applicant argues Specifically the examiner finds that Lovborg teaches on the use of temperature and ischemic time as preanalytical factors in Pg 2 Lines 15-20 Pg 32 Lines 28-32 and Pg 22 lines 15-21. The examiner will maintain prior art Bauer and Saltz and Lovborg and details of the rejection are below.
Claim Interpretation
Under MPEP 2143.03, "All words in a claim must be considered in judging the patentability of that claim against the prior art." In re Wilson, 424 F.2d 1382, 1385, 165 USPQ 494, 496 (CCPA 1970). As a general matter, the grammar and ordinary meaning of terms as understood by one having ordinary skill in the art used in a claim will dictate whether, and to what extent, the language limits the claim scope. Language that suggests or makes a feature or step optional but does not require that feature or step does not limit the scope of a claim under the broadest reasonable claim interpretation. In addition, when a claim requires selection of an element from a list of alternatives, the prior art teaches the element if one of the alternatives is taught by the prior art. See, e.g., Fresenius USA, Inc. v. Baxter Int’l, Inc., 582 F.3d 1288, 1298, 92 USPQ2d 1163, 1171 (Fed. Cir. 2009).
Claim 5 recite “at least one of” then listing “(i) an input image and additional metadata indicating a source preanalytical factor that has been abnormally processed, and metadata indicating a destination preanalytical factor that has been normally processed” and “(ii) an input image and further comprising providing a reference image from the destination set used to infer the destination of the input image.”. Since “at least one of” is disjunctive, any one of the elements found in the prior art is sufficient to reject the claim. While citations have been provided for completeness and rapid prosecution, only one element is required. Because, on balance, it appears the disjunctive interpretation enjoys the most specification support and for that reason the disjunctive interpretation (one of A, B OR C) is being adopted for the purposes of this Office Action. Applicant’s comments and/or amendments relating to this issue are invited to clarify the claim language and the prosecution history.
Claim 17 recite “at least one of” then listing “preanalytical factor is selected from a group consisting,”; “an indication of a quality of a stain of the pathological tissue of the slide, fixation time, tissue thickness obtained by sectioning of the FFPE block, fixative type, warm ischemic time, cold ischemic time, duration and delay of temperature during prefixation, fixative formula, fixative concentration, fixative pH, fixative age of reagent, fixative preparation source, tissue to fixative volume ratio, method of fixation, conditions of primary and secondary fixation, post fixation washing conditions and duration, post fixation storage reagent and duration, type of processor, frequency of servicing and reagent replacement, tissue to reagent volume ratio, number of position of co-processed specimens, dehydration and clearing reagent, dehydration and clearing temperature, dehydration and clearing number of changes, dehydration clearing duration, baking time, and temperature.”. Since “at least one of” is disjunctive, any one of the elements found in the prior art is sufficient to reject the claim. While citations have been provided for completeness and rapid prosecution, only one element is required. Because, on balance, it appears the disjunctive interpretation enjoys the most specification support and for that reason the disjunctive interpretation (one of A, B OR C) is being adopted for the purposes of this Office Action. Applicant’s comments and/or amendments relating to this issue are invited to clarify the claim language and the prosecution history.
Claim 18 and 20 recite “at least one of” then listing “(i) feeding the target image and the at least one target preanalytical factor into a secondary machine learning model, wherein the secondary machine learning model is trained on a secondary indication training dataset of a plurality of records, wherein a secondary indication record comprises: the image of the slide of pathological tissue of the subject processed with the at least one preanalytical factor, the at least one preanalytical factor, and a ground truth label indicating the secondary indication; and obtaining an outcome of a target secondary indication,” and “(ii) in response to classifying the at least one target preanalytical factor as abnormal, feeding the target image and the at least one target preanalytical factor into an image correction machine learning model, wherein the image correction machine learning model is trained on a corrected image training dataset of a plurality of records, wherein an image correction record comprises: the image of the slide of pathological tissue of the subject processed with the at least one preanalytical factor, wherein the at least one preanalytical factor is classified as abnormal, wherein the image of the slide depicts abnormally processed pathological tissue; the at least one preanalytical factor, and a ground truth label indicating a normal image of a slide of pathological tissue processed with at least one preanalytical factor classified as normal; and obtaining an outcome of a corrected image that simulates what the target image of the slide would look like when processed with the at least one preanalytical factor classified as normal;” and “(iii) in response to classifying the at least one target preanalytical factor as abnormal, feeding the target image and the at least one target preanalytical factor into an image translation machine learning model, wherein the image translation machine learning model is trained on an image translation training dataset, comprising two or more sets of image translation records, wherein a source image translation record of a source set of image translation records comprises: a source image of the slide of pathological tissue of the subject processed with the at least one preanalytical factor, and a ground truth indicating a source label, wherein a destination image translation record of a destination set of image translation records comprises: a destination image of the slide of pathological tissue of the subject processed with the at least one preanalytical factor, and a ground truth indicating a destination label; and obtaining an outcome destination image of a slide of pathological tissue of the destination set of image translation records that is a conversion of the abnormally processed target image into a normally processed image.” Since “at least one of” is disjunctive, any one of the elements found in the prior art is sufficient to reject the claim. While citations have been provided for completeness and rapid prosecution, only one element is required. Because, on balance, it appears the disjunctive interpretation enjoys the most specification support and for that reason the disjunctive interpretation (one of A, B OR C) is being adopted for the purposes of this Office Action. Applicant’s comments and/or amendments relating to this issue are invited to clarify the claim language and the prosecution history.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 9, 11-13, 15, 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Bauer et al (WO 2021/037875 A1, hereafter referred to as Bauer) in view of Saltz et al (U.S Patent Pub. No 2020/0388029, hereafter referred to as Saltz).
Regarding Claim 1, Bauer teaches a computer implemented method (Bauer ¶0097 discloses a computer implemented method) of training a preanalytical factor machine learning model, (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) comprising:
creating a preanalytical training dataset of a plurality of records (Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset), wherein a preanalytical record comprises:
an image of a slide of pathological tissue (Bauer ¶0180 discloses using images from 210 slides of tissues) of a subject processed with at least one preanalytical factor, (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) and a ground truth label indicating the at least one preanalytical factor (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time); and
Training (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) the preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) on the preanalytical training dataset(Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset) for generating an outcome of at least one target preanalytical factor (Bauer ¶0149 discloses the engine outputting a qualitative assessment of fixation quality) used to process tissue depicted in a target image in response to the input of the target image (Bauer ¶0123, ¶0149, discloses using input output pairs to train the model),
wherein the ground truth label indicating the at least one preanalytical factor (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time) comprises a ground truth label indicating correctly applied preanalytical factors (Bauer ¶0226 discloses labeling the fixation duration with annotations on the quality)
wherein training comprises training an implementation of the preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) for learning a distribution of inlier images labelled as correctly applied preanalytical factors (Bauer ¶0191, Fig 17D discloses using a box and whisker plot to determine if the model is well trained at predicting the fixation time of the specimen) for detecting an image as an outlier indicating incorrectly applied preanalytical factors. (Bauer ¶0191, Fig 17D discloses using a box and whisker plot to determine outliers if the model is well trained at predicting the fixation time of the specimen);
creating a secondary training dataset of a plurality of records (Bauer ¶0108 discloses training on a second tissue sample up to the nth tissue sample and ¶0139 discloses a second derivative of the training data) , wherein a secondary record comprises:
the image of the slide of pathological tissue of the subject (Bauer ¶0180 discloses using images from 210 slides of tissues) processed with the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) , the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration, and a ground truth label indicating a secondary indication (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second); and
training a secondary machine learning model (Bauer ¶0190 discloses an initial model and then further developing the model which can be interpreted as a second model) on the secondary training dataset (Bauer ¶0108 discloses training on a second tissue sample up to the nth tissue sample and ¶0139 discloses a second derivative of the training data) for generating an outcome of a target secondary indication (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second) in response to an input of a target image (Bauer ¶0132 discloses target values for the network) and at least one target preanalytical factor (Bauer ¶0149 discloses the engine outputting a qualitative assessment of fixation quality) used to process tissue depicted in the target image (Bauer ¶153 discloses processing the specimen),
wherein the input of the at least one preanalytical factor fed into the secondary machine learning model ( Bauer Figure 2, 201 and ¶0139 where the fixation estimation engine is adapted to operate in a training mode and provide the model with training data based on different algorithms) is obtained as the outcome of the preanalytical machine learning model fed the target image (Bauer ¶0133 discloses feeding data to the neural network), wherein the preanalytical machine learning model and the secondary machine learning model Bauer ¶0190 discloses an initial model and then further developing the model which can be interpreted as a second model) using at least common images and common labels of preanalytical factors (Bauer ¶0108 discloses training on a second tissue sample up to the nth tissue sample and ¶0139 discloses a second derivative of the training data) fed the image (Bauer ¶0133 discloses feeding data to the neural network) of the slide of pathological tissue (Bauer ¶0180 discloses using images from 210 slides of tissues) of a subject processed with at least one preanalytical factor, (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), and wherein the secondary machine learning model (Bauer ¶0190 discloses an initial model and then further developing the model which can be interpreted as a second model) generates the outcome of the target secondary indication (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second) in response to an input of a target image (Bauer ¶0132 discloses target values for the network).
Bauer does not explicitly disclose or anomalous application of preanalytical factors, are jointly trained, wherein the at least one preanalytical factor of the secondary record comprises at least one feature map extracted from a hidden layer of the preanalytical machine learning model, and a target feature map extracted from a hidden layer of the preanalytical machine learning model fed the target image.
Saltz is in the same field of are of image analysis of tissues samples. Further, Saltz teaches or anomalous application of preanalytical factors (Saltz ¶0102 discloses a preprocessing step that checks for quality which includes correct temperature storage of the specimens, a preanalytical factor), are jointly trained (Saltz ¶0249 discloses jointly training the model), wherein the at least one preanalytical factor of the secondary record comprises at least one feature map extracted from a hidden layer of the preanalytical machine learning model (Saltz et al. ¶0132 "The fully unsupervised autoencoder 70 in FIG. 2A first decomposes or segments an input histopathology image patch 71 into foreground (e.g. nuclei) 73 and background (e.g. cytoplasm) 74 during the sparse autoencoding step 72. The CAE then detects nuclei in the foreground by representing the locations of nuclei as a sparse feature map.") and a target feature map extracted from a hidden layer of the preanalytical machine learning model fed the target image (Saltz et al ¶0132 "The CAE then detects nuclei in the foreground by representing the locations of nuclei as a sparse feature map.").
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer by incorporating the anomalous application of preanalytical factors, input image patches, jointly and supervised training the model, nuclear segmentation machine, color normalization and resolution adjusting, and the feature map extracted from a hidden layer that is taught by Saltz, to make an invention that analyzes the tissues using multiple kinds of models and forms of inputs for maximum flexibility but also outputs a feature map to identify elements of the tissues to improve the training of the model; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for increased accuracy of the machine learning model to improve the end result of feature identification in the slides (Saltz et al. ¶0005 "Complex segmentation of nuclei in whole slide tissue images, is considered a common methodology in pathology image analysis and quality control of such algorithms are being implemented to improve segmentation results").
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
27. Regarding Claim 9, Bauer in view of Saltz teaches the computer implemented method of claim 1, further comprising training a baseline model using a self-supervised and/or unsupervised approach (Saltz et al. ¶0261 "The system next trains the CNNS on an initial supervised dataset") on an unlabeled training dataset of a plurality of unlabeled images of pathological tissues of a subject processed with at least one preanalytical factor (Saltz et al. ¶0254 "The dataset for unsupervised learning was collected as 0.5 million unlabeled small images randomly cropped from 400 lung adenocarcinoma histopathology images obtained from the public TCGA repository."), and wherein training comprises further training the baseline model on the preanalytical training dataset (Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset) for creating the preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets). See Claim 1 for rationale, its parent claim.
28. Regarding Claim 11, Bauer in view of Saltz teaches the computer implemented method of claim 1, further comprising, for each preanalytical record, feeding the image into a nuclear segmentation machine learning model to obtain an outcome of a segmentation of nuclei in the image, (Saltz et al. ¶0291 "As shown in the workflow 280, nuclear segmentation is implemented at the outset in step 281 for proper quality segmentation of nuclei in tumoral tissue.") creating a mask that masks out pixels external to the segmentation of the nuclei based on the outcome of the segmentation (Saltz et al. ¶0169 "The training set of the necrosis segmentation CNN consisted of 1,800 patches. Each patch was annotated with a necrosis region mask segmented by a pathologist. Sample were 2480 patches to create the test dataset."), and applying the mask to the image to create a masked image, wherein the image of the preanalytical record comprises the masked image, and wherein a target masked image created from the target image is fed into the preanalytical machine learning model trained on the preanalytical training dataset (Saltz et al. ¶0169 "The training set of the necrosis segmentation CNN consisted of 1,800 patches. Each patch was annotated with a necrosis region mask segmented by a pathologist. Sample were 2480 patches to create the test dataset."). See Claim 1 for rationale, its parent claim.
29. Regarding Claim 12, Bauer in view of Saltz teaches the computer implemented method of claim 1, further comprising, for each preanalytical record, feeding the image into a nuclear segmentation machine learning model to obtain an outcome of a segmentation of nuclei in the image (Saltz et al. ¶0291 "As shown in the workflow 280, nuclear segmentation is implemented at the outset in step 281 for proper quality segmentation of nuclei in tumoral tissue.") , and cropping a boundary around each segmentation to create single-nucleus patches (Saltz et al. ¶0296 "Step 4 shows a specific nucleus of interest being selected. Each dot represents a single nucleus."), wherein the image of the preanalytical record comprises a plurality of single-nucleus patches, and wherein a target segmentation of nuclei created from the target image is fed into the preanalytical machine learning model trained on the preanalytical training dataset. (Saltz et al. ¶0169 "The training set of the necrosis segmentation CNN consisted of 1,800 patches. Each patch was annotated with a necrosis region mask segmented by a pathologist. Sample were 2480 patches to create the test dataset."). See Claim 1 for rationale, its parent claim.
30. Regarding Claim 13, Bauer in view of Saltz teaches the computer implemented method of claim 1, further comprising, for each preanalytical record, converting a color version of the image to a gray-scale version of the image, and wherein a target gray-scale version of the target image is fed into the preanalytical machine learning model trained on the preanalytical training dataset. (Saltz et al. ¶0102 "This color normalization step ensures that artifacts due to over-staining or under-staining, as well as any operator or scanner errors are accounted for, highlighted and/or resolved, if possible. Next, the image is transmitted into a CNN trained model as shown in step 5."). See Claim 1 for rationale, its parent claim.
31. Regarding Claim 15, Bauer in view of Saltz teaches the computer implemented method of claim 1, wherein the preanalytical record further comprises metadata indicating at least one known preanalytical factor (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second), and wherein the ground truth label is for at least one unknown preanalytical factor (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time), wherein at least one known preanalytical factor associated with the target image (Bauer ¶0132 discloses target values for the network) is further fed into the preanalytical machine learning model (Bauer ¶0133 discloses feeding data to the neural network) trained on the preanalytical training dataset (Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset). See Claim 1 for rationale, its parent claim.
34. Regarding Claim 21 Bauer in view of Saltz teaches the computer implemented method (Bauer ¶0097 discloses a computer implemented method) of claim 1, wherein the at least one preanalytical factor comprises a fixation time (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration). See Claim 1 for rationale, its parent claim.
35. Regarding Claim 22, Bauer teaches a computer implemented method (Bauer ¶0097 discloses a computer implemented method) of training a preanalytical factor machine learning model, (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets), comprising:
creating a preanalytical training dataset of a plurality of records (Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset), wherein a preanalytical record comprises an image of a slide of pathological tissue (Bauer ¶0180 discloses using images from 210 slides of tissues) of a subject processed with at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), and a ground truth label indicating the at least one preanalytical factor (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time ; and
training the preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) on the preanalytical training dataset (Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset) for generating an outcome of at least one target preanalytical factor (Bauer ¶0149 discloses the engine outputting a qualitative assessment of fixation quality) used to process tissue depicted in a target image in response to the input of the target image (Bauer ¶0123, ¶0149, discloses using input output pairs to train the model);
creating a secondary training dataset of a plurality of records (Bauer ¶0108 discloses training on a second tissue sample up to the nth tissue sample and ¶0139 discloses a second derivative of the training data), wherein a secondary record comprises:
the image of the slide of pathological tissue of the subject (Bauer ¶0180 discloses using images from 210 slides of tissues) processed with the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), and a ground truth label indicating a secondary indication (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second); and
training a secondary machine learning model (Bauer ¶0190 discloses an initial model and then further developing the model which can be interpreted as a second model) on the secondary training dataset (Bauer ¶0108 discloses training on a second tissue sample up to the nth tissue sample and ¶0139 discloses a second derivative of the training data) for generating an outcome of a target secondary indication (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second) in response to an input of a target image (Bauer ¶0132 discloses target values for the network) and at least one target preanalytical factor (Bauer ¶0149 discloses the engine outputting a qualitative assessment of fixation quality) used to process tissue depicted in the target image (Bauer ¶153 discloses processing the specimen);
wherein the input of the at least one preanalytical factor fed into the secondary machine learning model (Bauer ¶0123, ¶0149, discloses using input output pairs to train the model) is obtained as the outcome of the preanalytical machine learning model fed the target image (Bauer ¶0133 discloses feeding data to the neural network), wherein the preanalytical machine learning model and the secondary machine learning model Bauer ¶0190 discloses an initial model and then further developing the model which can be interpreted as a second model) using at least common images and common labels of preanalytical factors (Bauer ¶0108 discloses training on a second tissue sample up to the nth tissue sample and ¶0139 discloses a second derivative of the training data) fed the image (Bauer ¶0133 discloses feeding data to the neural network) of the slide of pathological tissue (Bauer ¶0180 discloses using images from 210 slides of tissues) of the subject processed (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), and wherein the secondary machine learning model (Bauer ¶0190 discloses an initial model and then further developing the model which can be interpreted as a second model) generates the outcome of the target secondary indication (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second) in response to an input of a target image (Bauer ¶0132 discloses target values for the network).
Bauer does not explicitly disclose are jointly trained, wherein the at least one preanalytical factor of the secondary record comprises at least one feature map extracted from a hidden layer of the preanalytical machine learning model, )and a target feature map extracted from a hidden layer of the preanalytical machine learning model fed the target image.
Saltz is in the same field of are of image analysis of tissues samples. Further, Saltz teaches are jointly trained (Saltz ¶0249 discloses jointly training the model), and wherein the at least one preanalytical factor of the secondary record comprises at least one feature map extracted from a hidden layer of the preanalytical machine learning model (Saltz et al. ¶0132 "The fully unsupervised autoencoder 70 in FIG. 2A first decomposes or segments an input histopathology image patch 71 into foreground (e.g. nuclei) 73 and background (e.g. cytoplasm) 74 during the sparse autoencoding step 72. The CAE then detects nuclei in the foreground by representing the locations of nuclei as a sparse feature map.") and a target feature map extracted from a hidden layer of the preanalytical machine learning model fed the target image (Saltz et al ¶0132 "The CAE then detects nuclei in the foreground by representing the locations of nuclei as a sparse feature map.").
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer by incorporating the anomalous application of preanalytical factors, input image patches, jointly and supervised training the model, nuclear segmentation machine, color normalization and resolution adjusting, and the feature map extracted from a hidden layer that is taught by Saltz, to make an invention that analyzes the tissues using multiple kinds of models and forms of inputs for maximum flexibility but also outputs a feature map to identify elements of the tissues to improve the training of the model; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for increased accuracy of the machine learning model to improve the end result of feature identification in the slides (Saltz et al. ¶0005 "Complex segmentation of nuclei in whole slide tissue images, is considered a common methodology in pathology image analysis and quality control of such algorithms are being implemented to improve segmentation results").
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
36. Claims 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Bauer et al (WO 2021/037875 A1, hereafter referred to as Bauer) in view of Saltz et al (U.S Patent Pub. No 2020/0388029, hereafter referred to as Saltz) in further view of in view of Klaiman (U.S Patent Pub. No 20210005308 A1, hereafter referred to as Klaiman).
37. Regarding Claim 18, Bauer teaches a computer implemented method (Bauer ¶0097 discloses a computer implemented method) of processing a target image (Bauer ¶0035, Figs 9A and 9B discloses image processing) of a slide of pathological tissue of a subject (Bauer ¶0180 discloses using images from 210 slides of tissues), comprising:
feeding the target image (Bauer ¶0133 discloses feeding data to the neural network) into a preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) , wherein the preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) of a plurality of records (Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset), where a preanalytical record comprises:
an image of a slide of pathological tissue (Bauer ¶0180 discloses using images from 210 slides of tissues) of a subject processed with at least one preanalytical factor, (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) and a ground truth label indicating the at least one preanalytical factor (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time); and
obtaining an outcome of at least one target preanalytical factor (Bauer ¶0149 discloses the engine outputting a qualitative assessment of fixation quality) used to process the pathological tissue depicted in the target image (Bauer ¶0123, ¶0149, discloses using input output pairs to train the model)
wherein the ground truth label indicating the at least one preanalytical factor (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time) comprises a ground truth label indicating correctly applied preanalytical factors (Bauer ¶0226 discloses labeling the fixation duration with annotations on the quality)
wherein training comprises training an implementation of the preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) for learning a distribution of inlier images labelled as correctly applied preanalytical factors (Bauer ¶0191, Fig 17D discloses using a box and whisker plot to determine if the model is well trained at predicting the fixation time of the specimen) for detecting an image as an outlier indicating incorrectly applied preanalytical factors. (Bauer ¶0191, Fig 17D discloses using a box and whisker plot to determine outliers if the model is well trained at predicting the fixation time of the specimen).
feeding the target image and the at least one target preanalytical factor (Bauer ¶0133 discloses feeding data to the neural network) into a secondary machine learning model (Bauer ¶0190 discloses an initial model and then further developing the model which can be interpreted as a second model), wherein the secondary machine learning model is trained on a secondary indication training dataset of a plurality of records (Bauer ¶0108 discloses training on a second tissue sample up to the nth tissue sample and ¶0139 discloses a second derivative of the training data), wherein a secondary indication record comprises:
the image of the slide of pathological tissue of the subject (Bauer ¶0180 discloses using images from 210 slides of tissues) processed with the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), the at least one preanalytical factor , and a ground truth label indicating the secondary indication (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time),
i) feeding the target image and the at least one target preanalytical factor (Bauer ¶0133 discloses feeding data to the neural network) of pathological tissue of the subject processed with the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) , wherein the at least one preanalytical factor is classified as abnormal (Bauer ¶0003 discloses the presence of abnormalities),
the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) , and a ground truth label indicating a normal image of a slide of pathological tissue processed with at least one preanalytical factor classified as normal(Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time ; and
ii)feeding the target image and the at least one target preanalytical factor(Bauer ¶0133 discloses feeding data to the neural network).
Bauer does not explicitly disclose is trained on a preanalytical training dataset, or anomalous application of preanalytical factors, in response to classifying the at least one target preanalytical factor as abnormal, the image of the slide, of the slide depicts abnormally processed pathological tissue, a source image of the slide of pathological tissue of the subject.
Saltz is in the same field of are of image analysis of tissues samples. Further, Saltz teaches is trained on a preanalytical training dataset (Saltz ¶0203 discloses a training dataset),
or anomalous application of preanalytical factors (Saltz ¶0102 discloses a preprocessing step that checks for quality which includes correct temperature storage of the specimens, a preanalytical factor), in response to classifying the at least one target preanalytical factor as abnormal (Saltz Fig 8F discloses images of slide with abnormal tissue), the image of the slide (Saltz ¶0247 discloses input image patches),
wherein the image of the slide depicts abnormally processed pathological tissue (Saltz Fig 8F discloses images of slide with abnormal tissue);
a source image of the slide of pathological tissue of the subject (Saltz ¶0247 discloses input image patches).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer by incorporating the anomalous application of preanalytical factors, input image patches, jointly and supervised training the model, nuclear segmentation machine, color normalization and resolution adjusting, and the feature map extracted from a hidden layer that is taught by Saltz, to make an invention that analyzes the tissues using multiple kinds of models and forms of inputs for maximum flexibility but also outputs a feature map to identify elements of the tissues to improve the training of the model; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for increased accuracy of the machine learning model to improve the end result of feature identification in the slides (Saltz et al. ¶0005 "Complex segmentation of nuclei in whole slide tissue images, is considered a common methodology in pathology image analysis and quality control of such algorithms are being implemented to improve segmentation results").
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Bauer and Saltz in combination do not explicitly teach into an image correction machine learning model, wherein the image correction machine learning model is trained on a corrected image training dataset, into an image translation machine learning model, wherein the image translation machine learning model is trained on an image translation training dataset, comprising two or more sets of image translation records, wherein a source image translation record of a source set of image translation records comprises, translation record of a destination set of image translation records comprises: a destination image of the slide of pathological tissue of the subject processed with the at least one preanalytical factor, and a ground truth indicating a destination label,
and obtaining an outcome destination image of a slide of pathological tissue of the destination set of image translation records that is a conversion of the abnormally processed target image into a normally processed image.
Klaiman is in the same field of art of image analysis in tissue samples. Further Klaiman teaches into an image correction machine learning model, wherein the image correction machine learning model is trained on a corrected image training dataset (Klaiman ¶0113 "A "training image" as used herein is an image acquired from a training tissue sample. The training images are used for training an untrained version of the MLL for generating a trained MLL that is adapted to transform an acquired tissue image into an output image that highlights a specific biomarker although the tissue sample"),
into an image translation machine learning model, wherein the image translation machine learning model is trained on an image translation training dataset, comprising two or more sets of image translation records, wherein a source image translation record of a source set of image translation records comprises (Klaiman ¶0061 "The image transformation routine is adapted to transform each of the first training images into a virtual staining image that is identical or similar to the one of the second training images having been obtained for the same training tissue sample."),
translation record of a destination set of image translation records comprises: a destination image of the slide of pathological tissue of the subject processed with the at least one preanalytical factor, and a ground truth indicating a destination label (Klaiman ¶0061 "The image transformation routine is adapted to transform each of the first training images into a virtual staining image that is identical or similar to the one of the second training images having been obtained for the same training tissue sample."); and obtaining an outcome destination image of a slide of pathological tissue of the destination set of image translation records that is a conversion of the abnormally processed target image into a normally processed image.(Klaiman ¶0061 "Thus, a large number of different image transformation routines can be easily generated by generating a corresponding training data set and training an untrained MLL on said training data set.").
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer in view of Saltz by incorporating image translation into the preprocessing of the dataset that is taught by Klaiman, to make the invention of the machine learning model one that could process multiple different types of slides; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for increased testing of multiple different staining types of slides to reduce cost and burden of testing (Klaiman ¶0015 "This may be advantageous as the method may generate an output image highlighting the presence of a second biomarker although the acquired image used as input depicts a sample that was not stained at all or was stained by one or more first biomarkers adapted to selectively stain one or more respective first biomarkers, but not the second biomarker. Thus, by providing a MLL that was trained on information that is implicitly contained in an acquired image, explicit information on the presence of a second biomarker in a tissue can be obtained without the need of staining the tissue sample with a stain that is adapted to selectively stain the second biomarker. Thus, valuable time and the costs for the stain for selectively staining the second biomarker can be saved.")
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
38. Regarding Claim 20, Bauer teaches a device (Bauer ¶0205 discloses implementation with a device) for obtaining at least one preanalytical factor (Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset of a target image of a slide of pathological tissue of a subject (Bauer ¶0180 discloses using images from 210 slides of tissues), comprising:
at least one hardware processor (Bauer ¶0086 discloses a processor)
feeding the target image into a preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) , wherein the preanalytical machine learning model is trained on a preanalytical training dataset of a plurality of records (Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset), where a preanalytical record comprises:
an image of a slide of pathological tissue (Bauer ¶0180 discloses using images from 210 slides of tissues) of a subject processed with at least one preanalytical factor, (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) and a ground truth label indicating the at least one preanalytical factor (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time); and
obtaining an outcome of at least one target preanalytical factor(Bauer ¶0149 discloses the engine outputting a qualitative assessment of fixation quality) used to process the pathological tissue depicted in the target image (Bauer ¶0123, ¶0149, discloses using input output pairs to train the model);
wherein the ground truth label indicating the at least one preanalytical factor (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time) comprises a ground truth label indicating correctly applied preanalytical factors (Bauer ¶0226 discloses labeling the fixation duration with annotations on the quality)
wherein training comprises training an implementation of the preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) for learning a distribution of inlier images labelled as correctly applied preanalytical factors (Bauer ¶0191, Fig 17D discloses using a box and whisker plot to determine if the model is well trained at predicting the fixation time of the specimen) for detecting an image as an outlier indicating incorrectly applied preanalytical factors. (Bauer ¶0191, Fig 17D discloses using a box and whisker plot to determine outliers if the model is well trained at predicting the fixation time of the specimen);
feeding the target image and the at least one target preanalytical factor (Bauer ¶0133 discloses feeding data to the neural network) into a secondary machine learning model (Bauer ¶0190 discloses an initial model and then further developing the model which can be interpreted as a second model), wherein the secondary machine learning model is trained on a secondary indication training dataset of a plurality of records (Bauer ¶0108 discloses training on a second tissue sample up to the nth tissue sample and ¶0139 discloses a second derivative of the training data), wherein a secondary indication record comprises:
the image of the slide of pathological tissue of the subject (Bauer ¶0180 discloses using images from 210 slides of tissues) processed with the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), the at least one preanalytical factor , and a ground truth label indicating the secondary indication (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time),
i) feeding the target image and the at least one target preanalytical factor (Bauer ¶0133 discloses feeding data to the neural network)
of pathological tissue of the subject processed with the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) , wherein the at least one preanalytical factor is classified as abnormal (Bauer ¶0003 discloses the presence of abnormalities),
the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) , and a ground truth label indicating a normal image of a slide of pathological tissue processed with at least one preanalytical factor classified as normal(Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time ; and
ii)feeding the target image and the at least one target preanalytical factor(Bauer ¶0133 discloses feeding data to the neural network)
processed with the at least one preanalytical factor, and a ground truth indicating a source label (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), wherein a destination image (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second).
Bauer does not explicitly disclose executing a code, or anomalous application of preanalytical factors, in response to classifying the at least one target preanalytical factor as abnormal, the image of the slide, wherein the image of the slide depicts abnormally processed pathological tissue, a source image of the slide of pathological tissue of the subject.
Saltz is in the same field of are of image analysis of tissues samples. Further, Saltz teaches executing a code (Saltz ¶0396 discloses executing a set of instructions), or anomalous application of preanalytical factors (Saltz ¶0102 discloses a preprocessing step that checks for quality which includes correct temperature storage of the specimens, a preanalytical factor), in response to classifying the at least one target preanalytical factor as abnormal (Saltz Fig 8F discloses images of slide with abnormal tissue), the image of the slide (Saltz ¶0247 discloses input image patches), wherein the image of the slide depicts abnormally processed pathological tissue (Saltz Fig 8F discloses images of slide with abnormal tissue);
a source image of the slide of pathological tissue of the subject (Saltz ¶0247 discloses input image patches).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer by incorporating the anomalous application of preanalytical factors, input image patches, jointly and supervised training the model, nuclear segmentation machine, color normalization and resolution adjusting, and the feature map extracted from a hidden layer that is taught by Saltz, to make an invention that analyzes the tissues using multiple kinds of models and forms of inputs for maximum flexibility but also outputs a feature map to identify elements of the tissues to improve the training of the model; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for increased accuracy of the machine learning model to improve the end result of feature identification in the slides (Saltz et al. ¶0005 "Complex segmentation of nuclei in whole slide tissue images, is considered a common methodology in pathology image analysis and quality control of such algorithms are being implemented to improve segmentation results").
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Bauer and Saltz in combination do not explicitly teach into an image correction machine learning model, wherein the image correction machine learning model is trained on a corrected image training dataset, obtaining an outcome of a corrected image that simulates what the target image of the slide would look like when processed with the at least one preanalytical factor classified as normal, into an image translation machine learning model, wherein the image translation machine learning model is trained on an image translation training dataset, comprising two or more sets of image translation records, wherein a source image translation record of a source set of image translation records comprises, translation record of a destination set of image translation records comprises: a destination image of the slide of pathological tissue of the subject processed with the at least one preanalytical factor, and a ground truth indicating a destination label; and obtaining an outcome destination image of a slide of pathological tissue of the destination set of image translation records that is a conversion of the abnormally processed target image into a normally processed image.
Klaiman is in the same field of art of image analysis in tissue samples. Further Klaiman teaches into an image correction machine learning model, wherein the image correction machine learning model is trained on a corrected image training dataset (Klaiman ¶0113 "A "training image" as used herein is an image acquired from a training tissue sample. The training images are used for training an untrained version of the MLL for generating a trained MLL that is adapted to transform an acquired tissue image into an output image that highlights a specific biomarker although the tissue sample"),
into an image translation machine learning model, wherein the image translation machine learning model is trained on an image translation training dataset, comprising two or more sets of image translation records, wherein a source image translation record of a source set of image translation records comprises (Klaiman ¶0061 "The image transformation routine is adapted to transform each of the first training images into a virtual staining image that is identical or similar to the one of the second training images having been obtained for the same training tissue sample."):
obtaining an outcome of a corrected image that simulates what the target image of the slide would look like when processed with the at least one preanalytical factor classified as normal (Klaiman ¶0114"Thus, the pixel intensity and color values "simulate" the effect of said particular staining protocol.")
into an image translation machine learning model, wherein the image translation machine learning model is trained on an image translation training dataset, comprising two or more sets of image translation records, wherein a source image translation record of a source set of image translation records comprises (Klaiman ¶0061 "The image transformation routine is adapted to transform each of the first training images into a virtual staining image that is identical or similar to the one of the second training images having been obtained for the same training tissue sample.")
translation record of a destination set of image translation records comprises: a destination image of the slide of pathological tissue of the subject processed with the at least one preanalytical factor, and a ground truth indicating a destination label (Klaiman ¶0061 "The image transformation routine is adapted to transform each of the first training images into a virtual staining image that is identical or similar to the one of the second training images having been obtained for the same training tissue sample."); and obtaining an outcome destination image of a slide of pathological tissue of the destination set of image translation records that is a conversion of the abnormally processed target image into a normally processed image.(Klaiman ¶0061 "Thus, a large number of different image transformation routines can be easily generated by generating a corresponding training data set and training an untrained MLL on said training data set.").
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer in view of Saltz by incorporating image translation into the preprocessing of the dataset that is taught by Klaiman, to make the invention of the machine learning model one that could process multiple different types of slides; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for increased testing of multiple different staining types of slides to reduce cost and burden of testing (Klaiman ¶0015 "This may be advantageous as the method may generate an output image highlighting the presence of a second biomarker although the acquired image used as input depicts a sample that was not stained at all or was stained by one or more first biomarkers adapted to selectively stain one or more respective first biomarkers, but not the second biomarker. Thus, by providing a MLL that was trained on information that is implicitly contained in an acquired image, explicit information on the presence of a second biomarker in a tissue can be obtained without the need of staining the tissue sample with a stain that is adapted to selectively stain the second biomarker. Thus, valuable time and the costs for the stain for selectively staining the second biomarker can be saved.")
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
39. Claims 4-8 are rejected under 35 U.S.C. 103 as being unpatentable over Bauer in view of Saltz in further view of Klaiman (U.S Patent Pub. No 20210005308 A1, hereafter referred to as Klaiman).
40. Regarding Claim 4 Bauer in view of Saltz teaches the computer implemented method of claim 1, further comprising: a source image of the slide of pathological tissue (Bauer ¶0180 discloses using images from 210 slides of tissues) of the subject processed with the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), and a ground truth indicating a source label (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time),
of the slide of pathological tissue (Bauer ¶0180 discloses using images from 210 slides of tissues) of the subject processed with the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), and a ground truth indicating a destination label (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration);
Bauer in view of Saltz does not explicitly disclose creating an image translation training dataset, comprising two or more sets of image translation records, wherein a source image translation record of a source set of image translation records, wherein a destination image translation record of a destination set of image translation, a destination image, training an image translation machine learning model on the image translation training dataset for converting a target source image of a slide of pathological tissue of the source set of image translation records to an outcome destination of a slide of pathological tissue of the destination set of image translation record.
Klaiman is in the same field of art of image analysis in tissue samples. Further Klaiman teaches creating an image translation training dataset, comprising two or more sets of image translation records, wherein a source image translation record of a source set of image translation records (Klaiman ¶0061 "The image transformation routine is adapted to transform each of the first training images into a virtual staining image that is identical or similar to the one of the second training images having been obtained for the same training tissue sample.")
wherein a destination image translation record of a destination set of image translation (Klaiman ¶0083 discloses image to image translation resulting in an output image), a destination image (Klaiman ¶0083 discloses image to image translation resulting in an output image),
training an image translation machine learning model (Klaiman ¶0083 discloses image to image translation MLL resulting in an output image) on the image translation training dataset for converting a target source image of a slide of pathological tissue of the source set of image translation records to an outcome destination of a slide of pathological tissue of the destination set of image translation records (Klaiman ¶0061 "Thus, a large number of different image transformation routines can be easily generated by generating a corresponding training data set and training an untrained MLL on said training data set.").
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer in view of Saltz by incorporating image translation into the preprocessing of the dataset that is taught by Klaiman, to make the invention of the machine learning model one that could process multiple different types of slides; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for increased testing of multiple different staining types of slides to reduce cost and burden of testing (Klaiman ¶0015 "This may be advantageous as the method may generate an output image highlighting the presence of a second biomarker although the acquired image used as input depicts a sample that was not stained at all or was stained by one or more first biomarkers adapted to selectively stain one or more respective first biomarkers, but not the second biomarker. Thus, by providing a MLL that was trained on information that is implicitly contained in an acquired image, explicit information on the presence of a second biomarker in a tissue can be obtained without the need of staining the tissue sample with a stain that is adapted to selectively stain the second biomarker. Thus, valuable time and the costs for the stain for selectively staining the second biomarker can be saved.")
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention
41. Regarding Claim 5, Bauer in view of Saltz in further view of Klaiman teaches the computer implemented method of claim 4, wherein the source label (Bauer ¶0011 discloses labeling the samples with fixation times) indicates pathological tissue abnormally processed (Bauer ¶0003 discloses the presence of abnormalities) with the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), and the destination label indicates pathological tissue normally processed (Bauer ¶0011 discloses labeling the samples with fixation times) with the at least one preanalytical factor , wherein the target source image (Bauer ¶0123, ¶0149, discloses using input output pairs to train the model) comprises at least one of:
(i) an input image (Saltz ¶0247 discloses input image patches) and additional metadata indicating a source preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) that has been abnormally processed (Bauer ¶0003 discloses the presence of abnormalities), and metadata indicating a destination preanalytical factor that has been normally processed (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second), and
(ii) an input image (Saltz ¶0247 discloses input image patches) and further comprising providing a reference image (Bauer ¶0173 discloses using a reference value to judge fixation quality) from the destination set used to infer the destination (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second) of the input image (Saltz ¶0247 discloses input image patches).
See Claim 4 for rationale, its parent claim.
42. Regarding Claim 6, Bauer in view of Saltz in further view of Klaiman teaches the computer implemented method of claim 4, wherein the source set is selected according to an input (Klaiman ¶0061 "Thus, a large number of different image transformation routines can be easily generated by generating a corresponding training data set and training an untrained MLL on said training data set.") of the at least one preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) obtained as the outcome of the preanalytical machine learning model fed the target image(Bauer ¶0149 discloses the engine outputting a qualitative assessment of fixation quality). See Claim 4 for rationale, its parent claim.
43. Regarding Claim 7, Bauer in view of Saltz teaches the computer implemented method of claim 1, further comprising:
the image of the slide of pathological tissue (Bauer ¶0180 discloses using images from 210 slides of tissues) of a subject processed with at least one preanalytical factor, (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), wherein the at least one preanalytical factor is classified as abnormal (Bauer ¶0003 discloses the presence of abnormalities), wherein the image of the slide depicts abnormally processed pathological tissue (Saltz Fig 8F discloses images of slide with abnormal tissue);
the at least one preanalytical factor (Bauer ¶0180 discloses using images from 210 slides of tissues), and a ground truth label indicating a normal image of a slide of pathological tissue (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second) processed with at least one preanalytical factor classified as normal(Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second);
the at least one preanalytical factor (Bauer ¶0180 discloses using images from 210 slides of tissues), and a ground truth label indicating a normal image of a slide of pathological tissue (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second) processed with at least one preanalytical factor classified as normal(Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second);
in response to the target image of the slide processed with at least one target preanalytical factor (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) classified as abnormal (Bauer ¶0003 discloses the presence of abnormalities),
and the preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) are jointly trained (Saltz ¶0249 discloses jointly training the model) using common images and common ground truth labels of preanalytical factors (Bauer ¶0108 discloses training on a second tissue sample up to the nth tissue sample and ¶0139 discloses a second derivative of the training data).
Bauer in view of Saltz does not explicitly disclose creating an image correction training dataset of a plurality of records, wherein an image correction record comprises, and
training an image correction machine learning model on the image correction training dataset for generating an outcome of a synthesized corrected image of a slide of pathological tissue that simulates what a target image of the slide would look like when processed with the at least one preanalytical factor classified as normal, wherein the image correction machine learning model.
Klaiman is in the same field of art of image analysis in tissue samples. Further Klaiman teaches creating an image correction training dataset of a plurality of records, wherein an image correction record comprises (Klaiman ¶0089 "This optimization process can be implemented as a process of minimizing the number of events when the discriminators DG, DF correctly identify an image generated by the generators GG, GF as "fake"/"simulated" rather than "acquired".), and training an image correction machine learning model on the image correction training dataset for generating an outcome of a synthesized corrected image of a slide of pathological tissue that simulates what a target image of the slide would look like when processed with the at least one preanalytical factor classified as normal (Klaiman ¶0114" A "virtual staining image" is an image that is not captured by an image acquisition system, but is rather generated computationally de nova or by transforming an acquired image of a tissue sample into a new image. The new image looks like an image of a tissue sample having been stained according to a particular protocol although the tissue sample depicted in the acquired image from which the virtual staining image is derived, if any, was not stained in accordance with said protocol.")wherein the image correction machine learning model (Klaiman ¶0089 "This optimization process can be implemented as a process of minimizing the number of events when the discriminators DG, DF correctly identify an image generated by the generators GG, GF as "fake"/"simulated" rather than "acquired").
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer in view of Saltz by incorporating image translation into the preprocessing of the dataset that is taught by Klaiman, to make the invention of the machine learning model one that could process multiple different types of slides; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for increased testing of multiple different staining types of slides to reduce cost and burden of testing (Klaiman ¶0015 "This may be advantageous as the method may generate an output image highlighting the presence of a second biomarker although the acquired image used as input depicts a sample that was not stained at all or was stained by one or more first biomarkers adapted to selectively stain one or more respective first biomarkers, but not the second biomarker. Thus, by providing a MLL that was trained on information that is implicitly contained in an acquired image, explicit information on the presence of a second biomarker in a tissue can be obtained without the need of staining the tissue sample with a stain that is adapted to selectively stain the second biomarker. Thus, valuable time and the costs for the stain for selectively staining the second biomarker can be saved.")
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention
44. Regarding Claim 8, Bauer in view of Saltz in further view of Klaiman teaches the computer implemented method of claim 7, wherein the input of the at least one preanalytical factor fed into the image correction machine learning model (Klaiman ¶0114" A "virtual staining image" is an image that is not captured by an image acquisition system, but is rather generated computationally de nova or by transforming an acquired image of a tissue sample into a new image. The new image looks like an image of a tissue sample having been stained according to a particular protocol although the tissue sample depicted in the acquired image from which the virtual staining image is derived, if any, was not stained in accordance with said protocol.") is obtained as the outcome of the preanalytical machine learning model fed the target image (Bauer ¶0149 discloses the engine outputting a qualitative assessment of fixation quality). See Claim 7 for rationale, its parent claim.
45. Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Bauer in view of Saltz in further view of Madabhushi et al (US Patent Pub 2021/0241178 A1, hereafter referred to as Madabhushi).
46. Regarding Claim 14 Bauer in view of Saltz teaches the computer implemented method of claim 1, is fed (Bauer ¶0133 discloses feeding data to the neural network) into the preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) trained on the preanalytical training dataset (Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset).
Bauer in view of Saltz does not explicitly teach for each preanalytical record, feeding the image into a red blood cell (RBC) segmentation machine learning model to obtain an outcome of a segmentation of (RBC) in the image and/or patches that depict RBCs, wherein the image of the preanalytical record comprises the segmentations of RBC and/or patches that depict RBCs.
Madabhusi is in the same field of image analysis using machine learning in tissue samples. Further Madabhusi teaches further comprising, for each preanalytical record, feeding the image into a red blood cell (RBC) segmentation machine learning model to obtain an outcome of a segmentation of (RBC) in the image and/or patches that depict RBCs (Madabhushi et al. ¶0095" The first training step aimed to segment all WBCs against background, red blood cells, and artifacts. This step leveraged the similar appearance of blasts and other WBCs to produce a segmentation of candidate objects."), wherein the image of the preanalytical record comprises the segmentations of RBC and/or patches that depict RBCs, (Madabhushi et al. ¶0112 "CNN based segmentation methods are mainly divided into two types according to the segmentation process: a segmentation method based on patch-wise classification and an end-to-end segmentation method based on semantic segmentation.") and wherein a target segmentation of RBC and/or patches that depict RBC from the target image (Madabhushi et al. ¶0122 discloses the target image having the smallest error in distribution).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer in view of Saltz by incorporating a blood cell (RBC) segmentation machine learning model that is taught by Madabhusi, to make the invention of the machine learning model one that could specifically process red blood cells in the slides; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for increased segmentation of the red bloods cells to help with diagnosis efficiency. (Madabhushi et al. ¶0152 "This shows that the segmentation system can provide a very visual presentation for clinical application, which can assist clinicians in improving diagnosis efficiency.").
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
47. Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Bauer in view of Saltz in further view of Singh et al. Singh, A., Sengupta, S., & Lakshminarayanan, V. (2020). Explainable deep learning models in medical image analysis. Journal of imaging, 6(6), 52., hereafter referred to as Singh).
48. Regarding Claim 16 Bauer in view of Saltz teaches the computer implemented method of claim 1, to obtaining the at least one target preanalytical factor (Bauer ¶0149 discloses the engine outputting a qualitative assessment of fixation quality), wherein the target image is at low resolution (Saltz ¶0177 discloses the images being in a lower resolution for segmentation), and further comprising sampling a plurality of high resolution patches of the target image (Saltz ¶116 discloses better results with high resolution images), and feeding (Bauer ¶0133 discloses feeding data to the neural network) the plurality of high resolution patches (Saltz ¶116 discloses better results with high resolution images) into the preanalytical machine learning model (Bauer ¶0097 discloses a computer implemented method) to obtain the at least one target preanalytical factor (Bauer ¶0149 discloses the engine outputting a qualitative assessment of fixation quality).
Bauer in view of Saltz does not explicitly disclose further comprising training an interpretability machine learning model to generate an interpretability map indicating relative significance of pixels of the target image.
Singh is in the same field of art as image analysis in the medical field. Further Singh teaches further comprising training an interpretability machine learning model (Singh et al. Section 2.3 "Interpretability methods, integrated in the model itself, are called as in-model methods. Some methods are implemented after building model and hence these methods are termed as post model and these methods can potentially develop meaningful insights about what exactly a model learned during the training.") to generate an interpretability map indicating relative significance of pixels of the target image (Singh et al. Introduction "The terms explainability and interpretability are often used interchangeably in the literature. A distinction between these was provided in [6] where interpretation was defined as mapping an abstract concept like the output class into a domain example, while explanation was defined as a set of domain features such as pixels of an image the contribute to the output decision of the model.").
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer in view of Saltz by incorporating an interpretability model and an interpretability map that is taught by Singh, to make the invention of the machine learning model have a more easily understandable output; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need better understanding of the output to increase productivity and accuracy of diagnosis. (Singh et al. Discussion "This can simulate the diagnostic workflow of a clinician where both images and physical parameters of a patient are used to make a decision. This can potentially improve accuracy as well as explain the phenomena more comprehensively.").
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
49. Claim 17 and 26 is rejected under 35 U.S.C. 103 as being unpatentable over Bauer in view of Saltz in further view of Lovborg et al. (WO 2010/066252 A1, hereafter referred to as Lovborg).
50. Regarding Claim 17 Bauer in view of Saltz teaches the computer implemented method of claim 1.
Bauer in view of Saltz does not explicitly teach wherein the at least one preanalytical factor is selected from a group consisting of: an indication of a quality of a stain of the pathological tissue of the slide, fixation time, tissue thickness obtained by sectioning of the FFPE block, fixative type, warm ischemic time, cold ischemic time, duration and delay of temperature during prefixation, fixative formula, fixative concentration, fixative pH, fixative age of reagent, fixative preparation source, tissue to fixative volume ratio, method of fixation, conditions of primary and secondary fixation, post fixation washing conditions and duration, post fixation storage reagent and duration, type of processor, frequency of servicing and reagent replacement, tissue to reagent volume ratio, number of position of co-processed specimens, dehydration and clearing reagent, dehydration and clearing temperature, dehydration and clearing number of changes, dehydration clearing duration, baking time, and temperature.
Lovberg is in the same field of analysis of tissue samples. Further, Lovborg teaches wherein the at least one of preanalytical factor is selected from a group consisting of: an indication of a quality of a stain of the pathological tissue of the slide, fixation time (Lovborg et al. Pg 24 Line 10 "fixation time and the one or more antibody that demonstrates variations in antigen accessibility or variations in"), tissue thickness obtained by sectioning of the FFPE block, fixative type, (Lovborg et al. Pg. 24 Lines 28-30 "Formalin fixation is the pre-treatment where the tissue is fixed by formalin. During the fixation process the tissue is affected. The fixation stops the degradation process at the time point of fixation and is thus dependent on the thickness of the tissue which influences the time of fixative,") warm ischemic time, cold ischemic time,(Lovborg et al. Pg 40 Line 2 "variation in tissue pre-treatment is ischemic time") duration and delay of temperature during prefixation, fixative formula, fixative concentration, fixative pH, (Lovborg et al. Pg 34 Line 29 "time periods for reagent/sample and mixtures, temperature, buffer conditions and the like.") fixative age of reagent, fixative preparation source, tissue to fixative volume ratio, method of fixation, (Lovborg et al. Pg 34 Lines 25-29 "Such instructions typically include a tangible expression describing reagent concentrations and/or at least one assay method parameter such as the relative amounts of reagent and sample to be mixed, maintenance time periods for reagent/sample admixtures, temperature, buffer conditions and the like.") conditions of primary and secondary fixation,(Lovborg et al. Pg. 29 Lines 20-21 "as given above for primary antibodies of the same species. Processes using secondary or further antibodies are also enclosed.") post fixation washing conditions and duration, (Lovborg et al. Pg. 19 Lines 9-10 "The secondary immune complexes can then be generally washed to remove any non-specifically 10 bound labelled secondary antibodies,") postfixation storage reagent and duration, (Lovborg et al. Pg 34 Line 29 "time periods for reagent/sample and mixtures, temperature, buffer conditions and the like.") type of processor (Lovborg, Pg 35 Lines 26-27 "NBF before the ethanol dehydration step and paraffin infiltration in a
tissue processor (Shandon Tissue Processor ) .") frequency of servicing and reagent replacement, tissue to reagent volume ratio, number of position of co-processed specimens, dehydration and clearing reagent, dehydration and clearing temperature, dehydration and clearing number of changes, dehydration clearing duration, baking time, and temperature.(Lovborg et al. Pg 34 Lines 25-29 "Such instructions typically include a tangible expression describing reagent concentrations and/or at least one assay method parameter such as the relative amounts of reagent and sample to be mixed, maintenance time periods for reagent/sample admixtures, temperature, buffer conditions and the like.").
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer in view of Saltz by incorporating the preanalytical factors that affect the tissue stains including temperature and ischemic time that is taught by Lovborg, to make the invention of the machine learning model one that could specifically identify the preanalytical factor that effected the stain; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for standardization on how the tissue is handled before analysis. (Lovborg et al. Pg 11-12, Lines 33-2 "The present invention counteracts the lack of standardization and lack of knowledge of the treatment of individual tissues and enables testing of the pre-analytical parameters and provides guidance for which changes may have impact on the analysis. With the present invention it will be possible to realize if a specimen pre-treatment was correct with respect to the following immunohistochemical process, and, hence, whether results from a specific IHC test conducted on said tissue will be correct or not. Finally, guidance can be given to describe measures to obtain a correct result.").
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
51. Regarding Claim 26 Bauer in view of Saltz teaches the computer implemented method of claim 22, wherein the at least one preanalytical factor (Bauer ¶0180 discloses using images from 210 slides of tissues) comprises a fixation time (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including tissue thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), and additionally comprises a tissue thickness (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including tissue thickness, sample moisture and temperature).
Bauer in view of Saltz do not disclose or a warm ischemic time.
Lovborg is in the same field of analysis of tissue samples. Further, Lovborg teaches or a warm ischemic time (Lovborg Pg 2 Lines 15-20 disclose ischemic time as a preanalytical factor Pg 32 Lines 28-32 and Pg 22 lines 15-21 disclose the retrieval temperature and how it affects the sample).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer in view of Saltz by incorporating the preanalytical factors that affect the tissue stains including temperature and ischemic time that is taught by Lovborg, to make the invention of the machine learning model one that could specifically identify the preanalytical factor that effected the stain; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for standardization on how the tissue is handled before analysis. (Lovborg et al. Pg 11-12, Lines 33-2 "The present invention counteracts the lack of standardization and lack of knowledge of the treatment of individual tissues and enables testing of the pre-analytical parameters and provides guidance for which changes may have impact on the analysis. With the present invention it will be possible to realize if a specimen pre-treatment was correct with respect to the following immunohistochemical process, and, hence, whether results from a specific IHC test conducted on said tissue will be correct or not. Finally, guidance can be given to describe measures to obtain a correct result.").
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claims 23-24 are rejected under 35 U.S.C. 103 as being unpatentable over Bauer et al (WO 2021/037875 A1, hereafter referred to as Bauer) in view of Klaiman (U.S Patent Pub. No 20210005308 A1, hereafter referred to as Klaiman) in further view of in view of Lovborg et al. (WO 2010/066252 A1, hereafter referred to as Lovborg).
Regarding Claim 23 Bauer teaches comprises the image of the slide of pathological tissue (Bauer ¶0180 discloses using images from 210 slides of tissues) of a subject processed with at least one preanalytical factor, (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), wherein the at least one preanalytical factor is classified as abnormal (Bauer ¶0003 discloses the presence of abnormalities),
the at least one preanalytical factor (Bauer ¶0180 discloses using images from 210 slides of tissues), and a ground truth label indicating a normal image of a slide of pathological tissue (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second) processed with at least one preanalytical factor classified as normal(Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time being the first and fixation quality being the second) ; and
wherein the at least one preanalytical factor (Bauer ¶0180 discloses using images from 210 slides of tissues) comprises a fixation time (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including tissue thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), and additionally comprises a tissue thickness (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including tissue thickness, sample moisture and temperature).
Bauer does not explicitly disclose a computer implemented method of training an image correction machine learning model comprising:
creating an image correction training dataset of a plurality of records, wherein an image correction record
wherein the image of the slide depicts abnormally processed pathological tissue
training an image correction machine learning model on the image correction training dataset for generating an outcome of a synthesized corrected image of a slide of pathological tissue that simulates what a target image of the slide would look like when processed with the at least one preanalytical factor classified as normal.
Klaiman is in the same field of art of image analysis in tissue samples. Further Klaiman teaches a computer implemented method of training an image correction machine learning model (Klaiman ¶0089 "This optimization process can be implemented as a process of minimizing the number of events when the discriminators DG, DF correctly identify an image generated by the generators GG, GF as "fake"/"simulated" rather than "acquired".) comprising:
creating an image correction training dataset of a plurality of records, wherein an image correction record (Klaiman ¶0089 "This optimization process can be implemented as a process of minimizing the number of events when the discriminators DG, DF correctly identify an image generated by the generators GG, GF as "fake"/"simulated" rather than "acquired") wherein the image of the slide depicts abnormally processed pathological tissue (Klaiman Fig 2F discloses images of slide with abnormal tissue) ;
training an image correction machine learning model on the image correction training dataset for generating an outcome of a synthesized corrected image of a slide of pathological tissue that simulates what a target image of the slide would look like when processed with the at least one preanalytical factor classified as normal (Klaiman ¶0114" A "virtual staining image" is an image that is not captured by an image acquisition system, but is rather generated computationally de nova or by transforming an acquired image of a tissue sample into a new image. The new image looks like an image of a tissue sample having been stained according to a particular protocol although the tissue sample depicted in the acquired image from which the virtual staining image is derived, if any, was not stained in accordance with said protocol.");
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer in view of Saltz by incorporating image translation into the preprocessing of the dataset that is taught by Klaiman, to make the invention of the machine learning model one that could process multiple different types of slides; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for increased testing of multiple different staining types of slides to reduce cost and burden of testing (Klaiman ¶0015 "This may be advantageous as the method may generate an output image highlighting the presence of a second biomarker although the acquired image used as input depicts a sample that was not stained at all or was stained by one or more first biomarkers adapted to selectively stain one or more respective first biomarkers, but not the second biomarker. Thus, by providing a MLL that was trained on information that is implicitly contained in an acquired image, explicit information on the presence of a second biomarker in a tissue can be obtained without the need of staining the tissue sample with a stain that is adapted to selectively stain the second biomarker. Thus, valuable time and the costs for the stain for selectively staining the second biomarker can be saved.")
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Bauer and Klaiman in combination do not explicitly teach or a warm ischemic time.
Lovborg is in the same field of analysis of tissue samples. Further, Lovborg teaches or a warm ischemic time (Lovborg Pg 2 Lines 15-20 disclose ischemic time as a preanalytical factor Pg 32 Lines 28-32 and Pg 22 lines 15-21 disclose the retrieval temperature and how it affects the sample).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer in view of Saltz by incorporating the preanalytical factors that affect the tissue stains that is taught by Lovborg, to make the invention of the machine learning model one that could specifically identify the preanalytical factor that effected the stain; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for standardization on how the tissue is handled before analysis. (Lovborg et al. Pg 11-12, Lines 33-2 "The present invention counteracts the lack of standardization and lack of knowledge of the treatment of individual tissues and enables testing of the pre-analytical parameters and provides guidance for which changes may have impact on the analysis. With the present invention it will be possible to realize if a specimen pre-treatment was correct with respect to the following immunohistochemical process, and, hence, whether results from a specific IHC test conducted on said tissue will be correct or not. Finally, guidance can be given to describe measures to obtain a correct result.").
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 24 Bauer in view of Klaiman in view of Lovborg discloses the computer implemented method of claim 23, wherein the at least one preanalytical factor (Bauer ¶0180 discloses using images from 210 slides of tissues) further comprises a method of fixation (Bauer ¶0178 discloses formalin fixation and how it affects the sample). See rationale for Claim 23 its parent claim.
Claim 25 is rejected under 35 U.S.C. 103 as being unpatentable over Bauer et al (WO 2021/037875 A1, hereafter referred to as Bauer) in view of Saltz et al (U.S Patent Pub. No 2020/0388029, hereafter referred to as Saltz) in further view of in view of Lovborg et al. (WO 2010/066252 A1, hereafter referred to as Lovborg).
Regarding Claim 25, Bauer teaches a computer implemented method (Bauer ¶0097 discloses a computer implemented method) of training a preanalytical factor machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets), comprising:
creating a preanalytical training dataset of a plurality of records, (Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset), wherein a preanalytical record comprises an image of a slide of pathological tissue (Bauer ¶0180 discloses using images from 210 slides of tissues) of a subject processed with at least one preanalytical factor, (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration) and a ground truth label indicating the at least one preanalytical factor (Bauer ¶0084, ¶0109 discloses a ground truth labeling on the fixation time); and
training the preanalytical machine learning model (Bauer ¶0011 discloses a fixation estimation engine trained using training datasets) on the preanalytical training dataset (Bauer ¶0125 discloses using principle component analysis to create a dataset that retains variation but reduces dimensionality in the variables of the dataset, ¶0180 discloses 210 slides being used as the dataset) for generating an outcome of at least one target preanalytical factor (Bauer ¶0149 discloses the engine outputting a qualitative assessment of fixation quality) used to process tissue depicted in a target image in response to the input of the target image (Bauer ¶0123, ¶0149, discloses using input output pairs to train the model); and
wherein the at least one preanalytical factor (Bauer ¶0180 discloses using images from 210 slides of tissues) comprises a fixation time (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including tissue thickness, sample moisture and temperature, ¶0083 discloses fixation quality and duration), and additionally comprises a tissue thickness (Bauer ¶0004 discloses the tissues sections used were prepared with preanalytical factors including tissue thickness, sample moisture and temperature).
Bauer does not explicitly disclose for each preanalytical record, feeding the image into a nuclear segmentation machine learning model to obtain an outcome of a segmentation of nuclei in the image, creating a mask that masks out pixels external to the segmentation of the nuclei based on the outcome of the segmentation, and applying the mask to the image to create a masked image, wherein the image of the preanalytical record comprises the masked image, and wherein a target masked image created from the target image is fed into the preanalytical machine learning model trained on the preanalytical training dataset.
Saltz is in the same field of are of image analysis of tissues samples. Further, Saltz teaches for each preanalytical record, feeding the image into a nuclear segmentation machine learning model to obtain an outcome of a segmentation of nuclei in the image, (Saltz et al. ¶0291 "As shown in the workflow 280, nuclear segmentation is implemented at the outset in step 281 for proper quality segmentation of nuclei in tumoral tissue."), creating a mask that masks out pixels external to the segmentation of the nuclei based on the outcome of the segmentation(Saltz et al. ¶0169 "The training set of the necrosis segmentation CNN consisted of 1,800 patches. Each patch was annotated with a necrosis region mask segmented by a pathologist. Sample were 2480 patches to create the test dataset.") , and applying the mask to the image to create a masked image, wherein the image of the preanalytical record comprises the masked image, and wherein a target masked image created from the target image is fed into the preanalytical machine learning model trained on the preanalytical training dataset (Saltz et al. ¶0169 "The training set of the necrosis segmentation CNN consisted of 1,800 patches. Each patch was annotated with a necrosis region mask segmented by a pathologist. Sample were 2480 patches to create the test dataset.").
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer by incorporating the anomalous application of preanalytical factors, input image patches, jointly and supervised training the model, nuclear segmentation machine, color normalization and resolution adjusting, and the feature map extracted from a hidden layer that is taught by Saltz, to make an invention that analyzes the tissues using multiple kinds of models and forms of inputs for maximum flexibility but also outputs a feature map to identify elements of the tissues to improve the training of the model; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for increased accuracy of the machine learning model to improve the end result of feature identification in the slides (Saltz et al. ¶0005 "Complex segmentation of nuclei in whole slide tissue images, is considered a common methodology in pathology image analysis and quality control of such algorithms are being implemented to improve segmentation results").
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Bauer and Saltz in combination do not explicitly teach or a warm ischemic time.
Lovborg is in the same field of analysis of tissue samples. Further, Lovborg teaches or a warm ischemic time (Lovborg Pg 2 Lines 15-20 disclose ischemic time as a preanalytical factor Pg 32 Lines 28-32 and Pg 22 lines 15-21 disclose the retrieval temperature and how it affects the sample).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bauer in view of Saltz by incorporating the preanalytical factors that affect the tissue stains that is taught by Lovborg, to make the invention of the machine learning model one that could specifically identify the preanalytical factor that effected the stain; thus, one of ordinary skilled in the art would be motivated to combine the references since there is a need for standardization on how the tissue is handled before analysis. (Lovborg et al. Pg 11-12, Lines 33-2 "The present invention counteracts the lack of standardization and lack of knowledge of the treatment of individual tissues and enables testing of the pre-analytical parameters and provides guidance for which changes may have impact on the analysis. With the present invention it will be possible to realize if a specimen pre-treatment was correct with respect to the following immunohistochemical process, and, hence, whether results from a specific IHC test conducted on said tissue will be correct or not. Finally, guidance can be given to describe measures to obtain a correct result.").
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Conclusion
56. Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHEL LYNN ROBERTS whose telephone number is (571)272-6413. The examiner can normally be reached Monday- Friday 7:30am- 5:00pm.
57. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
58. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mistry Oneal can be reached on (313) 446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
59. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RACHEL L ROBERTS/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674