DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 12/13/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4, 11-13, 16 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Van Rijthoven et al. ("HookNet: Multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images") in view of Italiano et al. US PG-Pub (US 20230296606 A1).
Regarding Claim 1, Van Rijthoven teaches a method comprising, by a digital pathology image processing system: accessing a digital pathology image that depicts a tissue sample from a subject under a treatment(Fig. 3. shows pathological images of tissues were used as input to a dual encoder-decoder model); detecting, based on a machine-learning model, one or more tertiary lymphoid structures depicted within the digital pathology image of the tissue sample (Page 3, 1.4. Our Contribution, Last Paragraph, “this work contributes with two multi-class and multi-organ segmentation models addressing problems where tissue is subjected to high-resolution details and context. The first application focuses on DCIS, IDC, and ILC segmentation in histopathology breast tissue. The second application focuses on the segmentation of GC, TLS, and tumor in lung tissue. To the best of our knowledge, these tissue types have never been simultaneously and separately segmented with a single model.” As disclosed in this section of the prior art, a machine learning model is used to segment the tertiary lymphoid structure in the tissue image and Fig. 7 shows segmentation results on lung tissue for TLS and GC, Tumor, and Other.); determining, for the one or more detected tertiary lymphoid structures, descriptive information associated with the detected tertiary lymphoid structures (Page 2, Left Col, Paragraph 1, “A GC always lays within a TLS. A TLS contains a high density of lymphocytes with poorly visible cytoplasm, while GCs rather share similarities with other less dense tissues like tumor nests. To identify the TLS region and to differentiate between TLS and GC, both finegrained details, as well as contextual information, are needed.”, in this section of the prior art a germinal center (GC) needs to be identified in order to identify the TLS region from the GC and contextual information with fine-grained details are used to differentiate these regions.),
Van Rijthoven does not explicitly teach wherein the descriptive information comprises at least a maturation state associated with each of the detected tertiary lymphoid structures; and determining, based on the detected tertiary lymphoid structures and the descriptive information, an outcome of the subject in response to the treatment.
Italiano teaches wherein the descriptive information comprises at least a maturation state associated with each of the detected tertiary lymphoid structures(¶[0080]-¶[0082] determining the maturity status of TLS)and determining, based on the detected tertiary lymphoid structures and the descriptive information, an outcome of the subject in response to the treatment. (See paragraphs 0075, 0111-0118, 0121 where the maturity of the TLS is used to determine an outcome of the subject in response to the treatment; also see the abstract and claim 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven with Italiano in order to determine the maturation state of the TLS and use the information to predict an outcome in response to the treatment. One skilled in the art would have been motivated to modify Van Rijthoven in this manner in order to determine the appropriate therapeutic modalities for a given cancer type, to increase patient survival and quality of life. (Italiano, ¶[0002])
Regarding Claim 4, the combination of Van Rijthoven and Italiano teach the method of claim 1, Italiano further teaches wherein each of the one or more detected tertiary lymphoid structures is associated with a numeric representation generated by the machine-learning model, and wherein the method further comprises: determining the maturation state associated with each of the detected tertiary lymphoid structures based on their respective numeric representations. (¶[0079] discloses determining a number of TLS structures and ¶[0081] discloses determining a maturity stage based on TLS.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven with Italiano in order to determine the maturation state of the TLS based on the number of structures. One skilled in the art would have been motivated to modify Van Rijthoven in this manner in order to determine the appropriate therapeutic modalities for a given cancer type, to increase patient survival and quality of life. (Italiano, ¶[0002])
Regarding Claim 11, the combination of Van Rijthoven and Italiano teach the method of claim 1, Italiano further teaches wherein the tissue sample is associated with one or more tumors, and wherein the descriptive information further comprises one or more of:a number of the detected tertiary lymphoid structures; a number of detected tertiary lymphoid structures that are associated with image markers; a number of detected tertiary lymphoid structures outside a tumor region; a ratio between a number of detected tertiary lymphoid structures located inside the tumor region and a number of detected tertiary lymphoid structures located outside the tumor region; an average distance of the detected tertiary lymphoid structures to the tumor region or a boundary thereof; a size of each of the detected tertiary lymphoid structures; a percentage of a total tissue sample area that comprises the detected tertiary lymphoid structures; or an average distance between any given pair of detected tertiary lymphoid structures (¶[0079] discloses determining a number of TLS structures and ¶[0081] discloses determining a maturity stage based on TLS.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven with Italiano in order to determine the maturation state of the TLS based on the number of structures. One skilled in the art would have been motivated to modify Van Rijthoven in this manner in order to determine the appropriate therapeutic modalities for a given cancer type, to increase patient survival and quality of life. (Italiano, ¶[0002])
Regarding Claim 12, the combination of Van Rijthoven and Italiano teach the method of claim 1, where Van Rijthoven further teaches wherein the machine-learning model is trained based on a plurality of training data, wherein each training data point comprises a slide image of a tissue sample and a corresponding annotation of tertiary lymphoid structures identified within that tissue sample. (Page 3, 2.2. Lung Dataset, “We randomly selected 27 diagnostic H&E-stained digital slides from the cancer genome atlas lung squamous cell carcinoma (TCGA-LUSC) data collection, which is publicly available in genomic data commons (GDC) Data Portal (Grossman et al., 2016). For this dataset, sparse annotations of TLS, GC, tumor, and other lung parenchyma were made by a senior researcher (KS) with more than six years of experience in tumor immunology and histopathology, and checked by a resident pathologist (MB)”, in this section of the prior art, the TLS in the slide image is annotated.)
Regarding Claim 13, the combination of Van Rijthoven and Italiano teach the method of claim 12, where Van Rijthoven further teaches further comprising training the machine-learning model, wherein the training comprises: applying one or more data augmentations to each slide image of a training data set, wherein the one or more data augmentations are based on one or more of brightness, hue, saturation, cropping, clipping, flipping, rotation, or a mean pixel density in a color channel. (Page 5, 3.6. Model Training setup, “To increase the variation of the datasets and account for color changes induced by the variability of staining, we applied spatial, color, noise and stain augmentations”, in this section of the prior art, data augmentation is performed to increase the diversity of the training data.)
Regarding Claim 16, Van Rijthoven teaches One or more computer-readable non-transitory storage media embodying software(Page 8, Right Col, paragraph 1, “we limited HookNet, as well as models used in comparison to 50 million parameters, which allow model training using a single modern GPU with 11 GB of RAM.”, discloses the neural network is trained using a single GPU with 11 GB of RAM.) that is operable when executed to: access a digital pathology image that depicts a tissue sample from a subject under a treatment(Fig. 3. shows pathological images of tissues were used as input to a dual encoder-decoder model); detect, based on a machine-learning model, one or more tertiary lymphoid structures depicted within the digital pathology image of the tissue sample (Page 3, 1.4. Our Contribution, Last Paragraph, “this work contributes with two multi-class and multi-organ segmentation models addressing problems where tissue is subjected to high-resolution details and context. The first application focuses on DCIS, IDC, and ILC segmentation in histopathology breast tissue. The second application focuses on the segmentation of GC, TLS, and tumor in lung tissue. To the best of our knowledge, these tissue types have never been simultaneously and separately segmented with a single model.” As disclosed in this section of the prior art, a machine learning model is used to segment the tertiary lymphoid structure in the tissue image and Fig. 7 shows segmentation results on lung tissue for TLS and GC, Tumor, and Other.); determine, for the one or more detected tertiary lymphoid structures, descriptive information associated with the detected tertiary lymphoid structures (Page 2, Left Col, Paragraph 1, “A GC always lays within a TLS. A TLS contains a high density of lymphocytes with poorly visible cytoplasm, while GCs rather share similarities with other less dense tissues like tumor nests. To identify the TLS region and to differentiate between TLS and GC, both finegrained details, as well as contextual information, are needed.”, in this section of the prior art a germinal center (GC) needs to be identified in order to identify the TLS region from the GC and contextual information with fine-grained details are used to differentiate these regions.),
Van Rijthoven does not explicitly teach wherein the descriptive information comprises at least a maturation state associated with each of the detected tertiary lymphoid structures; and determining, based on the detected tertiary lymphoid structures and the descriptive information, an outcome of the subject in response to the treatment.
Italiano teaches wherein the descriptive information comprises at least a maturation state associated with each of the detected tertiary lymphoid structures(¶[0080]-¶[0082] determining the maturity status of TLS)and determining, based on the detected tertiary lymphoid structures and the descriptive information, an outcome of the subject in response to the treatment. (See paragraphs 0075, 0111-0118, 0121 where the maturity of the TLS is used to determine an outcome of the subject in response to the treatment; also see the abstract and claim 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven with Italiano in order to determine the maturation state of the TLS and use the information to predict an outcome in response to the treatment. One skilled in the art would have been motivated to modify Van Rijthoven in this manner in order to determine the appropriate therapeutic modalities for a given cancer type, to increase patient survival and quality of life. (Italiano, ¶[0002])
Regarding Claim 19, Van Rijthoven teaches a system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors(Page 8, Right Col, paragraph 1, “we limited HookNet, as well as models used in comparison to 50 million parameters, which allow model training using a single modern GPU with 11 GB of RAM.”, discloses the neural network is trained using a single GPU with 11 GB of RAM.), the processors operable when executing the instructions to:access a digital pathology image that depicts a tissue sample from a subject under a treatment(Fig. 3. shows pathological images of tissues were used as input to a dual encoder-decoder model); detect, based on a machine-learning model, one or more tertiary lymphoid structures depicted within the digital pathology image of the tissue sample (Page 3, 1.4. Our Contribution, Last Paragraph, “this work contributes with two multi-class and multi-organ segmentation models addressing problems where tissue is subjected to high-resolution details and context. The first application focuses on DCIS, IDC, and ILC segmentation in histopathology breast tissue. The second application focuses on the segmentation of GC, TLS, and tumor in lung tissue. To the best of our knowledge, these tissue types have never been simultaneously and separately segmented with a single model.” As disclosed in this section of the prior art, a machine learning model is used to segment the tertiary lymphoid structure in the tissue image and Fig. 7 shows segmentation results on lung tissue for TLS and GC, Tumor, and Other.); determine, for the one or more detected tertiary lymphoid structures, descriptive information associated with the detected tertiary lymphoid structures (Page 2, Left Col, Paragraph 1, “A GC always lays within a TLS. A TLS contains a high density of lymphocytes with poorly visible cytoplasm, while GCs rather share similarities with other less dense tissues like tumor nests. To identify the TLS region and to differentiate between TLS and GC, both finegrained details, as well as contextual information, are needed.”, in this section of the prior art a germinal center (GC) needs to be identified in order to identify the TLS region from the GC and contextual information with fine-grained details are used to differentiate these regions.),
Van Rijthoven does not explicitly teach wherein the descriptive information comprises at least a maturation state associated with each of the detected tertiary lymphoid structures; and determining, based on the detected tertiary lymphoid structures and the descriptive information, an outcome of the subject in response to the treatment.
Italiano teaches wherein the descriptive information comprises at least a maturation state associated with each of the detected tertiary lymphoid structures(¶[0080]-¶[0082] determining the maturity status of TLS)and determining, based on the detected tertiary lymphoid structures and the descriptive information, an outcome of the subject in response to the treatment. (See paragraphs 0075, 0111-0118, 0121 where the maturity of the TLS is used to determine an outcome of the subject in response to the treatment; also see the abstract and claim 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven with Italiano in order to determine the maturation state of the TLS and use the information to predict an outcome in response to the treatment. One skilled in the art would have been motivated to modify Van Rijthoven in this manner in order to determine the appropriate therapeutic modalities for a given cancer type, to increase patient survival and quality of life. (Italiano, ¶[0002])
Claims 2, 17 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Van Rijthoven et al. ("HookNet: Multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images") in view of Italiano et al. ("Tertiary lymphoid structures in the era of cancer immunotherapy") in view of Stumpe et al. US PG-Pub(US 20220296925 A1).
Regarding Claim 2, while the combination of Van Rijthoven and Italiano teaches the method of claim 1, where Van Rijthoven further teaches further comprising: identifying, for each of the one or more tertiary lymphoid structures, an area comprising the tertiary lymphoid structure(Page 2, Left Col, Paragraph 1, “to identify the presence of duct (both healthy and potentially cancerous) and other tissue structures. Additionally, they zoom-in into each region of interest, where the tissue is examined at a high-resolution, to obtain the details of the cancer cells, and characterize the tumor based on its local cellular composition.”, in this section of the prior art, cancerous regions are zoom-in to to determine potential TLS regions.);
However, Van Rijthoven and Italiano do not explicitly teach providing instructions for outlining the area comprising the tertiary lymphoid structure in the digital pathology image, wherein the outlining comprises a polygonal shape.
Stumpe teaches providing instructions for outlining the area comprising the tertiary lymphoid structure in the digital pathology image, wherein the outlining comprises a polygonal shape. (¶[0010],” The workstation is configured with either (a) user interface tools by which an operator inspecting the registered digital magnified images on the display may annotate a digital magnified image of the tissue specimen stained with the staining agent so as to form a closed polygon or other shape to mark a region of said image containing region of interest (e.g., tumor cells) to thereby create a mask.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven and Italiano with Stumpe in order to outline the cell of interest with a polygonal shape. One skilled in the art would have been motivated to modify Van Rijthoven and Italiano in this manner in order to identify regions of interest within novel unstained images. (Stumpe, Abstract)
Regarding Claim 17, while the combination of Van Rijthoven and Italiano teach the media of claim 16, where Van Rijthoven further teaches wherein the software is further operable when executed to: identify, for each of the one or more tertiary lymphoid structures, an area comprising the tertiary lymphoid structure (Page 2, Left Col, Paragraph 1, “to identify the presence of duct (both healthy and potentially cancerous) and other tissue structures. Additionally, they zoom-in into each region of interest, where the tissue is examined at a high-resolution, to obtain the details of the cancer cells, and characterize the tumor based on its local cellular composition.”, in this section of the prior art, cancerous regions are zoom-in to to determine potential TLS regions.);
However, Van Rijthoven and Italiano do not explicitly teach provide instructions for outlining the area comprising the tertiary lymphoid structure in the digital pathology image, wherein the outlining comprises a polygonal shape.
Stumpe teaches provide instructions for outlining the area comprising the tertiary lymphoid structure in the digital pathology image, wherein the outlining comprises a polygonal shape. (¶[0010],” The workstation is configured with either (a) user interface tools by which an operator inspecting the registered digital magnified images on the display may annotate a digital magnified image of the tissue specimen stained with the staining agent so as to form a closed polygon or other shape to mark a region of said image containing region of interest (e.g., tumor cells) to thereby create a mask.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven and Italiano with Stumpe in order to outline the cell of interest with a polygonal shape. One skilled in the art would have been motivated to modify Van Rijthoven and Italiano in this manner in order to identify regions of interest within novel unstained images. (Stumpe, Abstract)
Regarding Claim 20, while the combination of Van Rijthoven and Italiano teach the system of claim 19, where Van Rijthoven further teaches wherein the processors are further operable when executing the instructions to: identify, for each of the one or more tertiary lymphoid structures, an area comprising the tertiary lymphoid structure (Page 2, Left Col, Paragraph 1, “to identify the presence of duct (both healthy and potentially cancerous) and other tissue structures. Additionally, they zoom-in into each region of interest, where the tissue is examined at a high-resolution, to obtain the details of the cancer cells, and characterize the tumor based on its local cellular composition.”, in this section of the prior art, cancerous regions are zoom-in to to determine potential TLS regions.);
However, Van Rijthoven and Italiano do not explicitly teach provide instructions for outlining the area comprising the tertiary lymphoid structure in the digital pathology image, wherein the outlining comprises a polygonal shape.
Stumpe teaches provide instructions for outlining the area comprising the tertiary lymphoid structure in the digital pathology image, wherein the outlining comprises a polygonal shape. (¶[0010],” The workstation is configured with either (a) user interface tools by which an operator inspecting the registered digital magnified images on the display may annotate a digital magnified image of the tissue specimen stained with the staining agent so as to form a closed polygon or other shape to mark a region of said image containing region of interest (e.g., tumor cells) to thereby create a mask.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven and Italiano with Stumpe in order to outline the cell of interest with a polygonal shape. One skilled in the art would have been motivated to modify Van Rijthoven and Italiano in this manner in order to identify regions of interest within novel unstained images. (Stumpe, Abstract)
Claims 3 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Van Rijthoven et al. ("HookNet: Multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images") in view of Italiano et al. ("Tertiary lymphoid structures in the era of cancer immunotherapy") in view of Yoo et al. US PG-Pub(US 20220036971 A1).
Regarding Claim 3, while the combination of Van Rijthoven and Italiano teaches the method of claim 1, where Van Rijthoven further teaches wherein the digital pathology image of the tissue sample depicts one or more structures(Page 3, 1.4. Our Contribution, Last Paragraph, “this work contributes with two multi-class and multi-organ segmentation models addressing problems where tissue is subjected to high-resolution details and context. The first application focuses on DCIS, IDC, and ILC segmentation in histopathology breast tissue. The second application focuses on the segmentation of GC, TLS, and tumor in lung tissue. To the best of our knowledge, these tissue types have never been simultaneously and separately segmented with a single model.” As disclosed in this section of the prior art, a machine learning model is used to segment the tertiary lymphoid structure and cell regions in the tissue image and Fig. 7 shows segmentation results on lung tissue for TLS and GC, Tumor, and Other.);
However, Van Rijthoven and Italiano do not explicitly teach wherein the method further comprises: generating, for each of the one or more structures by the machine-learning model, a numeric representation, wherein detecting the one or more tertiary lymphoid structures comprises determining one or more of the structures as the one or more tertiary lymphoid structures based on the respective numeric representations associated with the one or more of the structures.
Yoo teaches wherein the method further comprises: generating, for each of the one or more structures by the machine-learning model, a numeric representation(¶[0064], “the feature value of the immune phenotype may include various vectors related to the immune phenotype, such as a score value (score or density value for each class of the classifier) corresponding to the class corresponding to the immune phenotype (e.g., immune inflamed, immune excluded, and immune desert) and/or a feature fed as an input to the classifier”, a score or confidence is fed into the classifier when detecting an object of interest in the slide image.), wherein detecting the one or more tertiary lymphoid structures comprises determining one or more of the structures as the one or more tertiary lymphoid structures based on the respective numeric representations associated with the one or more of the structures. (¶[0064], “information associated with immune phenotype may include … 4) scalar values or vector values including statistics (e.g., ratio of the number of immune cells to the cancer stroma region, and the like) or distributions (e.g., histogram vectors or graph expression vectors, and the like) for immune cells and cancer cells and adjacent regions (e.g., cancer area, cancer stroma region, tertiary lymphoid structure, normal area, necrosis, fat, blood vessel, high endothelial venule, lymphatic vessel, nerve, and the like), and the like.”, TLS can be identified using scalar values fed into the classifier.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven and Italiano with Yoo in order to determine TLS using a numerical representation. One skilled in the art would have been motivated to modify Van Rijthoven and Italiano in this manner in order to predict a response to an immune checkpoint inhibitor. (Yoo, Abstract)
Regarding Claim 18, while the combination of Van Rijthoven and Italiano teach the media of claim 16, where Van Rijthoven further teaches wherein the tissue sample comprises one or more structures (Page 3, 1.4. Our Contribution, Last Paragraph, “this work contributes with two multi-class and multi-organ segmentation models addressing problems where tissue is subjected to high-resolution details and context. The first application focuses on DCIS, IDC, and ILC segmentation in histopathology breast tissue. The second application focuses on the segmentation of GC, TLS, and tumor in lung tissue. To the best of our knowledge, these tissue types have never been simultaneously and separately segmented with a single model.” As disclosed in this section of the prior art, a machine learning model is used to segment the tertiary lymphoid structure and cell regions in the tissue image and Fig. 7 shows segmentation results on lung tissue for TLS and GC, Tumor, and Other.);
However, Van Rijthoven and Italiano do not explicitly teach wherein the software is further operable when executed to: generate, for each of the one or more structures by the machine-learning model, a numeric representation, wherein detecting the one or more tertiary lymphoid structures comprises determining one or more of the structures as the one or more tertiary lymphoid structures based on the respective numeric representations associated with the one or more of the structures.
Yoo teaches wherein the software is further operable when executed to: generate, for each of the one or more structures by the machine-learning model, a numeric representation (¶[0064], “the feature value of the immune phenotype may include various vectors related to the immune phenotype, such as a score value (score or density value for each class of the classifier) corresponding to the class corresponding to the immune phenotype (e.g., immune inflamed, immune excluded, and immune desert) and/or a feature fed as an input to the classifier”, a score or confidence is fed into the classifier when detecting an object of interest in the slide image.), wherein detecting the one or more tertiary lymphoid structures comprises determining one or more of the structures as the one or more tertiary lymphoid structures based on the respective numeric representations associated with the one or more of the structures. (¶[0064], “information associated with immune phenotype may include … 4) scalar values or vector values including statistics (e.g., ratio of the number of immune cells to the cancer stroma region, and the like) or distributions (e.g., histogram vectors or graph expression vectors, and the like) for immune cells and cancer cells and adjacent regions (e.g., cancer area, cancer stroma region, tertiary lymphoid structure, normal area, necrosis, fat, blood vessel, high endothelial venule, lymphatic vessel, nerve, and the like), and the like.”, TLS can be identified using scalar values fed into the classifier.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven and Italiano with Yoo in order to determine TLS using a numerical representation. One skilled in the art would have been motivated to modify Van Rijthoven and Italiano in this manner in order to predict a response to an immune checkpoint inhibitor. (Yoo, Abstract)
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Van Rijthoven et al. ("HookNet: Multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images") in view of Italiano et al. US PG-Pub (US 20230296606 A1) in view of Wedenberg et al. US PG-Pub(US 20220296925 A1).
Regarding Claim 9, the combination of Van Rijthoven and Italiano teach the method of claim 1, where Van Rijthoven further teaches wherein the method further comprises: generating, one or more clusters of the detected tertiary lymphoid structures based on the numeric representation associated with each of the detected tertiary lymphoid structures; (Page 2, Left Col, Paragraph 1, “they zoom-in into each region of interest, where the tissue is examined at a high-resolution, to obtain the details of the cancer cells, and characterize the tumor based on its local cellular composition. Another example where pathologists take advantage of both context and details is the spatial distribution of immune cells, which may be detected in the presence of inflammation inside the tumor or the stromal compartment of the cancer regions, as well as in specific clustered groups called tertiary lymphoid structures (TLS)”, in this section of the prior art, each region of interest is zoomed in to obtain the details of cancer cells and the spatial distributions of the immune cells in order to generate cancer regions with clustered groups of TLS in the image.)
Italiano teaches wherein each of the determined one or more tertiary lymphoid structures is associated with a numeric representation (Page 315, TLS during tumour progression, paragraph 1, “In a study of 80 patients with oral squamous cell carcinoma, TLSs were more abundant in T1 and T2 stages than in T3 and T4 stages. By contrast, TLS presence correlates with higher tumour grade, stage and TIL densities and does not significantly impact on survival in other cancers (Table 2). For example, the abundance of TLSs increases in advanced stages (II–IV) compared with stage I gastric cancer (171 patients analysed)” in this section of the prior art different stages values of maturation in tumors are associated with TLS as stage T1 and T2 show the most detected TLS.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven with Italiano in order to determine the maturation state of the TLS and use the information to predict an outcome in response to the treatment. One skilled in the art would have been motivated to modify Van Rijthoven in this manner in order to show that TLSs are a major player in antitumour immune responses. (Italiano, Page 322, Conclusion and perspective, Paragraph 1)
However, Van Rijthoven and Italiano do not explicitly teach wherein determining the outcome of the subject in response to the treatment is further based on the one or more clusters.
Wedenberg teaches wherein determining the outcome of the subject in response to the treatment is further based on the one or more clusters. ( [0042] “The treatment plan is generated with the purpose of providing radiotherapy treatment of a treatment volume of a subject (patient) which may be an organ and includes a target which may be a tumor or cluster of tumor cells. The treatment volume is defined using a plurality of voxels, as known in the art.”, ¶[0042] discloses a treatment plan is determined based on the clusters of tumor cells in the image.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven and Italiano with Wedenberg in order determine treatment for a subject based on a cluster of cells. One skilled in the art would have been motivated to modify Van Rijthoven and Italiano in this manner in order to generate a robust radiotherapy treatment plan for a treatment volume of a subject, the treatment volume being defined using a plurality of voxels. (Wedenberg, Abstract)
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Van Rijthoven et al. ("HookNet: Multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images") in view of Italiano et al. US PG-Pub (US 20230296606 A1) in view of Tunstall et al. US PG-Pub(US 20160070949 A1).
Regarding Claim 10, while the combination of Van Rijthoven and Italiano teach the method of claim 1, they do not explicitly teach wherein the tissue sample is associated with one or more tumors, wherein the method further comprises: determining a type of the tissue sample; generating a tissue image mask for the tissue sample; identifying at least one tumor region within the tissue image mask; and determining a type of the at least one tumor region within the tissue image mask; wherein determining the outcome of the subject in response to the treatment is further based on the type of the tissue sample, the tissue image mask, and the type of the at least one tumor within the tissue image mask.
Tunstall further teaches wherein the tissue sample is associated with one or more tumors([0035] “The processor then estimates 3006 the area of a tumor cell by selecting regions of image data at native resolution corresponding to the objects identified as tumors.”, ¶[0035] discloses the processor estimates an area of the tissue sample with tumor cells and selects a region that contains these cells.), wherein the method further comprises: determining a type of the tissue sample([0049] “The processor then selects 2010 parts of the candidate region for further analysis, for example at a higher resolution, and selects 2012 a data model for that type of tissue structure from the classification data”, ¶[0049] discloses determining a type of tissue structure from the classification data in the image.); generating a tissue image mask for the tissue sample(¶[0099] “The segmentation by masking may be based on a single component of the image data, such as the first (eosin) component as described above, or from one of the other components, or from the original image data, or from a combination of one or more of these. In some examples a predefined image mask may be used”, ¶[0099] discloses performing a segmentation on the image data and generating a mask for the tissue in the image.); identifying at least one tumor region within the tissue image mask([0072] “The processor is configured to combine 1012 the tiles to provide a two state map (e.g. binary) identifying tumor, and non-tumor regions of the tissue with the tissue/non-tissue mask generated by the segmentation 1002 to provide a spatial map of the image data which classifies regions of the image into one of three states e.g. background, tumor tissue, and non-tumor tissue.” ¶[0072] discloses determining tumor regions for the segmented mask of the image data.); and determining a type of the at least one tumor region within the tissue image mask([0105] “At or following the classification step 1010, tiles classified as representing tumor regions may be assigned a posterior probability of corresponding to a tumor region of the tissue sample, based on a selected threshold level. For example, when classifying the tile as tumor or non-tumor, a threshold level of 0.5 (50%) may be applied.”, ¶[0105] discloses classifying a region in the image data based on a probability of the region corresponding to a tumor region.); wherein determining the outcome of the subject in response to the treatment is further based on the type of the tissue sample, the tissue image mask, and the type of the at least one tumor within the tissue image mask. (Tunstall, ¶[0146], “provide a computer implemented method of cancer detection including carrying out the steps of any one of the methods described herein. Such methods may further comprise providing an indication of cancer diagnosis, or a prognostic indication and optionally also providing or selecting a medical intervention based on the indication.”, ¶[0146] discloses determining a treatment or diagnosis based on the tumor region detected in the image.)
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven and Italiano with Tunstall. in order to mask the image to segment the tumor regions. One skilled in the art would have been motivated to modify Van Rijthoven and Italiano in this manner in order to analyze and identify a tumor region of a tissue sample for the purpose of macrodissection of the tissue sample. (Tunstall, ¶[0001])
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Van Rijthoven et al. ("HookNet: Multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images") in view of Italiano et al. US PG-Pub (US 20230296606 A1) in view of Inoue et al. US PG-Pub(US 20220215543 A1).
Regarding Claim 15, while the combination of Van Rijthoven and Italiano teach the method of claim 12, where Van Rijthoven further teaches further comprising training the machine-learning model(Page 1, Abstract, We describe a framework to design and train HookNet for achieving high-resolution semantic segmentation.),
Van Rijthoven and Italiano do not explicitly teach wherein the training comprises: identifying at least two tertiary lymphoid structures within at least one slide image of the training data; generating a cropped image from the at least one slide image by cropping the at least one slide image to make the at least two tertiary lymphoid structures centered; and training the machine-learning model based in part on the cropped image.
Inoue teaches wherein the training comprises: identifying at least two tertiary lymphoid structures within at least one slide image of the training data (¶0071], “FIG. 7 is a view showing a generation method of the cropped image IMG.sub.C. For example, the image processor 114 selects one cell image IMG.sub.k from a cell image group captured in a time series for a certain nerve cell. The image processor 114 generates a plurality of cropped images IMG.sub.C(k) by scanning the region R of a predetermined aspect ratio while sliding the region R on the selected cell image IMG.sub.k, and by cutting out the pixel region overlapping the region R as the cropped image IMG.sub.C. ”, as disclosed in ¶[0071], the prior art performing cropping on a region of interest to contain a certain type of cell in the image.); generating a cropped image from the at least one slide image by cropping the at least one slide image to make the at least two tertiary lymphoid structures centered([0114] “The original size of the single-well image captured by using the time lapse imaging system (product name “Incucyte ZOOM”, Essen Bioscience, Ltd.) was 1,392×1,038 pixels, and four images were cut out from this image and extracted as the cropped image IMG.sub.C. The original image was divided evenly into four regions, and square images were taken out from the center of each region.”, ¶[0114] discloses that the images were cropped such that the regions where taken from the center of the image.); and training the machine-learning model based in part on the cropped image([0072] “Description will return to the flowchart in FIG. 6. Next, the learner 120 divides the plurality of cropped images IMG.sub.C generated from the cell images IMG of the training data by the image processor 114 into a training cropped image Tr_IMG.sub.C and a validation cropped image Va_IMG.sub.C, and inputs the training cropped image Tr_IMG.sub.C to an i-th model WL-i among the K models WL included in the prediction model MDL as weak learners (step S204).”, ¶[0072] discloses the model is trained with the cropped image data.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the claimed invention as taught by Van Rijthoven and Italiano with Inoue in order to crop areas of interest in the image data and train a model based on the cropped image. One skilled in the art would have been motivated to modify Van Rijthoven and Italiano in this manner in order to accurately predict future diseases of a subject based on images of cells differentiated from pluripotent stem cells derived from the subject. (Inoue, ¶[0007])
Allowable Subject Matter
Claims 5-8 and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claim 5, the primary reason for the allowance of the claim is the inclusion of the limitations, “wherein the digital pathology image of the tissue sample depicts one or more structures, and wherein detecting the one or more tertiary lymphoid structures depicted within the digital pathology image of the tissue sample comprises: determining whether each of the one or more structures comprises a germinal center; and based on the determining of whether each of the one or more structures comprises a germinal center: if at least one of the one or more structures comprises a germinal center, determining the at least one structure as a tertiary lymphoid structure; else: analyzing an interaction between a presence of each of the one or more structures and one or more features associated with the structure; and determining whether each of the one or more structures is a tertiary lymphoid structure based on the analyzed interaction.”, in all the claim which is not found in the prior art references. It is noted that the examiner has not found any other prior art to anticipate or obviate the quoted claim limitations supra, when read in light/combination of the other claimed limitations within the cited claims. Also, it is noted that the quoted limitations, in combination with the other claim limitations of the cited claims, deem the claim patentable, not just the consideration of the quoted limitations by themselves.
Regarding Claim 6, the primary reason for the allowance of the claim is the inclusion of the limitations, “wherein the digital pathology image of the tissue sample depicts one or more structures, and wherein detecting the one or more tertiary lymphoid structures depicted within the digital pathology image of the tissue sample comprises: calculating, for each of the one or more structures identified by the machine-learning model, a confidence score based on a precision and recall for the machine-learning model, wherein the confidence score indicates a probability of the structure being a tertiary lymphoid structure; and determining, based on the one or more confidence scores, that one or more of the structures are tertiary lymphoid structures.”, in all the claim which is not found in the prior art references. It is noted that the examiner has not found any other prior art to anticipate or obviate the quoted claim limitations supra, when read in light/combination of the other claimed limitations within the cited claims. Also, it is noted that the quoted limitations, in combination with the other claim limitations of the cited claims, deem the claim patentable, not just the consideration of the quoted limitations by themselves.
Regarding Claim 7 and 8, these claims depend on claim 6 and would be allowed by virtue of dependency.
Regarding Claim 14, the primary reason for the allowance of the claim is the inclusion of the limitations, “wherein the machine-learning model is based on one or more neural networks, and wherein the method further comprises training the machine-learning model, and wherein the training comprises: generating, by the one or more neural networks, one or more initial bounding boxes for each slide image, wherein each initial bounding box is associated with an initial aspect ratio and an initial size; adjusting, based on characteristics of tertiary lymphoid structures, the initial aspect ratio and the initial size associated with each of the initial bounding box; and training the machine-learning model based on slide image data using the adjusted bounding boxes.”, in all the claim which is not found in the prior art references. It is noted that the examiner has not found any other prior art to anticipate or obviate the quoted claim limitations supra, when read in light/combination of the other claimed limitations within the cited claims. Also, it is noted that the quoted limitations, in combination with the other claim limitations of the cited claims, deem the claim patentable, not just the consideration of the quoted limitations by themselves.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAN D HOANG whose telephone number is (571)272-4344. The examiner can normally be reached Monday-Friday 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JOHN M VILLECCO can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAN HOANG/Examiner, Art Unit 2661