DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
The present application claims benefit of foreign application JP 2023-148664 filed on 09/13/2023. While the certified copy (WIPO) is received, the translation is not on file. No translation is required at this time unless the status of the case changes.
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 08/27/2024 and 04/08/2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 2, 5-6, 9, 12, 13, and 16-17 are objected to because of the following informalities:
Claims 2 and 9 state “the plurality of second tomographic image groups includes the second tomographic image groups that include the first tomographic image groups in which the first image feature is equal to or greater than a first threshold value, and the second tomographic image groups that include the first tomographic image groups in which the first image feature is equal to or less than a second threshold value lower than the first threshold value,” but should read “the plurality of second tomographic image groups includes
Claims 5 and 12 read “wherein the calculating the integrated value includes calculating the integrated value by performing […].” This is grammatically incorrect. The claim should read “wherein
Claims 5 and 12 recites “as the second average value is lower.” The wording of this phrase is unclear and should be revised to more clearly express the intended relationship between the second average value and the higher weighting factor. If the applicant were to amend the phrase to read “…wherein a second lower average value corresponds to a higher weighting factor” or another clear alternative version, the objection would likely be resolved.
Claim 6, line 16, and claim 13, line 17, “the calculating the integrated value includes” should read “.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-14 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 1 and 8,
(issue) 1. The calculation of a “first image feature” is unclear which renders the claims indefinite. The limitation on lines 4-6 of the respective claims, states “calculating a first image feature, based on a plurality of first tomographic image groups obtained by imaging an inside of each of a plurality of first human bodies, for each of the first tomographic image groups.” It is unclear how a “first image feature” may both be (i) “based on a plurality of first tomographic image groups,” and (ii) calculated “for each of the first tomographic image groups.” The phrasing of the limitation further makes it unclear whether the “first image feature” is calculated independently for each of the image groups and the calculation is complete, whether the “first image feature” is first calculated independently for each group and then each group’s “first image feature” is aggregated to calculate the “first image feature,” or if another logical or mathematical modeled structure is performed to calculate a “first image feature.” The lack of clarity surrounding the calculation of “a first image feature,” as detailed above, makes subsequent processing that is reliant upon the “first image feature” unclear (e.g. comparing a first image feature to a first threshold value, as seen in claims 2 and 9). Therefore, claims 1, 8, and their dependent claims (claims 2-7 and 9-14) are rejected for being indefinite.
If applicant were to amend independent claims to further clarify the calculation of a first image feature, and to incorporate the features of claims 3 and 4 (i.e. describe the calculation of first and second image features as shown in claim 3, and include the first and second average value that indicates an average luminance in the regions as detailed in claim 4), the issue would likely be resolved. Doing so would provide further clarity to how a first image feature is calculated and represented as a numerical value for use in the range classification of tomographic images, used for performing threshold and value comparisons in subsequent claims, and other subsequent processing implementations.
2. The claim recites “calculating a second image feature of a same type as the first image feature, based on a plurality of first tomographic images obtained by imaging the inside of a second human body” and subsequently recites inputting “the plurality of first tomographic images” into the plurality of first lesion identification models. However, it is unclear what image set is encompassed by the recited “first tomographic images.” Specifically, the claim previously recites “first tomographic image groups” associated with the plurality of first human bodies during the training process, while the lesion detection process introduces “first tomographic images” in connection with a second human body. Accordingly, it is unclear whether the recited “first tomographic images” correspond to images associated with the first human bodies, images associated with the second human body, or another image set. Because the identity of the image set is unclear, the scope of the subsequent limitations cannot be determined with reasonable certainty. In particular, it is unclear how the second image feature corresponds to the probabilities acquired from the lesion identification models and subsequently integrated.
Regarding claims 1, 6, 8, and 13,
Claims 1 and 8 recite “for each of the unit image regions, each through machine learning that uses different ones of the plurality of second tomographic image groups from each other, as training data” (claims 1 and 8, lines 12-14) is unclear because it is ambiguous what antecedent basis the term “each” refers to and what claim element is modified by the phrase “for each of the unit image regions.” Claims 6 and 13 have similar issues with the unclear use of “each” and “for each of the unit image regions,” reciting “for each of the unit image regions, each through the machine learning that uses different ones of the plurality of third tomographic image groups from each other, as the training data.” Therefore, the scope of the limitations are not reasonably certain and the claims and their dependent claims are rejected for being indefinite.
Regarding claims 1, 7, 8, and 14,
The term “range” in claim claims 1, 7, 8, and 14 is a relative term which renders the claim indefinite. The term “range” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The claims fail to specify what logical and/or measurement structurally defines a “range,” “first range,” and/or “second range” in relation to a “first image feature” (i.e. is it a range relating to pixel values, distances, ratios, instances of features found, average luminance etc.). Paragraph [0142] of the specification indicates “a range of slice planes in which a lesion region is detected (slice planes corresponding to result images in which information indicating a lesion region is included in one or more pixel)” when discussing the lesion-detected region display portion 73. However, method claims 3 and 4, and their respective mirrored non-transitory computer-readable recording medium claims 10 and 11, claim a first image feature is calculated based on pixel information (see claims 3 and 10) and that the first image feature is a first average value (see claims 4 and 11) that indicates an average of luminance in the regions of a particular organ in the image regions of the respective tomographic images. Thus, it is unclear if a range is defined by slice planes (as indicated by paragraph [0142] of the specification), an average value of luminance based on pixel values, or another structure of ranges. Further, it is unclear what defines the lower and upper limits of a range, or if the limits of the range are indicative of another structured model. Accordingly, the use of “range” in claims 1, 7, 8, 14, and their dependent claims (claims 2-6 and 9-13) further renders the claims indefinite. Thus, claims 1-14 are rejected for being indefinite. If applicant were to amend claims to define the terms of the range or the metrics for the first image feature in order to clarify the relationship (e.g. “according to a range of the first image feature’s average luminance” for claims 1 and 8, “first image feature’s average luminance is included in a first average luminance range” and “for image feature’s average luminance is included in a second average luminance range” for claims 7 and 14, or another well-defined framework), the issue would likely be resolved.
Regarding claims 2 and 9,
As previously discussed (see issue 1 regarding claims 1 and 8 section above), the lack of clarity when calculating “the first image feature” makes the use of the term for comparison purposes, as seen in claims 2 and 9, unclear because one of ordinary skill in the art doesn’t clearly know how the information is derived.
The recited “first image feature” is also indefinite in claims 2 and 9 because the claim does not reasonably identify what characteristic, metric, or quantity the feature represents. While dependent claims 4 and 11 indicate that the first image feature may correspond to an average luminance value, claims 2 and 9, and their respective independent claims, do not impose such a limitation. Moreover, the specification describes multiple image features that may be compared to a threshold, including average luminance and a percentage of high luminance pixels (see [0116], [0183]), thereby leading to multiple reasonable interpretations of the claimed “first image feature.” As a result, it is also unclear what characteristic or quantity the recited threshold values correspond to, and the scope of the claimed comparisons (e.g. first image feature equal to or greater than a first threshold value) cannot be determined with reasonable certainty.
If applicant were to amend the claims to read the value used to represent first image feature (e.g. first image feature’s average luminance is equal to or greater than a first predetermined threshold, first image feature’s percentage of high luminance pixels is equal to or greater than a first predetermined threshold, etc.), the issue would likely be resolved.
Regarding claim 7 and 14,
Claims 7 and 14 recite the limitation “wherein the classifying the plurality of first tomographic images into the plurality of third tomographic image groups includes classifying the plurality of first tomographic images into the third tomographic image groups […].” in lines 1-4 of claim 7 and lines 1-5 of claim 14. The term “plurality of first tomographic images” lacks proper antecedent basis because the parent claim recites a plurality of “first tomographic image groups” rather than a plurality of first tomographic images when describing the classification process. As a result, it is unclear whether the recited first tomographic images correspond to the previously recited first tomographic image groups, images within the groups, or another image set.
All of the claims mentioned above and their dependent claims are rejected accordingly.
Applicant is reminded many of the issues addressed throughout all of the 112 rejections above impact other subsequent claims and corrections/amendments to corresponding issues or terms are required.
Obviousness Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-14 are provisionally rejected on the ground of non-statutory obvious-type double patenting as being unpatentable over claims 1-7 of co-pending Application No. 18/810,313 (reference application). Although the claims at issue are not identical, they are not patentably distinct from each other because both the instant application and the reference application are directed to a lesion detection method and non-transitory computer-readable recording medium storing a lesion detection program that (1) employs a plurality of lesion identification models trained using clustered groups of tomographic images, (2) acquires probabilities from the plurality of lesion identification models for unit image regions of test tomographic images, (2) integrates the probabilities based on an image feature amount calculated from the test tomographic images, and (3) detects lesion regions based on the resulting integrated value. The dependent claims of the instant application (claims 2-7 and 9-14) substantially correspond to the dependent claims of the reference application (reference claims 2-7), reciting analogous image feature definitions, weighting schemes, grouping conditions, and model generating operations. Accordingly, the claimed subject matter are obvious variants of one another and not patentably distinct, as shown below for the independent claims below.
Instant Application
Reference Application No. 18/810,313
Claim 1: A method for detecting lesion for a computer to execute a process comprising:
a training process that includes: calculating a first image feature, based on a plurality of first tomographic image groups obtained by imaging an inside of each of a plurality of first human bodies, for each of the first tomographic image groups;
classifying tomographic images included in the plurality of first tomographic image groups into a plurality of second tomographic image groups, according to a range of the first image feature;
and generating a plurality of first lesion identification models configured to identify whether or not unit image regions included in the tomographic images to be identified are regions of a particular lesion, for each of the unit image regions, each through machine learning that uses different ones of the plurality of second tomographic image groups from each other, as training data;
and a lesion detection process that includes: calculating a second image feature of a same type as the first image feature, based on a plurality of first tomographic images obtained by imaging the inside of a second human body;
acquiring, from each of the plurality of first lesion identification models, probabilities of being the regions of the particular lesion, for each of the unit image regions included in the plurality of first tomographic images, by inputting the plurality of first tomographic images to each of the plurality of first lesion identification models;
integrating, for each of the unit image regions included in the plurality of first tomographic images, the probabilities acquired from each of the plurality of first lesion identification models, based on the second image feature, to calculate an integrated value;
and detecting the regions of the particular lesion from each of the plurality of first tomographic images, based on the integrated value.
Claim 1: A lesion detection method for a computer to execute:
Claim 5: wherein the learning process includes a process of: calculating, for each second tomographic image group into which the plurality of first tomographic images are classified for each of the first human bodies, a second image feature amount of a same type as the first image feature amount
Claim 5: classifying the plurality of first tomographic images into a plurality of third tomographic image groups according to a range of the second image feature amount
Claim 5: generating a plurality of second lesion identification models for identifying whether or not each of the unit image regions included in the tomographic image as the identification target is the specific lesion region by machine learning which uses each of ones different from each other among the plurality of third tomographic image groups as learning data
Claim 1: a lesion detection process of: calculating a first image feature amount based on a plurality of second tomographic images obtained by imaging an inside of a second human body,
Claim 5: acquiring the probability for each of the unit image regions included in the plurality of second tomographic images from each of the plurality of second lesion identification models, by inputting the plurality of second tomographic images to each of the plurality of second lesion identification models
Claim 1: calculating, for each of the unit image regions included in the plurality of second tomographic images, an integration value by integrating the probabilities acquired from each of the plurality of first lesion identification models, based on the first image feature amount
Claim 5 further teaches: for each of the unit image regions included in the plurality of second tomographic images, the integration value is calculated by integrating the probabilities acquired from each of the plurality of first lesion identification models and each of the plurality of second lesion identification models, based on the first image feature amount.
Claim 1: detecting the specific lesion region from each of the plurality of second tomographic images based on the integration value.
Independent Claim 8: A non-transitory computer-readable recording medium storing a lesion detection program for causing a computer to execute a process comprising:The remaining limitations of claim 8 mirror the method limitations of claim 1. For sake of brevity, refer to mirrored method limitations above and corresponding limitations from Reference Application No. 18/810,313.
Claim 7: A non-transitory computer-readable recording medium storing a lesion detection program causing a computer to execute:
Accordingly, claims 1-14 of the instant application are not patentably distinct from claims 1-7 of the co-pending reference Application No. 18/810,313 and are provisionally rejected on the grounds of non-statutory obviousness double patenting. This a provisional non-statutory double patenting rejection because the reference application has not yet issued as a patent.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-4 and 8-11 are rejected under 35 U.S.C. 103 as being unpatentable over Bonakdar et al. (US 20220138932 A1; hereinafter “Bonakdar”) in view of Lai Y (CN 109635664 A; hereinafter “Lai”; provided by Applicant in IDS; translation relied upon provided by Examiner).
Regarding claims 1 (method) and 8 (non-transitory CRM),
Bonakdar teaches: A method (claim 1 of instant application; ¶ [0077] “The present invention may be…a method…”) for and a non-transitory computer-readable recording medium storing a program (claim 8 of instant application; ¶ [0077]-[0078] “The present invention may be …a computer program product…The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device…A computer readable storage medium…is not to be construed as being transitory signals per se…”)
for detecting lesion for a computer to execute a process comprising (see Abstract, title “Ensemble Machine Learning Model Architecture For Lesion Detection,” and ¶ [0044] “embodiments are specifically directed to an improved computing tool that provides automated computer driven artificial intelligence medical image analysis that is specifically trained, through machine learning/deep learning computer processes, to…detect lesions...”:
a training process that includes (¶ [0006] “a plurality of trained machine learning computer models”):
(Although Bonakdar fails to explicitly teach this calculation limitation, Bonakdar does teach the processing of tomographic images obtained by imaging an inside of each of a plurality of first human bodies. Specifically, Bonakdar teaches receiving input images described as input volume using CT images of internal organs like the liver, where the input volume is a 3D representation of the biological entity’s internal anatomical structure to identify lesions present in a patient’s organ (e.g. liver images and lesion detection) (see ¶ [0050], ¶ [0145], ¶ [0179], and ¶ [0236]).)
generating a plurality of first lesion identification models configured to identify whether or not unit image regions included in the tomographic images to be identified are regions of a particular lesion, for each of the unit image regions, each through machine learning that uses (Bonakdar teaches generating a plurality of lesion identification models using medical images (e.g. CT images) and machine learning (¶ [0061] “lesion detection stage of the AI pipeline uses an ensemble of ML/DL computer models to detect the liver and lesions in the liver as represented in the volume of input CT medical images. The ensemble of ML/DL computer models uses differently trained ML/DL computer models to perform liver and lesion detection, with the ML/DL computer models being trained and using loss functions to counterbalance false positives and false negatives in lesion detection.”), although Bonakdar’s model uses loss functions for training, rather than training group data.) and
a lesion detection process that includes:
acquiring, from each of the plurality of first lesion identification models, probabilities of being the regions of the particular lesion, for each of the unit image regions included in the plurality of first tomographic images, by inputting the plurality of first tomographic images to each of the plurality of first lesion identification models (Bonakdar teaches multiple lesion identification models that each process input medical image volume and generating lesion prediction outputs (i.e. probabilities) that are subsequently combined. Specifically, Bonakdar teaches that “input volume is also processed via a first trained ML/DL computer model of an ensemble, which is specifically configured and trained to perform lesion detection” and that “The first trained ML/DL computer model generates a first set of lesion detection prediction outputs based on its processing of the input volume” (¶ [0145]). Bonakdar further teaches that “A second trained ML/DL computer model of the ensemble” generates “two sets of lesion prediction outputs” (¶ [0146]), and that “the two ML/DL computer models 620, 630 process the input slices to generate lesion predictions that are averaged” (¶ [0143]), thereby teaching acquiring prediction outputs (i.e. probabilities) from multiple lesion identification models and subsequently combining those output to generate a final lesion probability.)
integrating, for each of the unit image regions included in the plurality of first tomographic images, the probabilities acquired from each of the plurality of first lesion identification models, (Bonakdar teaches integration/averaging of multiple outputs (¶ [0139] “The final detection map of the second ML/DL model 620 is combined with that of the third ML/DL model 630 by means of the average operation 640.”). Bonakdar further teaches that detections are aggregated, and that volume averaging logic computes the average of the two detection masks at the voxel level, stating “All the generated detections of the ML/DL model 620 for each slab of the input volume 105 are combined with the generated detections of the ML/DL model 630 via the volume averaging (VOL AVG) logic 640. This logic computes the average of the two detection masks at the voxel level” (¶ [0142]). Therefore, Bonkdar teaches integrating the prediction outputs (i.e. probabilities) acquired from a plurality of lesion identification models by averaging outputs to generate a final lesion prediction output, corresponding to the claimed calculation of an integrated value based on outputs from multiple lesion identification models.)
and detecting the regions of the particular lesion from each of the plurality of first tomographic images, based on the integrated value (Bonakdar teaches that the averaged detections (discussed in previous limitation, found above) produce “a Final Lesion mask 650 corresponding to the detected lesions in the input volume 105.” (¶ [0142]). Bonakdar also states that the models generate lesion predictions that are averaged, resulting in “a final lesion output” (¶ [0143]).
Bonakdar fails to explicitly disclose: (1.) calculating a first image feature, based on a plurality of first tomographic image groups obtained by imaging an inside of each of a plurality of first human bodies, for each of the first tomographic image groups;
(2.) classifying tomographic images included in the plurality of first tomographic image groups into a plurality of second tomographic image groups, according to a range of the first image feature;
(3.) machine learning that uses different ones of the plurality of second tomographic image groups from each other as training data;
(4.) calculating a second image feature of a same type as the first image feature, based on a plurality of first tomographic images obtained by imaging the inside of a second human body;
(5.) and using a second image feature during the integration process.
In a related art, Lai teaches: (1.) calculating a first image feature, based on a plurality of first (Lai teaches acquiring an eye image database (¶ [0030] “obtain a database based on human eye images”) and calculating a feature for images (¶ [0031] “For each image in the human eye image database, calculate its average brightness value.”). The eye image database taught by Lai corresponds to the claimed plurality of first image groups, and the average brightness value corresponds to the claimed first image feature.);
(2.) classifying (Lai teaches classifying the images into second image groups according to ranges of the calculated feature, stating that “Based on the range of the average brightness value, the human eye images are divided into three image datasets” and goes on to disclose ranges for each classification (¶ [0031]). The three brightness range datasets taught by Lai are interpreted by the Examiner to be equivalent to a plurality of second image groups);
(3.) machine learning that uses different ones of the plurality of second (Lai teaches training separate models from different feature-based image groups (¶ [0014] “…performing feature extraction and SVM classification training on each labeled dataset to obtain three SVM-based detection models.”)
(4.) calculating a second image feature of a same type as the first image feature, based on a plurality of first (Lai teaches calculating an average brightness value for a newly acquired image (i.e. a second image feature of the same type as the first image feature) (see ¶ [0037] “Calculate the average brightness of the human eye region image”) and that model selection is performed according to the calculated brightness value (¶ [0038] “Select which SVM model to use for detection based on the average brightness value.”). The average brightness value corresponds to the claimed second image feature and is of the same type as the average brightness value previously calculated for the training images.);
(5.) and using a second image feature during the (Lai teaches calculating a brightness average value for a newly acquired image (¶ [0037]) and using that image feature to determine which of a plurality of separately trained models is applied (¶ [0038] “Select which SVM model to use for detection based on the average brightness value. If the average brightness is less than 96, select M1; if it is greater than 176, select M3; otherwise, select M2.”). Therefore, Lai teaches using the calculated image feature to dictate application of feature specific models, while Bonakdar previously taught integrating outputs from multiple models (see Bonakdar teachings above and Bonakdar ¶ [0139] and ¶ [0142]).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the lesion detection ensemble of Bonakdar to incorporate Lai’s feature-based image grouping and model training because Lai teaches that classifying images according to a calculated image feature and training separate models for different feature ranges improves detection accuracy (see Lai Abstract, ¶ [0010], and ¶ [0018]). Doing so would have predictably improved the accuracy and robustness of Bonakdar’s lesion detection ensemble by allowing different models to be optimized for different image characteristics (e.g. average luminance pixel values) while retaining Bonakdar’s combined prediction approach. Both inventions are directed to machine learning-based image analysis for automated detection/classification tasks and improving image-based detection accuracy through the use of multiple models to output predictions.
Regarding claims 2 and 9,
Bonakdar and Lai teach the method according to claim 1 and the non-transitory computer-readable recording medium according to claim 8.
Lai further teaches: the plurality of second tomographic image groups includes the second tomographic image groups that include the first tomographic image groups in which the first image feature is equal to or greater than a first threshold value, and the second tomographic image groups that include the first tomographic image groups in which the first image feature is equal to or less than a second threshold value lower than the first threshold value (Lai teaches calculating a brightness average value for each image and classifying the images into groups according to threshold ranges of the calculated feature (¶ [0031] “Based on the range of the average brightness value, the human eye images are divided into three image datasets”; ¶ [0015] “the brightness threshold ranges of the image dataset are 0-96, 96-176 and 176-255”). Accordingly, the brightness average value taught by Lai corresponds to the claimed first image feature, and the brightness thresholds correspond to the claimed first and second threshold values used to define the second tomographic image groups).
Regarding claims 3 and 10,
Bonakdar and Lai teach the method according to claim 1 and the non-transitory computer-readable recording medium according to claim 8, including calculating a first and second image feature.
Lai further teaches: the first image feature is calculated based on pixel information on in the first tomographic image groups, and the second image feature is calculated based on the pixel information on (Lai teaches calculating an image feature from pixel information of a segmented human body image region during both training and detection. Specifically, Lai teaches obtaining an eye image database by segmenting the eye region (¶ [0030] “segment the eye region, and obtain a database based on human eye images.”) and calculating a brightness feature (i.e. first and second image feature) (¶ [0031] “For each image in the human eye image database, calculate its average brightness value.”). Lai further teaches calculating the same luminance feature during detection (¶ [0037] “Calculate the average brightness of the human eye region image”).
Bonakdar further teaches: identifying a particular organ (e.g. liver) and generating organ-specific image regions for subsequent lesion analysis (¶ [0145] “The result of the anatomical structure detection is a segmentation of the input volume to identify a mask for the anatomical structure, e.g., liver mask…”; ¶ [0146] “receives a masked input generated by applying the generated anatomical structure mask to the input volume and thereby identify portions of the medical images in the input volume that correspond to the anatomical structure of interest”). Bonakdar further teaches that lesion detection is performed using the mask input corresponding to the liver region (¶ [0146).
Bonakdar teaches identifying and segmenting a liver region for lesion analysis, while Lai teaches calculating image features from a segmented anatomical region and using those features for more selection. It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to calculate the first and second image features (taught by Lai) from Bonakdar’s segmented liver region because the liver is the anatomical region used for lesion detection and feature calculation from the relevant anatomical region would have predictably improved detection accuracy.
Regarding claims 4 and 11,
Bonakdar and Lai teach the method according to claim 3 and the non-transitory computer-readable recording medium according to claim 10, including a first and second image feature calculated based on regions of a particular organ in image regions of the plurality of first tomographic image groups and second tomographic images, respectively.
Lai further teaches: the calculated image features, previously taught to be the first image feature and the second image feature (see 103 rejections above for claims 1 and 8, and claims 3 and 10), are average brightness values (i.e. an average value that indicates the average of the luminance) (See ¶ [0030]-[0031] for training using average brightness value and Lai’s teaching of detection using average brightness value in ¶ [0037], as previously discussed above in the claim 3 and claim 10 103 rejection).
Accordingly, Bonakdar and Lai’s obviousness combination teaches: the first image feature is a first average value that indicates an average of luminance in the regions of the particular organ in the image regions of the respective tomographic images included in the first tomographic image groups, and the second image feature is a second average value that indicates the average of the luminance in the regions of the particular organ in the image regions of the plurality of first tomographic images.
Claim(s) 5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Bonakdar et al. (US 20220138932 A1; hereinafter “Bonakdar”) in view of Lai Y (CN 109635664 A; hereinafter “Lai”; provided by Applicant in IDS; translation relied upon provided by Examiner), and in further view of Odaibo et al. (US 20170357879 A1; hereinafter “Odaibo”).
Regarding claims 5 and 12,
Bonakdar and Lai teach the method according to claim 4 and the non-transitory computer-readable recording medium according to claim 11.
Bonakdar and Lai further teach: the calculating the integrated value includes calculating the integrated value by performing (Bonakdar teaches obtaining lesion probabilities from multiple lesion identification models and combining the outputs (Bonakdar ¶ [0139] “The final detection map of the second ML/DL model 620 is combined with that of the third ML/DL model 630 by means of the average operation 640”; Detections are aggregated, see Bonakdar ¶ [0142] and further discussion in claim 1 and 8 rejection above; Bonakdar ¶ [0143] “two ML/DL computer models 620, 630 process the input slices to generate lesion predictions that are averaged for the volume by the volume averaging logic 640. The result is a final lesion output 650 along with the liver mask output…”)
(Lai does teach generating separate models from image groups having different brightness (i.e. luminance) average ranges (Lai ¶ [0014] “dividing the human eye images into three image datasets based on the average brightness value of the images”) and calculating a runtime brightness average value, which is used to select a corresponding brightness-specific detection model (¶ [0037]-[0038]). Thus, Lai teaches that lower-average-value image groups correspond to lower brightness model conditions and that a runtime lower average value corresponds to the lower brightness model.).
Bonakdar and Lai fail to explicitly disclose: performing weighted addition of probabilities on the plurality of lesion identification models and a higher weighting factor is set for the probabilities from the first lesion identification models generated by using the second tomographic image groups that have a lower first average value, among the plurality of first lesion identification models, as the second average value is lower.
In a related art, Odaibo teaches: an ensemble classification framework for automatic detection of ophthalmic disease from images (see Odaibo’s Abstract and Title) and assigning weights to probabilities generated by multiple models and calculating an integrated value using weighted addition of those probabilities (Odaibo discloses an “image is presented to each model of the ensemble for classification, yielding a probabilistic classification score” from each model, and that “Using the model weights, a weighted-average of the individual model-generated probabilistic scores is computed…” (Abstract). Odaibo further teaches that “for each model, the model's assigned weight is multiplied by the class score of the subject image. The sum of all such products is taken…” such that “the weighted average of class scores is computed…” (¶ [0019).).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Bonakdar and Lai’s averaging of lesion model probabilities using the weighted ensemble technique of Odaibo because Odaibo teaches assigning weights to models and computing a weighted average of model generated probability scores. In view of Lai’s teachings that images are divided according to brightness average groups and that corresponding model is selected based on the runtime brightness average value to improve detection accuracy (see Lai Abstract, ¶ [0010], and ¶ [0018]), it would have been obvious to assign greater weight to the model trained on the luminance group most closely corresponding to the runtime image. Thus, when the runtime average value is lower, the lower average value model would receive a higher weighting factor, predictably improving the detection accuracy under the current image condition. All three inventions are in the medical field and are aimed at improving medical image analysis through the use of multiple machine learning models that generate prediction probabilities that are subsequently combined to produce more reliable diagnostic data.
Claim(s) 6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Bonakdar et al. (US 20220138932 A1; hereinafter “Bonakdar”) in view of Lai Y (CN 109635664 A; hereinafter “Lai”; provided by Applicant in IDS; translation relied upon provided by Examiner), and in further view of Gazit (US 20160300351 A1; hereinafter “Gazit”).
Regarding claims 6 and 13,
Bonakdar and Lai teach the method according to claim 1 and the non-transitory computer-readable recording medium according to claim 8. Claims 6 and 13 depends from claims 1 and 8 and includes limitations substantially similar to the training lesion detection, probability acquisition, and integration limitations discussed above with respect to claims 1 and 8. Bonakdar and Lai teach or suggest those limitations set forth in the rection of claim 1 (for sake of brevity refer to claim 1 and claim 8 rejections above and Bonakdar and Lai’s teachings).
Bonakdar and Lai fail to explicitly disclose: classifying the plurality of first tomographic image groups into a plurality of third tomographic image groups that have different medical findings;
and generating a plurality of second lesion identification models configured to identify whether or not the unit image regions included in the tomographic images to be identified are the regions of the particular lesion, for each of the unit image regions, each through the machine learning that uses different ones of the plurality of third tomographic image groups from each other, as the training data.
Specifically, while Bonakdar and Lai teach generating multiple lesion identification models from different image groups and combining output of those modes (see Bonakdar and Lai teachings in claim 1 and 8 rejections above), neither reference teaches grouping the training images according to different medical findings (e.g., diseases or disease states and additional set of lesion identification models based on those medical finding specific image groups. In a related art, Gazit teaches: that different image groups may correspond to different disease states and grouping training images into clusters based on organ intensity characteristics (¶ [0090] “different organ intensity characteristics are due to disease states, such as pneumonia in the lungs or cirrhosis in the liver”; ¶ [0091] “training images are grouped into the clusters based on the organ intensity characteristics of the image”). Gazit further teaches generating separate training data for the respective clusters (¶ [0108] “for each target organ, the training images are optionally divided into clusters which have different organ intensity characteristics…A different set of training data is then generated for each cluster”). Thus, Gazit teaches classifying image groups according to disease related characteristics (i.e. material findings) and generating separate training datasets for the respective groups.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to apply the disease state based clustering techniques of Gazit to the training process of Bonakdar and Lai because training separate models using image groups associated with different medical findings would have predictably improved model specialization and classification performance for images with differing characteristics, thereby improving the accuracy of lesion identification when images have different medical findings. All three inventions are in the medical field and are aimed at improving medical image analysis by specifying model training to characteristics of the training images.
Allowable Subject Matter
Claims 7 and 14 are rejected under double patenting and 35 U.S.C. 112(b), but
would be allowable if the double patenting and 112(b) issues were resolved and rewritten in independent form, including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAMUEL DAVID BAYNES whose telephone number is (571)272-0607. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen R Koziol can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SDB/
Samuel D. Baynes
Examiner | Art Unit 2665
/Stephen R Koziol/Supervisory Patent Examiner, Art Unit 2665
Prosecution Insights